There are `python-matplotlib` and `python-numpy` packages in the Debian/Ubuntu repos.
However, if you want to run in a virtualenv (with no-site-packages), and pip install these packages from PyPI, you need some system dependencies installed first to build with:
$ sudo apt-get install build-essential python-dev libfreetype6-dev libpng-dev python-virtualenv
Then, you can create a virtualenv, and the installers for Numpy and Matplotlib will work:
$ virtualenv env $ cd env $ source bin/activate (env)$ pip install numpy matplotlib ... ... Successfully installed numpy matplotlib Cleaning up...
(tested on Ubuntu Oneiric 11.10 and Ubuntu Precise 12.04 alpha)