
Plan
virtualenv, pip and Distribute are tree tools that help developers and packagers. In this short presentation we will see some virtualenv capabilities.
Please, keep in mind that all above stuff has been made using : Debian Lenny, python 2.5 and virtualenv 1.4.5.
Abstract
virtualenv builds python sandboxes where it is possible to do whatever you want as a simple user without putting in jeopardy your global environment.
virtualenv allows you to safety:
- install any python packages
- add debug lines everywhere (not only in your scripts)
- switch between python versions
- try your code as you are a final user
- and so on ...
Install and usage
Install
Prefered way
Just download the virtualenv python script at http://bitbucket.org/ianb/virtualenv/raw/tip/virtualenv.py and call it using python (e.g. python virtualenv.py).
For conveinience, we will refers to this script using virtualenv.
Other ways
For Debian (ubuntu as well) addicts, just do :
$ sudo aptitude install python-virtualenv
Fedora users would do:
$ sudo yum install python-virtualenv
And others can install from PyPI (as superuser):
$ pip install virtualenv
or
$ easy_install pip && pip install virtualenv
You could also get the source here.
Quick Guide
To work in a python sandbox, do as follow:
$ virtualenv my_py_env $ source my_py_env/bin/activate (my_py_env)$ python
"That's all Folks !"
Once you have finished just do:
(my_py_env)$ deactivate
or quit the tty.
What does virtualenv actually do ?
At creation time
Let's start again ... more slowly. Consider the following environment:
$ pwd /home/you/some/where $ ls
Now create a sandbox called my-sandbox:
$ virtualenv my-sandbox New python executable in "my-sandbox/bin/python" Installing setuptools............done.
The output said that you have a new python executable and specific install tools. Your current directory now looks like:
$ ls -Cl my-sandbox/ README $ tree -L 3 my-sandbox my-sandbox/ |-- bin | |-- activate | |-- activate_this.py | |-- easy_install | |-- easy_install-2.5 | |-- pip | `-- python |-- include | `-- python2.5 -> /usr/include/python2.5 `-- lib `-- python2.5 |-- ... |-- orig-prefix.txt |-- os.py -> /usr/lib/python2.5/os.py |-- re.py -> /usr/lib/python2.5/re.py |-- ... |-- site-packages | |-- easy-install.pth | |-- pip-0.6.3-py2.5.egg | |-- setuptools-0.6c11-py2.5.egg | `-- setuptools.pth |-- ...
In addition to the new python executable and the install tools you have an whole new python environment containing libraries, a site-packages/ (where your packages will be installed), a bin directory, ...
- Note:
- virtualenv does not create every file needed to get a whole new python environment. It uses links to global environment files instead in order to save disk space end speed up the sandbox creation. Therefore, there must already have an active python environment installed on your system.
At activation time
At this point you have to activate the sandbox in order to use your custom python. Once activated, python still has access to the global environment but will look at your sandbox first for python's modules:
$ source my-sandbox/bin/activate (my-sandbox)$ which python /home/you/some/where/my-sandbox/bin/python $ echo $PATH /home/you/some/where/my-sandbox/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games (pyver)$ python -c 'import sys;print sys.prefix;' /home/you/some/where/my-sandbox (pyver)$ python -c 'import sys;print "\n".join(sys.path)' /home/you/some/where/my-sandbox/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg [...] /home/you/some/where/my-sandbox /home/you/personal/PYTHONPATH /home/you/some/where/my-sandbox/lib/python2.5/ [...] /usr/lib/python2.5 [...] /home/you/some/where/my-sandbox/lib/python2.5/site-packages [...] /usr/local/lib/python2.5/site-packages /usr/lib/python2.5/site-packages [...]
First of all, a (my-sandbox) message is automatically added to your prompt in order to make it clear that you're using a python sandbox environment.
Secondly, my-sandbox/bin/ is added to your PATH. So, running python calls the specific python executable placed in my-sandbox/bin.
- Note
- It is possible to improve the sandbox isolation by ignoring the global paths and your PYTHONPATH (see Improve isolation section).
Installing package
It is possible to install any packages in the sandbox without any superuser privilege. For instance, we will install the pylint development revision in the sandbox.
Suppose that you have the pylint stable version already installed in your global environment:
(my-sandbox)$ deactivate $ python -c 'from pylint.__pkginfo__ import version;print version' 0.18.0
Once your sandbox activated, install the development revision of pylint as an update:
$ source /home/you/some/where/my-sandbox/bin/activate (my-sandbox)$ pip install -U hg+http://www.logilab.org/hg/pylint#egg=pylint-0.19
The new package and its dependencies are only installed in the sandbox:
(my-sandbox)$ python -c 'import pylint.__pkginfo__ as p;print p.version, p.__file__' 0.19.0 /home/you/some/where/my-sandbox/lib/python2.6/site-packages/pylint/__pkginfo__.pyc (my-sandbox)$ deactivate $ python -c 'import pylint.__pkginfo__ as p;print p.version, p.__file__' 0.18.0 /usr/lib/pymodules/python2.6/pylint/__pkginfo__.pyc
You can safely do any change in the new pylint code or in others sandboxed packages because your global environment is still unchanged.
Useful options
Improve isolation
As said before, your sandboxed python sys.path still references the global system path. You can however hide them by:
- either use the --no-site-packages that do not give access to the global site-packages directory to the sandbox
- or change your PYTHONPATH in my-sandbox/bin/activate in the same way as for PATH (see tips)
$ virtualenv --no-site-packages closedPy $ sed -i '9i PYTHONPATH="$_OLD_PYTHON_PATH" 9i export PYTHONPATH 9i unset _OLD_PYTHON_PATH 40i _OLD_PYTHON_PATH="$PYTHONPATH" 40i PYTHONPATH="." 40i export PYTHONPATH' closedPy/bin/activate $ source closedPy/bin/activate (closedPy)$ python -c 'import sys; print "\n".join(sys.path)' /home/you/some/where/closedPy/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg /home/you/some/where/closedPy /home/you/some/where/closedPy/lib/python2.5 /home/you/some/where/closedPy/lib/python2.5/plat-linux2 /home/you/some/where/closedPy/lib/python2.5/lib-tk /home/you/some/where/closedPy/lib/python2.5/lib-dynload /usr/lib/python2.5 /usr/lib64/python2.5 /usr/lib/python2.5/lib-tk /home/you/some/where/closedPy/lib/python2.5/site-packages $ deactivate
This way, you'll get an even more isolated sandbox, just as with a brand new python environment.
Work with different versions of Python
It is possible to dedicate a sandbox to a particular version of python by using the --python=PYTHON_EXE which specifies the interpreter that virtualenv was installed with (default is /usr/bin/python):
$ virtualenv --python=python2.4 pyver24 $ source pyver24/bin/activate (pyver24)$ python -V Python 2.4.6 $ deactivate $ virtualenv --python=python2.5 pyver25 $ source pyver25/bin/activate (pyver25)$ python -V Python 2.5.2 $ deactivate
Distribute a sandbox
To distribute your sandbox, you must use the --relocatable option that makes an existing sandbox relocatable. This fixes up scripts and makes all .pth files relative This option should be called just before you distribute the sandbox (each time you have changed something in your sandbox).
An important point is that the host system should be similar to your own.
Tips
Speed up sandbox manipulation
Add these scripts to your .bashrc in order to help you using virtualenv and automate the creation and activation processes.
rel2abs() { #from http://unix.derkeiler.com/Newsgroups/comp.unix.programmer/2005-01/0206.html [ "$#" -eq 1 ] || return 1 ls -Ld -- "$1" > /dev/null || return dir=$(dirname -- "$1" && echo .) || return dir=$(cd -P -- "${dir%??}" && pwd -P && echo .) || return dir=${dir%??} file=$(basename -- "$1" && echo .) || return file=${file%??} case $dir in /) printf '%s\n' "/$file";; /*) printf '%s\n' "$dir/$file";; *) return 1;; esac return 0 } function activate(){ if [[ "$1" == "--help" ]]; then echo -e "usage: activate PATH\n" echo -e "Activate the sandbox where PATH points inside of.\n" return fi if [[ "$1" == '' ]]; then local target=$(pwd) else local target=$(rel2abs "$1") fi until [[ "$target" == '/' ]]; do if test -e "$target/bin/activate"; then source "$target/bin/activate" echo "$target sandbox activated" return fi target=$(dirname "$target") done echo 'no sandbox found' } function mksandbox(){ if [[ "$1" == "--help" ]]; then echo -e "usage: mksandbox NAME\n" echo -e "Create and activate a highly isaolated sandbox named NAME.\n" return fi local name='sandbox' if [[ "$1" != "" ]]; then name="$1" fi if [[ -e "$1/bin/activate" ]]; then echo "$1 is already a sandbox" return fi virtualenv --no-site-packages --clear --distribute "$name" sed -i '9i PYTHONPATH="$_OLD_PYTHON_PATH" 9i export PYTHONPATH 9i unset _OLD_PYTHON_PATH 40i _OLD_PYTHON_PATH="$PYTHONPATH" 40i PYTHONPATH="." 40i export PYTHONPATH' "$name/bin/activate" activate "$name" }
- Note:
- The virtualenv-commands and virtualenvwrapper projects add some very interesting features to virtualenv. So, put on eye on them for more advanced features than the above ones.
Conclusion
I found it to be irreplaceable for testing new configurations or working on projects with different dependencies. Moreover, I use it to learn about other python projects, how my project exactly interacts with its dependencies (during debugging) or to test the final user experience.
All of this stuff can be done without virtualenv but not in such an easy and secure way.
I will continue the series by introducing other useful projects to enhance your productivity : pip and Distribute. See you soon.
Other Documentation and Links
virtualenv homepage: http://virtualenv.openplans.org/
pip homepage: http://pip.openplans.org/
Distribute homepage: http://packages.python.org/distribute/
virtualenv-commands extension for virtualenv : http://thisismedium.com/tech/extending-virtualenv/
virtualenvwrapper extension for virtualenv : http://www.doughellmann.com/projects/virtualenvwrapper
(Image under creative commons by-nc-nd by minlynn071502 / Mindy Roth)
- Keeping score in test-driven development with Python, PyLint, unittest, doctest, and PyRate (http://www.freesoftwaremagazine.com/columns/keeping_score_test_driven_development_python_pylint_unittest_doctest_and_pyrate)
- Apycot for Mercurial
- Windows, fichiers ouverts et tests unitaires
- The Python Package Index is not a "Software Distribution"
- pyreverse http://www.logilab.org/project/pylint a set of tools for reverse engineering Python code.
Comments
Note: virtualenv, pip and Distribute are included with ActivePython-2.6.
i recommend simply downloading and compiling your own python where you want to have it... and call the custom python executable where needed. when i hear that virtualenv uses links for some files ‘to save disk space’ i go YUCK! there are uncounted bytes completely wasted in any installation of ubuntu and similar for games i don’t want and never play but am too lazy to remove, and untold numbers of strange and obscure libraries i will never touch. how much can a single python installation change that? when i see that activation script i get the feeling you could control a small nuclear power plant or a jet plane with that much code. see, installing a custom stackless python just means you have to wget http://www.stackless.com/binaries/stackless-311-export.tar.bz2
&& tar jxvf stackless-311-export.tar.bz2 && cd python-3.1.1-stackless && sudo ./configure --prefix=/flower && sudo make all && sudo make install. then, call /flower/bin/python3.1 and you're good to go. to me, virtualenv is a cure in search of a problem.
Virtualenv is basically equivalent to compiling your own Python (using "./configure --prefix=DIR"). But many people have a dozen or more virtualenv environments in their workspace. If you look at virtualenvwrapper (a set of shell scripts implementing a workflow for virtualenv) it's there to answer the question of how to keep a whole bunch of environments straight, and quickly create and destroy those environments -- if you had one or two environments that tool wouldn't be helpful.
Also I regularly use virtualenv in testing, to create isolated and repeatable installations of software stacks (e.g., for integration testing). Compiling a new interpreter for a test would be too time consuming. virtualenv environments can be created quickly.
I actually recommend creating a virtualenv environment without any kind of installation. Grab this file: http://bitbucket.org/ianb/virtualenv/raw/tip/virtualenv.py -- and you can run it like "python virtualenv.py new-environment/"
This installs Setuptools (or Distribute if you pass it a --distribute flag) and pip, and virtualenv doesn't need any installation itself. Also you don't need any root access on the machine. Also you can download and unpack the tarball and run the virtualenv.py script from within there (the tarball comes with all the source code for Setuptools, Distribute, and pip, so creating new environments then doesn't require downloading anything additional).