blog entries (Logilab.org) RSS Feed

MiniDebConf Paris 2010

Arthur Lutz — 2010-09-09T11:33-01:00

Debian France organise le 30 et 31 octobre prochain une minidebconf à Paris. Le wiki de la conférence est en train de s'étoffer, et pour le moment c'est là qu'il faut s'inscrire. À Logilab nous sommes utilisateurs et contributeurs de Debian, c'est donc naturellement que nous allons essayer d'aller participer à cette conférence. Alexandre Fayolle, développeur Debian ira assister (entre autres) à la présentation de Carl Chenet sur l'état de Python dans Debian.

Discovering logilab-common Part 1 - deprecation module

Stéphanie Marcu — 2010-09-02T12:42-01:00

logilab-common library contains a lot of utilities which are often unknown. I will write a series of blog entries to explore nice features of this library.

We will begin with the logilab.common.deprecation module which contains utilities to warn users when:

a function or a method is deprecated

a class has been moved into another module

a class has been renamed

a callable has been moved to a new module

deprecated

When a function or a method is deprecated, you can use the deprecated decorator. It will print a message to warn the user that the function is deprecated.

The decorator takes two optional arguments:

reason: the deprecation message. A good practice is to specify at the beginning of the message, between brackets, the version number from which the function is deprecated. The default message is 'The function "[function name]" is deprecated'.

stacklevel: This is the option of the warnings.warn function which is used by the decorator. The default value is 2.

We have a class Person defined in a file person.py. The get_surname method is deprecated, we must use the get_lastname method instead. For that, we use the deprecated decorator on the get_surname method.

from logilab.common.deprecation import deprecated

class Person(object):

    def __init__(self, firstname, lastname):
        self._firstname = firstname
        self._lastname = lastname

    def get_firstname(self):
        return self._firstname

    def get_lastname(self):
        return self._lastname

    @deprecated('[1.2] use get_lastname instead')
    def get_surname(self):
        return self.get_lastname()

def create_user(firstname, lastname):
    return Person(firstname, lastname)

if __name__ == '__main__':
    person = create_user('Paul', 'Smith')
    surname = person.get_surname()

When running person.py we have the message below:

person.py:22: DeprecationWarning: [1.2] use get_lastname instead

surname = person.get_surname()

class_moved

Now we moved the class Person in a new_person.py file. We notice in the person.py file that the class has been moved:

from logilab.common.deprecation import class_moved
import new_person
Person = class_moved(new_person.Person)

if __name__ == '__main__':
    person = Person('Paul', 'Smith')

When we run the person.py file, we have the following message:

person.py:6: DeprecationWarning: class Person is now available as new_person.Person

person = Person('Paul', 'Smith')

The class_moved function takes one mandatory argument and two optional:

new_class: this mandatory argument is the new class

old_name: this optional argument specify the old class name. By default it is the same name than the new class. This argument is used in the default printed message.

message: with this optional argument, you can specify a custom message

class_renamed

The class_renamed function automatically creates a class which fires a DeprecationWarning when instantiated.

The function takes two mandatory arguments and one optional:

old_name: a string which contains the old class name

new_class: the new class

message: an optional message. The default one is '[old class name] is deprecated, use [new class name]'

We now rename the Person class into User class in the new_person.py file. Here is the new person.py file:

from logilab.common.deprecation import class_renamed
from new_person import User

Person = class_renamed('Person', User)

if __name__ == '__main__':
    person = Person('Paul', 'Smith')

When running person.py, we have the following message:

person.py:5: DeprecationWarning: Person is deprecated, use User

person = Person('Paul', 'Smith')

moved

The moved function is used to tell that a callable has been moved to a new module. It returns a callable wrapper, so that when the wrapper is called, a warning is printed telling where the object can be found. Then the import is done (and not before) and the actual object is called.

Note

The usage is somewhat limited on classes since it will fail if the wrapper is used in a class ancestors list: use the class_moved function instead (which has no lazy import feature though).

The moved function takes two mandatory parameters:

modpath: a string representing the path to the new module

objname: the name of the new callable

We will use in person.py, the create_user function which is now defined in the new_person.py file:

from logilab.common.deprecation import moved

create_user = moved('new_person', 'create_user')

if __name__ == '__main__':
    person = create_user('Paul', 'Smith')

When running person.py, we have the following message:

person.py:4: DeprecationWarning: object create_user has been moved to module new_person

person = create_user('Paul', 'Smith')

pdb.set_trace no longer working: problem solved

Alexandre Fayolle — 2010-08-12T16:17-01:00

I had a bad case of bug hunting today which took me > 5 hours to track down (with the help of Adrien in the end).

I was trying to start a CubicWeb instance on my computer, and was encountering some strange pyro error at startup. So I edited some source file to add a pdb.set_trace() statement and restarted the instance, waiting for Python's debugger to kick in. But that did not happen. I was baffled. I first checked for standard problems:

no pdb.py or pdb.pyc was lying around in my Python sys.path
the pdb.set_trace function had not been silently redefined
no other thread was bugging me
the standard input and output were what they were supposed to be
I was not able to reproduce the issue on other machines

After triple checking everything, grepping everywhere, I asked a question on StackOverflow before taking a lunch break (if you go there, you'll see the answer). After lunch, no useful answer had come in, so I asked Adrien for help, because two pairs of eyes are better than one in some cases. We dutifully traced down the pdb module's code to the underlying bdb and cmd modules and learned some interesting things on the way down there. Finally, we found out that the Python code frames which should have been identical where not. This discovery caused further bafflement. We looked at the frames, and saw that one of those frames's class was a psyco generated wrapper.

It turned out that CubicWeb can use two implementation of the RQL module: one which uses gecode (a C++ library for constraint based programming) and one which uses logilab.constraint (a pure python library for constraint solving). The former is the default, but it would not load on my computer, because the gecode library had been replaced by a more recent version during an upgrade. The pure python implementation tries to use psyco to speed up things. Installing the correct version of libgecode solved the issue. End of story.

When I checked out StackOverflow, Ned Batchelder had provided an answer. I didn't get the satisfaction of answering the question myself...

Once this was figured out, solving the initial pyro issue took 2 minutes...

EuroSciPy'10

Adrien Chauve — 2010-07-13T14:12-01:00

The EuroSciPy2010 conference was held in Paris at the Ecole Normale Supérieure from July 8th to 11th and was organized and sponsored by Logilab and other companies.

July, 8-9: Tutorials

The first two days were dedicated to tutorials and I had the chance to talk about SciPy with André Espaze, Gaël Varoquaux and Emanuelle Gouillart in the introductory track. This was nice but it was a bit tricky to present SciPy in such a short time while trying to illustrate the material with real and interesting examples. One very nice thing for the introductory track is that all the material was contributed by different speakers and is freely available in a github repository (licensed under CC BY).

July, 10-11: Scientific track

The next two days were dedicated to scientific presentations and why python is such a great tool to develop scientific software and carry out research.

Keynotes

I had a very great time listening to the presentations, starting with the two very nice keynotes given by Hans Petter Langtangen and Konrad Hinsen. The latter gave us a very nice summary of what happened in the scientific python world during the past 15 years, what is happening now and of course what could happen during the next 15 years. Using a crystal ball and a very humorous tone, he made it very clear that the challenge in the next years will be about how using our hundreds, thousands or even more cores in a bug-free and efficient way. Functional programming may be a very good solution to this challenge as it provides a deterministic way of parallelizing our programs. Konrad also provided some hints about future versions of python that could provide a deeper and more efficient support of functional programming and maybe the addition of a keyword 'async' to handle the computation of a function in another core.

In fact, the PEP 3148 entitled "Futures - execute computations asynchronously" was just accepted two days ago. This PEP describes the new package called "futures" designed to facilitate the evaluation of callables using threads and processes in future versions of python. A full implementation is already available.

Parallelization

Parallelization was indeed a very popular issue across presentations, and as for resolving embarrassingly parallel problems, several solutions were presented.

Playdoh: Distributes computations over computers connected to a secure network (see playdoh presentation).

Distributing the computation of a function over two machines is as simple as:
```
import playdoh
result1, result2 = playdoh.map(fun, [arg1, arg2], _machines = ['machine1.network.com', 'machine2.network.com'])
```
Theano: Allows to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. In particular it can use GPU transparently and generate optimized C code (see theano presentation).
joblib: Provides among other things helpers for embarrassingly parallel problems. It's built over the multiprocessing package introduced in python 2.6 and brings more readable code and easier debugging.

Speed

Concerning speed, Fransesc Alted has showed us interesting tools for memory optimization currently successfully used in PyTables 2.2. You can read more details on these kind of optimizations in EuroSciPy'09 (part 1/2): The Need For Speed.

SCons

Last but not least, I talked with Cristophe Pradal who is one of the core developer of OpenAlea. He convinced me that SCons is worth using once you have built a nice extension for it: SConsX. I'm looking forward to testing it.

HOWTO install lodgeit pastebin under Debian/Ubuntu

Arthur Lutz — 2010-06-24T16:46-01:00

Lodge it is a simple open source pastebin... and it's written in Python!

The installation under debian/ubuntu goes as follows:

sudo apt-get update
sudo apt-get -uVf install python-imaging python-sqlalchemy python-jinja2 python-pybabel python-werkzeug python-simplejson
cd local
hg clone http://dev.pocoo.org/hg/lodgeit-main
cd lodgeit-main
vim manage.py

For debian squeeze you have to downgrade python-werkzeug, so get the old version of python-werkzeug from snapshot.debian.org at http://snapshot.debian.org/package/python-werkzeug/0.5.1-1/

wget http://snapshot.debian.org/archive/debian/20090808T041155Z/pool/main/p/python-werkzeug/python-werkzeug_0.5.1-1_all.deb

Modify the dburi and the SECRET_KEY. And launch application:

python manage.py runserver

Then off you go configure your apache or lighthttpd.

An easy (and dirty) way of running it at startup is to add the following command to the www-data crontab

@reboot cd /tmp/; nohup /usr/bin/python /usr/local/lodgeit-main/manage.py runserver &

This should of course be done in an init script.

Hopefully we'll find some time to package this nice webapp for debian/ubuntu.

EuroSciPy 2010 schedule is out !

Nicolas Chauvat — 2010-06-06T19:12-01:00

The EuroSciPy 2010 conference will be held in Paris from july 8th to 11th at Ecole Normale Supérieure. Two days of tutorials, two days of conference, two interesting keynotes, a lightning talk session, an open space for collaboration and sprinting, thirty quality talks in the schedule and already 100 delegates registered.

If you are doing science and using Python, you want to be there!

Salomé accepted into Debian unstable

Andre Espaze — 2010-06-03T18:28-01:00

Salomé is a platform for pre and post-processing of numerical simulation available at http://salome-platform.org/. It is now available as a Debian package http://packages.debian.org/source/sid/salome and should soon appear in Ubuntu https://launchpad.net/ubuntu/+source/salome as well.

A difficult packaging work

A first package of Salomé 3 was made by the courageous Debian developper Adam C. Powell, IV on January 2008. Such packaging is very resources intensive because of the building of many modules. But the most difficult part was to bring Salomé to an unported environment. Even today, Salomé 5 binaries are only provided by upstream as a stand-alone piece of software ready to unpack on a Debian Sarge/Etch or a Mandriva 2006/2008. This is the first reason why several patches were required for adapting the code to new versions of the dependencies. The version 3 of Salomé was so difficult and time consuming to package that Adam decided to stop during two years.

The packaging of Salomé started back with the version 5.1.3 in January 2010. Thanks to Logilab and the OpenHPC project, I could join him during 14 weeks of work for adapting every module to Debian unstable. Porting to the new versions of the dependencies was a first step, but we had also to adapt the code to the Debian packaging philosophy with binaries, librairies and data shipped to dedicated directories.

A promising future

Salomé being accepted to Debian unstable means that porting it to Ubuntu should follow in a near future. Moreover the work done for adapting Salomé to a GNU/Linux distribution may help developpers on others platforms as well.

That is excellent news for all people involved in numerical simulation because they are going to have access to Salomé services by using their packages management tools. It will help the spreading of Salomé code on any fresh install and moreover keep it up to date.

Join the fun

For mechanical engineers, a derived product called Salomé-Méca has recently been published. The goal is to bring the functionalities from the Code Aster finite element solver to Salomé in order to ease simulation workflows. If you are as well interested in Debian packages for those tools, you are invited to come with us and join the fun.

I have submitted a proposal to talk about Salomé at EuroSciPy 2010. I look forward to meet other interested parties during this conference that will take place in Paris on July 8th-11th.

Une mise en place de l’eXtreme Programming - ce que j'en retiens

Arthur Lutz — 2010-05-20T12:19-01:00

Je suis allé à la présentation de "Une mise en place de l’eXtreme Programming" présenté par Karine Sabatier dans le cadre d'Agile Nantes. On y a parlé plutôt Ruby on Rails que python, mais surtout de méthodes agiles et XP.

Voici quelques points que j'ai retenu de cette présentation tout à fait pragmatique d'une mise en pratique des principes XP :

Le principe de "Convention over configuration" : préférer la convention (notamment pour la programmation) plutôt que la contrainte par la configuration. Dans Ruby On Rails, les conventions sont très fortes, pour faire une application on ne peut pas s'éloigner du modèle MVC : les modèles sont dans "model", les views sont dans "views" etc... À ce sujet, la conférencière a fait référence à deux types de designs que je ne connaissait pas : le DDD Domain-driven Design et Behavior Driven Development.
Utilisation de métaphores : trouver un langage commun avec le client mais aussi avec les utilisateurs
Application de déploiement ruby on rails : Capistrano, bien qu'à Logilab nous privilégions le déploiement par paquets et dépôts debian, en python on pourra jeter un coup d'œil à Fabric.

Leur projet XP utilisait le logiciel de gestion de projet Retrospectiva. Celui-ci ressemble sur bien des points à JPL (Jeux de Planification Logiciel) disponible sur le framework CubicWeb (http://www.cubicweb.org). Coté intégration continue : CruiseControl , en python nous privilégions apycot.
Ce projet a essayé l'utilisation de Selenium pour ces tests web. Le constat est le même que chez Logilab : la première fois que ca marche c'est utile et apporte une grande satisfaction, dans un deuxième temps ca reste très difficile à maintenir. Nous regardons à présent plutôt du coté de Windmill qui a été intégré à la version 3.9 de cubicweb.
Une mention a été fait d'une société fonctionnement uniquement en mode agile Pyxis.

Enable and disable encrypted swap - Ubuntu

Arthur Lutz — 2010-05-18T16:31-01:00

With the release of Ubuntu Lucid Lynx, the use of an encrypted /home is becoming a pretty common and simple to setup thing. This is good news for privacy reasons obviously. The next step which a lot of users are reluctant to accomplish is the use of an encrypted swap. One of the most obvious reasons is that in most cases it breaks the suspend and hibernate functions.

Here is a little HOWTO on how to switch from normal swap to encrypted swap and back. That way, when you need a secure laptop (trip to a conference, or situtation with risk of theft) you can active it, and then deactivate it when you're at home for example.

Turn it on

That is pretty simple

sudo ecryptfs-setup-swap

Turn it off

The idea is to turn off swap, remove the ecryptfs layer, reformat your partition with normal swap and enable it. We use sda5 as an example for the swap partition, please use your own (fdisk -l will tell you which swap partition you are using - or in /etc/crypttab)

sudo swapoff -a
sudo cryptsetup remove /dev/mapper/cryptswap1
sudo vim /etc/crypttab
*remove the /dev/sda5 line*
sudo /sbin/mkswap /dev/sda5
sudo swapon /dev/sda5
sudo vim /etc/fstab
*replace /dev/mapper/cryptswap1 with /dev/sda5*

If this is is useful, you can probably stick it in a script to turn on and off... maybe we could get an ecryptfs-unsetup-swap into ecryptfs.

The DEBSIGN_KEYID trick

Nicolas Chauvat — 2010-05-12T18:23-01:00

I have been wondering for some time why debsign would not use the DEBSIGN_KEYID environment variable that I exported from my bashrc. Debian bug 444641 explains the trick: debsign ignores environment variables and sources ~/.devscripts instead. A simple export DEBSIGN_KEYID=ABCDEFG in ~/.devscripts is enough to get rid of the -k argument once and for good.

Sprint CubicWeb chez Logilab - annonce de dernière minute

Arthur Lutz — 2010-04-29T18:50-01:00

Logilab est en ce moment en train d'acceuillir un sprint autour de la plateforme CubicWeb. L'objectif principal de ce sprint de 5 jours est d'améliorer l'usage de javascript et des css dans CubicWeb :

avoir une API javascript propre, testée et documentée
pouvoir facilement changer le style d'une application cubicweb
gestion de bundle pour javascript et CSS
une documentation sur les standards d'écriture des fichiers JS et CSS pour cubicweb

Ce sprint aura lieu du jeudi 29 avril 2010 au 5 mai 2010 (weekend exlus - les bureaux seront fermés). Vous êtes les bienvenus pour contribuer, filer un coup de main, faire du pair programming... ou simplement rencontrer des développeurs cubicweb. Vous pouvez même venir une après-midi ou une seule journée. Pour ceux avec des portables, il y aura du réseau disponible pour être connecté.

Adresse : 104 Boulevard Auguste-Blanqui, Paris. Sonnez à "Logilab".

Métro : St Jacques or Corvisart (Glacière est la station la plus proche mais sera fermée à partir de lundi)

Contact : http://www.logilab.fr/contact

Dates : du 29/04/2010 au 30/04/2010 et du 03/05/2010 au 05/05/2010

pylint bug days #2 report

Sylvain Thenault — 2010-04-19T17:12-01:00

First of all, I've to say that pylint bugs day wasn't that successful in term of 'community event': I've been sprinting almost alone. My Logilab's felows were tied to customer projects, and no outside people shown up on jabber. Fortunatly Tarek Ziade came to visit us, and that was a nice opportunity to talk about pylint, distribute, etc ... Thank you Tarek, you saved my day ;)

As I felt a bit alone, I decided to work on somethings funnier than bug fixing: refactoring!

First, I've greatly simplified the command line: enable-msg/enable-msg-cat/enable-checker/enable-report and their disable-* counterparts were all merged into single --enable/--disable options.

I've also simplified "pylint --help" output, providing a --long-help option to get what we had before. Generic support in `logilab.common.configuration of course.

And last but not least, I refactored pylint so we can have multiple checkers with the same name. The idea behind this is that we can split checker into smaller chunks, basically only responsible for one or a few related messages. When pylint runs, it only uses necessary checkers according to activated messages and reports. When all checkers will be splitted, it should improve performance of "pylint --error-only".

So, I can say I'm finally happy with the results of that pylint bugs day! And hopefuly we will be more people for the next edition...

Virtualenv - Play safely with a Python

Alain Leufroy — 2010-03-26T10:19-01:00

Plan

virtualenv, pip and Distribute are tree tools that help developers and packagers. In this short presentation we will see some virtualenv capabilities.

Please, keep in mind that all above stuff has been made using : Debian Lenny, python 2.5 and virtualenv 1.4.5.

Abstract

virtualenv builds python sandboxes where it is possible to do whatever you want as a simple user without putting in jeopardy your global environment.

virtualenv allows you to safety:

install any python packages
add debug lines everywhere (not only in your scripts)
switch between python versions
try your code as you are a final user
and so on ...

Install and usage

Install

Prefered way

Just download the virtualenv python script at http://bitbucket.org/ianb/virtualenv/raw/tip/virtualenv.py and call it using python (e.g. python virtualenv.py).

For conveinience, we will refers to this script using virtualenv.

Other ways

For Debian (ubuntu as well) addicts, just do :

$ sudo aptitude install python-virtualenv

Fedora users would do:

$ sudo yum install python-virtualenv

And others can install from PyPI (as superuser):

$ pip install virtualenv

$ easy_install pip && pip install virtualenv

You could also get the source here.

Quick Guide

To work in a python sandbox, do as follow:

$ virtualenv my_py_env
$ source my_py_env/bin/activate
(my_py_env)$ python

"That's all Folks !"

Once you have finished just do:

(my_py_env)$ deactivate

or quit the tty.

What does `virtualenv` actually do ?

At creation time

Let's start again ... more slowly. Consider the following environment:

$ pwd
/home/you/some/where
$ ls

Now create a sandbox called my-sandbox:

$ virtualenv my-sandbox
New python executable in "my-sandbox/bin/python"
Installing setuptools............done.

The output said that you have a new python executable and specific install tools. Your current directory now looks like:

$ ls -Cl
my-sandbox/ README
$ tree -L 3 my-sandbox
my-sandbox/
|-- bin
|   |-- activate
|   |-- activate_this.py
|   |-- easy_install
|   |-- easy_install-2.5
|   |-- pip
|   `-- python
|-- include
|   `-- python2.5 -> /usr/include/python2.5
`-- lib
    `-- python2.5
        |-- ...
        |-- orig-prefix.txt
        |-- os.py -> /usr/lib/python2.5/os.py
        |-- re.py -> /usr/lib/python2.5/re.py
        |-- ...
        |-- site-packages
        |   |-- easy-install.pth
        |   |-- pip-0.6.3-py2.5.egg
        |   |-- setuptools-0.6c11-py2.5.egg
        |   `-- setuptools.pth
        |-- ...

In addition to the new python executable and the install tools you have an whole new python environment containing libraries, a site-packages/ (where your packages will be installed), a bin directory, ...

Note:: virtualenv does not create every file needed to get a whole new python environment. It uses links to global environment files instead in order to save disk space end speed up the sandbox creation. Therefore, there must already have an active python environment installed on your system.

At activation time

At this point you have to activate the sandbox in order to use your custom python. Once activated, python still has access to the global environment but will look at your sandbox first for python's modules:

$ source my-sandbox/bin/activate
(my-sandbox)$ which python
/home/you/some/where/my-sandbox/bin/python
$ echo $PATH
/home/you/some/where/my-sandbox/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
(pyver)$ python -c 'import sys;print sys.prefix;'
/home/you/some/where/my-sandbox
(pyver)$ python -c 'import sys;print "\n".join(sys.path)'
/home/you/some/where/my-sandbox/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg
[...]
/home/you/some/where/my-sandbox
/home/you/personal/PYTHONPATH
/home/you/some/where/my-sandbox/lib/python2.5/
[...]
/usr/lib/python2.5
[...]
/home/you/some/where/my-sandbox/lib/python2.5/site-packages
[...]
/usr/local/lib/python2.5/site-packages
/usr/lib/python2.5/site-packages
[...]

First of all, a (my-sandbox) message is automatically added to your prompt in order to make it clear that you're using a python sandbox environment.

Secondly, my-sandbox/bin/ is added to your PATH. So, running python calls the specific python executable placed in my-sandbox/bin.

Note: It is possible to improve the sandbox isolation by ignoring the global paths and your PYTHONPATH (see Improve isolation section).

Installing package

It is possible to install any packages in the sandbox without any superuser privilege. For instance, we will install the pylint development revision in the sandbox.

Suppose that you have the pylint stable version already installed in your global environment:

(my-sandbox)$ deactivate
$ python -c 'from pylint.__pkginfo__ import version;print version'
0.18.0

Once your sandbox activated, install the development revision of pylint as an update:

$ source /home/you/some/where/my-sandbox/bin/activate
(my-sandbox)$ pip install -U hg+http://www.logilab.org/hg/pylint#egg=pylint-0.19

The new package and its dependencies are only installed in the sandbox:

(my-sandbox)$ python -c 'import pylint.__pkginfo__ as p;print p.version, p.__file__'
0.19.0 /home/you/some/where/my-sandbox/lib/python2.6/site-packages/pylint/__pkginfo__.pyc
(my-sandbox)$ deactivate
$ python -c 'import pylint.__pkginfo__ as p;print p.version, p.__file__'
0.18.0 /usr/lib/pymodules/python2.6/pylint/__pkginfo__.pyc

You can safely do any change in the new pylint code or in others sandboxed packages because your global environment is still unchanged.

Useful options

Improve isolation

As said before, your sandboxed python sys.path still references the global system path. You can however hide them by:

either use the --no-site-packages that do not give access to the global site-packages directory to the sandbox
or change your PYTHONPATH in my-sandbox/bin/activate in the same way as for PATH (see tips)

$ virtualenv --no-site-packages closedPy
$ sed -i '9i PYTHONPATH="$_OLD_PYTHON_PATH"
      9i export PYTHONPATH
      9i unset _OLD_PYTHON_PATH
      40i _OLD_PYTHON_PATH="$PYTHONPATH"
      40i PYTHONPATH="."
      40i export PYTHONPATH' closedPy/bin/activate
$ source closedPy/bin/activate
(closedPy)$ python -c 'import sys; print "\n".join(sys.path)'
/home/you/some/where/closedPy/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg
/home/you/some/where/closedPy
/home/you/some/where/closedPy/lib/python2.5
/home/you/some/where/closedPy/lib/python2.5/plat-linux2
/home/you/some/where/closedPy/lib/python2.5/lib-tk
/home/you/some/where/closedPy/lib/python2.5/lib-dynload
/usr/lib/python2.5
/usr/lib64/python2.5
/usr/lib/python2.5/lib-tk
/home/you/some/where/closedPy/lib/python2.5/site-packages
$ deactivate

This way, you'll get an even more isolated sandbox, just as with a brand new python environment.

Work with different versions of Python

It is possible to dedicate a sandbox to a particular version of python by using the --python=PYTHON_EXE which specifies the interpreter that virtualenv was installed with (default is /usr/bin/python):

$ virtualenv --python=python2.4 pyver24
$ source pyver24/bin/activate
(pyver24)$ python -V
Python 2.4.6
$ deactivate
$ virtualenv --python=python2.5 pyver25
$ source pyver25/bin/activate
(pyver25)$ python -V
Python 2.5.2
$ deactivate

Distribute a sandbox

To distribute your sandbox, you must use the --relocatable option that makes an existing sandbox relocatable. This fixes up scripts and makes all .pth files relative This option should be called just before you distribute the sandbox (each time you have changed something in your sandbox).

An important point is that the host system should be similar to your own.

Tips

Speed up sandbox manipulation

Add these scripts to your .bashrc in order to help you using virtualenv and automate the creation and activation processes.

rel2abs() {
#from http://unix.derkeiler.com/Newsgroups/comp.unix.programmer/2005-01/0206.html
  [ "$#" -eq 1 ] || return 1
  ls -Ld -- "$1" > /dev/null || return
  dir=$(dirname -- "$1" && echo .) || return
  dir=$(cd -P -- "${dir%??}" && pwd -P && echo .) || return
  dir=${dir%??}
  file=$(basename -- "$1" && echo .) || return
  file=${file%??}
  case $dir in
    /) printf '%s\n' "/$file";;
    /*) printf '%s\n' "$dir/$file";;
    *) return 1;;
  esac
  return 0
}
function activate(){
    if [[ "$1" == "--help" ]]; then
        echo -e "usage: activate PATH\n"
        echo -e "Activate the sandbox where PATH points inside of.\n"
        return
    fi
    if [[ "$1" == '' ]]; then
        local target=$(pwd)
    else
        local target=$(rel2abs "$1")
    fi
    until  [[ "$target" == '/' ]]; do
        if test -e "$target/bin/activate"; then
            source "$target/bin/activate"
            echo "$target sandbox activated"
            return
        fi
        target=$(dirname "$target")
    done
    echo 'no sandbox found'
}
function mksandbox(){
    if [[ "$1" == "--help" ]]; then
        echo -e "usage: mksandbox NAME\n"
        echo -e "Create and activate a highly isaolated sandbox named NAME.\n"
        return
    fi
    local name='sandbox'
    if [[ "$1" != "" ]]; then
        name="$1"
    fi
    if [[ -e "$1/bin/activate" ]]; then
        echo "$1 is already a sandbox"
        return
    fi
    virtualenv --no-site-packages --clear --distribute "$name"
    sed -i '9i PYTHONPATH="$_OLD_PYTHON_PATH"
            9i export PYTHONPATH
            9i unset _OLD_PYTHON_PATH
           40i _OLD_PYTHON_PATH="$PYTHONPATH"
           40i PYTHONPATH="."
           40i export PYTHONPATH' "$name/bin/activate"
    activate "$name"
}

Note:: The virtualenv-commands and virtualenvwrapper projects add some very interesting features to virtualenv. So, put on eye on them for more advanced features than the above ones.

Conclusion

I found it to be irreplaceable for testing new configurations or working on projects with different dependencies. Moreover, I use it to learn about other python projects, how my project exactly interacts with its dependencies (during debugging) or to test the final user experience.

All of this stuff can be done without virtualenv but not in such an easy and secure way.

I will continue the series by introducing other useful projects to enhance your productivity : pip and Distribute. See you soon.

Astng 0.20.0 and Pylint 0.20.0 releases

Emile Anclin — 2010-03-24T11:59-01:00

We are happy to announce the Astng 0.20.0 and Pylint 0.20.0 releases.

Pylint is a static code checker based on Astng, both depending on logilab-common 0.49.

Astng

Astng 0.20.0 is a major refactoring: instead of parsing and modifying the syntax tree generated from python's _ast or compiler.ast modules, the syntax tree is rebuilt. Thus the code becomes much clearer, and all monkey patching will eventually disappear from this module.

Speed improvement is achieved by caching the parsed modules earlier to avoid double parsing, and avoiding some repeated inferences, all along fixing a lot of important bugs.

Pylint

Pylint 0.20.0 uses the new Astng, and fixes a lot of bugs too, adding some new functionality:

parameters with leading "_" shouldn't count as "local" variables
warn on assert( a, b )
warning if return or break inside a finally
specific message for NotImplemented exception

We would like to thank Chmouel Boudjnah, Johnson Fletcher, Daniel Harding, Jonathan Hartley, Colin Moris, Winfried Plapper, Edward K. Ream and Pierre Rouleau for their contributions, and all other people helping the project to progress.

pylint bugs day #2 on april 16, 2010

Sylvain Thenault — 2010-03-22T14:08-01:00

Hey guys,

we'll hold the next pylint bugs day on april 16th 2010 (friday). If some of you want to come and work with us in our Paris office, you'll be much welcome.

Else you can still join us on jabber / irc:

jabber: chat room public@jabber.logilab.org
irc: #public on irc://irc.logilab.org

See you then!