blog entries (Logilab.org) RSS Feed

Discovering logilab-common Part 1 - deprecation module

Stéphanie Marcu — 2010-09-02T12:42-01:00

logilab-common library contains a lot of utilities which are often unknown. I will write a series of blog entries to explore nice features of this library.

We will begin with the logilab.common.deprecation module which contains utilities to warn users when:

a function or a method is deprecated

a class has been moved into another module

a class has been renamed

a callable has been moved to a new module

deprecated

When a function or a method is deprecated, you can use the deprecated decorator. It will print a message to warn the user that the function is deprecated.

The decorator takes two optional arguments:

reason: the deprecation message. A good practice is to specify at the beginning of the message, between brackets, the version number from which the function is deprecated. The default message is 'The function "[function name]" is deprecated'.

stacklevel: This is the option of the warnings.warn function which is used by the decorator. The default value is 2.

We have a class Person defined in a file person.py. The get_surname method is deprecated, we must use the get_lastname method instead. For that, we use the deprecated decorator on the get_surname method.

from logilab.common.deprecation import deprecated

class Person(object):

    def __init__(self, firstname, lastname):
        self._firstname = firstname
        self._lastname = lastname

    def get_firstname(self):
        return self._firstname

    def get_lastname(self):
        return self._lastname

    @deprecated('[1.2] use get_lastname instead')
    def get_surname(self):
        return self.get_lastname()

def create_user(firstname, lastname):
    return Person(firstname, lastname)

if __name__ == '__main__':
    person = create_user('Paul', 'Smith')
    surname = person.get_surname()

When running person.py we have the message below:

person.py:22: DeprecationWarning: [1.2] use get_lastname instead

surname = person.get_surname()

class_moved

Now we moved the class Person in a new_person.py file. We notice in the person.py file that the class has been moved:

from logilab.common.deprecation import class_moved
import new_person
Person = class_moved(new_person.Person)

if __name__ == '__main__':
    person = Person('Paul', 'Smith')

When we run the person.py file, we have the following message:

person.py:6: DeprecationWarning: class Person is now available as new_person.Person

person = Person('Paul', 'Smith')

The class_moved function takes one mandatory argument and two optional:

new_class: this mandatory argument is the new class

old_name: this optional argument specify the old class name. By default it is the same name than the new class. This argument is used in the default printed message.

message: with this optional argument, you can specify a custom message

class_renamed

The class_renamed function automatically creates a class which fires a DeprecationWarning when instantiated.

The function takes two mandatory arguments and one optional:

old_name: a string which contains the old class name

new_class: the new class

message: an optional message. The default one is '[old class name] is deprecated, use [new class name]'

We now rename the Person class into User class in the new_person.py file. Here is the new person.py file:

from logilab.common.deprecation import class_renamed
from new_person import User

Person = class_renamed('Person', User)

if __name__ == '__main__':
    person = Person('Paul', 'Smith')

When running person.py, we have the following message:

person.py:5: DeprecationWarning: Person is deprecated, use User

person = Person('Paul', 'Smith')

moved

The moved function is used to tell that a callable has been moved to a new module. It returns a callable wrapper, so that when the wrapper is called, a warning is printed telling where the object can be found. Then the import is done (and not before) and the actual object is called.

Note

The usage is somewhat limited on classes since it will fail if the wrapper is used in a class ancestors list: use the class_moved function instead (which has no lazy import feature though).

The moved function takes two mandatory parameters:

modpath: a string representing the path to the new module

objname: the name of the new callable

We will use in person.py, the create_user function which is now defined in the new_person.py file:

from logilab.common.deprecation import moved

create_user = moved('new_person', 'create_user')

if __name__ == '__main__':
    person = create_user('Paul', 'Smith')

When running person.py, we have the following message:

person.py:4: DeprecationWarning: object create_user has been moved to module new_person

person = create_user('Paul', 'Smith')

pdb.set_trace no longer working: problem solved

Alexandre Fayolle — 2010-08-12T16:17-01:00

I had a bad case of bug hunting today which took me > 5 hours to track down (with the help of Adrien in the end).

I was trying to start a CubicWeb instance on my computer, and was encountering some strange pyro error at startup. So I edited some source file to add a pdb.set_trace() statement and restarted the instance, waiting for Python's debugger to kick in. But that did not happen. I was baffled. I first checked for standard problems:

no pdb.py or pdb.pyc was lying around in my Python sys.path
the pdb.set_trace function had not been silently redefined
no other thread was bugging me
the standard input and output were what they were supposed to be
I was not able to reproduce the issue on other machines

After triple checking everything, grepping everywhere, I asked a question on StackOverflow before taking a lunch break (if you go there, you'll see the answer). After lunch, no useful answer had come in, so I asked Adrien for help, because two pairs of eyes are better than one in some cases. We dutifully traced down the pdb module's code to the underlying bdb and cmd modules and learned some interesting things on the way down there. Finally, we found out that the Python code frames which should have been identical where not. This discovery caused further bafflement. We looked at the frames, and saw that one of those frames's class was a psyco generated wrapper.

It turned out that CubicWeb can use two implementation of the RQL module: one which uses gecode (a C++ library for constraint based programming) and one which uses logilab.constraint (a pure python library for constraint solving). The former is the default, but it would not load on my computer, because the gecode library had been replaced by a more recent version during an upgrade. The pure python implementation tries to use psyco to speed up things. Installing the correct version of libgecode solved the issue. End of story.

When I checked out StackOverflow, Ned Batchelder had provided an answer. I didn't get the satisfaction of answering the question myself...

Once this was figured out, solving the initial pyro issue took 2 minutes...

EuroSciPy'10

Adrien Chauve — 2010-07-13T14:12-01:00

The EuroSciPy2010 conference was held in Paris at the Ecole Normale Supérieure from July 8th to 11th and was organized and sponsored by Logilab and other companies.

July, 8-9: Tutorials

The first two days were dedicated to tutorials and I had the chance to talk about SciPy with André Espaze, Gaël Varoquaux and Emanuelle Gouillart in the introductory track. This was nice but it was a bit tricky to present SciPy in such a short time while trying to illustrate the material with real and interesting examples. One very nice thing for the introductory track is that all the material was contributed by different speakers and is freely available in a github repository (licensed under CC BY).

July, 10-11: Scientific track

The next two days were dedicated to scientific presentations and why python is such a great tool to develop scientific software and carry out research.

Keynotes

I had a very great time listening to the presentations, starting with the two very nice keynotes given by Hans Petter Langtangen and Konrad Hinsen. The latter gave us a very nice summary of what happened in the scientific python world during the past 15 years, what is happening now and of course what could happen during the next 15 years. Using a crystal ball and a very humorous tone, he made it very clear that the challenge in the next years will be about how using our hundreds, thousands or even more cores in a bug-free and efficient way. Functional programming may be a very good solution to this challenge as it provides a deterministic way of parallelizing our programs. Konrad also provided some hints about future versions of python that could provide a deeper and more efficient support of functional programming and maybe the addition of a keyword 'async' to handle the computation of a function in another core.

In fact, the PEP 3148 entitled "Futures - execute computations asynchronously" was just accepted two days ago. This PEP describes the new package called "futures" designed to facilitate the evaluation of callables using threads and processes in future versions of python. A full implementation is already available.

Parallelization

Parallelization was indeed a very popular issue across presentations, and as for resolving embarrassingly parallel problems, several solutions were presented.

Playdoh: Distributes computations over computers connected to a secure network (see playdoh presentation).

Distributing the computation of a function over two machines is as simple as:
```
import playdoh
result1, result2 = playdoh.map(fun, [arg1, arg2], _machines = ['machine1.network.com', 'machine2.network.com'])
```
Theano: Allows to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. In particular it can use GPU transparently and generate optimized C code (see theano presentation).
joblib: Provides among other things helpers for embarrassingly parallel problems. It's built over the multiprocessing package introduced in python 2.6 and brings more readable code and easier debugging.

Speed

Concerning speed, Fransesc Alted has showed us interesting tools for memory optimization currently successfully used in PyTables 2.2. You can read more details on these kind of optimizations in EuroSciPy'09 (part 1/2): The Need For Speed.

SCons

Last but not least, I talked with Cristophe Pradal who is one of the core developer of OpenAlea. He convinced me that SCons is worth using once you have built a nice extension for it: SConsX. I'm looking forward to testing it.

HOWTO install lodgeit pastebin under Debian/Ubuntu

Arthur Lutz — 2010-06-24T16:46-01:00

Lodge it is a simple open source pastebin... and it's written in Python!

The installation under debian/ubuntu goes as follows:

sudo apt-get update
sudo apt-get -uVf install python-imaging python-sqlalchemy python-jinja2 python-pybabel python-werkzeug python-simplejson
cd local
hg clone http://dev.pocoo.org/hg/lodgeit-main
cd lodgeit-main
vim manage.py

For debian squeeze you have to downgrade python-werkzeug, so get the old version of python-werkzeug from snapshot.debian.org at http://snapshot.debian.org/package/python-werkzeug/0.5.1-1/

wget http://snapshot.debian.org/archive/debian/20090808T041155Z/pool/main/p/python-werkzeug/python-werkzeug_0.5.1-1_all.deb

Modify the dburi and the SECRET_KEY. And launch application:

python manage.py runserver

Then off you go configure your apache or lighthttpd.

An easy (and dirty) way of running it at startup is to add the following command to the www-data crontab

@reboot cd /tmp/; nohup /usr/bin/python /usr/local/lodgeit-main/manage.py runserver &

This should of course be done in an init script.

Hopefully we'll find some time to package this nice webapp for debian/ubuntu.

EuroSciPy 2010 schedule is out !

Nicolas Chauvat — 2010-06-06T19:12-01:00

The EuroSciPy 2010 conference will be held in Paris from july 8th to 11th at Ecole Normale Supérieure. Two days of tutorials, two days of conference, two interesting keynotes, a lightning talk session, an open space for collaboration and sprinting, thirty quality talks in the schedule and already 100 delegates registered.

If you are doing science and using Python, you want to be there!

Salomé accepted into Debian unstable

Andre Espaze — 2010-06-03T18:28-01:00

Salomé is a platform for pre and post-processing of numerical simulation available at http://salome-platform.org/. It is now available as a Debian package http://packages.debian.org/source/sid/salome and should soon appear in Ubuntu https://launchpad.net/ubuntu/+source/salome as well.

A difficult packaging work

A first package of Salomé 3 was made by the courageous Debian developper Adam C. Powell, IV on January 2008. Such packaging is very resources intensive because of the building of many modules. But the most difficult part was to bring Salomé to an unported environment. Even today, Salomé 5 binaries are only provided by upstream as a stand-alone piece of software ready to unpack on a Debian Sarge/Etch or a Mandriva 2006/2008. This is the first reason why several patches were required for adapting the code to new versions of the dependencies. The version 3 of Salomé was so difficult and time consuming to package that Adam decided to stop during two years.

The packaging of Salomé started back with the version 5.1.3 in January 2010. Thanks to Logilab and the OpenHPC project, I could join him during 14 weeks of work for adapting every module to Debian unstable. Porting to the new versions of the dependencies was a first step, but we had also to adapt the code to the Debian packaging philosophy with binaries, librairies and data shipped to dedicated directories.

A promising future

Salomé being accepted to Debian unstable means that porting it to Ubuntu should follow in a near future. Moreover the work done for adapting Salomé to a GNU/Linux distribution may help developpers on others platforms as well.

That is excellent news for all people involved in numerical simulation because they are going to have access to Salomé services by using their packages management tools. It will help the spreading of Salomé code on any fresh install and moreover keep it up to date.

Join the fun

For mechanical engineers, a derived product called Salomé-Méca has recently been published. The goal is to bring the functionalities from the Code Aster finite element solver to Salomé in order to ease simulation workflows. If you are as well interested in Debian packages for those tools, you are invited to come with us and join the fun.

I have submitted a proposal to talk about Salomé at EuroSciPy 2010. I look forward to meet other interested parties during this conference that will take place in Paris on July 8th-11th.

Une mise en place de l’eXtreme Programming - ce que j'en retiens

Arthur Lutz — 2010-05-20T12:19-01:00

Je suis allé à la présentation de "Une mise en place de l’eXtreme Programming" présenté par Karine Sabatier dans le cadre d'Agile Nantes. On y a parlé plutôt Ruby on Rails que python, mais surtout de méthodes agiles et XP.

Voici quelques points que j'ai retenu de cette présentation tout à fait pragmatique d'une mise en pratique des principes XP :

Le principe de "Convention over configuration" : préférer la convention (notamment pour la programmation) plutôt que la contrainte par la configuration. Dans Ruby On Rails, les conventions sont très fortes, pour faire une application on ne peut pas s'éloigner du modèle MVC : les modèles sont dans "model", les views sont dans "views" etc... À ce sujet, la conférencière a fait référence à deux types de designs que je ne connaissait pas : le DDD Domain-driven Design et Behavior Driven Development.
Utilisation de métaphores : trouver un langage commun avec le client mais aussi avec les utilisateurs
Application de déploiement ruby on rails : Capistrano, bien qu'à Logilab nous privilégions le déploiement par paquets et dépôts debian, en python on pourra jeter un coup d'œil à Fabric.

Leur projet XP utilisait le logiciel de gestion de projet Retrospectiva. Celui-ci ressemble sur bien des points à JPL (Jeux de Planification Logiciel) disponible sur le framework CubicWeb (http://www.cubicweb.org). Coté intégration continue : CruiseControl , en python nous privilégions apycot.
Ce projet a essayé l'utilisation de Selenium pour ces tests web. Le constat est le même que chez Logilab : la première fois que ca marche c'est utile et apporte une grande satisfaction, dans un deuxième temps ca reste très difficile à maintenir. Nous regardons à présent plutôt du coté de Windmill qui a été intégré à la version 3.9 de cubicweb.
Une mention a été fait d'une société fonctionnement uniquement en mode agile Pyxis.

Enable and disable encrypted swap - Ubuntu

Arthur Lutz — 2010-05-18T16:31-01:00

With the release of Ubuntu Lucid Lynx, the use of an encrypted /home is becoming a pretty common and simple to setup thing. This is good news for privacy reasons obviously. The next step which a lot of users are reluctant to accomplish is the use of an encrypted swap. One of the most obvious reasons is that in most cases it breaks the suspend and hibernate functions.

Here is a little HOWTO on how to switch from normal swap to encrypted swap and back. That way, when you need a secure laptop (trip to a conference, or situtation with risk of theft) you can active it, and then deactivate it when you're at home for example.

Turn it on

That is pretty simple

sudo ecryptfs-setup-swap

Turn it off

The idea is to turn off swap, remove the ecryptfs layer, reformat your partition with normal swap and enable it. We use sda5 as an example for the swap partition, please use your own (fdisk -l will tell you which swap partition you are using - or in /etc/crypttab)

sudo swapoff -a
sudo cryptsetup remove /dev/mapper/cryptswap1
sudo vim /etc/crypttab
*remove the /dev/sda5 line*
sudo /sbin/mkswap /dev/sda5
sudo swapon /dev/sda5
sudo vim /etc/fstab
*replace /dev/mapper/cryptswap1 with /dev/sda5*

If this is is useful, you can probably stick it in a script to turn on and off... maybe we could get an ecryptfs-unsetup-swap into ecryptfs.

The DEBSIGN_KEYID trick

Nicolas Chauvat — 2010-05-12T18:23-01:00

I have been wondering for some time why debsign would not use the DEBSIGN_KEYID environment variable that I exported from my bashrc. Debian bug 444641 explains the trick: debsign ignores environment variables and sources ~/.devscripts instead. A simple export DEBSIGN_KEYID=ABCDEFG in ~/.devscripts is enough to get rid of the -k argument once and for good.

Sprint CubicWeb chez Logilab - annonce de dernière minute

Arthur Lutz — 2010-04-29T18:50-01:00

Logilab est en ce moment en train d'acceuillir un sprint autour de la plateforme CubicWeb. L'objectif principal de ce sprint de 5 jours est d'améliorer l'usage de javascript et des css dans CubicWeb :

avoir une API javascript propre, testée et documentée
pouvoir facilement changer le style d'une application cubicweb
gestion de bundle pour javascript et CSS
une documentation sur les standards d'écriture des fichiers JS et CSS pour cubicweb

Ce sprint aura lieu du jeudi 29 avril 2010 au 5 mai 2010 (weekend exlus - les bureaux seront fermés). Vous êtes les bienvenus pour contribuer, filer un coup de main, faire du pair programming... ou simplement rencontrer des développeurs cubicweb. Vous pouvez même venir une après-midi ou une seule journée. Pour ceux avec des portables, il y aura du réseau disponible pour être connecté.

Adresse : 104 Boulevard Auguste-Blanqui, Paris. Sonnez à "Logilab".

Métro : St Jacques or Corvisart (Glacière est la station la plus proche mais sera fermée à partir de lundi)

Contact : http://www.logilab.fr/contact

Dates : du 29/04/2010 au 30/04/2010 et du 03/05/2010 au 05/05/2010

pylint bug days #2 report

Sylvain Thenault — 2010-04-19T17:12-01:00

First of all, I've to say that pylint bugs day wasn't that successful in term of 'community event': I've been sprinting almost alone. My Logilab's felows were tied to customer projects, and no outside people shown up on jabber. Fortunatly Tarek Ziade came to visit us, and that was a nice opportunity to talk about pylint, distribute, etc ... Thank you Tarek, you saved my day ;)

As I felt a bit alone, I decided to work on somethings funnier than bug fixing: refactoring!

First, I've greatly simplified the command line: enable-msg/enable-msg-cat/enable-checker/enable-report and their disable-* counterparts were all merged into single --enable/--disable options.

I've also simplified "pylint --help" output, providing a --long-help option to get what we had before. Generic support in `logilab.common.configuration of course.

And last but not least, I refactored pylint so we can have multiple checkers with the same name. The idea behind this is that we can split checker into smaller chunks, basically only responsible for one or a few related messages. When pylint runs, it only uses necessary checkers according to activated messages and reports. When all checkers will be splitted, it should improve performance of "pylint --error-only".

So, I can say I'm finally happy with the results of that pylint bugs day! And hopefuly we will be more people for the next edition...

Virtualenv - Play safely with a Python

Alain Leufroy — 2010-03-26T10:19-01:00

Plan

virtualenv, pip and Distribute are tree tools that help developers and packagers. In this short presentation we will see some virtualenv capabilities.

Please, keep in mind that all above stuff has been made using : Debian Lenny, python 2.5 and virtualenv 1.4.5.

Abstract

virtualenv builds python sandboxes where it is possible to do whatever you want as a simple user without putting in jeopardy your global environment.

virtualenv allows you to safety:

install any python packages
add debug lines everywhere (not only in your scripts)
switch between python versions
try your code as you are a final user
and so on ...

Install and usage

Install

Prefered way

Just download the virtualenv python script at http://bitbucket.org/ianb/virtualenv/raw/tip/virtualenv.py and call it using python (e.g. python virtualenv.py).

For conveinience, we will refers to this script using virtualenv.

Other ways

For Debian (ubuntu as well) addicts, just do :

$ sudo aptitude install python-virtualenv

Fedora users would do:

$ sudo yum install python-virtualenv

And others can install from PyPI (as superuser):

$ pip install virtualenv

$ easy_install pip && pip install virtualenv

You could also get the source here.

Quick Guide

To work in a python sandbox, do as follow:

$ virtualenv my_py_env
$ source my_py_env/bin/activate
(my_py_env)$ python

"That's all Folks !"

Once you have finished just do:

(my_py_env)$ deactivate

or quit the tty.

What does `virtualenv` actually do ?

At creation time

Let's start again ... more slowly. Consider the following environment:

$ pwd
/home/you/some/where
$ ls

Now create a sandbox called my-sandbox:

$ virtualenv my-sandbox
New python executable in "my-sandbox/bin/python"
Installing setuptools............done.

The output said that you have a new python executable and specific install tools. Your current directory now looks like:

$ ls -Cl
my-sandbox/ README
$ tree -L 3 my-sandbox
my-sandbox/
|-- bin
|   |-- activate
|   |-- activate_this.py
|   |-- easy_install
|   |-- easy_install-2.5
|   |-- pip
|   `-- python
|-- include
|   `-- python2.5 -> /usr/include/python2.5
`-- lib
    `-- python2.5
        |-- ...
        |-- orig-prefix.txt
        |-- os.py -> /usr/lib/python2.5/os.py
        |-- re.py -> /usr/lib/python2.5/re.py
        |-- ...
        |-- site-packages
        |   |-- easy-install.pth
        |   |-- pip-0.6.3-py2.5.egg
        |   |-- setuptools-0.6c11-py2.5.egg
        |   `-- setuptools.pth
        |-- ...

In addition to the new python executable and the install tools you have an whole new python environment containing libraries, a site-packages/ (where your packages will be installed), a bin directory, ...

Note:: virtualenv does not create every file needed to get a whole new python environment. It uses links to global environment files instead in order to save disk space end speed up the sandbox creation. Therefore, there must already have an active python environment installed on your system.

At activation time

At this point you have to activate the sandbox in order to use your custom python. Once activated, python still has access to the global environment but will look at your sandbox first for python's modules:

$ source my-sandbox/bin/activate
(my-sandbox)$ which python
/home/you/some/where/my-sandbox/bin/python
$ echo $PATH
/home/you/some/where/my-sandbox/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
(pyver)$ python -c 'import sys;print sys.prefix;'
/home/you/some/where/my-sandbox
(pyver)$ python -c 'import sys;print "\n".join(sys.path)'
/home/you/some/where/my-sandbox/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg
[...]
/home/you/some/where/my-sandbox
/home/you/personal/PYTHONPATH
/home/you/some/where/my-sandbox/lib/python2.5/
[...]
/usr/lib/python2.5
[...]
/home/you/some/where/my-sandbox/lib/python2.5/site-packages
[...]
/usr/local/lib/python2.5/site-packages
/usr/lib/python2.5/site-packages
[...]

First of all, a (my-sandbox) message is automatically added to your prompt in order to make it clear that you're using a python sandbox environment.

Secondly, my-sandbox/bin/ is added to your PATH. So, running python calls the specific python executable placed in my-sandbox/bin.

Note: It is possible to improve the sandbox isolation by ignoring the global paths and your PYTHONPATH (see Improve isolation section).

Installing package

It is possible to install any packages in the sandbox without any superuser privilege. For instance, we will install the pylint development revision in the sandbox.

Suppose that you have the pylint stable version already installed in your global environment:

(my-sandbox)$ deactivate
$ python -c 'from pylint.__pkginfo__ import version;print version'
0.18.0

Once your sandbox activated, install the development revision of pylint as an update:

$ source /home/you/some/where/my-sandbox/bin/activate
(my-sandbox)$ pip install -U hg+http://www.logilab.org/hg/pylint#egg=pylint-0.19

The new package and its dependencies are only installed in the sandbox:

(my-sandbox)$ python -c 'import pylint.__pkginfo__ as p;print p.version, p.__file__'
0.19.0 /home/you/some/where/my-sandbox/lib/python2.6/site-packages/pylint/__pkginfo__.pyc
(my-sandbox)$ deactivate
$ python -c 'import pylint.__pkginfo__ as p;print p.version, p.__file__'
0.18.0 /usr/lib/pymodules/python2.6/pylint/__pkginfo__.pyc

You can safely do any change in the new pylint code or in others sandboxed packages because your global environment is still unchanged.

Useful options

Improve isolation

As said before, your sandboxed python sys.path still references the global system path. You can however hide them by:

either use the --no-site-packages that do not give access to the global site-packages directory to the sandbox
or change your PYTHONPATH in my-sandbox/bin/activate in the same way as for PATH (see tips)

$ virtualenv --no-site-packages closedPy
$ sed -i '9i PYTHONPATH="$_OLD_PYTHON_PATH"
      9i export PYTHONPATH
      9i unset _OLD_PYTHON_PATH
      40i _OLD_PYTHON_PATH="$PYTHONPATH"
      40i PYTHONPATH="."
      40i export PYTHONPATH' closedPy/bin/activate
$ source closedPy/bin/activate
(closedPy)$ python -c 'import sys; print "\n".join(sys.path)'
/home/you/some/where/closedPy/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg
/home/you/some/where/closedPy
/home/you/some/where/closedPy/lib/python2.5
/home/you/some/where/closedPy/lib/python2.5/plat-linux2
/home/you/some/where/closedPy/lib/python2.5/lib-tk
/home/you/some/where/closedPy/lib/python2.5/lib-dynload
/usr/lib/python2.5
/usr/lib64/python2.5
/usr/lib/python2.5/lib-tk
/home/you/some/where/closedPy/lib/python2.5/site-packages
$ deactivate

This way, you'll get an even more isolated sandbox, just as with a brand new python environment.

Work with different versions of Python

It is possible to dedicate a sandbox to a particular version of python by using the --python=PYTHON_EXE which specifies the interpreter that virtualenv was installed with (default is /usr/bin/python):

$ virtualenv --python=python2.4 pyver24
$ source pyver24/bin/activate
(pyver24)$ python -V
Python 2.4.6
$ deactivate
$ virtualenv --python=python2.5 pyver25
$ source pyver25/bin/activate
(pyver25)$ python -V
Python 2.5.2
$ deactivate

Distribute a sandbox

To distribute your sandbox, you must use the --relocatable option that makes an existing sandbox relocatable. This fixes up scripts and makes all .pth files relative This option should be called just before you distribute the sandbox (each time you have changed something in your sandbox).

An important point is that the host system should be similar to your own.

Tips

Speed up sandbox manipulation

Add these scripts to your .bashrc in order to help you using virtualenv and automate the creation and activation processes.

rel2abs() {
#from http://unix.derkeiler.com/Newsgroups/comp.unix.programmer/2005-01/0206.html
  [ "$#" -eq 1 ] || return 1
  ls -Ld -- "$1" > /dev/null || return
  dir=$(dirname -- "$1" && echo .) || return
  dir=$(cd -P -- "${dir%??}" && pwd -P && echo .) || return
  dir=${dir%??}
  file=$(basename -- "$1" && echo .) || return
  file=${file%??}
  case $dir in
    /) printf '%s\n' "/$file";;
    /*) printf '%s\n' "$dir/$file";;
    *) return 1;;
  esac
  return 0
}
function activate(){
    if [[ "$1" == "--help" ]]; then
        echo -e "usage: activate PATH\n"
        echo -e "Activate the sandbox where PATH points inside of.\n"
        return
    fi
    if [[ "$1" == '' ]]; then
        local target=$(pwd)
    else
        local target=$(rel2abs "$1")
    fi
    until  [[ "$target" == '/' ]]; do
        if test -e "$target/bin/activate"; then
            source "$target/bin/activate"
            echo "$target sandbox activated"
            return
        fi
        target=$(dirname "$target")
    done
    echo 'no sandbox found'
}
function mksandbox(){
    if [[ "$1" == "--help" ]]; then
        echo -e "usage: mksandbox NAME\n"
        echo -e "Create and activate a highly isaolated sandbox named NAME.\n"
        return
    fi
    local name='sandbox'
    if [[ "$1" != "" ]]; then
        name="$1"
    fi
    if [[ -e "$1/bin/activate" ]]; then
        echo "$1 is already a sandbox"
        return
    fi
    virtualenv --no-site-packages --clear --distribute "$name"
    sed -i '9i PYTHONPATH="$_OLD_PYTHON_PATH"
            9i export PYTHONPATH
            9i unset _OLD_PYTHON_PATH
           40i _OLD_PYTHON_PATH="$PYTHONPATH"
           40i PYTHONPATH="."
           40i export PYTHONPATH' "$name/bin/activate"
    activate "$name"
}

Note:: The virtualenv-commands and virtualenvwrapper projects add some very interesting features to virtualenv. So, put on eye on them for more advanced features than the above ones.

Conclusion

I found it to be irreplaceable for testing new configurations or working on projects with different dependencies. Moreover, I use it to learn about other python projects, how my project exactly interacts with its dependencies (during debugging) or to test the final user experience.

All of this stuff can be done without virtualenv but not in such an easy and secure way.

I will continue the series by introducing other useful projects to enhance your productivity : pip and Distribute. See you soon.

Astng 0.20.0 and Pylint 0.20.0 releases

Emile Anclin — 2010-03-24T11:59-01:00

We are happy to announce the Astng 0.20.0 and Pylint 0.20.0 releases.

Pylint is a static code checker based on Astng, both depending on logilab-common 0.49.

Astng

Astng 0.20.0 is a major refactoring: instead of parsing and modifying the syntax tree generated from python's _ast or compiler.ast modules, the syntax tree is rebuilt. Thus the code becomes much clearer, and all monkey patching will eventually disappear from this module.

Speed improvement is achieved by caching the parsed modules earlier to avoid double parsing, and avoiding some repeated inferences, all along fixing a lot of important bugs.

Pylint

Pylint 0.20.0 uses the new Astng, and fixes a lot of bugs too, adding some new functionality:

parameters with leading "_" shouldn't count as "local" variables
warn on assert( a, b )
warning if return or break inside a finally
specific message for NotImplemented exception

We would like to thank Chmouel Boudjnah, Johnson Fletcher, Daniel Harding, Jonathan Hartley, Colin Moris, Winfried Plapper, Edward K. Ream and Pierre Rouleau for their contributions, and all other people helping the project to progress.

pylint bugs day #2 on april 16, 2010

Sylvain Thenault — 2010-03-22T14:08-01:00

Hey guys,

we'll hold the next pylint bugs day on april 16th 2010 (friday). If some of you want to come and work with us in our Paris office, you'll be much welcome.

Else you can still join us on jabber / irc:

jabber: chat room public@jabber.logilab.org
irc: #public on irc://irc.logilab.org

See you then!

PostgreSQL on windows : plpythonu and "specified module could not be found" error

Alexandre Fayolle — 2010-03-22T13:37-01:00

I recently had to (remotely) debug an issue on windows involving PostgreSQL and PL/Python. Basically, two very similar computers, with Python2.5 installed via python(x,y), PostgreSQL 8.3.8 installed via the binary installer. On the first machine create language plpythonu; worked like a charm, and on the other one, it failed with C:\\Program Files\\Postgresql\\8.3\\plpython.dll: specified module could not be found. This is caused by the dynamic linker not finding some DLL. Using Depends.exe showed that plpython.dll looks for python25.dll (the one it was built against in the 8.3.8 installer), but that the DLL was there.

I'll save the various things we tried and jump directly to the solution. After much head scratching, it turned out that the first computer had TortoiseHg installed. This caused C:\\Program Files\\TortoiseHg to be included in the System PATH environment variable, and that directory contains python25.dll. On the other hand C:\\Python25 was in the user's PATH environment variable on both computers. As the database Windows service runs using a dedicated local account (typically with login postgres), it would not have C:\\Python25 in its PATH, but if TortoiseHg was there, it would find the DLL in some other directory. So the solution was to add C:\\Python25 to the system PATH.

L'intégration continue présenté par Agiles Nantes

Arthur Lutz — 2010-03-18T09:59-01:00

Hier, Logilab était à nouveau présent aux rencontres organisées par Agiles Nantes, il s'agissait d'une présentation très fournie sur l'intégration continue (présenté par la société Netapsys). Malheureusement, la principale technologie utilisée était Java dont nous ne sommes pas des grands fans, préférant python. Néanmoins cela donne une bonne idée des possibilités qu'offrent les outils autour du développement java, notamment en terme d'intégration continue (voir Maven, Hudson, Sonar, etc.).

On retrouve donc un certain nombre de similarités avec le monde python, notamment avec Artifactory qui reproduit en partie les fonctionnalités de pypi avec easy_install ou buildout ou apt-cacher-ng dans son coté proxy. A Logilab, nous privilégions l'utilisation de paquets debian pour distribuer nos logiciels, ce qui permet, notamment, de ne pas réinventer la roue pour chaque langage de programmation.

Voici quelques une des notions avancées lors de la présentation qui nous semblent intéressantes sur l'intégration continue :

celle-ci permet de mettre en place des environnements de test mis à disposition du client tout le long du projet.
cette notion de prototype utilisable en continu doit être présente le plus tôt possible dans un projet.
idéalement, un serveur de déploiement/intégration doit être quasiment à l'identique de l'environnement client (utilisation de machines virtuelles)
l'intégration continue est souvent plus utilisées par les développeurs et les chefs de projets que par les clients. Mettre en place des rapports utiles au client requiert un soin particulier
pour une émulation collective, certaines notations des développeurs sont possible. Par exemple un plugin Hudson donne un point par build réussi, un point par test ajouté, moins dix points pour un build cassé.
la visualisation des données peut motiver les développeurs, l'outil Sonar semble être très complet et propose des visualisation assez léchées. A noter, des graphes sur la complexité du code, par classe, par méthode etc. Voir aussi la visualisation de l'arbre des dépendances des librairies avec Radiator.

J'y ai mentionné apycot et buildbot qui sont tous les deux des outils plutôt orientés python (mais pas seulement).

Pour conclure, tout ça m'a fait penser au concept plus poussé de "Continuous Delivery - BlueGreenDelivery" développé par Martin Fowler, une lecture recommandée pour ceux qui ont déjà pris le pas de l'intégration continue.

Now publishing blog entries under creative commons

Arthur Lutz — 2010-03-15T12:57-01:00

Logilab is proud to announce that the blog entries published on the blogs of http://www.logilab.org and http://www.cubicweb.org are now licensed under a Creative Commons Attribution-Share Alike 2.0 License (check out the footer).

We often use creative commons licensed photographs to illustrate this blog, and felt that being developers of open source software it was quite logical that some of our content should be published under a similar license. Some of the documentation that we release also uses this license, for example the "Building Salome" documentation. This license footer has been integrated to the cubicweb-blog package that is used to publish our sites (as part of cubicweb-forge).

On a fait un tour au Coding Dojo à Nantes

Arthur Lutz — 2010-02-18T12:34-01:00

L'association de promotion des méthodes Agiles sur Nantes et sa région [1] organisait hier soir un "Coding Dojo". C'est une session de codage en public avec des créneaux de 15 minutes par codeur (explication sur codingdojo.org). Le focus de la session était le TDD (Test Driven Developpement).

Annonce de l'événement sur leur site : http://www.agilenantes.org/2010/01/25/coding-dojo-mercredi-17-fevrier-2010/

Photo par yepyep sous licence creative commons

Launching Python scripts via Condor

Alexandre Fayolle — 2010-02-17T08:56-01:00

As part of an ongoing customer project, I've been learning about the Condor queue management system (actually it is more than just a batch queue management system, tacking the High-throughput computing problem, but in my current project, we're not using the full possibilities of Condor, and the choice was dictated by other considerations outside the scope of this note). The documentation is excellent, and the features of the product are really amazing (pity the project runs on Windows, and we cannot use 90% of these...).

To launch a job on a computer participating in the Condor farm, you just have to write a job file which looks like this:

Universe=vanilla
Executable=$path_to_executabe
Arguments=$arguments_to_the_executable
InitialDir=$working_directory
Log=$local_logfile_name
Output=$local_file_for_job_stdout
Error=$local_file_for_job_stderr
Queue

and then run condor_submit my_job_file and use condor_q to monitor the status your job (queued, running...)

My program is generating Condor job files and submitting them, and I've spent hours yesterday trying to understand why they were all failing : the stderr file contained a message from Python complaining that it could not import site and exiting.

A point which was not clear in the documentation I read (but I probably overlooked it) is that the executable mentionned in the job file is supposed to be a local file on the submission host which is copied to the computer running the job. In the jobs generated by my code, I was using sys.executable for the Executable field, and a path to the python script I wanted to run in the Arguments field. This resulted in the Python interpreter being copied on the execution host and not being able to run because it was not able to find the standard files it needs at startup.

Once I figured this out, the fix was easy: I made my program write a batch script which launched the Python script and changed the job to run that script.

UPDATE : I'm told there is a Transfer_executable=False line I could have put in the script to achieve the same thing.

(photo by gudi&cris licenced under CC-BY-ND)

Adding Mercurial build identification to Python

Andre Espaze — 2010-02-15T16:41-01:00

This work is a part of the build identification task found in the PEP 385, Migrating from svn to Mercurial: http://www.python.org/dev/peps/pep-0385/ It was done during the Mercurial sprint hosted at Logilab. If you would like to see the result, just follow the steps:

hg clone http://hg.xavamedia.nl/cpython/pymigr/
cd pymigr/build-identification

Setting up the environment

The current Python development branch is first checkout:

svn co http://svn.python.org/projects/python/trunk

A patch will be applied for adding the 'sys.mercurial' attribute and modifying the build informations:

cp add-hg-build-id.diff trunk/
cd trunk
svn up -r 78019
patch -p0 < add-hg-build-id.diff

The changed made to 'configure.in' need then to be propagated to the configure script:

autoconf

The configuration is then done by:

./configure --enable-shared --prefix=/dev/null

You should now see changes propagated to the Makefile for finding the revision, the tag and the branch:

grep MERCURIAL Makefile

Finally, Python can be built:

make

The sys.mercurial attribute should already be present:

LD_LIBRARY_PATH=. ./python
>>> import sys
>>> sys.mercurial
('CPython', '', '')

No tag nor revision have been found as there was no mercurial repository. A test by the Py_GetBuildInfo() in the C API will also be built:

gcc -o show-build-info -I. -IInclude -L. -lpython2.7 ../show-build-info.c

You can test its result by:

LD_LIBRARY_PATH=. ./show-build-info
-> default, Feb  7 2010, 15:07:46

Manual test

First a fake mercurial tree is built:

hg init
hg add README
hg ci -m "Initial repo"
hg id
-> 84a6de74e48f tip

Now Python needs to be built with the given mercurial information:

rm Modules/getbuildinfo.o
make

You should then see the current revision number:

LD_LIBRARY_PATH=. ./python
>>> import sys
>>> sys.mercurial
('CPython', 'default', '84a6de74e48f')

and the C API can be tested by:

LD_LIBRARY_PATH=. ./show-build-info
-> default:84a6de74e48f, Feb  7 2010, 15:10:13

The fake mercurial repository can now be cleaned:

rm -rf .hg

Automatic tests

Automatic tests for checking the behavior for every cases will now work build Python and clean afterward. Those tests work only when run from the trunk svn directory of Python:

python ../test_build_identification.py

Further work

The current work is only an attempt to add the mercurial build identification to Python, it still needs to be checked on production cases. Moreover the build identification on Windows has not been started yet, it will need to be integrated to the Microsoft Visual Studio building process.

Why you shoud get rid of os.system, os.popen, etc. in your code

Alexandre Fayolle — 2010-02-12T12:43-01:00

I regularly come across code such as:

output = os.popen('diff -u %s %s' % (appl_file, ref_file), 'r')

Code like this might well work machine but it is buggy and will fail (preferably during the demo or once shipped).

Where is the bug?

It is in the use of %s, which can inject in your command any string you want and also strings you don't want. The problem is that you probably did not check appl_file and ref_file for weird things (spaces, quotes, semi colons...). Putting quotes around the %s in the string will not solve the issue.

So what should you do? The answer is "use the subprocess module": subprocess.Popen takes a list of arguments as first parameter, which are passed as-is to the new process creation system call of your platform, and not interpreted by the shell:

pipe = subprocess.Popen(['diff', '-u', appl_file, ref_file], stdout=subprocess.PIPE)
output = pipe.stdout

By now, you should have guessed that the shell=True parameter of subprocess.Popen should not be used unless you really really need it (and even them, I encourage you to question that need).

Apycot for Mercurial

Pierre-Yves David — 2010-02-11T16:30-01:00

What is apycot

apycot is a highly extensible test automatization tool used for Continuous Integration. It can:

download the project from a version controlled repository (like SVN or Hg);
install it from scratch with all dependencies;
run various checkers;
store the results in a CubicWeb database;
post-process the results;
display the results in various format (html, xml, pdf, mail, RSS...);
repeat the whole procedure with various configurations;
get triggered by new changesets or run periodically.

For an example, take a look at the "test reports" tab of the logilab-common project.

Setting up an apycot for Mercurial

During the mercurial sprint, we set up a proof-of-concept environment running six different checkers:

Check syntax of all python files.
Check syntax of all documentation files.
Run pylint on the mercurial source code with the mercurial pylintrc.
Run the check-code.py script included in mercurial checking style and python errors
Run the Mercurial's test suite.
Run Mercurial's benchmark on a reference repository.

The first three checkers, shipped with apycot, were set up quickly. The last three are mercurial specific and required few additional tweaks to be integrated to apycot.

The bot was setup to run with all public mercurial repositories. Five checkers immediately proved useful as they pointed out some errors or warnings (on some rarely used contrib files it even found a syntax error).

Prospectives

A public instance is being set up. It will provide features that the community is looking forward to:

testing all python versions;
running pure python or the C variant;
code coverage of the test suite;
performance history.

Conclusion

apycot proved to be highly flexible and could quickly be adapted to Mercurial's test suite even for people new to apycot. The advantages of continuously running different long running tests is obvious. So apycot seems to be a very valuable tool for improving the software development process.

SCons presentation in 5 minutes

Andre Espaze — 2010-02-09T15:26-01:00

Building software with SCons requires to have Python and SCons installed.

As SCons is only made of Python modules, the sources may be shipped with your project if your clients can not install dependencies. All the following exemples can be downloaded at the end of that blog.

A building tool for every file extension

First a Fortran 77 program will be built made of two files:

$ cd fortran-project
$ scons -Q
gfortran -o cfib.o -c cfib.f
gfortran -o fib.o -c fib.f
gfortran -o compute-fib cfib.o fib.o
$ ./compute-fib
 First 10 Fibonacci numbers:
  0.  1.  1.  2.  3.  5.  8. 13. 21. 34.

The '-Q' option tell to Scons to be less verbose. For cleaning the project, add the '-c' option:

$ scons -Qc
Removed cfib.o
Removed fib.o
Removed compute-fib

From this first example, it can been seen that SCons find the 'gfortran' tool from the file extension. Then have a look at the user's manual if you want to set a particular tool.

Describing the construction with Python objects

A second C program will directly run the execution from the SCons file by adding a test command:

$ cd c-project
$ scons -Q run-test
gcc -o test.o -c test.c
gcc -o fact.o -c fact.c
ar rc libfact.a fact.o
ranlib libfact.a
gcc -o test-fact test.o libfact.a
run_test(["run-test"], ["test-fact"])
OK

However running scons alone builds only the main program:

$ scons -Q
gcc -o main.o -c main.c
gcc -o compute-fact main.o libfact.a
$ ./compute-fact
Computing factorial for: 5
Result: 120

This second example shows that the construction dependency is described by passing Python objects. An interesting point is the possibility to add your own Python functions in the build process.

Hierarchical build with environment

A third C++ program will create a shared library used for two different programs: the main application and a test suite. The main application can be built by:

$ cd cxx-project
$ scons -Q
g++ -o main.o -c -Imbdyn-src main.cxx
g++ -o mbdyn-src/nodes.os -c -fPIC -Imbdyn-src mbdyn-src/nodes.cxx
g++ -o mbdyn-src/solver.os -c -fPIC -Imbdyn-src mbdyn-src/solver.cxx
g++ -o mbdyn-src/libmbdyn.so -shared mbdyn-src/nodes.os mbdyn-src/solver.os
g++ -o mbdyn main.o -Lmbdyn-src -lmbdyn

It shows that SCons handles for us the compilation flags for creating a shared library according to the tool (-fPIC). Moreover extra environment variables have been given (CPPPATH, LIBPATH, LIBS), which are all translated for the chosen tool. All those variables can be found in the user's manual or in the man page. The building and running of the test suite is made by giving an extra variable:

$ TEST_CMD="LD_LIBRARY_PATH=mbdyn-src ./%s" scons -Q run-tests
g++ -o tests/run_all_tests.o -c -Imbdyn-src tests/run_all_tests.cxx
g++ -o tests/test_solver.o -c -Imbdyn-src tests/test_solver.cxx
g++ -o tests/all-tests tests/run_all_tests.o tests/test_solver.o -Lmbdyn-src -lmbdyn
run_test(["tests/run-tests"], ["tests/all-tests"])
OK

Conclusion

That is rather convenient to build softwares by manipulating Python objects, moreover custom actions can be added in the process. SCons has also a configuration mechanism working like autotools macros that can be discovered in the user's manual.

Extended 256 colors in bash prompt

Nicolas Chauvat — 2010-02-07T11:07-01:00

The Mercurial 1.5 sprint is taking place in our offices this week-end and pair-programming with Steve made me want a better looking terminal. Have you seen his extravagant zsh prompt ? I used to have only 8 colors to decorate my shell prompt, but thanks to some time spent playing around, I now have 256.

Here is what I used to have in my bashrc for 8 colors:

NO_COLOUR="\[\033[0m\]"
LIGHT_WHITE="\[\033[1;37m\]"
WHITE="\[\033[0;37m\]"
GRAY="\[\033[1;30m\]"
BLACK="\[\033[0;30m\]"

RED="\[\033[0;31m\]"
LIGHT_RED="\[\033[1;31m\]"
GREEN="\[\033[0;32m\]"
LIGHT_GREEN="\[\033[1;32m\]"
YELLOW="\[\033[0;33m\]"
LIGHT_YELLOW="\[\033[1;33m\]"
BLUE="\[\033[0;34m\]"
LIGHT_BLUE="\[\033[1;34m\]"
MAGENTA="\[\033[0;35m\]"
LIGHT_MAGENTA="\[\033[1;35m\]"
CYAN="\[\033[0;36m\]"
LIGHT_CYAN="\[\033[1;36m\]"

# set a fancy prompt
export PS1="${RED}[\u@\h \W]\$${NO_COLOUR} "

Just put the following lines in your bashrc to get the 256 colors:

function EXT_COLOR () { echo -ne "\[\033[38;5;$1m\]"; }

# set a fancy prompt
export PS1="`EXT_COLOR 172`[\u@\h \W]\$${NO_COLOUR} "

Yay, I now have an orange prompt! I now need to write a script that will display useful information depending on the context. Displaying the status of the mercurial repository I am in might be my next step.

We're happy to host the mercurial Sprint

Arthur Lutz — 2010-02-02T10:30-01:00

We're very happy to be hosting the next mercurial sprint in our brand new offices in central Paris. It is quite an honor to be chosen when the other contender was Google.

So a bunch of mercurial developers are heading out to our offices this coming Friday to sprint for three days on mercurial. We use mercurial a lot here over at Logilab and we also contribute a tool to visualize and manipulate a mercurial repository : hgview.

To check out the things that we will be working on with the mercurial crew, check out the program of the sprint on their wiki.

What is a sprint? "A sprint (sometimes called a Code Jam or hack-a-thon) is a short time period (three to five days) during which software developers work on a particular chunk of functionality. "The whole idea is to have a focused group of people make progress by the end of the week," explains Jeff Whatcott" [source]. For geographically distributed open source communities, it is also a way of physically meeting and working in the same room for a period of time.

Sprinting is a practice that we encourage at Logilab, with CubicWeb we organize as often as possible open sprints, which is an opportunity for users and developers to come and code with us. We even use the sprint format for some internal stuff.

photo by Sebastian Mary under creative commons licence.

hgview 1.2.0 released

David Douard — 2010-01-21T18:13-01:00

Here is at last the release of the version 1.2.0 of hgview.

In a nutshell, this release includes:

a basic support for mq extension,
a basic support for hg-bfiles extension,
working directory is now displayed as a node of the graph (if there are local modifications of course),
it's now possible to display only the subtree from a given revision (a bit like hg log -f)
it's also possible to activate an annotate view (make navigation slower however),
several improvements in the graph filling and rendering mecanisms,
I also added toolbar icons for the search and goto "quickbars" so they are not "hidden" any more to the one reluctant to user manuals,
it's now possible to go directly to the common ancestor of 2 revisions,
when on a merge node, it's now possible to choose the parent the diff is computed against,
make search also search in commit messages (it used to search only in diff contents),
and several bugfixes of course.

Notes:: there are packages for debian lenny, squeeze and sid, and for ubuntu hardy, interpid, jaunty and karmic. However, for lenny and hardy, provided packages won't work on pure distribs since hgview 1.2 depends on mercurial 1.1. Thus for these 2 distributions, packages will only work if you have installed backported mercurial packages.

New supported repositories for Debian and Ubuntu

Arthur Lutz — 2010-01-21T17:14-01:00

For the release of hgview 1.2.0 in our Karmic Ubuntu repository, we would like to announce that we are now going to generate packages for the following distributions :

Debian Lenny (because it's stable)
Debian Sid (because it's the dev branch)
Ubuntu Hardy (because it has Long Term Support)
Ubuntu Karmic (because it's the current stable)
Ubuntu Lucid (because it's the next stable) - no repo yet, but soon...

The old packages in the previously supported architectures are still accessible (etch, jaunty, intrepid), but new versions will not be generated for these repositories. Packages will be coming in as versions get released, if before that you need a package, give us a shout and we'll see what we can do.

For instructions on how to use the repositories for Ubuntu or Debian, go to the following page : http://www.logilab.org/card/LogilabDebianRepository

Open Source/Design Hardware

Nicolas Chauvat — 2009-12-13T15:32-01:00

I have been doing free software since I discovered it existed. I bought an OpenMoko some time ago, since I am interested in anything that is open, including artwork like books, music, movies and... hardware.

I just learned about two lists, one at Wikipedia and another one at MakeOnline, but Google has more. Explore and enjoy!

Solution to a common Mercurial task

David Douard — 2009-12-10T12:09-01:00

An interesting question has just been sent by Greg Ward on the Mercurial devel mailing-list (as a funny coincidence, it happened that I had to solve this problem a few days ago).

Let me quote his message:

here's my problem: imagine a customer is running software built from
changeset A, and we want to upgrade them to a new version, built from
changeset B.  So I need to know what bugs are fixed in B that were not
fixed in A.  I have already implemented a changeset/bug mapping, so I
can trivially lookup the bugs fixed by any changeset.  (It even handles
"ongoing" and "reverted" bugs in addition to "fixed".)

And he gives an example of situation where a tricky case may be found:

                +--- 75 -- 78 -- 79 ------------+
               /                                 \
              /     +-- 77 -- 80 ---------- 84 -- 85
             /     /                        /
0 -- ... -- 74 -- 76                       /
                   \                      /
                    +-- 81 -- 82 -- 83 --+

So what is the problem?

Imagine the lastest distributed stable release is built on rev 81. Now, I need to publish a new bugfix release based on this latest stable version, including every changeset that is a bugfix, but that have not yet been applied at revision 81.

So the first problem we need to solve is answering: what are the revisions ancestors of revision 85 that are not ancestor of revision 81?

Command line solution

Using hg commands, the solution is proposed by Steve Losh:

hg log --template '{rev}\n' --rev 85:0 --follow --prune 81

or better, as suggested by Matt:

hg log -q --template '{rev}\n' --rev 85:0 --follow --prune 81

The second is better since it does only read the index, and thus is much faster. But on big repositories, this command remains quite slow (with Greg's situation, a repo of more than 100000 revisions, the command takes more than 2 minutes).

Python solution

Using Python, one may think about using revlog.nodesbetween(), but it won't work as wanted here, not listing revisions 75, 78 and 79.

On the mailing list, Matt gave the most simple and efficient solution:

cl = repo.changelog
a = set(cl.ancestors(81))
b = set(cl.ancestors(85))
revs = b - a

Idea for a new extension

Using this simple python code, it should be easy to write a nice Mercurial extension (which could be named missingrevisions) to do this job.

Then, it should be interesting to also implement some filtering feature. For example, if there are simple conventions used in commit messages, eg. using something like "[fix #1245]" or "[close #1245]" in the commit message when the changeset is a fix for a bug listed in the bugtracker, then we may type commands like:

hg missingrevs REV -f bugfix

or:

hg missingrevs REV -h HEADREV -f bugfix

to find bugfix revisions ancestors of HEADREV that are not ancestors of REV.

With filters (bugfix here) may be configurables in hgrc using regexps.

pylint bug day report

Pierre-Yves David — 2009-12-04T15:56-01:00

The first pylint bug day took place on wednesday 25th. Four members of the Logilab crew and two other people spent the day working on pylint.

Several patches submitted before the bug day were processed and some tickets were closed.

Charles Hébert added James Lingard's patches for string formatting and is working on several improvements. Vincent Férotin submitted a patch for simple message listings. Sylvain Thenault fixed significant inference bugs in astng (an underlying module of pylint managing the syntax tree). Émile Anclin began a major astng refactoring to take advantage of new python2.6 functionality. For my part, I made several improvements to the test suite. I applied James Lingard patches for ++ operator and generalised it to -- too. I also added a new checker for function call arguments submitted by James Lingard once again. Finally I improved the message filtering of the --errors-only options.

We thank Maarten ter Huurne, Vincent Férotin for their participation and of course James Lingard for submitting numerous patches.

Another pylint bug day will be held in a few months.

image under creative commons by smccann

Resume of the first Coccinelle users day

Andre Espaze — 2009-11-30T12:05-01:00

A matching and transformation tool for systems code

The Coccinelle's goal is to ease code maintenance by first revealing code smells based on design patterns and second easing an API (Application Programming Interface) change for a heavily used library. Coccinelle can thus be seen as two tools inside one. The first one matches patterns, the second applies transformations. However facing such a big problem, the project needed to define boundaries in order to increase chances of success. The building motivation was thus to target the Linux kernel. This choice has implied a tool working on the C programming language before the preprocessor step. Moreover the Linux code base adds interesing constraints as it is huge, contains many possible configurations depending on C macros, may contain many bugs and evolves a lot. What was the Coccinelle solution for easing the kernel maintenance?

Generating diff files from the semantic patch langage

The Linux community reads lot of diff files for following the kernel evolution. As a consequence the diff file syntax is widely spread and commonly understood. However this syntax concerns a particular change between two files, its does not allow to match a generic pattern.

The Coccinelle's solution is to build its own langage allowing to declare rules describing a code pattern and a possible transformation. This langage is the Semantic Patch Langage (SmPL), based on the declarative approach of the diff file syntax. It allows to propagate a change rule to many files by generating diff files. Then those results can be directly applied by using the patch command but most of the time they will be reviewed and may be slightly adapted to the programmer's need.

A Coccinelle's rule is made of two parts: metavariable declaration and a code pattern match followed by a possible transformation. A metavariable means a control flow variable, its possibles names inside the program do not matter. Then the code pattern will describe a particular control flow in the program by using the C and SmPL syntaxes manipulating the metavariables. As a result, Coccinelle succeeds to generate diff files because it works on the C program control flow.

A complete SmPL description will not be given here because it can be found in the Coccinelle's documentation. However a brief introduction will be made on a rule declaration. The metavariable part will look like this:

@@
expression E;
constant C;
@@

'expression' means a variable or the result of a function. However 'constant' means a C constant. Then for negating the result of an and operation between an expression and a constant instead of negating the expression first, the transformation part will be:

- !E & C
+ !(E & C)

A file containing several rules like that will be called a semantic patch. It will be applied by using the Coccinelle 'spatch' command that will generate a change written in the diff file syntax each time the above pattern is matched. The next section will illustrate this way of work.

A working example on the Linux kernel 2.6.30

You can download and install Coccinelle 'spatch' command from its website: http://coccinelle.lip6.fr/ if you want to execute the following example. Let's first consider the following structure with accessors in the header 'device.h':

struct device {
    void *driver_data;
};

static inline void *dev_get_drvdata(const struct device *dev)
{
    return dev->driver_data;
}

static inline void dev_set_drvdata(struct device *dev, void* data)
{
    dev->driver_data = data;
}

it imitates the 2.6.30 kernel header 'include/linux/device.h'. Let's now consider the following client code that does not make use of the accessors:

#include <stdlib.h>
#include <assert.h>

#include "device.h"

int main()
{
    struct device devs[2], *dev_ptr;
    int data[2] = {3, 7};
    void *a = NULL, *b = NULL;

    devs[0].driver_data = (void*)(&data[0]);
    a = devs[0].driver_data;

    dev_ptr = &devs[1];
    dev_ptr->driver_data = (void*)(&data[1]);
    b = dev_ptr->driver_data;

    assert(*((int*)a) == 3);
    assert(*((int*)b) == 7);
    return 0;
}

Once this code saved in the file 'fake_device.c', we can check that the code compiles and runs by:

$ gcc fake_device.c && ./a.out

We will now create a semantic patch 'device_data.cocci' trying to add the getter accessor with this first rule:

@@
struct device dev;
@@
- dev.driver_data
+ dev_get_drvdata(&dev)

The 'spatch' command is then run by:

$ spatch -sp_file device_data.cocci fake_device.c

producing the following change in a diff file:

-    devs[0].driver_data = (void*)(&data[0]);
-    a = devs[0].driver_data;
+    dev_get_drvdata(&devs[0]) = (void*)(&data[0]);
+    a = dev_get_drvdata(&devs[0]);

which illustrates the great Coccinelle's way of work on program flow control. However the transformation has also matched code where the setter accessor should be used. We will thus add a rule above the previous one, the semantic patch becomes:

@@
struct device dev;
expression data;
@@
- dev.driver_data = data
+ dev_set_drvdata(&dev, data)

@@
struct device dev;
@@
- dev.driver_data
+ dev_get_drvdata(&dev)

Running the command again will produce the wanted output:

$ spatch -sp_file device_data.cocci fake_device.c
-    devs[0].driver_data = (void*)(&data[0]);
-    a = devs[0].driver_data;
+    dev_set_drvdata(&devs[0], (void *)(&data[0]));
+    a = dev_get_drvdata(&devs[0]);

It is important to write the setter rule before the getter rule else the getter rule will be applied first to the whole file.

At this point our semantic patch is still incomplete because it does not work on 'device' structure pointers. By using the same logic, let's add it to the 'device_data.cocci' semantic patch:

@@
struct device dev;
expression data;
@@
- dev.driver_data = data
+ dev_set_drvdata(&dev, data)

@@
struct device * dev;
expression data;
@@
- dev->driver_data = data
+ dev_set_drvdata(dev, data)

@@
struct device dev;
@@
- dev.driver_data
+ dev_get_drvdata(&dev)

@@
struct device * dev;
@@
- dev->driver_data
+ dev_get_drvdata(dev)

Running Coccinelle again:

$ spatch -sp_file device_data.cocci fake_device.c

will add the remaining transformations for the 'fake_device.c' file:

-    dev_ptr->driver_data = (void*)(&data[1]);
-    b = dev_ptr->driver_data;
+    dev_set_drvdata(dev_ptr, (void *)(&data[1]));
+    b = dev_get_drvdata(dev_ptr);

but a new problem appears: the 'device.h' header is also modified. We meet here an important point of the Coccinelle's philosophy described in the first section. 'spatch' is a tool to ease code maintenance by propagating a code pattern change to many files. However the resulting diff files are supposed to be reviewed and in our case the unwanted modification should be removed. Note that it would be possible to avoid the 'device.h' header modification by using SmPL syntax but the explanation would be too much for a starting tutorial. Instead, we will simply cut the unwanted part:

$ spatch -sp_file device_data.cocci fake_device.c | cut -d $'\n' -f 16-34

This result will now be kept in a diff file by moreover asking 'spatch' to produce it for the current working directory:

$ spatch -sp_file device_data.cocci -patch "" fake_device.c | \
cut -d $'\n' -f 16-34 > device_data.patch

It is now time to apply the change for getting a working C code using accessors:

$ patch -p1 < device_data.patch

The final result for 'fake_device.c' should be:

#include <stdlib.h>
#include <assert.h>

#include "device.h"

int main()
{
    struct device devs[2], *dev_ptr;
    int data[2] = {3, 7};
    void *a = NULL, *b = NULL;

    dev_set_drvdata(&devs[0], (void *)(&data[0]));
    a = dev_get_drvdata(&devs[0]);

    dev_ptr = &devs[1];
    dev_set_drvdata(dev_ptr, (void *)(&data[1]));
    b = dev_get_drvdata(dev_ptr);

    assert(*((int*)a) == 3);
    assert(*((int*)b) == 7);
    return 0;
}

Finally, we can test that the code compiles and runs:

.. sourcecode:: sh

$ gcc fake_device.c && ./a.out

The semantic patch is now ready to be used on the Linux's 2.6.30 kernel:

$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.30.tar.bz2
$ tar xjf linux-2.6.30.tar.bz2
$ spatch -sp_file device_data.cocci -dir linux-2.6.30/drivers/net/ \
  > device_drivers_net.patch
$ wc -l device_drivers_net.patch
642

You may also try the 'drivers/ieee1394' directory.

Conclusion

Coccinelle is made of around 60 thousands lines of Objective Caml. As illustrated by the above example on the linux kernel, the 'spatch' command succeeds to ease code maintenance. For the Coccinelle's team working on the kernel code base, a semantic patch is usually around 100 lines and will generated diff files to sometimes hundred of files. Moreover the processing is rather fast, the average time per file is said to be 0.7s.

Two tools using the 'spatch' engine have already been built: 'spdiff' and 'herodotos'. With the first one you could almost avoid to learn the SmPL language because the idea is to generate a semantic patch by looking to transformations between files pairs. The second allows to correlate defects over software versions once the corresponding code smells have been described in SmPL.

One of the Coccinelle's problem is to not being easily extendable to another language as the engine was designed for analyzing control flows on C programs. The C++ langage may be added but required obviously lot of work. It would be great to also have such a tool on dynamic languages like Python.

image under creative commons by Rémi Vannier

pylint bug day next wednesday!

Sylvain Thenault — 2009-11-23T17:01-01:00

Remember that the first pylint bug day will be held on wednesday, november 25, from around 8am to 8pm in the Paris (France) time zone.

We'll be a few people at Logilab and hopefuly a lot of other guys all around the world, trying to make pylint better.

Join us on the #public conference room of conference.jabber.logilab.org, or if you prefer using an IRC client, join #public on irc.logilab.org which is a gateway to the jabber forum. And if you're in Paris, come to work with us in our office.

People willing to help but without knowledge of pylint internals are welcome, it's the perfect occasion to learn a lot about it, and to be able to hack on pylint in the future!

First contact with pupynere

Pierre-Yves David — 2009-11-06T17:36-01:00

I spent some time this week evaluating Pupynere, the PUre PYthon NEtcdf REader written by Roberto De Almeida. I see several advantages in pupynere.

First it's a pure Python module with no external dependency. It doesn't even depend on the NetCDF lib and it is therefore very easy to deploy.

Second, it offers the same interface as Scientific Python's NetCDF bindings which makes transitioning from one module to another very easy.

Third pupynere is being integrated into Scipy as the scypi.io.netcdf module. Once integrated, this could ensure a wide adoption by the python community.

Finally it's easy to dig in this clear and small code base of about 600 lines. I have just sent several fixes and bug reports to the author.

However pupynere isn't mature yet. First it seems pupynere has been only used for simple cases so far. Many common cases are broken. Moreover there is no support for new NetCDF formats such as long-NetCDF and NetCDF4, and important features such as file update are still missing. In addition, The lack of a test suite is a serious issue. In my opinion, various bugs could already have been detected and fixed with simple unit tests. Contributions would be much more comfortable with the safety net offered by a test suite. I am not certain that the fixes and improvements I made this week did not introduce regressions.

To conclude, pupynere seems too young for production use. But I invite people to try it and provide feedback and fixes to the author. I'm looking forward to using this project in production in the future.

First Pylint Bug Day on Nov 25th, 2009 !

Sylvain Thenault — 2009-10-21T15:43-01:00

Since we don't stop being overloaded here at Logilab, and we've got some encouraging feedback after the "Pylint needs you" post, we decided to take some time to introduce more "community" in pylint.

And the easiest thing to do, rather sooner than later, is a irc/jabber synchronized bug day, which will be held on Wednesday november 25. We're based in France, so main developpers will be there between around 8am and 19pm UTC+1. If a few of you guys are around Paris at this time and wish to come at Logilab to sprint with us, contact us and we'll try to make this possible.

The focus for this bug killing day could be:

using logilab.org tracker : getting an account, submitting tickets, triaging existing tickets...
using mercurial to develop pylint / astng
guide people in the code so they're able to fix simple bugs

We will of course also try to kill a hella-lotta bugs, but the main idea is to help whoever wants to contribute to pylint... and plan for the next bug-killing day !

As we are in the process of moving to another place, we can't organize a sprint yet, but we should have some room available for the next time, so stay tuned :)

Projman 0.14.0 includes a Graphical User Interface

Emile Anclin — 2009-10-19T15:39-01:00

Introduction

Projman is a project manager. With projman 0.14.0, the first sketch of a GUI has been updated, and important functionalities added. You can now easily see and edit task dependencies and test the resulting scheduling. Furthermore, a begin-after-end-previous constraint has been added which should really simplify the edition of the scheduling.

The GUI can be used the two following ways:

$ projman-gui
$ projman-gui <path/to/project.xml>

The file <path/to/project.xml> is the well known main file of a projman project. Starting projman-gui with no project.xml specified, or after opening a project, you can open an existing project simply with "File->Open". (For now, you can't create a new project with projman-gui.) You can edit the tasks and then save the modifications to the task file with "File->Save".

The Project tab

The Project tab shows simply the four needed files of a projman project for resources, activities, tasks and schedule.

Resources

The Resources tab presents the different resources:

human resources
resource roles describing the different roles that resources can play
Different calendars for different resources with their "offdays"

Activities

For now, the Activities tab is not implemented. It should show the planning of the activities for each resource and the progress of the project.

Tasks

The Tasks tab is for now the most important one; it shows a tree view of the task hierarchy, and for each task:

the title of the task,
the role for that task,
the load (time in days),
the scheduling type,
the list of the constraints for the scheduling,
and the description of the task,

each of which can be edited. You easily can drag and drop tasks inside the task tree and add and delete tasks and constraints.

See the attached screenshot of the projman-gui task panel.

Scheduling

In the Scheduling tab you can simply test your scheduling by clicking "START". If you expect the scheduling to take a longer time, you can modify the maximum time of searching a solution.

Known bugs

The begin-after-end-previous constraint does not work for a task having subtasks.
Deleting a task doesn't check for depending tasks, so scheduling won't work anymore.

hgview 1.1.0 released

David Douard — 2009-09-25T15:16-01:00

I am pleased to announce the latest release of hgview 1.1.0.

What is it?

For the ones from the back of the classroom near the radiator, let me remind you that hgview is a very helpful tool for daily work using the excellent DVCS Mercurial (which we heavily use at Logilab). It allows to easily and visually navigate your hg repository revision graphlog. It is written in Python and pyqt.

What's new

user can now configure colors used in the diff area (and they now defaults to white on black)
indicate current working directory position by a square node
add many other configuration options (listed when typing hg help hgview)
removed 'hg hgview-options' command in favor of 'hg help hgview'
add ability to choose which parent to diff with for merge nodes
dramatically improved UI behaviour (shortcuts)
improved help and make it accessible from the GUI
make it possible not to display the diffstat column of the file list (which can dramatically improve performances on big repositories)
standalone application: improved command line options
indicate working directory position in the graph
add auto-reload feature (when the repo is modified due to a pull, a commit, etc., hgview detects it, reloads the repo and updates the graph)
fix many bugs, especially the file log navigator should now display the whole graph

Download and installation

The source code is available as a tarball, or using our public hg repository of course.

To use it from the sources, you just have to add a line in your .hgrc file, in the [extensions] section:

hgext.hgview=/path/to/hgview/hgext/hgview.py

Debian and Ubuntu users can also easily install hgview (and Logilab other free software tools) using our deb package repositories.

Pylint a besoin de vous

Alexandre Fayolle — 2009-09-17T09:20-01:00

Après plusieurs mois au point mort ou presque, Sylvain a pu hier soir publier des versions corrigeant un certain nombre de bogues dans pylint et astng ([1] et [2]).

Il n'en demeure pas moins qu'à Logilab, nous manquons de temps pour faire baisser la pile de tickets ouverts dans le tracker de pylint. Si vous jetez un œuil dans l'onglet Tickets, vous y trouverez un grand nombre de bogues en souffrance et de fonctionalités indispensables (certaines peut-être un peu moins que d'autres...) Il est déjà possible de contribuer en utilisant mercurial pour fournir des patches, ou en signalant des bogues (aaaaaaaaaarg ! encore des tickets !) et certains s'y sont mis, qu'ils en soient remerciés.

Maintenant, nous nous demandions ce que nous pourrions faire pour faire avance Pylint, et nos premières idées sont :

organiser un petit sprint de 3 jours environ
organiser des jours de "tuage de ticket", comme ça se pratique sur différents projets OSS

Mais pour que ça soit utile, nous avons besoin de votre aide. Voici donc quelques questions :

est-ce que vous participeriez à un sprint à Logilab (à Paris, France), ce qui nous permettrait de nous rencontrer, de vous apprendre plein de choses sur le fonctionnement de Pylint et de travailler ensemble sur des tickets qui vous aideraient dans votre travail ?
si la France c'est trop loin, où est-ce que ça vous arrangerait ?
seriez-vous prêt à vous joindre à nous sur le serveur jabber de Logilab ou sur IRC, pour participer à une chasse au ticket (à une date à déterminer). Si oui, quel est votre degré de connaissance du fonctionnement interne de Pylint et astng ?

Vous pouvez répondre en commentant sur ce blog (pensez à vous enregistrer en utilisant le lien en haut à droite sur cette page) ou en écrivant à sylvain.thenault@logilab.fr. Si nous avons suffisamment de réponses positives nous organiserons quelque chose.

Using tempfile.mkstemp correctly

Alexandre Fayolle — 2009-09-10T09:40-01:00

The mkstemp function in the tempfile module returns a tuple of 2 values:

an OS-level handle to an open file (as would be returned by os.open())
the absolute pathname of that file.

I often see code using mkstemp only to get the filename to the temporary file, following a pattern such as:

from tempfile import mkstemp
import os

def need_temp_storage():
    _, temp_path = mkstemp()
    os.system('some_commande --output %s' % temp_path)
    file = open(temp_path, 'r')
    data = file.read()
    file.close()
    os.remove(temp_path)
    return data

This seems to be working fine, but there is a bug hiding in there. The bug will show up on Linux if you call this functions many time in a long running process, and on the first call on Windows. We have leaked a file descriptor.

The first element of the tuple returned by mkstemp is typically an integer used to refer to a file by the OS. In Python, not closing a file is usually no big deal because the garbage collector will ultimately close the file for you, but here we are not dealing with file objects, but with OS-level handles. The interpreter sees an integer and has no way of knowing that the integer is connected to a file. On Linux, calling the above function repeatedly will eventually exhaust the available file descriptors. The program will stop with:

IOError: [Errno 24] Too many open files: '/tmp/tmpJ6g4Ke'

On Windows, it is not possible to remove a file which is still opened by another process, and you will get:

Windows Error [Error 32]

Fixing the above function requires closing the file descriptor using os.close_():

from tempfile import mkstemp
import os

def need_temp_storage():
    fd, temp_path = mkstemp()
    os.system('some_commande --output %s' % temp_path)
    file = open(temp_path, 'r')
    data = file.read()
    file.close()
    os.close(fd)
    os.remove(temp_path)
    return data

If you need your process to write directly in the temporary file, you don't need to call os.write_(fd, data). The function os.fdopen_(fd) will return a Python file object using the same file descriptor. Closing that file object will close the OS-level file descriptor.

You can now register on our sites

Arthur Lutz — 2009-09-03T17:43-01:00

With the new version of CubicWeb deployed on our "public" sites, we would like to welcome a new (much awaited) functionality : you can now register directly on our websites. Getting an account with give you access to a bunch of functionalities :

registering to a project's activity with get you automated email reports of what is happening on that project
you can directly add tickets on projects instead of talking about it on the mailing lists
you can bookmark content
tag stuff
and much more...

This is also a way of testing out the CubicWeb framework (in this case the forge cube) which you can take home and host yourself (debian recommended). Just click on the "register" link on the top right, or here.

Photo by wa7son under creative commons.

New pylint/astng release, but... pylint needs you !

Sylvain Thenault — 2009-08-27T18:51-01:00

After several months with no time to fix/enhance pylint beside answering email and filing tickets, I've finally tackled some tasks yesterday night to publish bug fixes releases ([1] and [2]).

The problem is that we don't have enough free time at Logilab to lower the number of tickets in pylint tracker page . If you take a look at the ticket tab, you'll see a lot of pendings bug and must-have features (well, and some other less necessary...). You can already easily contribute thanks to the great mercurial dvcs, and some of you do, either by providing patches or by reporting bugs (more tickets, iiirk ! ;) Thank you all btw !!

Now I was wondering what could be done to make pylint going further, and the first ideas which came to my mind was :

do ~3 days sprint
do some 'tickets killing' days, as done in some popular oss projects

But for this to be useful, we need your support, so here are some questions for you:

would you come to a sprint at Logilab (in Paris, France), so you can meet us, learn a lot about pylint, and work on tickets you wish to have in pylint?
if France is too far away for most people, would you have another location to propose?
would you be on jabber for a tickets killing day, providing it's ok with your agenda? if so, what's your knowledge of pylint/astng internals?

you may answer by adding a comment to this blog (please register first by using the link at the top right of this page) or by mail to sylvain.thenault@logilab.fr. If we've enough positive answers, we'll take the time to organize such a thing.

Looking for a Windows Package Manager

Nicolas Chauvat — 2009-07-31T15:01-01:00

As said in a previous article, I am convinced that part of the motivation for making package sub-systems like the Python one, which includes distutils, setuptools, etc, is that Windows users and Mac users never had the chance to use a tool that properly manages the configuration of their computer system. They just do not know what it would be like if they had at least a good package management system and do not miss it in their daily work.

I looked for Windows package managers that claim to provide features similar to Debian's dpkg+apt-get and here is what I found in alphabetical order.

AppSnap

AppSnap is written in Python and uses wxPython, PyCurl and PyYAML. It is packaged using Py2Exe, compressed with UPX and installed using NSIS.

It has not seen activity in the svn or on its blog since the end of 2008.

Appupdater

Appupdater provides functionality similar to apt-get or yum. It automates the process of installing and maintaining up to date versions of programs. It claims to be fully customizable and is licensed under the GPL.

It seems under active development at SourceForge.

QWinApt

QWinApt is a Synaptic clone written in C# that has not evolved since september 2007.

WinAptic

WinAptic is another Synaptic clone written this time in Pascal that has not evolved since the end of 2007.

Win-Get

Win-get is an automated install system and software repository for Microsoft Windows. It is similar to apt-get: it connects to a link repository, finds an application and downloads it before performing the installation routine (silent or standard) and deleting the install file.

It is written in pascal and is set up as a SourceForge project, but not much has been done lately.

WinLibre

WinLibre is a Windows free software distribution that provides a repository of packages and a tool to automate and simplify their installation.

WinLibre was selected for Google Summer of Code 2009.

ZeroInstall

ZeroInstall started as a "non-admin" package manager for Linux distributions and is now extending its reach to work on windows.

Conclusion

I have not used any of these tools, the above is just the result of some time spent searching the web.

A more limited approach is to notify the user of the newer versions:

App-Get will show you a list of your installed Applications. When an update is available for one of them, it will highlighted and you will be able to update the specific applications in seconds.

GetIt is not an application-getter/installer. When you want to install a program, you can look it up in GetIt to choose which program to install from a master list of all programs made available by the various apt-get clones.

The appupdater project also compares itself to the programs automating the installation of software on Windows.

Some columists expect the creation of application stores replicating the iPhone one.

I once read about a project to get the Windows kernel into the Debian distribution, but can not find any trace of it... Remember that Debian is not limited to the Linux kernel, so why not think about a very improbable apt-get install windows-vista ?

The Configuration Management Problem

Nicolas Chauvat — 2009-07-31T13:54-01:00

Today I felt like summing up my opinion on a topic that was discussed this year on the Python mailing lists, at PyCon-FR, at EuroPython and EuroSciPy... packaging software! Let us discuss the two main use cases.

The first use case is to maintain computer systems in production. A trait of production systems, is that they can not afford failures and are often deployed on a large scale. It leaves little room for manually fixing problems. Either the installation process works or the system fails. Reaching that level of quality takes a lot of work.

The second use case is to facilitate the life of software developers and computer users by making it easy for them to give a try to new pieces of software without much work.

The first use case has to be addressed as a configuration management problem. There is no way around it. The best way I know of managing the configuration of a computer system is called Debian. Its package format and its tool chain provide a very extensive and efficient set of features for system development and maintenance. Of course it is not perfect and there are missing bits and open issues that could be tackled, like the dependencies between hardware and software. For example, nothing will prevent you from installing on your Debian system a version of a driver that conflicts with the version of the chip found in your hardware. That problem could be solved, but I do not think the Debian project is there yet and I do not count it as a reason to reject Debian since I have not seen any other competitor at the level as Debian.

The second use case is kind of a trap, for it concerns most computer users and most of those users are either convinced the first use case has nothing in common with their problem or convinced that the solution is easy and requires little work.

The situation is made more complicated by the fact that most of those users never had the chance to use a system with proper package management tools. They simply do not know the difference and do not feel like they are missing when using their system-that-comes-with-a-windowing-system-included.

Since many software developers have never had to maintain computer systems in production (often considered a lower sysadmin job) and never developed packages for computer systems that are maintained in production, they tend to think that the operating system and their software are perfectly decoupled. They have no problem trying to create a new layer on top of existing operating systems and transforming an operating system issue (managing software installation) into a programming langage issue (see CPAN, Python eggs and so many others).

Creating a sub-system specific to a language and hosting it on an operating system works well as long as the language boundary is not crossed and there is no competition between the sub-system and the system itself. In the Python world, distutils, setuptools, eggs and the like more or less work with pure Python code. They create a square wheel that was made round years ago by dpkg+apt-get and others, but they help a lot of their users do something they would not know how to do another way.

A wall is quickly hit though, as the approach becomes overly complex as soon as they try to depend on things that do not belong to their Python sub-system. What if your application needs a database? What if your application needs to link to libraries? What if your application needs to reuse data from or provide data to other applications? What if your application needs to work on different architectures?

The software developers that never had to maintain computer systems in production wish these tasks were easy. Unfortunately they are not easy and cannot be. As I said, there is no way around configuration management for the one who wants a stable system. Configuration management requires both project management work and software development work. One can have a system where packaging software is less work, but that comes at the price of stability and reduced functionnality and ease of maintenance.

Since none of the two use cases will disappear any time soon, the only solution to the problem is to share as much data as possible between the different tools and let each one decide how to install software on his computer system.

Some links to continue your readings on the same topic:

The every Linux distribution should have the same package manager fallacy by D. Cournapeau
Python Packaging: a few observations and a cabal for a solution by D. Cournapeau
Words on distribute, distutils and PEP 376, 386, 345 by Tarek Ziadé
Looking for a Windows Package Manager on Logilab's blog

EuroSciPy'09 (part 1/2): The Need For Speed

Nicolas Chauvat — 2009-07-29T20:43-01:00

The EuroSciPy2009 conference was held in Leipzig at the end of July and was sponsored by Logilab and other companies. It started with three talks about speed.

Starving CPUs

In his keynote, Fransesc Alted talked about starving CPUs. Thirty years back, memory and CPU frequencies where about the same. Memory speed kept up for about ten years with the evolution of CPU speed before falling behind. Nowadays, memory is about a hundred times slower than the cache which is itself about twenty times slower than the CPU. The direct consequence is that CPUs are starving and spend many clock cycles waiting for data to process.

In order to improve the performance of programs, it is now required to know about the multiple layers of computer memory, from disk storage to CPU. The common architecture will soon count six levels: mechanical disk, solid state disk, ram, cache level 3, cache level 2, cache level 1.

Using optimized array operations, taking striding into account, processing data blocks of the right size and using compression to diminish the amount of data that is transfered from one layer to the next are four techniques that go a long way on the road to high performance. Compression algorithms like Blosc increase throughput for they strike the right balance between being fast and providing good compression ratios. Blosc compression will soon be available in PyTables.

Fransesc also mentions the numexpr extension to numpy, and its combination with PyTables named tables.Expr, that nicely and easily accelerates the computation of some expressions involving numpy arrays. In his list of references, Fransesc cites Ulrich Drepper article What every programmer should know about memory.

Using PyPy's JIT for science

Maciej Fijalkowski started his talk with a general presentation of the PyPy framework. One uses PyPy to describe an interpreter in RPython, then generate the actual interpreter code and its JIT.

Since PyPy is has become more of a framework to write interpreters than a reimplementation of Python in Python, I suggested to change its misleading name to something like gcgc the Generic Compiler for Generating Compilers. Maciej answered that there are discussions on the mailing list to split the project in two and make the implementation of the Python interpreter distinct from the GcGc framework.

Maciej then focused his talk on his recent effort to rewrite in RPython the part of numpy that exposes the underlying C library to Python. He says the benefits of using PyPy's JIT to speedup that wrapping layer are already visible. He has details on the PyPy blog. Gaël Varoquaux added that David Cournapeau has started working on making the C/Python split in numpy cleaner, which would further ease the job of rewriting it in RPython.

CrossTwine Linker

Damien Diederen talked about his work on CrossTwine Linker and compared it with the many projects that are actively attacking the problem of speed that dynamic and interpreted languages have been dragging along for years. Parrot tries to be the über virtual machine. Psyco offers very nice acceleration, but currently only on 32bits system. PyPy might be what he calls the Right Approach, but still needs a lot of work. Jython and IronPython modify the language a bit but benefit from the qualities of the JVM or the CLR. Unladen Swallow is probably the one that's most similar to CrossTwine.

CrossTwine considers CPython as a library and uses a set of C++ classes to generate efficient interpreters that make calls to CPython's internals. CrossTwine is a tool that helps improving performance by hand-replacing some code paths with very efficient code that does the same operations but bypasses the interpreter and its overhead. An interpreter built with CrossTwine can be viewed as a JIT'ed branch of the official Python interpreter that should be feature-compatible (and bug-compatible) with CPython. Damien calls he approach "punching holes in C substrate to get more speed" and says it could probably be combined with Psyco for even better results.

CrossTwine works on 64bit systems, but it is not (yet?) free software. It focuses on some use cases to greatly improve speed and is not to be considered a general purpose interpreter able to make any Python code faster.

Logilab at OSCON 2009

Sandrine Ribeau — 2009-07-28T00:52-01:00

OSCON, Open Source CONvention, takes place every year and promotes Open Source for technology. It is one of the meeting hubs for the growing open source community. This was the occasion for us to learn about new projects and to present CubicWeb during a BAYPIGgies meeting hosted by OSCON.

I had the chance to talk with some of the folks working at OpenLina where they presented LINA. LINA is a thin virtual layer that enables developers to write and compile code using ordinary Linux tools, then package that code into a single executable that runs on a variety of operating systems. LINA runs invisibly in the background, enabling the user to install and run LINAfied Linux applications as if they were native to that user's operating system. They were curious about CubicWeb and took as a challenge to package it with LINA... maybe soon on LINA's applications list.

Two open sources projects catched my attention as potential semantic data publishers. The first one is Family search where they provide a tool to search for family history and genealogy. Also they are working to define a standard format to exchange citation with Open Library. Democracy Lab provide an application to collect votes and build geographic statitics based on political interests. They will at some point publish data semantically so that their application data could be consumed.

It also was for us the occasion of introducing CubicWeb to the BayPIGgies folks. The same presentation as the one held at Europython 2009. I'd like to take the opportunity to answer a question I did not manage to answer at that time. The question was: how different is CubicWeb from Freebase Parallax in terms of interface and views filters? Before answering this question let's detail what Freebase Parallax is.

Freebase Parallax provides a new way to browse and explore data in Freebase. It allows to browse data from a set of data to a related set of data. This interface enables to aggregate visualization. For instance, given the set of US presidents, different types of views could be applied, such as a timeline view, where the user could set up which start and end date to use to draw the timeline. So generic views (which applies to any data) are customizable by the user.

The search powered by Parallax is very similar to CubicWeb faceted search, except that Parallax provides the user with a list of suggested filters to add in addition to the default one, the user can even remove a filter. That is something we could think about for CubicWeb: provide a generated faceted search so that the user could decide which filters to choose.

Parallax also provides related topics to the current data set which ease navigation between sets of data. The main difference I could see with the view filter offered by Parallax and CubicWeb is that Parallax provides the same views to any type of data whereas CubicWeb has specific views depending on the data type and generic views that applies to any type of data. This is a nice Web interface to browse data and it could be a good source of inspiration for CubicWeb.

During this talk, I mentionned that CubicWeb now understands SPARQL queries thanks to the fyzz parser.

Quizz WolframAlpha

Nicolas Chauvat — 2009-07-10T16:00-01:00

Wolfram Alpha is a web front-end to huge database of information covering very different topics ranging from mathematical functions to genetics, geography, astronomy, etc.

When you search for a word, it will try to match it with one of the objects it as in its database and display all the information it has concerning that object. For example it can tell you a lot about the Halley Comet, including where it is at the moment you ask the query. This is the main difference with, say Wikipedia, that will know a lot about that comet in general, but is not meant to compute its location in the sky at the moment you enter your query.

Searches are not limited to words. One can key in commands like weather in Paris in june 2009 or x^2+sin(x) and get results for those precise queries. The processing of the input query is far from bad, since it returns results to questions like what are the cities of France, but I would not call it state of the art natural language processing since that query returns the largest cities instead of just the cities it knows about and the question what are the smallest cities of France will not return any result. Natural language processing is a very difficult problem, though, especially when done in the open world as it is the case there with a engine available to the wide public on the internet.

For more examples, visit the WolframAlpha website, where you will also be able to post feature requests or, if you are a developer, get documentation about the WolframAlpha API and maybe use it as a web service in your application when you need to answer certain types of questions.

EuroPython 2009

Nicolas Chauvat — 2009-07-06T17:35-01:00

Once again Logilab sponsored the EuroPython conference. We would like to thank the organization team (especially John Pinner and Laura Creighton) for their hard work. The Conservatoire is a very central location in Birmingham and walking around the city center and along the canals was nice. The website was helpful when preparing the trip and made it easy to find places where to eat and stay. The conference program was full of talks about interesting topics.

I presented CubicWeb and spent a large part of my talk explaining what is the semantic web and what features we need in the tools we will use to be part of that web of data. I insisted on the fact that CubicWeb is made of two parts, the web engine and the data repository, and that the repository can be used without the web engine. I demonstrated this with a TurboGears application that used the CubicWeb repository as its persistence layer. RQL in TurboGears! See my slides and Reinout Van Rees' write-up.

Christian Tismer took over the development of Psyco a few months ago. He said he recently removed some bugs that were show stoppers, including one that was generating way too many recompilations. His new version looks very promising. Performance improved, long numbers are supported, 64bit support may become possible, generators work... and Stackless is about to be rebuilt on top of Psyco! Psyco 2.0 should be out today.

I had a nice chat with Cosmin Basca about the Semantic Web. He suggested using Mako as a templating language for CubicWeb. Cosmin is doing his PhD at DERI and develops SurfRDF which is an Object-RDF mapper that wraps a SPARQL endpoint to provide "discoverable" objects. See his slides and Reinout Van Rees' summary of his talk.

I saw a lightning talk about the Nagare framework which refuses to use templating languages, for the same reason we do not use them in CubicWeb. Is their h.something the right way of doing things? The example reminds me of the C++ concatenation operator. I am not really convinced with the continuation idea since I have been for years a happy user of the reactor model that's implemented in frameworks liked Twisted. Read the blog and documentation for more information.

I had a chat with Jasper Op de Coul about Infrae's OAI Server and the work he did to manage RDF data in Subversion and a relational database before publishing it within a web app based on YUI. We commented code that handles books and library catalogs. Part of my CubicWeb demo was about books in DBpedia and cubicweb-book. He gave me a nice link to the WorldCat API.

Souheil Chelfouh showed me his work on Dolmen and Menhir. For several design problems and framework architecture issues, we compared the solutions offered by the Zope Toolkit library with the ones found by CubicWeb. I will have to read more about Martian and Grok to make sure I understand the details of that component architecture.

I had a chat with Martijn Faassen about packaging Python modules. A one sentence summary would be that the Python community should agree on a meta-data format that describes packages and their dependencies, then let everyone use the tool he likes most to manage the installation and removal of software on his system. I hope the work done during the last PyConUS and led by Tarek Ziadé arrived at the same conclusion. Read David Cournapeau's blog entry about Python Packaging for a detailed explanation of why the meta-data format is the way to go. By the way, Martijn is the lead developer of Grok and Martian.

Godefroid Chapelle and I talked a lot about Zope Toolkit (ZTK) and CubicWeb. We compared the way the two frameworks deal with pluggable components. ZTK has adapters and a registry. CubicWeb does not use adapters as ZTK does, but has a view selection mechanism that required a registry with more features than the one used in ZTK. The ZTK registry only has to match a tuple (Interface, Class) when looking for an adapter, whereas CubicWeb's registry has to find the views that can be applied to a result set by checking various properties:

interfaces: all items of first column implement the Calendar Interface,
dimensions: more than one line, more than two columns,
types: items of first column are numbers or dates,
form: form contains key XYZ that has a value lower than 10,
session: user is authenticated,
etc.

As for Grok and Martian, I will have to look into the details to make sure nothing evil is hinding there. I should also find time to compare zope.schema and yams and write about it on this blog.

And if you want more information about the conference:

photos tagged europython in flickr and gallery on the website
blogs reporting on europython
recorded talks and presentation materials

Semantic web technology conference 2009

Sandrine Ribeau — 2009-06-17T23:48-01:00

The semantic web technology conference is taking place every year in San Jose, California. It is meant to be the world's symposium on the business of semantic technologies. Essentially here we discuss about semantic search, how to improve access to the data and how we make sense of structured, but mainly unstructured content. Some exhibitors were more NLP oriented, concepts extraction (such as SemanticV), others were more focused on providing a scalable storage (essentially RDF storage). Most of the solutions includes a data aggregator/unifier in order to combine multi-sources data into a single storage from which ontologies could be defined. Then on top of that is the enhanced search engine. They concentrate on internal data within the enterprise and not that much about using the Web as a resource. For those who built a web application on top of the data, they choosed Flex as their framework (Metatomix).

From all the exhibitors, the ones that kept my attention were The Anzo suite (open source project), ORDI and Allegrograph RDF store.

Developped by Cambridge Semantics, in Java, Anzo suite, especially, Anzo on the web and Anzo collaboration server, is the closest tools to CubicWeb, providing a multi source data server and an AJAX/HTML interface to develop semantic web applications, customize views of the data using a templating language. It is available in open source. The feature that I think was interesting is an assistant to load data into their application that then helps the users define the data model based on that data. The internal representation of the content is totally transparent to the user, types are inferred by the application, as well as relations.

I did not get a demo of ORDI, but it was just mentionned to me as an open source equivalent to CubicWeb, which I am not too sure about after looking at their web site. It does data integration into RDF.

Allegrograph RDF store is a potential candidate for another source type in CubicWeb . It is already supported by Jena and Sesame framework. They developped a Python client API to attract pythonist in the Java world.

They all agreed on one thing : the use of SPARQL should be the standard query language. I quickly heard about Knowledge Interface Format (KIF) which seems to be an interesting representation of knowledge used for multi-lingual applications. If there was one buzz word to recall from the conference, I would choose ontology :)

IPMI plugin for Munin python code published

Arthur Lutz — 2009-06-17T16:52-01:00

As you might have noticed we quite like munin. We use it quite a bit to monitor how our servers and services are doing. One of the things we like about munin is obviously that the plugins can be written in python (and perl, bash and ruby).

On a few recent servers we started playing with IPMI to sensor the temperature, watts, fan's rpms etc. So we went out looking for a munin plugin for that. We found Peter Palfrader's ruby plugins. There was one small glitch though, we came across a simple bug : the "ipmitool -I open sensor" can be real long to execute on certain machines, so configuring the plugin was a bit painful and running it too. Changing the ruby code was a bit tricky since we don't really know ruby... so we did a quick rewrite of the plugin in python... with a few optimizations.

It's not really complete but works for us, and might be useful to you, so we're publishing the hg repo. You can get the tgz or browse the source.

Google I/O 2009

Sandrine Ribeau — 2009-06-10T07:17-01:00

The big event of the conference was the annoucement of Google Wave, a new online communication and collaboration tool, built on the Google Web Toolkit (GWT). Another big thing the GWT, lots of application built with it, a delightful tool for Java developers.

It was interesting to see that Google App Engine (GAE) will shortly provide an API to do offline processing, with objects called Task Queue. Task queue are web hooks, tasks are pushed to the server, queued and they are pushed until the task is executed (which overpass the annoying well-known time out issue with Google App Engine). It was introduced as asynchronous, enabling low latency, reliable and scalable (where are the buzz words?). Of course, this news has been a huge relief for most of the developers in the public as that was a big missing part from GAE.

An nice presentation from the founders of FrontSeat.org about their current project Walk score. They introduced themselves as civic software developers, writing software in a civic manner. They explained how they use GAE, and why they had to use Amazon EC2 to compensate GAE gaps. The gaps they listed here were the ranking non-ability of GAE, the too long reponse time for such computation they do, the fact that no cron jobs can be done (the arrival of Task Queue might change their opinion).

All the sessions have been recorded and are available here.

And yes, as all the participants of this conference, I went back home with an Android phone :)

hgview 1.0.0 released!

David Douard — 2009-06-05T21:55-01:00

I am pleased to introduce you to the latest kid of the Logilab team: hgview 1.0.0.

hgview is a very helpful tool for daily work using the excellent DVCS Mercurial (which we heavily use at Logilab). It allows to easily and visually navigate your hg repository revision graphlog. It is written in Python and pyqt.

This version is an almost complete rewrite of hgview 0.x which had two GUI backends, gtk and qt4. This 1.0 release drops the gtk backend (we may consider reintroducing it, we haven't decided yet... by the way, patches are always welcome). Some may not like this choice, but the immediate benefit of using qt4 is that hgview works like a charm on MacOS X systems.

Edit: there was a bug in hgview 1.0.0 on Ubuntu hardy. It's now fixed, and I've uploaded a 1.0.1 version deb package for hardy.

Features

4 different viewers:
- repository navigator that displays the graphlog efficiently (works well with 10,000 changesets),
- filelog navigator that displays the filelog of a file (follows files through renames),
- filelog diff navigator that displays the filelog in diff mode to easily track changes between two revisions of a file,
- manifest viewer that navigates in the files hierarchy as it was at a given revision.
Each viewer offers:
- easy keyboard navigation:
  - up/down to change revision,
  - left/right to change file (for the repo navigator only),
  - return to display the diff viewer of the selected file,
- search quickbar (Ctrl+F or /): search in graphlog (search as you type in the currently displayed file or diff, plus a cancellable background search in the revision tree),
- goto quickbar (Ctrl+G): go to the given revision (accepts id or tag, with completion for tags),
- navigation history: alt+left/alt+right to navigate backward/forward in the history,
can be used alone or as a hg extension,
can be configured using standard hg rc files (system, user or per repository),
possibility to declare users (with multiple mail addresses) and assign them a given color to make a given user look the same in all your repositories,

Download and installation

The source code is available as a tarball, or using our public hg repository of course.

To use it from the sources, you just have to add a line in your .hgrc file, in the [extensions] section:

hgext.hgview=/path/to/hgview/hgext/hgview.py

Debian and Ubuntu users can also easily install hgview (and Logilab other free software tools) using our deb package repositories.

The Web is reaching version 3

Nicolas Chauvat — 2009-06-05T20:18-01:00

I presented CubicWeb at several conferences recently and I used the following as an introduction.

Web version numbers:

version 0 = the internet links computers
version 1 = the web links documents
version 2 = web applications
version 3 = the semantic web links data [we are here!]
version 4 = more personnalization and fix problems with privacy and security
... reach into physical world, bits of AI, etc.

In his blog at MIT, Tim Berners-Lee calls version 0 the International Information Infrastructure, version 1 the World Wide Web and version 3 the Giant Global Graph. Read the details about the Giant Global Graph on his blog.

Nous allons à PyConFr 2009

Arthur Lutz — 2009-05-25T15:05-01:00

Le 30 et 31 mai prochain (samedi et dimanche prochain) nous allons être présents à PyConFr édition 2009, nous sommes partenaire de l'évènement et allons y parler de CubicWeb. Pour être plus précis, Nicolas Chauvat y présentera "CubicWeb pour publier DBpedia et OpenLibrary". Il avait déjà évoqué ces sujets sur ce site : Fetching book descriptions and covers et DBpedia 3.2 released.

Si vous comptez y aller, n'hésitez pas à venir nous dire bonjour.

Almost reached 1000 tickets

Arthur Lutz — 2009-05-13T10:30-01:00

Logilab.org has almost reached a thousand tickets on the Logilab's open source projects. To be exact there are 940 tickets right now. What kind of tickets are they ?

Here is a quick graph of the state of the tickets in our tracker :

Graphing is neat. Maybe soon we'll get this kind of feature automatically in the CubicWeb forge, see this ticket.

Reading SPE files

Andre Espaze — 2009-05-11T15:37-01:00

If you would like to read SPE files from charge-coupled device (CCD) cameras, I have contributed a recipe to the SciPy cookbook, see Reading SPE files.

Fetching book descriptions and covers

Nicolas Chauvat — 2009-05-11T06:34-01:00

We recently added the book cube to our intranet in order for books available in our library to show up in the search results. Entering a couple books using the default HTML form, even with the help of copy/paste from Google Book or Amazon, is boring enough to make one seek out other options.

As a Python and Debian user, I put the python-gdata package on my list of options, but quickly realized that the version in Debian is not current and that the books service is not yet accessible with the python gdata client. Both problems could be easily overcome since I could update Debian's version from 1.1.1 to the latest 1.3.1 and patch it with the book search support that will be included in the next release, but I went on exploring other options.

Amazon is the first answer that comes to mind when speaking of books on the net and pyAWS looks like a nice wrapper around the Amazon Web Service. The quickstart example on the home page does almost exactly what I was looking for. Trying to find a Debian package of pyAWS, I only came accross boto which appears to be general purpose.

Registering with Amazon and Google to get a key and use their web services is doable, but one wonders why something as common as books and public libraries would have to be accessed through private companies. It turns out Wikipedia knows of many book catalogs on the net, but I was looking for a site publishing data as RDF or part of the Linked Open Data initiative. I ended up with almost exactly what I needed.

The Open Library features millions of books and covers, publicly accessible as JSON using its API. There is even a dump of the database. End of search, be happy.

Next step is to use this service to enhance the cubicweb-book cube by allowing a user to add a new book to its collection by simply entering a ISBN. All data about the book can be fetched from the OpenLibrary, including the cover and information about the author. You can expect such a new version soon... and we will probably get a new demo of CubicWeb online in the process, since all that data available as a dump is screaming for reuse as others have already found out by making it available as RDF on AppEngine!

iclassmethod decorator to define both a class and an instance method in one go

Sylvain Thenault — 2009-04-28T09:27-01:00

You'll find in the logilab.common.decorators module the iclassmethod decorator which may be pretty handy in some cases as it allows methods to be both called as class methods or as instance methods. In the first case the first argument will be the class and the second case it will be the instance.

Example extracted (and adapted for simplicity) from CubicWeb:

from logilab.common.decorators import iclassmethod

class Form(object):
  _fields_ = []

  def __init__(self):
      self.fields = list(self._fields_)

  @iclassmethod
  def field_by_name(cls_or_self, name):
      """return field with the given name and role"""
      if isinstance(cls_or_self, type):
          fields = cls_or_self._fields_
      else:
          fields = cls_or_self.fields
      for field in fields:
          if field.name == name:
              return field
      raise Exception('FieldNotFound: %s' % name)

Example session:

>>> from logilab.common import attrdict
>>> f = Form()
>>> f.fields.append(attrdict({'name': 'something', 'value': 1})
>>> f.field_by_name('something')
{'name': 'something', 'value': 1}
>>> Form.field_by_name('something')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 15, in field_by_name
Exception: FieldNotFound: something

So we get a field_by_name method which will act differently (actually use different input data) when called as instance method or as class method.

Also notice the attrdict trick that can also be achieved with the Python 2.6 named tuple.

12 ans de l'April: bilan et objectifs

Nicolas Chauvat — 2009-04-05T18:45-01:00

L'April fête cette année ses douze ans. L'association de promotion et de défense du logiciel libre approche maintenant les 5000 membres, toutes catégories confondues: personnes physiques, entreprises commerciales, collectivités, associations. Elle vient de publier son rapport moral 2008 et sa feuille de route 2014 qui fixe des objectis pour les cinq années à venir. On notera en particulier la lutte contre les quatre dangers que sont les brevets sur les algorithmes, les dispositifs de contrôle d'usage, la vente liée et l'informatique déloyale. Consultez le site de l'April pour en savoir plus sur ses actions et n'hésitez pas à adhérer ou à donner quelques heures de votre temps.

Emacs and mercurial trick

Ludovic Aubry — 2009-04-03T15:01-01:00

While using emacs I always find the need to use grep on a terminal to search for things within a specific project. This is not ideal, even within an embedded emacs shell.

Since I recently discovered the emacs command grep-find I decided to make it work nicely with mercurial projects, and here's the result:

(defun grep-project (s)
  (interactive "sSearch project for: ")
   (grep-find (concat "files=(`hg manifest`);grep -nH -e \"" s "\" ${files[@]/#/`hg root`/}"))
)
(global-set-key (quote [f4]) (quote grep-project))

Venez nous rendre visite à Solution Linux 2009

Arthur Lutz — 2009-03-31T10:00-01:00

Nous sommes dès ce matin, pendant 3 jours, présents au salon Solutions Linux 2009 au stand du pôle de compétition System@tic dont nous faisons parti. C'est le stand B4/B8, assez prêt de l'entrée sur la gauche (détails).

Nous allons présenter CubicWeb à plusieurs reprises sur le stand, ainsi que lors des conférences sur le Web2 ce mardi 31 mars de 14h à 17h30 :

Adrien présentera "Simile Widgets, des composants de haut niveau pour IHM web"
Sylvain présentera "Cubic 3.0 - une plateforme pour les applications web sémantique"

pour plus de détails consultez le programme.

Pylint at BayPIGgies

Sandrine Ribeau — 2009-03-31T00:42-01:00

I am pleased to announce that Pylint was presented during a Tools night meeting organized by BayPIGgies on thursday march 26th. This meeting has been recorded and you can enjoy the video.

One point was missing from the presentation and I'll take the opportunity now to mention it. Flymake, an on-the-fly syntax checker for GNU Emacs which has been discussed, does work in combination with Pylint (please see EmacsWiki for more informations).

photo by ten safe frogs under creative commons

new pylint / astng / common releases

Sylvain Thenault — 2009-03-25T11:37-01:00

I'm pleased to announce releases of pylint 0.18, logilab-astng 0.19 and logilab-common 0.39. All these packages should now be cleanly available through easy install.

Also, happy pylint users will get:

fixed python 2.6 support (pylint/astng tested from 2.4 to 2.6)
get source code (and so astng) for zip/egg imports
some understanding of the property decorator and of unbound methods
some false positives fixed and others minor improvments

See projects home page and ChangeLog for more information:

http://www.logilab.org/project/pylint http://www.logilab.org/project/logilab-astng http://www.logilab.org/project/logilab-common

Please report any problem / question to the python-projects@lists.logilab.org mailing-list.

Enjoy!

broken easy_install support

Sylvain Thenault — 2009-03-25T10:12-01:00

I recently understood why easy_install wasn't able to find so many of our packages anymore.

The problem was due to a recent change on our website. The project page was ajaxified, and since easy_install uses some screenscrapping techniques to get distribution archives, it can not find the files it is looking for.

To fix this, we should make our tarballs downloadable from PyPI, by using

python setup.py register sdist upload

instead of the current:

python setup.py register

Uploading our public python software packages to PyPI will make them easy_installable in a breeze !

Pylint and Astng support for the _ast module

Emile Anclin — 2009-03-19T18:16-01:00

Supporting _ast and compiler

Python 2.5 introduces a new module _ast for Abstract Syntax Tree (AST) representation of python code. This module is quite faster than the compiler.ast representation that logilab-astng (and therefore pylint) used until now and the compiler module was removed in Python 3.0.

Faster is good, but the representations of python code are quite different in _ast and in compiler : some nodes exist in one AST but not the other and almost all child nodes have different names.

We had to engage in a big refactoring to use the new _ast module, since we wanted to stay compatible with python version < 2.5, which meant keeping the compiler module support. A lot of work was done to find a common representation for the two different trees. In most cases we used _ast-like representations and names, but in some cases we kept ideas or attribute names of compiler.

Abstract Syntax Trees

Let's look at an example to compare both representations. Here is a seamingly harmless snippet of code:

CODE = """
if cond:
    del delvar
elif next:
    print
"""

Now, compare the respective _ast and compiler representations (nodes are in upper case and their attributes are in lower case).

compiler representation

Module
    node =
    Stmt
        nodes = [
        If
            tests = [
            Name
                name = 'cond'
            Stmt
                nodes = [
                AssName
                    flags = 'OP_DELETE'
                    name = 'delvar'
                ]
            Name
                name = 'next'
            Stmt
                nodes = [
                Printnl
                ]
            ]

_ast representation

Module
    body = [
    If
        test =
        Name
            id = 'cond'
        body = [
        Delete
            targets = [
            Name
                id = 'delvar'
            ]
        ]
        orelse = [
        If
            test =
            Name
                id = 'next'
            body = [
            Print
                nl = True
            ]
        ]
    ]

Can you spot any differences? I would say, they differ quite a lot... For instance, compiler turns a "elif" statements into a list called 'tests' when _ast treats "elif cond:" as if it were "else:if cond:".

Tree Rebuilding

We transform these trees by renaming attributes and nodes, or removing or introducing new ones: with compiler, we remove the Stmt node, introduce a Delete node, and recursively build the If nodes coming from an "elif"; and with _ast, we reintroduce the AssName node. This might be only a temporary step towards full _ast like representation.

This is done by the TreeRebuilder Visitors, one for each representation, which are respectively in astng._nodes_compiler and astng._ast.

In the simplest case, the TreeRebuilder method looks like this (_nodes_compiler):

def visit_list(self, node):
    node.elts = node.nodes
    del node.nodes

(and nothing to do for _ast).

So, after doing all this and a lot more, we get the following representation from both input trees:

Module()
    body = [
    If()
        test =
        Name(cond)
        body = [
        Delete()
            targets = [
            DelName(delvar)
            ]
        ]
        orelse = [
        If()
            test =
            Name(next)
            body = [
            Print()
                dest =
                None
                values = [
                ]
            ]
            orelse = [
            ]
        ]
    ]

Faster towards Py3k

Of course, you can imagine these modifications had some API repercussions, and thus required a lot of smaller Pylint modifications. But all was done so that you should see no difference in Pylint's behavior using either python <2.5 or python >=2.5, except that with the _ast module pylint is around two times faster!

Oh, and we fixed small bugs on the way and maybe introduced a few new ones...

Finally, it is a major step towards Pylint Py3k!

hgview 0.10.2 released

Graziella Toutoungis — 2009-03-13T09:13-01:00

I have the pleasure of announcing that the version hgview 0.10.2 was posted on this site and is available for downloading. In this version we added some new functionalities like :

Change the search behavior: the button "Next" will move the focus on the next found searched text.
Diff works on the merge node.
The command --version shows the current version of hgview
Fix a bug when the file's name contains space.

New version of projman

2009-03-10T14:14-01:00

Projman is a project management tool. It reads project descriptions and activity logs to schedule tasks and to generate gantt diagrams.

The version 0.13.6 fix some gantt diagram generation bugs. Graphics library helper is now Cairo instead of matplotlib. Some examples, according to the new model (use of resources type and roles in tasks) have been added to the project.

And, of course, the compulsory screenshot.

Belier - le ssh par hops

Arthur Lutz — 2009-02-17T16:54-01:00

On vient de découvrir belier qui permet de se connecter facilement à des machines auquelles on doit accéder par des machines ssh intermédiaires. Ca peut s'avérer utile. En plus, c'est en python. En plus, il a fait des paquets debian... et en plus il mentionne pylint. Du coup il mérite mention ici.

Debian Lenny release date - almost there ?

Arthur Lutz — 2009-02-09T15:38-01:00

Being big fans of debian, we are impatiently awaiting the new stable release of the distribution : lenny. Finding it pretty difficult to find information about when they were expecting to release it, I asked a colleague if he knew. He's a debian developer so I though he might have the info. And he did : according to the debian.devel mailing list we should be having the release for the 14th of February 2009. In other words : in 5 days!

http://thread.gmane.org/gmane.linux.debian.devel.announce/1318

There's a few geeky emails on the release date if you have time to read the threads.

LUTIN77: Logilab Unit Test IN fortran 77

Andre Espaze — 2009-01-28T15:19-01:00

We've just released a new project on logilab.org : lutin77. It's a test framework for Fortran77.

The goal of this framework is to make unit tests in fortran 77 by having few dependencies: a POSIX environment with C and fortran 77 compilers. Of course, you can use it for making integration or acceptance tests too. The 0.1 version has just been released here: http://www.logilab.org/project/lutin77

If you are new to the unit tests way of building software, I must admit it lacks examples. For an introduction to the techniques involved, you can have a look at Growing Object-Oriented Software, Guided by Tests even if mocked subroutines will be for later. But remember that if you do not like to write tests, you are probably not writing unit tests.

Résumé de la rencontre francophone autour des forges du 21 janvier 2009

Julien Jehannet — 2009-01-27T20:02-01:00

Définition d'une forge

Je vous propose tout d'abord une définition assez complète d'une forge logicielle.

« Une forge ou plate-forme d'hébergement de projets logiciels est un ensemble réunissant les technologies du travail coopératif et du génie logiciel pour permettre le développement coordonné de logiciels en équipe. Les services de base d'une telle plate-forme sont axés sur le partage de fichiers (code source, données et exécutables) et l'animation du groupe. Ils permettent la rédaction et la programmation collaborative, et facilitent la communication dans le groupe grâce à des outils associés aux projets tels que des gestionnaires de listes de messagerie, des logiciels de suivi des tâches et de gestion des rapports d'anomalies. L'utilisation d'une plate-forme de ce type améliore la qualité des réalisations en accélérant les processus d'échange entre développeurs et les cycles de version des logiciels, tout en facilitant l'implication des utilisateurs dans la détection des erreurs ou la mise en lumière des fonctionnalités pertinentes des logiciels.»

Tirée de http://overcrowded.anoptique.org/ForgesEtatArt (licence Art Libre, créateur initial Benoît Sibaud, diffusable sous LAL, CC By, CC By-SA et GFDL).

Voir:

Problèmes recensés

Je liste ici quelques problèmes qui m'ont semblé émerger pendant les discussions de la journée.

forks nombreux du projet GForge avec peu de collaboration entre les projets
périmètre technique souvent imposé dans le débat; ce qui gêne la définition fonctionnelle d'une forge
forte hiérarchie des responsabilités avec un rôle 'développeur' limité au profit de celui du 'chef de projet'
très orienté web avec une stratégie d'intégration de produits tiers; d'où une situation délicate pour l'homogénéité des solutions à cause des limitations du protocole HTTP
la méthodologie de conception des logiciels n'est pas évoquée dans le choix d'une solution (et pas d'approche Agile en général).

Présentations

ALM, cycle de vie des logiciels et poste client

Voir:

http://en.wikipedia.org/wiki/Application_Lifecycle_Management

Projet NovaForge

Forge très aboutie où une approche MDA a été mise en oeuvre pour la génération de cas de tests à partir des cas d'utilisation décrits en UML.

Intégration continue

Plusieurs outils ont été nommés. J'ai placé la liste sur la page du wiki dédiée.

Interopérabilité sémantique

Projet xfmigration

Ce projet vise à migrer le contenu d'une forge (BerliOS) vers GForge sans passer par le backend SGBD. L'idée est alors d'utiliser différentes ontologies pour 'mapper' les concepts des 2 forges.

Cross-Forge Migration Tool is a concept for facilitating the migration process of project metadata between forge platforms. It uses SWRL rules for mapping and OWL-S standard for exporting mapping results.

Voir:

Projet HELIOS

L'exposé présentait un cas d'utilisation du web sémantique pour la recherche et le tri de rapport d'anomalies sur des forges séparées. L'exemple utilisé a été de pouvoir rapatrier; puis sélectionner des tickets relatifs au projet Konqueror à travers le bugzilla de Debian et celui du projet lui-même. Il est ainsi possible de détecter certains doublons ou incohérences de versions.

Projet CubicWeb forge

J'ai pu présenter le framework CubicWeb et son architecture de composants. La présentation n'étant pas prévue initialement mais j'ai pû faire une démonstration à partir de la forge publique de Logilab.

À partir d'un schéma enrichi des composants installés, il est possible de faire des requêtes RQL (similaire à SPARQL) sur des entités et des sources distantes (Base de données, LDAP, subversion, ...). Logilab travaille aujourd'hui à l'ajout d'ontologies dans la manipulation de ce schéma.

Remerciements

Je remercie les organisateurs de cette rencontre édition 2009 ainsi que l'ensemble des intervenants propices à l'échange de points de vue.

Apycot big version change

Arthur Lutz — 2009-01-26T17:12-01:00

The version convention that we use is pretty straight forward and standard : it's composed of 3 numbers separated by dots. What are the rules to incrementing each on of these numbers ?

The last number is a incremented when bugs are corrected
The middle number is incremented when stories (functionalities) are implemented to the software
The first number is incremented when we have a major change of technology

Well... if you've been paying attention, apycot just turned 1.0.0, the major change of technology is that it is now integrated to CubicWeb (instead of just generating html files). So for a project in your forge, you describe the apycot configuration for it, and the tests for quality assurance are launched on a regular basis. We're still in the process of stabilizing it (latest right now it 1.0.5), but it already runs on the CubicWeb projects, see the screenshot below :

You should also know that now apycot has two components : the apycotbot which runs the tests and an cubicweb-apycot which displays the results in cubicweb (download cubicweb-apycot-1.0.5.tar.gz and apycotbot-1.0.5.tar.gz).

We're now publishing for Ubuntu aswell

Arthur Lutz — 2009-01-26T15:55-01:00

We've always been big fans of debian here at Logilab. So publishing debian packages for our open source software has always been a priority.

We're now a bit involved with Ubuntu, work with it on some client projects, have a few Ubuntu machines lying around, and we like it too. So we've decided to publish our packages for Ubuntu as well as for debian.

In the 0.12.1 version of logilab-devtools we introduced publishing of Ubuntu packages with lgp (Logilab Packaging) - see ticket. Since then, you can add the following Ubuntu source to your Ubuntu system

deb http://ftp.logilab.org/dists hardy/

For now, only hardy is up and running, give us a shout if you want something else!

Rencontre francophone autour des forges

Julien Jehannet — 2009-01-19T12:12-01:00

Logilab sera présent le Mercredi 21 janvier à la rencontre francophone autour des forges. Les présentations et les discussions porteront sur :

échange de données entre forges, interopérabilité
définition d’un modèle d’intégration ouvert
recherche multi-forges
gestion des permissions et partage d’identités
interaction entre la forge et le poste client

Logilab espère ainsi bénéficier de l'expérience de chacun et des pistes d'évolutions pour, à terme, les intégrer dans son projet de forge basé sur CubicWeb.

Toutes les informations de l'événement sont disponibles ici.

photo par StormyDog sous licence Creative Commons.

Release of CubicWeb 3.0

Nicolas Chauvat — 2009-01-05T09:48-01:00

As some readers of this blog may be aware of, Logilab has been developing its own framework since 2001. It evolved over the years trying to reach the main goal (managing and publishing data with style) and to incorporate the goods ideas seen in other Python frameworks Logilab developers had used. Now, companies other than Logilab have started providing services for this framework and it is stable enough for the core team to be confident in recommending it to third parties willing to build on it without suffering from the tasmanian devil syndrom.

CubicWeb version 3.0 was released on the last day of 2008. That's 7 years of research and development and (at least) three rewrites that were needed to get this in shape. Enjoy it at http://www.cubicweb.org/ !

hgview 0.10.0

Graziella Toutoungis — 2008-12-30T10:44-01:00

I have the pleasure of announcing that the version hgview 0.10.0 was posted on this site and is available for downloading. In this version we added some new functionalities like :

The possibility to order all revisions by date or author or description.....
Support for localtime.
Improve the message header when hg mv is used and fix the author base color
Integration of bboissin's fixes

Finally : We have taken into account older versions. As pointed out by some users, mercurial version 1.1.x wasn't working very well with hgview, so we created patches which have to be applied according to the version of mercurial you are using.

Pyreverse : UML Diagrams for Python

Emile Anclin — 2008-12-23T17:39-01:00

Pyreverse analyses Python code and extracts UML class diagrams and package depenndencies. Since september 2008 it has been integrated with Pylint (0.15).

Introduction

Pyreverse builds a diagram representation of the source code with:

class attributes, if possible with their type
class methods
inheritance links between classes
association links between classes
representation of Exceptions and Interfaces

Generation of UML diagrams with Pyreverse

The command pyreverse generates the diagrams in all formats that graphviz/dot knows, or in VCG :

The following command shows what dot knows:

$ dot -Txxx
Format: "xxx" not recognized. Use one of: canon cmap cmapx cmapx_np dia dot
eps fig gd gd2 gif hpgl imap imap_np ismap jpe jpeg jpg mif mp pcl pdf pic
plain plain-ext png ps ps2 svg svgz tk vml vmlz vrml vtx wbmp xdot xlib

pyreverse creates by default two diagrams:

$ pyreverse -o png -p Pyreverse pylint/pyreverse/
[...]
creating diagram packages_Pyreverse.png
creating diagram classes_Pyreverse.png

-o : sets the output format
-p name : yields the output files packages_name.png and classes_name.png

Package Diagram

Class Diagram

class diagram full size image

Options

One can modify the output with following options:

-a N, -A    depth of research for ancestors
-s N, -S    depth of research for associated classes
-A, -S      all ancestors, resp. all associated
-m[yn]      add or remove the module name
-f MOD      filter the attributes : PUB_ONLY/SPECIAL/OTHER/ALL
-k          show only the classes (no attributes and methods)
-b          show 'builtin' objects

Examples:

General Vue on a Module

pyreverse -ASmy -k -o png pyreverse/main.py -p Main

full size image

With these options you can have a quick vue of the dependencies without being lost in endless lists of methods and attributes.

Detailed Vue on a Module

pyreverse -c PyreverseCommand -a1 -s1 -f ALL -o png  pyreverse/main.py

module in full size image

Show all methods and attributes of the class (-f ALL). By default, the class diagram option -c uses the options -A, -S, -my, but here we desactivate them to get a reasonably small image.

Configuration File

You can put some options into the file ".pyreverserc" in your home directory.

Exemple:

--filter-mode=PUB_ONLY --ignore doc --ignore test

This will exclude documentation and test files in the doc and test directories. Also, we will see only "public" methods.

Fusionner dans le bon sens avec mercurial

Nicolas Chauvat — 2008-11-27T14:16-01:00

Contrairement à ce que l'on pourrait penser a priori, il y a un bon et un mauvais sens pour fusionner deux têtes dans en entrepôt mercurial.

Prenons un exemple et admettons que l'on parte d'une révision 123. Alfred, Barnabé et Coruscant font des changements qui produisent l'arbre suivant:

123 -> 124 -> 125 ---> 126 -> 129 --> 130
                   \-> 127 -> 128 /

Si dans le même temps, Zoé part de la révision 123 et produit l'arbre:

123 -> 131

quand Zoé va vouloir faire son hg push sur l'entrepôt partagé, elle va avoir le message "ça crée des têtes, est-ce que vous êtes sûr de vouloir faire ce push". Zoé va se dire qu'il vaut mieux commencer par un 'pull' et un 'merge', mais c'est là qu'il faut se méfier.

En effet, si Zoé est sur la révision 131 et qu'elle fusionne avec 130, le changeset sera énorme car il contiendra tous les changements qui permettent de passer de 123 à 130. En revanche, si Zoé passe sur la 130 et fusionne avec la 131, elle n'aura que les changements qu'elle vient d'apporter. Dans le premier cas, le 'merge' sera difficile à comprendre, alors qu'il sera bien plus simple dans le second.

Au final, comment faire la fusion dans le "bon" sens ?

1/ hg push -> ah tient, ça crée des têtes, donc je ne le fais pas

2/ hg pull -u ou hg pull suivi de hg merge -> y'a eu un merge

3/ hg status -> beaucoup de fichiers modifiés... oulàlà, c'est bizarre je vais regarder

4/ hgview -> ah oui, sur l'autre tête y'a beaucoup plus de changements, donc je vais plutôt faire le 'merge' dans l'autre sens

5/ hg up -C ab123_autre_tete -> je me retrouve sur l'autre tête

6/ hg merge cd456_ma_tete -> un joli 'merge'

7/ hg status -> seulement quelques fichiers modifés, voilà qui est mieux

8/ hg ci -m merge -> youpi, je valide cette fusion

9/ hg push -> tagada, je partage avec mes voisins

Moralité: faites des hg status et des hgview aussi souvent que nécessaire pour préparer vos commits et parvenir à un historique facile à comprendre.

Remarque: on peut aussi remplacer les étapes 5, 6 et 7 par hg diff -r ab123_autre_tete -r cd456_ma_tete pour afficher le diff de ma_tete par rapport à autre_tete, car l'opération de fusion doit être symétrique et donner le même résultat qu'on la lance depuis l'une ou l'autre tête, même si l'affichage par défaut de hg status et hg diff dépend de la tête qui constitue le premier parent (l'étape 5 ayant justement pour effet de changer ce premier parent).

Javascript date support

Adrien Di Mascio — 2008-11-27T09:51-01:00

Coming from the python and mx.DateTime world, the javascript Date object is not really appealing. For me, the most disturbing things are :

The year parameter in the Date constructor is always considered as a XXe century year if year < 100. (this goes along with the getYear / getFullYear distinction).
The inconsistency between months and days indexes : months indexes starts at 0 whereas days indexes starts at 1.
The lack of decent strptime / strftime functions (even basic ones not taking locales into account).

Recently, I've worked with the great Timeline project which makes an heavy use of dates and I had the need for a very basic strptime implementation. This can by no mean be considered as a comprehensive implementation, but it might help so here it is:

var _DATE_FORMAT_REGXES = {
    'Y': new RegExp('^-?[0-9]+'),
    'd': new RegExp('^[0-9]{1,2}'),
    'm': new RegExp('^[0-9]{1,2}'),
    'H': new RegExp('^[0-9]{1,2}'),
    'M': new RegExp('^[0-9]{1,2}')
}

/*
 * _parseData does the actual parsing job needed by `strptime`
 */
function _parseDate(datestring, format) {
    var parsed = {};
    for (var i1=0,i2=0;i1<format.length;i1++,i2++) {
    var c1 = format[i1];
    var c2 = datestring[i2];
    if (c1 == '%') {
        c1 = format[++i1];
        var data = _DATE_FORMAT_REGXES[c1].exec(datestring.substring(i2));
        if (!data.length) {
            return null;
        }
        data = data[0];
        i2 += data.length-1;
        var value = parseInt(data, 10);
        if (isNaN(value)) {
            return null;
        }
        parsed[c1] = value;
        continue;
    }
    if (c1 != c2) {
        return null;
    }
    }
    return parsed;
}

/*
 * basic implementation of strptime. The only recognized formats
 * defined in _DATE_FORMAT_REGEXES (i.e. %Y, %d, %m, %H, %M)
 */
function strptime(datestring, format) {
    var parsed = _parseDate(datestring, format);
    if (!parsed) {
    return null;
    }
    // create initial date (!!! year=0 means 1900 !!!)
    var date = new Date(0, 0, 1, 0, 0);
    date.setFullYear(0); // reset to year 0
    if (parsed.Y) {
    date.setFullYear(parsed.Y);
    }
    if (parsed.m) {
    if (parsed.m < 1 || parsed.m > 12) {
        return null;
    }
    // !!! month indexes start at 0 in javascript !!!
    date.setMonth(parsed.m - 1);
    }
    if (parsed.d) {
    if (parsed.m < 1 || parsed.m > 31) {
        return null;
    }
    date.setDate(parsed.d);
    }
    if (parsed.H) {
    if (parsed.H < 0 || parsed.H > 23) {
        return null;
    }
    date.setHours(parsed.H);
    }
    if (parsed.M) {
    if (parsed.M < 0 || parsed.M > 59) {
        return null;
    }
    date.setMinutes(parsed.M);
    }
    return date;
}

// and now monkey patch the Timeline's parser ...
/* provide our own custom date parser since the default
 * one only understands iso8601 and gregorian dates
 */
Timeline.NativeDateUnit.getParser = function(format) {
    if (typeof format == "string") {
    if (format.indexOf('%') != -1) {
        return function(datestring) {
            if (datestring) {
                return strptime(datestring, format);
            }
            return null;
        };
    }
        format = format.toLowerCase();
    }
    if (format == "iso8601" || format == "iso 8601") {
    return Timeline.DateTime.parseIso8601DateTime;
    }
    return Timeline.DateTime.parseGregorianDateTime;
};

We're open for a chat

Arthur Lutz — 2008-11-25T17:40-01:00

We have a public forum that is accessible both using XMPP (jabber) or IRC.

Jabber / XMPP

Our jabber server is jabber.logilab.org.

If you don't have a jabber account, create one on a server such as jabber.org (here is a list of free jabber services) or use our web based client.

Once you have a jabber account, come and join us at xmpp://public@conference.jabber.logilab.org

If you do not know what jabber is, read the wikipedia page about jabber

IRC / International Relay Chat

Connect to irc://irc.logilab.org and join #public

If you do not know what irc is, read the wikipedia page about irc.

DBpedia 3.2 released

Nicolas Chauvat — 2008-11-19T22:59-01:00

For those interested in the Semantic Web as much as we are at Logilab, the announce of the new DBpedia release is very good news. Version 3.2 is extracted from the October 2008 Wikipedia dumps and provides three mayor improvements: the DBpedia Schema which is a restricted vocabulary extracted from the Wikipedia infoboxes ; RDF links from DBpedia to Freebase, the open-license database providing about a million of things from various domains ; cleaner abstracts without the traces of Wikipedia markup that made them difficult to reuse.

DBpedia can be downloaded, queried with SPARQL or linked to via the Linked Data interface. See the about page for details.

It is important to note that ontologies are usually more of a common language for data exchange, meant for broad re-use, which means that they can not enforce too many restrictions. On the opposite, database schemas are more restrictive and allow for more interesting inferences. For example, a database schema may enforce that the Publisher of a Document is a Person, whereas a more general ontology will have to allow for Publisher to be a Person or a Company.

DBpedia provides its schema and moves forward by adding a mapping from that schema to actual ontologies like UMBEL, OpenCyc and Yago. This enables DBpedia users to infer from facts fetched from different databases, like DBpedia + Freebase + OpenCyc. Moreover 'checking' DBpedia's data against ontologies will help detect mistakes or weirdnesses in Wikipedia's pages. For example, if data extracted from Wikipedia's infoboxes states that "Paris was_born_in New_York", reasoning and consistency checking tools will be able to point out that a person may be born in a city, but not a city, hence the above fact is probably an error and should be reviewed.

With CubicWeb, one can easily define a schema specific to his domain, then quickly set up a web application and easily publish the content of its database as RDF for a known ontology. In other words, CubicWeb makes almost no difference between a web application and a database accessible thru the web.

Semantic Roundup - infos glanées au NextMediaCamp

Arthur Lutz — 2008-11-12T12:42-01:00

Par manque de temps voici les infos en brut glanées jeudi soir dernier au NextMediaBarCamp :

Un BarCamp c'est assez rigolo, un peu trop de jeune cadres dynamiques en cravate à mon goût, mais bon. Parmi les trois mini-conférences auxquelles j'ai participé, il y avait une sur le web sémantique. Animée en partie par Fabrice Epelboin qui écrit pour la version française de ReadWriteWeb, j'ai appris des choses. Pour résumer ce que j'y ai compris : le web sémantique il y a deux approches :

soit on pompe le contenu existant et on le transforme en contenu lisible par les machines avec des algorithmes de la mort, comme opencalais le fait (top-down)
soit on écrit nous même en web sémantique avec des microformats ensuite les machines liront de plus en plus le web (bottom-up)

Dans le deuxième cas la difficulté est de faciliter la tache des rédacteurs du web pour qu'ils puissent facilement publier du contenu en web sémantique. Pour cela ces outils sont mentionnés : Zemanta, Glue (de la société AdaptiveBlue).

Tout ça m'a fait penser au fait que si CubicWeb publie déjà en microformats, il lui manque une interface d'édition à la hauteur des enjeux. Par exemple lorsque l'on tape un article et que l'application a les coordonnées d'une personne metionnée, il fait automagiquement une relation à cet élément. A creuser...

Sinon sur les autres confs et le futur des médias, selon les personnes présentes, c'est assez glauque : des medias publicitaires, custom-made pour le bonheur de l'individu, où l'on met des sous dans les agrégateurs de contenu plutôt sur des journalistes de terrain. Pour ceux que ça intéresse, j'ai aussi découvert lors de ce BarCamp un petit film "rigolo" qui traite ce sujet préoccupant.

Using branches in mercurial

Arthur Lutz — 2008-10-14T12:52-01:00

The more we use mercurial to manage our code repositories, the more we enjoy its extended functionalities. Lately we've been playing and using branches which end up being very useful. We also use hgview instead of the built-in "hg view" command. And its latest release supports the branches functionality, you can filter out the branch you want to look at. Update your installation (apt-get upgrade ?) to enjoy this new functionality... or download it.

A new way of distributing Python code ?

Julien Jehannet — 2008-09-28T17:02-01:00

On distutils-sig, the question of distutils/setuptools replacing is frequently raised and a lot of effort is made to find what would be the best way to build and distribute python code.

I don't understand the reason why we have a massive coupling between build and distribution (setuptools and pypi to be more precise) and I'm not convinced about this "global" approach. I hope the python community will examine the possibility to change that and split the problem in two distinct projects.

One of the most successful ideas of Python is its power in extending other languages. And in fact, that's the major problem to solve for the build area. I'm pretty sure it will take a long time before obtaining a valuable (and widely adopted) solution and this is so complicated that the choice of the building chain should be kept under the responsibility of the upstream maintainers for now (distutils, setuptools, makefile, SCons, ...).

Concerning the distribution, here are the mandatory features I expect:

installing source code managing dependencies with foreign contribution
have binary builds without interaction with the primary host system
be multi-platform agnostic (Linux, BSD, Windows, Mac, ...)
clean upgrade/uninstall
kind of sandboxes for testing and development mode
no administrator privilege required

I found the http://0install.net project homepage and was really impressed by the tons of functionalities already available and the other numerous advantages, like:

multiple version installation
reuse external distribution effort (integrate deb, rpm, ...)
digital signatures
basic mirroring solution
notification about software updates
command line oriented but various GUI exist
try to follow standards (XDG specifications on freedesktop.org))

I'm questioning seriously why this project could not be considered as a clean and build-independent python packages index system. Moreover, 0install has already some build capabilities (see 0compile) but the ultimate reason is that it will largely facilitate migrations when a new python build standard will emerge.

Conclusion

0install looks like a mature project driven by smart people and already included in modern distributions. I'll definitively give it a try soon.

Converting excel files to CSV using OpenOffice.org and pyuno

Alexandre Fayolle — 2008-09-19T14:35-01:00

The Task

I recently received from a customer a fairly large amount of data, organized in dozens of xls documents, each having dozens of sheets. I need to process this, and in order to ease the manipulation of the documents, I'd rather use standard text files in CSV (Comma Separated Values) format. Of course I didn't want to spend hours manually converting each sheet of each file to CSV, so I thought this would be a good time to get my hands in pyUno.

So I gazed over the documentation, found the Calc page on the OpenOffice.org wiki, read some sample code and got started.

The easy bit

The first few lines I wrote were (all imports are here, though some were actually added later).

import logging
import sys
import os.path as osp
import os
import time

import uno

def convert_spreadsheet(filename):
    pass

def run():
    for filename in sys.argv[1:]:
        convert_spreadsheet(filename)

def configure_log():
    logger = logging.getLogger('')
    logger.setLevel(logging.DEBUG)
    handler = logging.StreamHandler(sys.stdout)
    logger.addHandler(handler)
    format = "%(asctime)s %(levelname)-7s [%(name)s] %(message)s"
    handler.setFormatter(logging.Formatter(format))

if __name__ == '__main__':
    configure_log()
    run()

That was the easy part. In order to write the convert_spreadsheet function, I needed to open the document. And to do that, I need to start OpenOffice.org.

Starting OOo

I started by copy-pasting some code I found in another project, which expected OpenOffice.org to be already started with the -accept option. I changed that code a bit, so that the function would launch soffice with the correct options if it could not contact an existing instance:

def _uno_init(_try_start=True):
    """init python-uno bridge infrastructure"""
    try:
        # Get the uno component context from the PyUNO runtime
        local_context = uno.getComponentContext()
        # Get the local Service Manager
        local_service_manager = local_context.ServiceManager
        # Create the UnoUrlResolver on the Python side.
        local_resolver = local_service_manager.createInstanceWithContext(
            "com.sun.star.bridge.UnoUrlResolver", local_context)
        # Connect to the running OpenOffice.org and get its context.
        # XXX make host/port configurable
        context = local_resolver.resolve("uno:socket,host=localhost,port=2002;urp;StarOffice.ComponentContext")
        # Get the ServiceManager object
        service_manager = context.ServiceManager
        # Create the Desktop instance
        desktop = service_manager.createInstance("com.sun.star.frame.Desktop")
        return service_manager, desktop
    except Exception, exc:
        if exc.__class__.__name__.endswith('NoConnectException') and _try_start:
            logging.info('Trying to start UNO server')
            status = os.system('soffice -invisible -accept="socket,host=localhost,port=2002;urp;"')
            time.sleep(2)
            logging.info('status = %d', status)
            return _uno_init(False)
        else:
            logging.exception("UNO server not started, you should fix that now. "
                              "`soffice \"-accept=socket,host=localhost,port=2002;urp;\"` "
                              "or maybe `unoconv -l` might suffice")
            raise

Spreadsheet conversion

Now the easy (sort of, once you start understanding the OOo API): to load a document, use desktop.loadComponentFromURL(). To get the sheets of a Calc document, use document.getSheets() (that one was easy...). To iterate over the sheets, I used a sample from the SpreadsheetCommon page on the OpenOffice.org wiki.

Exporting the CSV was a bit more tricky. The function to use is document.storeToURL(). There are two gotchas, however. The first one, is that we need to specify a filter, and to parameterize it correctly. The second one is that the CSV export filter is only able to export the active sheet, so we need to change the active sheet as we iterate over the sheets.

Parametrizing the export filter

The parameters are passed in a tuple of PropertyValue uno structures, as the second argument to the storeToURL method. I wrote a helper function which accepts any named arguments and convert them to such a tuple:

def make_property_array(**kwargs):
    """convert the keyword arguments to a tuple of PropertyValue uno
    structures"""
    array = []
    for name, value in kwargs.iteritems():
        prop = uno.createUnoStruct("com.sun.star.beans.PropertyValue")
        prop.Name = name
        prop.Value = value
        array.append(prop)
    return tuple(array)

Now, what do we put in that array? The answer is in the FilterOptions page of the wiki : The FilterName property is "Text - txt - csv (StarCalc)". We also need to configure the filter by using the FilterOptions property. This is a string of comma separated values

ASCII code of field separator
ASCII code of text delimiter
character set, use 0 for "system character set", 76 seems to be UTF-8
number of first line (1-based)
Cell format codes for the different columns (optional)

I used the value "59,34,76,1", meaning I wanted semicolons for separators, and double quotes for text delimiters.

Here's the code:

def convert_spreadsheet(filename):
    """load a spreadsheet document, and convert all sheets to
    individual CSV files"""
    logging.info('processing %s', filename)
    url = "file://%s" % osp.abspath(filename)
    export_mask = make_export_mask(url)
    # initialize Uno, get a Desktop object
    service_manager, desktop = _uno_init()
    try:
        # load the Document
        document = desktop.loadComponentFromURL(url, "_blank", 0, ())
        controller = document.getCurrentController()
        sheets = document.getSheets()
        logging.info('found %d sheets', sheets.getCount())

        # iterate on all the spreadsheets in the document
        enumeration = sheets.createEnumeration()
        while enumeration.hasMoreElements():
            sheet = enumeration.nextElement()
            name = sheet.getName()
            logging.info('current sheet name is %s', name)
            controller.setActiveSheet(sheet)
            outfilename = export_mask % name.replace(' ', '_')
            document.storeToURL(outfilename,
                                make_property_array(FilterName="Text - txt - csv (StarCalc)",
                                                    FilterOptions="59,34,76,1" ))
    finally:
        document.close(True)

def make_export_mask(url):
    """convert the url of the input document to a mask for the written
    CSV file, with a substitution for the sheet name

    >>> make_export_mask('file:///home/foobar/somedoc.xls')
    'file:///home/foobar/somedoc$%s.csv'
    """

    components = url.split('.')
    components[-2] += '$%s'
    components[-1] = 'csv'
    return '.'.join(components)

qgpibplotter is (hopefully) working

David Douard — 2008-09-05T00:41-01:00

My latest personal project, pygpibtoolkit, holds a simple HPGL plotter trying to emulate the HP7470A GPIB plotter, using the very nice and cheap Prologix USB-GPIB dongle. This tool is (for now) called qgpibplotter (since it is using the Qt4 toolkit).

Tonight, I took (at last) the time to make it work nicely. Well, nicely with the only device I own which is capable of plotting on the GPIB bus, my HP3562A DSA.

Now, you just have to press the "Plot" button of your test equipment, and bingo! you can see the plot on your computer.

gajim, dbus and wmii

Adrien Di Mascio — 2008-09-02T17:06-01:00

I've been using for a long time a custom version of gajim in order to make it interact with wmii. More precisely, I have, in my wmii status bar, a dedicated log zone where I print notification messages such as new incoming emails or text received from gajim (with different colors if special words were cited, etc.).

I recently decided to throw away my custom gajim and use python and dbus to achieve the same goal in a cleaner way. A very basic version can be found in the simpled project. As of now, the only way to get the code is trhough mercurial:

hg clone http://www.logilab.org/hg/simpled

The source file is named gajimnotifier.py. In this file, you'll also find a version sending messages to Ion's status bar.

Command-line graphical user interfaces

Nicolas Chauvat — 2008-09-01T16:07-01:00

Graphical user interfaces help command discovery, while command-line interfaces help command efficiency. This article tries to explain why. I reached it when reading the list of references from the introduction to Ubiquity, which is the best extension to firefox I have seen so far. I expect to start writing Ubiquity commands soon, since I have already been using extensively the 'keyword shorcut' functionnality of firefox's bookmarks and we have already done work in the area of 'language interaction', as they call it at Mozilla Labs, when working with Narval. Our Logilab Simple Desktop project, aka simpled, also goes in the same direction since it tries to unify different applications into a coherent work environment by defining basic commands and shorcuts that can be applied everywhere and accessing the rest of the functionnalities via a command-line interface.

Is the Openmoko freerunner a computer or a phone ?

Nicolas Chauvat — 2008-08-27T14:13-01:00

The Openmoko Freerunner is a computer with embedded GSM, accelerometer and GPS. I got mine last week after waiting for a month for the batch to get from Taiwan to the french company I bought it from. The first thing I had to admit was that some time will pass before it gets confortable to use it as a phone. The current version of the system has many weird things in its user interface and the phone works, but the other end of the call suffers a very unpleasant echo.

I will try to install Debian, Qtopia and Om2008.8 to compare them. I also want to quickly get Python scripts to run on it and get back to Narval hacking. I had an agent running on a bulky Palm+GPS+radionetwork back in 1999 and I look forward to run on this device the same kind of funny things I was doing in AI research ten years ago.

simpled - Simple Desktop project started !

Nicolas Chauvat — 2008-08-11T14:48-01:00

I bought last week a new laptop computer that can drive a 24" LCD monitor, which means I do not need my desktop computer any more. In the process of setting up that new laptop, I did what I have been wanting to do for years without finding the time: spending time on my ion3 config to make it more generic and create a small python setup utility that can regenerate it from a template file and a keyboard layout.

The simpled project was born!

If you take a look at the list of pending tickets, you will guess that I am using a limited number of pieces of software during my work day and tried to configure them so that they share common action/shortcuts. This is what simpled is about: given a keyboard layout generate the config files for the common tools so that action/shortcuts are always on the same key.

I use ion3, xterm+bash, emacs, mutt, firefox, gajim. Common actions are: open, save, close, move up/down/left/right, new frame or tab, close frame or tab, move to previous or next tab, etc.

I will give news in this blog from time to time and announce it on mailing lists when version 0.1 will be out. If you want to give it a try, get the code from the mercurial repository.

Simile-Widgets

Nicolas Chauvat — 2008-08-07T11:17-01:00

While working on knowledge management and semantic web technologies, I came across the Simile project at MIT a few years back. I even had a demo of the Exhibit widget fetching then displaying data from our semantic web application framework back in 2006 at the Web2 track of Solutions Linux in Paris.

Now that we are using these widgets when implementing web apps for clients, I was happy to see that the projects got a life of their own outside of MIT and became full-fledged free-software projects hosted on Google Code. See Simile-Widgets for more details and expect us to provide a debian package soon unless someone does it first.

Speaking of Debian, here is a nice demo a the Timeline widget presenting the Debian history.

SciPy and TimeSeries

Nicolas Chauvat — 2008-08-04T10:56-01:00

We have been using many different tools for doing statistical analysis with Python, including R, SciPy, specific C++ code, etc. It looks like the growing audience of SciPy is now in movement to have dedicated modules in SciPy (lets call them SciKits). See this thread in SciPy-user mailing-list.

Google Custom Search Engine, for Python

Alexandre Fayolle — 2008-07-31T10:32-01:00

A Google custom search engine for Python has been made available by Gerard Flanagan, indexing:

About http://thehazeltree.org

The Hazel Tree is a collection of popular Python texts that I have converted to reStructuredText and put together using Sphinx. It's in a publishable state, but not as polished as I'd like, and since I'll be mostly offline for the next month it will have to remain as it is for the present. However, the search engine is ready now and the clock is ticking on its subscription (one year, renewal depending on success of site), so if it's useful to anyone, it's all yours (and if you use it on your own site a link back to http://thehazeltree.org would be appreciated).

Python for applied Mathematics

Nicolas Chauvat — 2008-07-29T10:17-01:00

The presentation of Python as a tool for applied mathematics got highlighted at the 2008 annual meeting of the american Society for Industrial and Applied Mathematics (SIAM). For more information, read this blogpost and the slides.

Windows, fichiers ouverts et tests unitaires

Alexandre Fayolle — 2008-07-22T10:51-01:00

Un problème rencontré hier : un test unitaire plante sous Windows, après avoir créé un objet qui garde des fichiers ouverts. le tearDown du test est appelé, mais il plante car Windows refuse de supprimer des fichiers ouverts, et le framework de test garde une référence sur la fonction de test pour qu'on puisse examiner la pile d'appels. Sous Linux, pas de problème (on a le droit du supprimer du disque un fichier ouvert, et donc pas de soucis dans le teardown).

Quelques pistes pour contourner le problème:

mettre le test dans un try...finally avec un del sur l'objet qui garde les fichiers ouverts dans le finally. Inconvénient : quand le test ne passe pas, pdb ne permet plus de voir grand chose
au lieu de nettoyer dans le tearDown, nettoyer plus tard dans un atexit par exemple. Il faut voir comment ça se passe si plusieurs tests veulent écrire dans les mêmes fichiers (je pense qu'il faudrait un répertoire temporaire par test, si on veut pouvoir avoir plusieurs tests qui foirent et examiner leurs données, mais il faut tester pour être sûr)
coller un try...except dans le tearDown autour de la suppression de chaque fichier, et mettre les fichiers qui posent problème dans une liste qui sera traitée à la sortie du programme (avec atexit par exemple).

Ça ressemble à du bricolage, mais on a un comportement de windows sur lequel on n'a pas de contrôle (même avec des privilèges Administrateur ou System, on ne peut pas contourner cette impossibilité de supprimer un fichier ouvert, à ma connaissance).

Une autre approche, nettement plus lourde, serait de virtualiser la création de fichiers pour travailler en mémoire (au minimum surcharger os.mkdir et le builtin open, voire dans le cas qui nous intéresse les modules qui travaillent avec des fichiers zip). Il y a peut-être des choses comme ça en circulation. Poser la question sur la liste TIP apportera peut-être des réponses (une rapide recherche dans les archives n'a rien donné).

Voir aussi ces enfilades de mars 2004 et novembre 2004 sur comp.lang.python.

ion, dock and screen configuration

David Douard — 2008-07-04T16:21-01:00

I have a laptop I use at work (with a docking station), in the train and at home (with an external display), on which my environment is ion3.

As I use suspend-to-RAM all the time, I have added some keybindings to automatically reconfigure my screen when I plug/unplug an external display (on the dock as well as direct VGA connection).

The lua code to paste in your .ion3/cfg_ion.lua for the bindings looks like:

function autoscreen_on()
        local f = io.popen('/home/david/bin/autoscreen -c', 'r')
      if not f then
          return
      end
      local s = f:read('*a')
      f:close()
    ioncore.restart()
end

function autoscreen_off()
        local f = io.popen('/home/david/bin/autoscreen -d', 'r')
      if not f then
          return
      end
      local s = f:read('*a')
      f:close()
    ioncore.restart()
end

defbindings("WMPlex.toplevel", {
    bdoc("Turn on any external display and tell ion to reconfigure itself"),
    kpress(META.."F10",
           "autoscreen_on()"),
})

defbindings("WMPlex.toplevel", {
    bdoc("Turn off any external display and tell ion to reconfigure itself"),
    kpress(META.."F11",
           "autoscreen_off()"),
})

It makes use of the following python script (named /home/david/bin/autoscreen in the lua code above):

#!/usr/bin/env python

import sys
import os
import re
from subprocess import Popen, PIPE
import optparse
parser = optparse.OptionParser("A simple automatic screen configurator (using xrandr)")
parser.add_option('-c', '--connect', action="store_true",
                  dest='connect',
                  default=False,
                  help="configure every connected screens")
parser.add_option('-d', '--disconnect', action="store_true",
                  dest='disconnect',
                  default=False,
                  help="unconfigure every connected screens other than LVDS (laptop screen)")
parser.add_option('', '--main-display',
                  dest='maindisplay',
                  default="LVDS",
                  help="main display identifier (typically, the laptop LCD screen; defaults to LVDS)")

options, args = parser.parse_args()

if int(options.connect) + int(options.disconnect) > 1:
    print "ERROR: only one option -c or -d at a time"
    parser.print_help()
    sys.exit(1)


xrandr = Popen("xrandr", shell=True, bufsize=0, stdout=PIPE).stdout.read()

connected = re.findall(r'([a-zA-Z0-9-]*) connected', xrandr)
connected = [c for c in connected if c != options.maindisplay]

cmd = "xrandr --output %s %s"

if options.connect or options.disconnect:
    for c in connected:
        if options.connect:
            action = "--auto"
        elif options.disconnect:
            action = "--off"

        p = Popen(cmd % (c, action), shell=True)
        sts = os.waitpid(p.pid, 0)

We're going to Europython'08

Arthur Lutz — 2008-07-02T11:28-01:00

Hey,

We've decided to go to Europython this year. We're obviously going to give a talk about the exciting things we're doing with LAX and GoogleAppEngine. We're on wednesday at midday in the alfa room, check out the schedule here. Since we think it's important that these events take place, we're also chipping in and sponsoring the event.

We hope to see you there. Drop us a note if you want to meet up.

Munin Plugins for Zope

Arthur Lutz — 2008-07-01T12:15-01:00

Here at Logilab we find Munin pretty useful. We monitor a lots of machines and a lot of services with it, and it usually gives us pretty useful indicators over time that guide us through to optimizations.

One of the reasons we adopted this technology is it's modular approach with the plugin architecture. And when we realized we could write plugins in python, we knew we'd like it. After years of using it, we're now actually writing plugins for it. Optimizing zope and zeo servers is not an easy task so we're developping plugins to be able to see the difference between before and after changing things.

You check out the project here, and download it from the ftp.

apycot 0.12.1 released

Arthur Lutz — 2008-06-24T11:35-01:00

After one month of internship at logilab, I'm pleased to announce the 0.12.1 release of apycot.

for more information read the apycot 0.12.1 release note

You can also check the new sample configuration.

Pierre-Yves David

Instrumentation of google appengine's datastore.

Sylvain Thenault — 2008-06-23T09:11-01:00

Here is a piece of code I've written which I thought may be useful to some other people...

You'll find here a simple python module to use with the Google AppEngine SDK to monkey patch the datastore API in order to get an idea of the calls performed by your application.

To instrument of the datastore, put at the top level of your handler file

import instrdatastore

Note that it's important to have this before any other import in your application or in the google package to avoid that some modules will use the unpatched version of datastore functions (and hence calls to those functions wouldn't be considered).

Then add at the end of your handler function

instrdatastore.print_info()

The handler file should look like this:

"""my handler file with datastore instrumenting activated"""
import instrdatastore

# ... other initialization code

# main function so this handler module is cached
def main():
  from wsgiref.handlers import CGIHandler
  from ginco.wsgi.handler import ErudiWSGIApplication
  application = ErudiWSGIApplication(config, vreg=vreg)
  CGIHandler().run(application)
  instrdatastore.print_info()

if __name__ == "__main__":
  main()

Now you should see in your logs the number of Get/Put/Delete/Query which has been done during request processing

2008-06-23 06:59:12 - (root) WARNING: datastore access information
2008-06-23 06:59:12 - (root) WARNING: nb Get: 2
2008-06-23 06:59:12 - (root) WARNING: arguments (args, kwargs):
((datastore_types.Key.from_path('EGroup', u'key_users', _app=u'winecellar'),), {})
((datastore_types.Key.from_path('EUser', u'key_test@example.com', _app=u'winecellar'),), {})
2008-06-23 06:59:12 - (root) WARNING: nb Query: 1
2008-06-23 06:59:12 - (root) WARNING: arguments (args, kwargs):
(({'for_user =': None}, 'EProperty'), {})
2008-06-23 06:59:58 - (root) WARNING: nb Put: 1
2008-06-23 06:59:58 - (root) WARNING: arguments (args, kwargs):
(({u'login': None, u'last_usage_time': 1214204398.2022741, u'data': ""},), {})

I'll probably extend this as the time goes. Also notice you may encounter some problems with the automatic reloading feature of the dev app server when instrumentation is activated, in which case you should simply restart the web server.

SciLab passe en logiciel libre

Arthur Lutz — 2008-06-16T17:33-01:00

Bienvenue à SciLab version 5.0 dans le monde du logiciel libre. SciLab 5.0, plateforme open source de calcul scientifique sous licence CeCill, est une alternative crédible et maintenant reconnue comme telle à Matlab. Pour assurer le développement pérenne de Scilab, le consortium Scilab rejoint DIGITEO, parc de recherche d'envergure mondiale dans le domaine des sciences et technologies de l'information en Île-de-France.

First version of LAX Book

Arthur Lutz — 2008-06-16T17:12-01:00

Previous documentation was merged into a LAX Book now featuring step-by-step screenshots to get up and running faster.

http://lax.logilab.org/lax-book

Don't we all like screenshots...

Update: LAX is now included in the CubicWeb semantic web framework.

Implementing scalable applications with AppEngine

Nicolas Chauvat — 2008-06-11T11:08-01:00

At Google IO, a large part of the Tools track was dedicated to AppEngine. Brett Slatkin gave a talk titled Building scalable Web Applications with Google AppEngine which focused on optimizing the server part of web apps. As other presenters demonstrated it, like Steve Souders in his talk Even Faster Websites, optimizing the browser part of webapps is not to be neglected either.

Webscale applications require man-made optimisation

First of all, I must confess I am used to repeat that "early optimisation is the root of all evil" and "delay commitment until the last responsible time". But reading about AppEngine and listening to the Google IO talks, it appears that the tools we have today ask for human intervention to reach web-scale performance, even when "we" stands for "Google".

In order for web-scale applications to handle the kind of load they are facing, they must be designed and implemented carefully. As carefully as any application was designed before the exponential growth of PC computation power let us move away from low-level implementation details and made some inefficiencies acceptable as long as the time spent developing was short enough.

It all depends on the parameters of your cost function, but for web-scale applications, it seems like we have not enough computer-time and can not trade it for human-time.

Writes are more expensive than reads

To get a better idea of the work constraints, one should know that a disk seek is about 10ms, which means there will be a maximum of 100 accesses per second. On the other hand, if we need consistent data as opposed to transactional data (the latter implying that data is fetched each time it is asked for), data can be read from disk once then cached. Following reads are done from memory at a rate of about 4GB/sec, which means 4000 accesses per second if entities are around 1MB in size. Result of this back of the envelope approximation is 40 reads equals one write.

It follows that, although the actual time depends on the size and shape of data, writes are very expensive compared to reads and both are better done in batches to optimise disk access.

Entity groups in AppEngine

The AppEngine Datastore was designed with this constraints in mind. Entities are sets of property name/value pairs. Each entity may have a parent. An entity without a parent is the root of a hierarchy called an entity group.

Entities of the same group are stored on disk close to each other, but two distinct entity groups may be stored on different computers. Read access to entities of the same group is thus faster than read access to entities of different groups.

Write access is serialized per entity group. As opposed to a traditionnal RDBMS that provides row locking, the datastore only provides entity group locking. Writes to the a single entity group will always happen in sequence, even though changes concern different entities.

There is no limit to the number of entity groups or to the number of entities per group, but because of the locking strategy, large entity groups will cause high contention and a lot of failed transactions. Since writes are expensive, not thinking about write throughput is a very bad idea when designing an AppEngine application if one want it to scale.

On the other hand, the parallel nature of the datastore make it scale wide and there is no limit to the number of entity groups that can be written to in parallel, nor to the number of reads that can be done in parallel.

To understand this design in details, you will have to read about GFS, BigTable and other technologies developed by Google to implement large-scale clustering.

Example of counters

Counters are a good example to address when discussing write throughput, because the datastore locking strategy makes writing to global data very expensive.

Let us assume that we want to display on the main page of a wiki application the total number of comments posted.

A global counter would serialize all its updates. If 100 users were to add comments at the same time, some of them would have to wait several seconds for their action to complete: one write for the comment, one write for the counter, at most 100 writes per second for the counter and a lot of time lost due to failed transaction that need to be restarted.

The solution to make the counter scale is to partition it among all entity groups then sum these partial counters when the global value is needed.

Since chances are low that a given user will write more than one comment at a time, comment entities for a user can be grouped together and a partial counter can be added to the same entity group. Creating a new comment and increasing the partial counter will be done in the same batch.

When a new request for the main page is received, the counter total is looked up in the cache. If it is not found, all partial counters are fetched and summed up, then the cache is refreshed with a short timeout, for example one minute.

During the next minute, the counter will be "consistent", read no too far-off, and served extremely fast from the cache.

Prevent repeated or unneeded work

To sum things up, when implementing applications on top of AppEngine with web-scale usage as a goal, everything that can be done to save time should be considered. Including the following:

importing python modules as late as possible will minimize the python runtime overhead
retrieving data that is not going to be used is a waste
repeated queries and queries returning large result sets must be avoided
when Get() if sufficient, do not spend time on Query()
landing pages are traffic intensive and would better use the same query for everyone
entity groups have to be designed to match the load and aim at low write contention
caching must be used aggressively (it is no surprise that memcache was the first improvement that followed within a month of the AppEngine public release)

Conclusion

As a conclusion, the interface AppEngine is exhibiting today requires to optimize early, but I would bet that in the years to come, new languages and domain-specific compilers or database engines will take part of that burden off the hands of the developers.

Did not Yahoo and Google start developping PigLatin and Sawzall to make it easier to write parallel data-processing programs ? The same could happen with describe a data-model in a high-level language and get a tool to optimize it for write contention and web-scale application.

Google App Engine future directions

Nicolas Chauvat — 2008-06-09T22:57-01:00

Several of us went to San Francisco last week to attend Google IO. As usual with conferences, meeting people was more interesting than listening to most talks. The AppEngine Fireside Chat was a Q&A session that lasted about an hour. Here is what I learned from this session and various chats with AppEngineers.

Google has decided to provide its scalable datastore architecture as a service. At this point, the datastore is the product and the goal it to make it as widely accessible as possible.
The google.appengine.api.datastore API alone would not have made for a very sexy launch. In order to attract more people and lower the bar the beginners would have to jump over, they looked for a higher level programming interface.
Since some people working at Google have been using Django and know it, they reimplemented part of its interface for defining data models. Late in the project, they added GQL because Django-like queries were a bit too difficult. In both case, the goal was to make it easier for external developers to get started.
But Google is not in the business of providing web application frameworks and AppEngineers made explicit that they would not be officially supporting a specific framework or a specific version of a given framework (not even Django 0.96, although there is a django-appengine-helper project on code.google.com). They expect frameworks to be provided by communities of developers.

My conclusion is twofold:

They will be focusing on supporting other languages in AppEngine (I would bet on Java being the next one available) rather than extending Python frameworks support.
Anyone is free to join with his own framework and provide support for it, the One True Interface being the one defined by google.appengine.api.datastore, not the one defined by db.model and GQL.

This is why Logilab published its own framework running on App Engine as free software and is providing support for it: Logilab Appengine eXtension.

Nouvelle version de LAX - Logilab Appengine eXtension

Arthur Lutz — 2008-06-09T10:27-01:00

La version 0.3.0 de LAX est sortie aujourd'hui voir : http://lax.logilab.org/

Il suffit de 10 petites minutes pour avoir une application qui tourne, suivez le guide :

Mise à jour: LAX est maintenant inclus dans CubicWeb.

LAX - Logilab Appengine eXtension is a full-featured web application framework running on AppEngine

Arthur Lutz — 2008-06-09T09:58-01:00

LAX version 0.3.0 was released today, see http://lax.logilab.org/

Get a new application running in ten minutes with the install guide and the tutorial:

Enjoy!

Update: LAX is now included in the CubicWeb semantic web framework.

Browsers strangeness ...

Adrien Di Mascio — 2008-06-07T18:21-01:00

... or when inverting two lines of code in your HTML's HEAD can speed up your web page rendering !

If you have the following HTML page:

<html>
  <head>
    <link rel="stylesheet" type="text/css" href="http://yourdomain.com/css1.css" />
    <script type="text/javascript">
      var somearray = [1, 2, 3];
    </script>
    <link rel="stylesheet" type="text/css" href="http://yourdomain.com/css2.css" />
  </head>
  <body>
    <h1>Hello</h1>
  </body>
</html>

Firefox3 [1] will download the CSS sequentially, hence if both CSS get 250ms to download, this page will approximatively appear in more or less half a second.

Now, if you just move the inline script before the two CSS declarations:

<html>
  <head>
    <script type="text/javascript">
      var somearray = [1, 2, 3];
    </script>
    <link rel="stylesheet" type="text/css" href="http://yourdomain.com/css1.css" />
    <link rel="stylesheet" type="text/css" href="http://yourdomain.com/css2.css" />
  </head>
  <body>
    <h1>Hello</h1>
  </body>
</html>

The two CSS files are now downloaded in parallel, and your page now take about half time to render !

One of the lessons here is that optimizing your website's backend is great and necessary, but is a quite long term and hard job. On the other hand, optimizing the frontend is often easier and pays off immediatly (well, so to speak...). Don't forget that in complex and rich web sites, most of the time can be spent on the client side.

[1] It seems that Firefox 2 doesn't event try to download CSS in parallel.

Going further

Of course, this is quite browser-dependant ! It would be simpler if all browsers behaved the same way but fortunately, there is a very nice tool named cuzillion developed by Steve Souders at Google (formerly Chief performance at Yahoo and developer of Yslow, a firebug's extension which is able to point out performance problems of your site). This tool lets you create web pages online by inserting inline scripts, CSS, images, etc. and then test how long the page takes to be rendered in your browser. You can control the order of the inserted elements as well as customize their properties (how long it shoud take to download, choose another domain to download, if a script is defined with a script tag, an XHR, an iframe, etc.)

New apycot release

Arthur Lutz — 2008-06-02T12:40-01:00

After almost 2 years of inactivity, here is a new release of apycot the "Automated Pythonic Code Tester". We use it everyday to maintain our software quality, and we hope this tool can help you as well.

Admittedly it's not trivial to setup, but once it's running you'll be able to count on it. We're working on getting it to work "out-of-the-box"...

Here's what's in the ChangeLog :

2008-05-19 -- 0.11.0

updated documentation
new pylintrc option for the pyhton_lint checker.
Added code to disabled checker with missing required option with the proper ERROR statut
removed the catalog option of the xml_valid checker this feature can now be handle with the XML_CATALOG_FILE environement variable (see libxml2 doc for details)
moved xml tool from python-xml to lxml
new 'hourly' mode for running tests
new 'test_activity_report' report
pylint checker support new disable_msg and show_categories options (show_categories default to Error and Fatal categories to avoid reports polution)
activity option "days" has been renamed to "time" and correspond to a number of day in daily mode but to a number of hour in hourly mode
fixed debian_lint and debian_piuparts to actually do something...
fixed docutils checker for recent docutils versions
dropped python 2.2/2.3 compat (to run apycot itself)
added output redirectors to the debian preprocessor to avoid parsing errors
can use regular expressions in <pp>_match_* options

Flying to Google I/O

Arthur Lutz — 2008-05-27T15:35-01:00

Three of us from Logilab are going to San Francisco to listen, share and discuss at Google I/O.

It's a two day developer gathering in San Francisco, with various talks about google technologies : http://code.google.com/events/io/

We're hoping to show and talk about LAX (http://lax.logilab.org) which uses Google AppEngine

Présentation PyCON FR 2008 - Assurance qualité

Julien Jehannet — 2008-05-27T14:55-01:00

Une présentation sur l'assurance-qualité a été présentée le 17 mai 2008 pour les journées Python organisées par l'Association Francophone Python (AFPy).

Le but visé est de décrire quelques notions et pratiques simples pour améliorer la lisibilité et la maintenabilité de votre code python.

Quelques outils standards de python sont décrits en première partie; pour finir par une revue de projets plus ambitieux mais indispensables pour la création de code de qualité.

Photo sous licence creative commons By-Nc-Nd par : yota

Pour accéder au diaporama :

http://fr.pycon.org/presentations_2008/julien-jehannet-assurance-qualite/slides.html

Présentation de Gendb à PyconFR

Andre Espaze — 2008-05-27T14:44-01:00

Ma présentation de 30 minutes est disponible à l'adresse suivante :

http://fr.pycon.org/presentations_2008/andre-espaze-recherche-de-gene.pdf

La journée s'est bien passée, j'ai eu quelques questions. Un chercheur en génétique a demandé si le projet serait continué sur la recherche des peptides (car ce sont les gènes qui codent les peptides d'après ce que j'ai compris). J'ai transféré cette demande à Eric Eveno, je n'ai pas de réponse pour l'instant. Normalement, la vidéo devrait être disponible sur :

http://fr.pycon.org/

Vous pouvez me contactez si vous avez des questions : andre.espaze@logilab.fr

Testing for NaN without depending on Numpy

Alexandre Fayolle — 2008-05-27T11:23-01:00

How can I test if a python float is "not a number" without depending on numpy? Simple, a nan value is different to any other value, including itself:

def isnan(x):
    return isinstance(x, float) and x!=x

Profiter pleinement des CPUs avec Zope/Zeo/Debian

Arthur Lutz — 2008-05-27T09:52-01:00

Voici vite fait comment on profite du quad-core bi-proc multicoeurs avec zope/zeo/pound ... le tout en commandes debian.

Inspiré de : http://plone.org/documentation/how-to/simple-zope-clustering-with-squid-and-pound

apt-get -uVf install plone-site pound
dzhandle -z 2.10 make-zeoinstance sgel_zeo
dzhandle -z 2.10 make-instance sgel2 --zeo-server=localhost:8100 -m all
dzhandle -z 2.10 make-instance sgel3 --zeo-server=localhost:8100 -m all
dzhandle -z 2.10 make-instance sgel1 --zeo-server=localhost:8100 -m all
dzhandle -z 2.10 make-instance sgel4 --zeo-server=localhost:8100 -m all
modifiez les ports de chaque instance (par exemple 9673, 9674, 9675, 9676)
vim ~/zope/instances/sgel*/etc/zope.conf
dzhandle add-product sgel1 CMFPlone
dzhandle add-product sgel2 CMFPlone
dzhandle add-product sgel3 CMFPlone
dzhandle add-product sgel4 CMFPlone
dzhandle zeoctl sgel_zeo start
dzhandle zopectl sgel1 start
dzhandle zopectl sgel2 start
dzhandle zopectl sgel3 start
dzhandle zopectl sgel4 start

vim /etc/pound/pound.cfg pour remplacer

BackEnd
        Address 127.0.0.1
        Port    8080
End

par

Service
        BackEnd
                Address 127.0.0.1
                Port    9673
        End
        BackEnd
                Address 127.0.0.1
                Port    9674
        End
        BackEnd
                Address 127.0.0.1
                Port    9675
        End
        BackEnd
                Address 127.0.0.1
                Port    9676
        End
End

/etc/init.d/pound restart
tapez sur http://localhost:8080
ajoutez un site plone

pour tester, lancez htop pour voir l'activité et regardez la différence entre :

apt-get -uVf install apache2-utils
/usr/sbin/ab -n 100 -c 100 localhost:8080/plone

/usr/sbin/ab -n 100 -c 100 localhost:9673/plone

nice!

Reinteract: un outil intéressant pour faire du numpy/scipy

Arthur Lutz — 2008-05-27T09:48-01:00

Il existe un outil, Reinteract, qui permet d'avoir une sorte de d'éditeur/shell Python, où l'on peut aisément modifier et réinterpreter une ligne de code.

Sachant qu'il sait aussi afficher des plots, etc, il est possible de s'en servir avantageusement pour faire des sessions Matlab-like.

Je pense donc que c'est un outil à présenter à nos chers apprenants qui sont intéressés par le couple python/numpy comme substitut à Matlab ©®.

Ex:

écrit par David Douard

Petit raccourci pratique avec ion3

Arthur Lutz — 2008-05-27T09:45-01:00

Un petit raccourci pratique pour ion3, qui permet, sur la combinaison de touches de son choix, de prendre le texte actuellement sélectionné (surligné) dans sa session X11, et, en fonction de son contenu :

d'ouvrir un onglet Firefox avec l'url sélectionnée,
d'ouvrir un xpdf si c'est une URL de fichier PDF,
lancer OpenOffice.org si c'est un fichier OOo,
ouvrir le fichier dans emacs si c'est un .py, .po, etc.
etc.

Pour cela, il faut le script magique ci-dessous, et configurer ion pour appeler ce script sur la bonne combinaison de touches. Par ex. ajouter dans votre ~/.ion3/cfg_ion.lua les lignes

defbindings("WMPlex.toplevel", {
  bdoc("Automagically view the selected string"),
  kpress(META.."F7",
         "ioncore.exec_on(_, '/home/user/bin/view')"),
})

Ici, j'ai mappé "Meta+F7", et le script est /home/user/bin/view

#!/usr/bin/env python

from mimetypes import guess_type
import sys
from os.path import abspath
from os import system, popen

import re
RGX = re.compile

EMACS = 'emacsclient --no-wait %(uri)s'
EMACS_WITH_LINE = 'emacsclient --no-wait +%(lineno)s %(uri)s'
FIREFOX = 'firefox -remote "openURL(%(uri)s, new-tab)"'
WGET = 'cd /home/adim/tmp && wget %(uri)s & '

commands = [
    ('text/html', FIREFOX),
    ('application/xml', EMACS),
    ('text', EMACS),
    ('image', 'display %(uri)s &'),
    ('application/pdf', 'xpdf %(uri)s &'),
    ('application/postcript', 'gv %(uri)s &'),
    ('application/vnd.sun.xml', 'ooffice %(uri)s &'), # matches writer, impress, etc.
    ('application/vnd.oasis.opendocument', 'ooffice %(uri)s &'),
    ('application/msword', 'ooffice %(uri)s &'),
    ('application/vnd.ms-', 'ooffice %(uri)s &'),
    ]

patterns = [
    (RGX(r'https?://.*?(zip|gz|pdf|ods|doc|odt|ppt|sxw|sxi)$'), WGET),
    (RGX(r'.*(?P<uri>https?://[^ ()]*)( .*)?'), FIREFOX),
    (RGX('.*?\.conf$'), EMACS),
    (RGX('.*?\.po$'), EMACS),
    (RGX('.*?\.xslt$'), EMACS),
    (RGX('.*?\.pot$'), EMACS),
    (RGX(r'\s*F?i?l?e? ?"?(?P<uri>.*?\.py)", line (?P<lineno>\d+)[a-zA-Z_:-]*'), EMACS_WITH_LINE),
    (RGX('.*?(readme|changelog|depends|manifest|makefile)(\.in|\.gz|\.bz2)?$', re.I), EMACS),
    # others might come here ...
    ]

def find_command(selection):
    for rgx, cmd in patterns:
        m = rgx.match(selection)
        if m:
            args = m.groupdict() or {'uri' : selection}
            return cmd, args
    mimetype, encoding = guess_type(selection)
    # XXX: encodings like zip, or gz could be handled
    if mimetype is not None:
        selection = abspath(selection)
        for registered_type, cmd in commands:
            if mimetype.startswith(registered_type):
                return cmd, {'uri' : selection}
    raise ValueError('nothing matched')

if len(sys.argv)>1:
    selected = ' '.join(sys.argv[1:])
else:
    selected = popen('xclip -o').read()

if selected:
    try:
        cmd, args = find_command(selected)
    except ValueError:
        # system('wmiijabber error viewing %s' % ' '.join(sys.argv[1:]))
        # XXX print a message in wmii status bar ?
        pass
    else:
        #print "yooo =>", repr(cmd), repr(args)
        system(cmd % args)

Pour que cela fonctionne, il ne faut pas oublier d'installer xclip (sous debian, apt-get install xclip).

-- écrit par David Douard sur un script de Adrien diMascio

Petites astuces de vim

Arthur Lutz — 2008-05-27T09:37-01:00

J'ai découvert vim-addons (qui est apparu dans debian récement) et ce petit outil permet de faire d'etendre les fonctionnalités de vim assez facilement. Voici une utilisation parmi tant d'autres :

mode gnupg

pour installer le mode gnupg faites

vim-addons install gnupg

Ensuite vim pourra ouvrir directement des fichiers encryptés et les réncrypter lorsque vous sauvez. Donc simplement

vim mot-de-passe-envoyé-par-client.gpg

Voilà, le tour est joué.

J'avoue que je reste avec emacs pour le code, mais ce genre de petit raccourcis est bien pratique.

LAX - Logilab Google AppEngin Sprint at Pycon-FR

Arthur Lutz — 2008-05-20T10:31-01:00

Here are a few pictures from the sprint we organized at Pycon-FR

We got a few people to install Google AppEngine and LAX on their machines, and explained the concepts at hand to a bunch of other people.

Update: LAX is now included in the CubicWeb semantic web framework.

HOWTO quickly get lax running on linux

Arthur Lutz — 2008-05-19T13:45-01:00

This is how easy it is to get lax to run on your linux machine :

hg clone http://www.logilab.org/hg/lax/
wget http://googleappengine.googlecode.com/files/google_appengine_1.0.2.zip
unzip google_appengine_1.0.2.zip
./google_appengine/dev_appserver.py lax/skel/

Point your favorite browser to http://localhost:8080/

UPDATE: LAX is now included in the CubicWeb semantic web framework.

Another step towards the semantic web

Nicolas Chauvat — 2007-02-06T17:24-01:00

I co-organized the Web2.0 conference track that was held at Solutions Linux 2007 in Paris last week . Researching to prepare the talk I gave, I came accross microformats and GRDDL. Both try to add semantics on top of (X)HTML.

Microformats uses the class attribute and the "invisibility" of `div` and `span` to insert semantic information, as in ::

  <li class="vevent">
    <a class="url" href="http://www.solutionslinux.fr/">
     <span class="summary">Solutions Linux Web 2.0 Conference</span>: 
     <abbr class="dtstart" title="20070201T143000Z">February 1st 2:30pm</abbr>-
     <abbr class="dtend" title="20070201T18000Z">6pm</abbr>, at the 
     <span class="location">CNIT, La Défense</span>
    </a>
  </li>

GRDDL information is added to the `head` of the XHTML page and points to an XSL that can extract the information from the page and output it as RDF.

Another option is to add `link` to the `head` of the page, pointing to an alternate representations like a RDF formatted one.

Firefox has add-ons that help you spot semantic enabled web pages: Tails detects microformats and the semantic radar detects RDF. Operator is an option I found too invasive.

As for my talk, it involved demonstrating CubicWeb, the engine behind logilab.org, and querying the data stored at logilab.org to reuse it with Exhibit.

Install Python Cartographic Library

Nicolas Chauvat — 2006-12-07T16:31-01:00

This is how we setup PCL <http://trac.gispython.org/projects/PCL> under debian. Of course having a package would be better...

Install dependencies

apt-get install mapserver-bin python-mapscript mapserver-cgi
gdal-bin python-gdal python-dev libgeos-dev libgdal-dev
libgd2-xpm-dev python-setuptools zope-interface
python-elementtree

Now link libgdal.a to libgdal1.3.2.a

Install PCL from sources

svn co  http://svn.gispython.org/svn/gispy/PCL/trunk

cd PCL/

cd externals/OWSLib
python setup.py install --prefix=my_python_library

cd externals/QuadTree
python setup.py install --prefix=my_python_library

cd PCL-Core
python setup.py install --prefix=my_python_library

cd PCL-GDAL
python setup.py install --prefix=my_python_library

cd PCL-MapServer

Put in a directory the source code of mapserver corresponding to the package version installed above. Get it from http://mapserver.gis.umn.edu/download or with dpkg-src.

Edit setup.py to set ms_home to the path of the above mapserver sources.

python setup.py install --prefix=my_python_library

my_python_library now contains Python eggs and directories with Python and C libraries.

New Logilab.org is launched

Arthur Lutz — 2006-10-24T12:15-01:00

** Insert announce text here **

CMFProjman is being ported to Plone2

Arthur Lutz — 2006-10-18T16:44-01:00

CMFProjman has been asleep for quite a while, and is now being reanimated to work with Plone2. We will release it as soon as we see it's stable.