subscribe to this blog

Logilab.org - en

News from Logilab and our Free Software projects, as well as on topics dear to our hearts (Python, Debian, Linux, the semantic web, scientific computing...)

show 207 results
  • pdb.set_trace no longer working: problem solved

    2010/08/12

    I had a bad case of bug hunting today which took me > 5 hours to track down (with the help of Adrien in the end).

    I was trying to start a CubicWeb instance on my computer, and was encountering some strange pyro error at startup. So I edited some source file to add a pdb.set_trace() statement and restarted the instance, waiting for Python's debugger to kick in. But that did not happen. I was baffled. I first checked for standard problems:

    • no pdb.py or pdb.pyc was lying around in my Python sys.path
    • the pdb.set_trace function had not been silently redefined
    • no other thread was bugging me
    • the standard input and output were what they were supposed to be
    • I was not able to reproduce the issue on other machines

    After triple checking everything, grepping everywhere, I asked a question on StackOverflow before taking a lunch break (if you go there, you'll see the answer). After lunch, no useful answer had come in, so I asked Adrien for help, because two pairs of eyes are better than one in some cases. We dutifully traced down the pdb module's code to the underlying bdb and cmd modules and learned some interesting things on the way down there. Finally, we found out that the Python code frames which should have been identical where not. This discovery caused further bafflement. We looked at the frames, and saw that one of those frames's class was a psyco generated wrapper.

    It turned out that CubicWeb can use two implementation of the RQL module: one which uses gecode (a C++ library for constraint based programming) and one which uses logilab.constraint (a pure python library for constraint solving). The former is the default, but it would not load on my computer, because the gecode library had been replaced by a more recent version during an upgrade. The pure python implementation tries to use psyco to speed up things. Installing the correct version of libgecode solved the issue. End of story.

    When I checked out StackOverflow, Ned Batchelder had provided an answer. I didn't get the satisfaction of answering the question myself...

    Once this was figured out, solving the initial pyro issue took 2 minutes...


  • EuroSciPy'10

    2010/07/13 by Adrien Chauve
    http://www.logilab.org/image/9852?vid=download

    The EuroSciPy2010 conference was held in Paris at the Ecole Normale Supérieure from July 8th to 11th and was organized and sponsored by Logilab and other companies.

    July, 8-9: Tutorials

    The first two days were dedicated to tutorials and I had the chance to talk about SciPy with André Espaze, Gaël Varoquaux and Emanuelle Gouillart in the introductory track. This was nice but it was a bit tricky to present SciPy in such a short time while trying to illustrate the material with real and interesting examples. One very nice thing for the introductory track is that all the material was contributed by different speakers and is freely available in a github repository (licensed under CC BY).

    July, 10-11: Scientific track

    The next two days were dedicated to scientific presentations and why python is such a great tool to develop scientific software and carry out research.

    Keynotes

    I had a very great time listening to the presentations, starting with the two very nice keynotes given by Hans Petter Langtangen and Konrad Hinsen. The latter gave us a very nice summary of what happened in the scientific python world during the past 15 years, what is happening now and of course what could happen during the next 15 years. Using a crystal ball and a very humorous tone, he made it very clear that the challenge in the next years will be about how using our hundreds, thousands or even more cores in a bug-free and efficient way. Functional programming may be a very good solution to this challenge as it provides a deterministic way of parallelizing our programs. Konrad also provided some hints about future versions of python that could provide a deeper and more efficient support of functional programming and maybe the addition of a keyword 'async' to handle the computation of a function in another core.

    In fact, the PEP 3148 entitled "Futures - execute computations asynchronously" was just accepted two days ago. This PEP describes the new package called "futures" designed to facilitate the evaluation of callables using threads and processes in future versions of python. A full implementation is already available.

    Parallelization

    Parallelization was indeed a very popular issue across presentations, and as for resolving embarrassingly parallel problems, several solutions were presented.

    • Playdoh: Distributes computations over computers connected to a secure network (see playdoh presentation).

      Distributing the computation of a function over two machines is as simple as:

      import playdoh
      result1, result2 = playdoh.map(fun, [arg1, arg2], _machines = ['machine1.network.com', 'machine2.network.com'])
      
    • Theano: Allows to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. In particular it can use GPU transparently and generate optimized C code (see theano presentation).

    • joblib: Provides among other things helpers for embarrassingly parallel problems. It's built over the multiprocessing package introduced in python 2.6 and brings more readable code and easier debugging.

    Speed

    Concerning speed, Fransesc Alted has showed us interesting tools for memory optimization currently successfully used in PyTables 2.2. You can read more details on these kind of optimizations in EuroSciPy'09 (part 1/2): The Need For Speed.

    SCons

    Last but not least, I talked with Cristophe Pradal who is one of the core developer of OpenAlea. He convinced me that SCons is worth using once you have built a nice extension for it: SConsX. I'm looking forward to testing it.


  • HOWTO install lodgeit pastebin under Debian/Ubuntu

    2010/06/24 by Arthur Lutz

    Lodge it is a simple open source pastebin... and it's written in Python!

    The installation under debian/ubuntu goes as follows:

    sudo apt-get update
    sudo apt-get -uVf install python-imaging python-sqlalchemy python-jinja2 python-pybabel python-werkzeug python-simplejson
    cd local
    hg clone http://dev.pocoo.org/hg/lodgeit-main
    cd lodgeit-main
    vim manage.py
    

    For debian squeeze you have to downgrade python-werkzeug, so get the old version of python-werkzeug from snapshot.debian.org at http://snapshot.debian.org/package/python-werkzeug/0.5.1-1/

    wget http://snapshot.debian.org/archive/debian/20090808T041155Z/pool/main/p/python-werkzeug/python-werkzeug_0.5.1-1_all.deb
    

    Modify the dburi and the SECRET_KEY. And launch application:

    python manage.py runserver
    

    Then off you go configure your apache or lighthttpd.

    An easy (and dirty) way of running it at startup is to add the following command to the www-data crontab

    @reboot cd /tmp/; nohup /usr/bin/python /usr/local/lodgeit-main/manage.py runserver &
    

    This should of course be done in an init script.

    http://rn0.ru/static/help/advanced_features.png

    Hopefully we'll find some time to package this nice webapp for debian/ubuntu.


  • EuroSciPy 2010 schedule is out !

    2010/06/06 by Nicolas Chauvat
    https://www.euroscipy.org/data/logo.png

    The EuroSciPy 2010 conference will be held in Paris from july 8th to 11th at Ecole Normale Supérieure. Two days of tutorials, two days of conference, two interesting keynotes, a lightning talk session, an open space for collaboration and sprinting, thirty quality talks in the schedule and already 100 delegates registered.

    If you are doing science and using Python, you want to be there!


  • Salomé accepted into Debian unstable

    2010/06/03 by Andre Espaze

    Salomé is a platform for pre and post-processing of numerical simulation available at http://salome-platform.org/. It is now available as a Debian package http://packages.debian.org/source/sid/salome and should soon appear in Ubuntu https://launchpad.net/ubuntu/+source/salome as well.

    http://salome-platform.org/salome_screens.png/image_preview

    A difficult packaging work

    A first package of Salomé 3 was made by the courageous Debian developper Adam C. Powell, IV on January 2008. Such packaging is very resources intensive because of the building of many modules. But the most difficult part was to bring Salomé to an unported environment. Even today, Salomé 5 binaries are only provided by upstream as a stand-alone piece of software ready to unpack on a Debian Sarge/Etch or a Mandriva 2006/2008. This is the first reason why several patches were required for adapting the code to new versions of the dependencies. The version 3 of Salomé was so difficult and time consuming to package that Adam decided to stop during two years.

    The packaging of Salomé started back with the version 5.1.3 in January 2010. Thanks to Logilab and the OpenHPC project, I could join him during 14 weeks of work for adapting every module to Debian unstable. Porting to the new versions of the dependencies was a first step, but we had also to adapt the code to the Debian packaging philosophy with binaries, librairies and data shipped to dedicated directories.

    A promising future

    Salomé being accepted to Debian unstable means that porting it to Ubuntu should follow in a near future. Moreover the work done for adapting Salomé to a GNU/Linux distribution may help developpers on others platforms as well.

    That is excellent news for all people involved in numerical simulation because they are going to have access to Salomé services by using their packages management tools. It will help the spreading of Salomé code on any fresh install and moreover keep it up to date.

    Join the fun

    For mechanical engineers, a derived product called Salomé-Méca has recently been published. The goal is to bring the functionalities from the Code Aster finite element solver to Salomé in order to ease simulation workflows. If you are as well interested in Debian packages for those tools, you are invited to come with us and join the fun.

    I have submitted a proposal to talk about Salomé at EuroSciPy 2010. I look forward to meet other interested parties during this conference that will take place in Paris on July 8th-11th.


  • Enable and disable encrypted swap - Ubuntu

    2010/05/18 by Arthur Lutz
    http://ubuntu-party.org/wp-content/themes/ubuntu-party/scripts/timthumb.php?src=//wp-content/uploads/2010/04/evl-pochette21.png&w=210&h=192&zc=1&q=100

    With the release of Ubuntu Lucid Lynx, the use of an encrypted /home is becoming a pretty common and simple to setup thing. This is good news for privacy reasons obviously. The next step which a lot of users are reluctant to accomplish is the use of an encrypted swap. One of the most obvious reasons is that in most cases it breaks the suspend and hibernate functions.

    Here is a little HOWTO on how to switch from normal swap to encrypted swap and back. That way, when you need a secure laptop (trip to a conference, or situtation with risk of theft) you can active it, and then deactivate it when you're at home for example.

    Turn it on

    That is pretty simple

    sudo ecryptfs-setup-swap
    

    Turn it off

    https://launchpadlibrarian.net/17699584/ecryptfs_64.png

    The idea is to turn off swap, remove the ecryptfs layer, reformat your partition with normal swap and enable it. We use sda5 as an example for the swap partition, please use your own (fdisk -l will tell you which swap partition you are using - or in /etc/crypttab)

    sudo swapoff -a
    sudo cryptsetup remove /dev/mapper/cryptswap1
    sudo vim /etc/crypttab
    *remove the /dev/sda5 line*
    sudo /sbin/mkswap /dev/sda5
    sudo swapon /dev/sda5
    sudo vim /etc/fstab
    *replace /dev/mapper/cryptswap1 with /dev/sda5*
    

    If this is is useful, you can probably stick it in a script to turn on and off... maybe we could get an ecryptfs-unsetup-swap into ecryptfs.


  • The DEBSIGN_KEYID trick

    2010/05/12 by Nicolas Chauvat

    I have been wondering for some time why debsign would not use the DEBSIGN_KEYID environment variable that I exported from my bashrc. Debian bug 444641 explains the trick: debsign ignores environment variables and sources ~/.devscripts instead. A simple export DEBSIGN_KEYID=ABCDEFG in ~/.devscripts is enough to get rid of the -k argument once and for good.


  • pylint bug days #2 report

    2010/04/19 by Sylvain Thenault

    First of all, I've to say that pylint bugs day wasn't that successful in term of 'community event': I've been sprinting almost alone. My Logilab's felows were tied to customer projects, and no outside people shown up on jabber. Fortunatly Tarek Ziade came to visit us, and that was a nice opportunity to talk about pylint, distribute, etc ... Thank you Tarek, you saved my day ;)

    As I felt a bit alone, I decided to work on somethings funnier than bug fixing: refactoring!

    First, I've greatly simplified the command line: enable-msg/enable-msg-cat/enable-checker/enable-report and their disable-* counterparts were all merged into single --enable/--disable options.

    I've also simplified "pylint --help" output, providing a --long-help option to get what we had before. Generic support in `logilab.common.configuration of course.

    And last but not least, I refactored pylint so we can have multiple checkers with the same name. The idea behind this is that we can split checker into smaller chunks, basically only responsible for one or a few related messages. When pylint runs, it only uses necessary checkers according to activated messages and reports. When all checkers will be splitted, it should improve performance of "pylint --error-only".

    So, I can say I'm finally happy with the results of that pylint bugs day! And hopefuly we will be more people for the next edition...


  • Virtualenv - Play safely with a Python

    2010/03/26 by Alain Leufroy
    http://farm5.static.flickr.com/4031/4255910934_80090f65d7.jpg

    virtualenv, pip and Distribute are tree tools that help developers and packagers. In this short presentation we will see some virtualenv capabilities.

    Please, keep in mind that all above stuff has been made using : Debian Lenny, python 2.5 and virtualenv 1.4.5.

    Abstract

    virtualenv builds python sandboxes where it is possible to do whatever you want as a simple user without putting in jeopardy your global environment.

    virtualenv allows you to safety:

    • install any python packages
    • add debug lines everywhere (not only in your scripts)
    • switch between python versions
    • try your code as you are a final user
    • and so on ...

    Install and usage

    Install

    Prefered way

    Just download the virtualenv python script at http://bitbucket.org/ianb/virtualenv/raw/tip/virtualenv.py and call it using python (e.g. python virtualenv.py).

    For conveinience, we will refers to this script using virtualenv.

    Other ways

    For Debian (ubuntu as well) addicts, just do :

    $ sudo aptitude install python-virtualenv
    

    Fedora users would do:

    $ sudo yum install python-virtualenv
    

    And others can install from PyPI (as superuser):

    $ pip install virtualenv
    

    or

    $ easy_install pip && pip install virtualenv
    

    You could also get the source here.

    Quick Guide

    To work in a python sandbox, do as follow:

    $ virtualenv my_py_env
    $ source my_py_env/bin/activate
    (my_py_env)$ python
    

    "That's all Folks !"

    Once you have finished just do:

    (my_py_env)$ deactivate
    

    or quit the tty.

    What does virtualenv actually do ?

    At creation time

    Let's start again ... more slowly. Consider the following environment:

    $ pwd
    /home/you/some/where
    $ ls
    

    Now create a sandbox called my-sandbox:

    $ virtualenv my-sandbox
    New python executable in "my-sandbox/bin/python"
    Installing setuptools............done.
    

    The output said that you have a new python executable and specific install tools. Your current directory now looks like:

    $ ls -Cl
    my-sandbox/ README
    $ tree -L 3 my-sandbox
    my-sandbox/
    |-- bin
    |   |-- activate
    |   |-- activate_this.py
    |   |-- easy_install
    |   |-- easy_install-2.5
    |   |-- pip
    |   `-- python
    |-- include
    |   `-- python2.5 -> /usr/include/python2.5
    `-- lib
        `-- python2.5
            |-- ...
            |-- orig-prefix.txt
            |-- os.py -> /usr/lib/python2.5/os.py
            |-- re.py -> /usr/lib/python2.5/re.py
            |-- ...
            |-- site-packages
            |   |-- easy-install.pth
            |   |-- pip-0.6.3-py2.5.egg
            |   |-- setuptools-0.6c11-py2.5.egg
            |   `-- setuptools.pth
            |-- ...
    

    In addition to the new python executable and the install tools you have an whole new python environment containing libraries, a site-packages/ (where your packages will be installed), a bin directory, ...

    Note:
    virtualenv does not create every file needed to get a whole new python environment. It uses links to global environment files instead in order to save disk space end speed up the sandbox creation. Therefore, there must already have an active python environment installed on your system.

    At activation time

    At this point you have to activate the sandbox in order to use your custom python. Once activated, python still has access to the global environment but will look at your sandbox first for python's modules:

    $ source my-sandbox/bin/activate
    (my-sandbox)$ which python
    /home/you/some/where/my-sandbox/bin/python
    $ echo $PATH
    /home/you/some/where/my-sandbox/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
    (pyver)$ python -c 'import sys;print sys.prefix;'
    /home/you/some/where/my-sandbox
    (pyver)$ python -c 'import sys;print "\n".join(sys.path)'
    /home/you/some/where/my-sandbox/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg
    [...]
    /home/you/some/where/my-sandbox
    /home/you/personal/PYTHONPATH
    /home/you/some/where/my-sandbox/lib/python2.5/
    [...]
    /usr/lib/python2.5
    [...]
    /home/you/some/where/my-sandbox/lib/python2.5/site-packages
    [...]
    /usr/local/lib/python2.5/site-packages
    /usr/lib/python2.5/site-packages
    [...]
    

    First of all, a (my-sandbox) message is automatically added to your prompt in order to make it clear that you're using a python sandbox environment.

    Secondly, my-sandbox/bin/ is added to your PATH. So, running python calls the specific python executable placed in my-sandbox/bin.

    Note
    It is possible to improve the sandbox isolation by ignoring the global paths and your PYTHONPATH (see Improve isolation section).

    Installing package

    It is possible to install any packages in the sandbox without any superuser privilege. For instance, we will install the pylint development revision in the sandbox.

    Suppose that you have the pylint stable version already installed in your global environment:

    (my-sandbox)$ deactivate
    $ python -c 'from pylint.__pkginfo__ import version;print version'
    0.18.0
    

    Once your sandbox activated, install the development revision of pylint as an update:

    $ source /home/you/some/where/my-sandbox/bin/activate
    (my-sandbox)$ pip install -U hg+http://www.logilab.org/hg/pylint#egg=pylint-0.19
    

    The new package and its dependencies are only installed in the sandbox:

    (my-sandbox)$ python -c 'import pylint.__pkginfo__ as p;print p.version, p.__file__'
    0.19.0 /home/you/some/where/my-sandbox/lib/python2.6/site-packages/pylint/__pkginfo__.pyc
    (my-sandbox)$ deactivate
    $ python -c 'import pylint.__pkginfo__ as p;print p.version, p.__file__'
    0.18.0 /usr/lib/pymodules/python2.6/pylint/__pkginfo__.pyc
    

    You can safely do any change in the new pylint code or in others sandboxed packages because your global environment is still unchanged.

    Useful options

    Improve isolation

    As said before, your sandboxed python sys.path still references the global system path. You can however hide them by:

    • either use the --no-site-packages that do not give access to the global site-packages directory to the sandbox
    • or change your PYTHONPATH in my-sandbox/bin/activate in the same way as for PATH (see tips)
    $ virtualenv --no-site-packages closedPy
    $ sed -i '9i PYTHONPATH="$_OLD_PYTHON_PATH"
          9i export PYTHONPATH
          9i unset _OLD_PYTHON_PATH
          40i _OLD_PYTHON_PATH="$PYTHONPATH"
          40i PYTHONPATH="."
          40i export PYTHONPATH' closedPy/bin/activate
    $ source closedPy/bin/activate
    (closedPy)$ python -c 'import sys; print "\n".join(sys.path)'
    /home/you/some/where/closedPy/lib/python2.5/site-packages/setuptools-0.6c8-py2.5.egg
    /home/you/some/where/closedPy
    /home/you/some/where/closedPy/lib/python2.5
    /home/you/some/where/closedPy/lib/python2.5/plat-linux2
    /home/you/some/where/closedPy/lib/python2.5/lib-tk
    /home/you/some/where/closedPy/lib/python2.5/lib-dynload
    /usr/lib/python2.5
    /usr/lib64/python2.5
    /usr/lib/python2.5/lib-tk
    /home/you/some/where/closedPy/lib/python2.5/site-packages
    $ deactivate
    

    This way, you'll get an even more isolated sandbox, just as with a brand new python environment.

    Work with different versions of Python

    It is possible to dedicate a sandbox to a particular version of python by using the --python=PYTHON_EXE which specifies the interpreter that virtualenv was installed with (default is /usr/bin/python):

    $ virtualenv --python=python2.4 pyver24
    $ source pyver24/bin/activate
    (pyver24)$ python -V
    Python 2.4.6
    $ deactivate
    $ virtualenv --python=python2.5 pyver25
    $ source pyver25/bin/activate
    (pyver25)$ python -V
    Python 2.5.2
    $ deactivate
    

    Distribute a sandbox

    To distribute your sandbox, you must use the --relocatable option that makes an existing sandbox relocatable. This fixes up scripts and makes all .pth files relative This option should be called just before you distribute the sandbox (each time you have changed something in your sandbox).

    An important point is that the host system should be similar to your own.

    Tips

    Speed up sandbox manipulation

    Add these scripts to your .bashrc in order to help you using virtualenv and automate the creation and activation processes.

    rel2abs() {
    #from http://unix.derkeiler.com/Newsgroups/comp.unix.programmer/2005-01/0206.html
      [ "$#" -eq 1 ] || return 1
      ls -Ld -- "$1" > /dev/null || return
      dir=$(dirname -- "$1" && echo .) || return
      dir=$(cd -P -- "${dir%??}" && pwd -P && echo .) || return
      dir=${dir%??}
      file=$(basename -- "$1" && echo .) || return
      file=${file%??}
      case $dir in
        /) printf '%s\n' "/$file";;
        /*) printf '%s\n' "$dir/$file";;
        *) return 1;;
      esac
      return 0
    }
    function activate(){
        if [[ "$1" == "--help" ]]; then
            echo -e "usage: activate PATH\n"
            echo -e "Activate the sandbox where PATH points inside of.\n"
            return
        fi
        if [[ "$1" == '' ]]; then
            local target=$(pwd)
        else
            local target=$(rel2abs "$1")
        fi
        until  [[ "$target" == '/' ]]; do
            if test -e "$target/bin/activate"; then
                source "$target/bin/activate"
                echo "$target sandbox activated"
                return
            fi
            target=$(dirname "$target")
        done
        echo 'no sandbox found'
    }
    function mksandbox(){
        if [[ "$1" == "--help" ]]; then
            echo -e "usage: mksandbox NAME\n"
            echo -e "Create and activate a highly isaolated sandbox named NAME.\n"
            return
        fi
        local name='sandbox'
        if [[ "$1" != "" ]]; then
            name="$1"
        fi
        if [[ -e "$1/bin/activate" ]]; then
            echo "$1 is already a sandbox"
            return
        fi
        virtualenv --no-site-packages --clear --distribute "$name"
        sed -i '9i PYTHONPATH="$_OLD_PYTHON_PATH"
                9i export PYTHONPATH
                9i unset _OLD_PYTHON_PATH
               40i _OLD_PYTHON_PATH="$PYTHONPATH"
               40i PYTHONPATH="."
               40i export PYTHONPATH' "$name/bin/activate"
        activate "$name"
    }
    
    Note:
    The virtualenv-commands and virtualenvwrapper projects add some very interesting features to virtualenv. So, put on eye on them for more advanced features than the above ones.

    Conclusion

    I found it to be irreplaceable for testing new configurations or working on projects with different dependencies. Moreover, I use it to learn about other python projects, how my project exactly interacts with its dependencies (during debugging) or to test the final user experience.

    All of this stuff can be done without virtualenv but not in such an easy and secure way.

    I will continue the series by introducing other useful projects to enhance your productivity : pip and Distribute. See you soon.


  • Astng 0.20.0 and Pylint 0.20.0 releases

    2010/03/24 by Emile Anclin

    We are happy to announce the Astng 0.20.0 and Pylint 0.20.0 releases.

    Pylint is a static code checker based on Astng, both depending on logilab-common 0.49.

    Astng

    Astng 0.20.0 is a major refactoring: instead of parsing and modifying the syntax tree generated from python's _ast or compiler.ast modules, the syntax tree is rebuilt. Thus the code becomes much clearer, and all monkey patching will eventually disappear from this module.

    Speed improvement is achieved by caching the parsed modules earlier to avoid double parsing, and avoiding some repeated inferences, all along fixing a lot of important bugs.

    Pylint

    Pylint 0.20.0 uses the new Astng, and fixes a lot of bugs too, adding some new functionality:

    • parameters with leading "_" shouldn't count as "local" variables
    • warn on assert( a, b )
    • warning if return or break inside a finally
    • specific message for NotImplemented exception

    We would like to thank Chmouel Boudjnah, Johnson Fletcher, Daniel Harding, Jonathan Hartley, Colin Moris, Winfried Plapper, Edward K. Ream and Pierre Rouleau for their contributions, and all other people helping the project to progress.


show 207 results