subscribe to this blog

Logilab.org - en

News from Logilab and our Free Software projects, as well as on topics dear to our hearts (Python, Debian, Linux, the semantic web, scientific computing...)

show 207 results
  • pylint bugs day #2 on april 16, 2010

    2010/03/22 by Sylvain Thenault

    Hey guys,

    we'll hold the next pylint bugs day on april 16th 2010 (friday). If some of you want to come and work with us in our Paris office, you'll be much welcome.

    Else you can still join us on jabber / irc:

    See you then!


  • PostgreSQL on windows : plpythonu and "specified module could not be found" error

    2010/03/22

    I recently had to (remotely) debug an issue on windows involving PostgreSQL and PL/Python. Basically, two very similar computers, with Python2.5 installed via python(x,y), PostgreSQL 8.3.8 installed via the binary installer. On the first machine create language plpythonu; worked like a charm, and on the other one, it failed with C:\\Program Files\\Postgresql\\8.3\\plpython.dll: specified module could not be found. This is caused by the dynamic linker not finding some DLL. Using Depends.exe showed that plpython.dll looks for python25.dll (the one it was built against in the 8.3.8 installer), but that the DLL was there.

    I'll save the various things we tried and jump directly to the solution. After much head scratching, it turned out that the first computer had TortoiseHg installed. This caused C:\\Program Files\\TortoiseHg to be included in the System PATH environment variable, and that directory contains python25.dll. On the other hand C:\\Python25 was in the user's PATH environment variable on both computers. As the database Windows service runs using a dedicated local account (typically with login postgres), it would not have C:\\Python25 in its PATH, but if TortoiseHg was there, it would find the DLL in some other directory. So the solution was to add C:\\Python25 to the system PATH.


  • Now publishing blog entries under creative commons

    2010/03/15 by Arthur Lutz

    Logilab is proud to announce that the blog entries published on the blogs of http://www.logilab.org and http://www.cubicweb.org are now licensed under a Creative Commons Attribution-Share Alike 2.0 License (check out the footer).

    http://creativecommons.org/images/deed/seal.png

    We often use creative commons licensed photographs to illustrate this blog, and felt that being developers of open source software it was quite logical that some of our content should be published under a similar license. Some of the documentation that we release also uses this license, for example the "Building Salome" documentation. This license footer has been integrated to the cubicweb-blog package that is used to publish our sites (as part of cubicweb-forge).


  • Launching Python scripts via Condor

    2010/02/17
    http://farm2.static.flickr.com/1362/1402963775_0185d2e62f.jpg

    As part of an ongoing customer project, I've been learning about the Condor queue management system (actually it is more than just a batch queue management system, tacking the High-throughput computing problem, but in my current project, we're not using the full possibilities of Condor, and the choice was dictated by other considerations outside the scope of this note). The documentation is excellent, and the features of the product are really amazing (pity the project runs on Windows, and we cannot use 90% of these...).

    To launch a job on a computer participating in the Condor farm, you just have to write a job file which looks like this:

    Universe=vanilla
    Executable=$path_to_executabe
    Arguments=$arguments_to_the_executable
    InitialDir=$working_directory
    Log=$local_logfile_name
    Output=$local_file_for_job_stdout
    Error=$local_file_for_job_stderr
    Queue
    

    and then run condor_submit my_job_file and use condor_q to monitor the status your job (queued, running...)

    My program is generating Condor job files and submitting them, and I've spent hours yesterday trying to understand why they were all failing : the stderr file contained a message from Python complaining that it could not import site and exiting.

    A point which was not clear in the documentation I read (but I probably overlooked it) is that the executable mentionned in the job file is supposed to be a local file on the submission host which is copied to the computer running the job. In the jobs generated by my code, I was using sys.executable for the Executable field, and a path to the python script I wanted to run in the Arguments field. This resulted in the Python interpreter being copied on the execution host and not being able to run because it was not able to find the standard files it needs at startup.

    Once I figured this out, the fix was easy: I made my program write a batch script which launched the Python script and changed the job to run that script.

    UPDATE : I'm told there is a Transfer_executable=False line I could have put in the script to achieve the same thing.

    (photo by gudi&cris licenced under CC-BY-ND)


  • Adding Mercurial build identification to Python

    2010/02/15 by Andre Espaze

    This work is a part of the build identification task found in the PEP 385, Migrating from svn to Mercurial: http://www.python.org/dev/peps/pep-0385/ It was done during the Mercurial sprint hosted at Logilab. If you would like to see the result, just follow the steps:

    hg clone http://hg.xavamedia.nl/cpython/pymigr/
    cd pymigr/build-identification
    

    Setting up the environment

    The current Python development branch is first checkout:

    svn co http://svn.python.org/projects/python/trunk
    

    A patch will be applied for adding the 'sys.mercurial' attribute and modifying the build informations:

    cp add-hg-build-id.diff trunk/
    cd trunk
    svn up -r 78019
    patch -p0 < add-hg-build-id.diff
    

    The changed made to 'configure.in' need then to be propagated to the configure script:

    autoconf
    

    The configuration is then done by:

    ./configure --enable-shared --prefix=/dev/null
    

    You should now see changes propagated to the Makefile for finding the revision, the tag and the branch:

    grep MERCURIAL Makefile
    

    Finally, Python can be built:

    make
    

    The sys.mercurial attribute should already be present:

    LD_LIBRARY_PATH=. ./python
    >>> import sys
    >>> sys.mercurial
    ('CPython', '', '')
    

    No tag nor revision have been found as there was no mercurial repository. A test by the Py_GetBuildInfo() in the C API will also be built:

    gcc -o show-build-info -I. -IInclude -L. -lpython2.7 ../show-build-info.c
    

    You can test its result by:

    LD_LIBRARY_PATH=. ./show-build-info
    -> default, Feb  7 2010, 15:07:46
    

    Manual test

    First a fake mercurial tree is built:

    hg init
    hg add README
    hg ci -m "Initial repo"
    hg id
    -> 84a6de74e48f tip
    

    Now Python needs to be built with the given mercurial information:

    rm Modules/getbuildinfo.o
    make
    

    You should then see the current revision number:

    LD_LIBRARY_PATH=. ./python
    >>> import sys
    >>> sys.mercurial
    ('CPython', 'default', '84a6de74e48f')
    

    and the C API can be tested by:

    LD_LIBRARY_PATH=. ./show-build-info
    -> default:84a6de74e48f, Feb  7 2010, 15:10:13
    

    The fake mercurial repository can now be cleaned:

    rm -rf .hg
    

    Automatic tests

    Automatic tests for checking the behavior for every cases will now work build Python and clean afterward. Those tests work only when run from the trunk svn directory of Python:

    python ../test_build_identification.py
    

    Further work

    The current work is only an attempt to add the mercurial build identification to Python, it still needs to be checked on production cases. Moreover the build identification on Windows has not been started yet, it will need to be integrated to the Microsoft Visual Studio building process.


  • Why you shoud get rid of os.system, os.popen, etc. in your code

    2010/02/12

    I regularly come across code such as:

    output = os.popen('diff -u %s %s' % (appl_file, ref_file), 'r')
    

    Code like this might well work machine but it is buggy and will fail (preferably during the demo or once shipped).

    Where is the bug?

    It is in the use of %s, which can inject in your command any string you want and also strings you don't want. The problem is that you probably did not check appl_file and ref_file for weird things (spaces, quotes, semi colons...). Putting quotes around the %s in the string will not solve the issue.

    So what should you do? The answer is "use the subprocess module": subprocess.Popen takes a list of arguments as first parameter, which are passed as-is to the new process creation system call of your platform, and not interpreted by the shell:

    pipe = subprocess.Popen(['diff', '-u', appl_file, ref_file], stdout=subprocess.PIPE)
    output = pipe.stdout
    

    By now, you should have guessed that the shell=True parameter of subprocess.Popen should not be used unless you really really need it (and even them, I encourage you to question that need).


  • Apycot for Mercurial

    2010/02/11 by Pierre-Yves David
    http://www.logilab.org/image/20439?vid=download

    What is apycot

    apycot is a highly extensible test automatization tool used for Continuous Integration. It can:

    • download the project from a version controlled repository (like SVN or Hg);
    • install it from scratch with all dependencies;
    • run various checkers;
    • store the results in a CubicWeb database;
    • post-process the results;
    • display the results in various format (html, xml, pdf, mail, RSS...);
    • repeat the whole procedure with various configurations;
    • get triggered by new changesets or run periodically.

    For an example, take a look at the "test reports" tab of the logilab-common project.

    Setting up an apycot for Mercurial

    During the mercurial sprint, we set up a proof-of-concept environment running six different checkers:

    • Check syntax of all python files.
    • Check syntax of all documentation files.
    • Run pylint on the mercurial source code with the mercurial pylintrc.
    • Run the check-code.py script included in mercurial checking style and python errors
    • Run the Mercurial's test suite.
    • Run Mercurial's benchmark on a reference repository.

    The first three checkers, shipped with apycot, were set up quickly. The last three are mercurial specific and required few additional tweaks to be integrated to apycot.

    The bot was setup to run with all public mercurial repositories. Five checkers immediately proved useful as they pointed out some errors or warnings (on some rarely used contrib files it even found a syntax error).

    Prospectives

    A public instance is being set up. It will provide features that the community is looking forward to:

    • testing all python versions;
    • running pure python or the C variant;
    • code coverage of the test suite;
    • performance history.

    Conclusion

    apycot proved to be highly flexible and could quickly be adapted to Mercurial's test suite even for people new to apycot. The advantages of continuously running different long running tests is obvious. So apycot seems to be a very valuable tool for improving the software development process.


  • SCons presentation in 5 minutes

    2010/02/09 by Andre Espaze
    http://www.scons.org/scons-logo-transparent.png

    Building software with SCons requires to have Python and SCons installed.

    As SCons is only made of Python modules, the sources may be shipped with your project if your clients can not install dependencies. All the following exemples can be downloaded at the end of that blog.

    A building tool for every file extension

    First a Fortran 77 program will be built made of two files:

    $ cd fortran-project
    $ scons -Q
    gfortran -o cfib.o -c cfib.f
    gfortran -o fib.o -c fib.f
    gfortran -o compute-fib cfib.o fib.o
    $ ./compute-fib
     First 10 Fibonacci numbers:
      0.  1.  1.  2.  3.  5.  8. 13. 21. 34.
    

    The '-Q' option tell to Scons to be less verbose. For cleaning the project, add the '-c' option:

    $ scons -Qc
    Removed cfib.o
    Removed fib.o
    Removed compute-fib
    

    From this first example, it can been seen that SCons find the 'gfortran' tool from the file extension. Then have a look at the user's manual if you want to set a particular tool.

    Describing the construction with Python objects

    A second C program will directly run the execution from the SCons file by adding a test command:

    $ cd c-project
    $ scons -Q run-test
    gcc -o test.o -c test.c
    gcc -o fact.o -c fact.c
    ar rc libfact.a fact.o
    ranlib libfact.a
    gcc -o test-fact test.o libfact.a
    run_test(["run-test"], ["test-fact"])
    OK
    

    However running scons alone builds only the main program:

    $ scons -Q
    gcc -o main.o -c main.c
    gcc -o compute-fact main.o libfact.a
    $ ./compute-fact
    Computing factorial for: 5
    Result: 120
    

    This second example shows that the construction dependency is described by passing Python objects. An interesting point is the possibility to add your own Python functions in the build process.

    Hierarchical build with environment

    A third C++ program will create a shared library used for two different programs: the main application and a test suite. The main application can be built by:

    $ cd cxx-project
    $ scons -Q
    g++ -o main.o -c -Imbdyn-src main.cxx
    g++ -o mbdyn-src/nodes.os -c -fPIC -Imbdyn-src mbdyn-src/nodes.cxx
    g++ -o mbdyn-src/solver.os -c -fPIC -Imbdyn-src mbdyn-src/solver.cxx
    g++ -o mbdyn-src/libmbdyn.so -shared mbdyn-src/nodes.os mbdyn-src/solver.os
    g++ -o mbdyn main.o -Lmbdyn-src -lmbdyn
    

    It shows that SCons handles for us the compilation flags for creating a shared library according to the tool (-fPIC). Moreover extra environment variables have been given (CPPPATH, LIBPATH, LIBS), which are all translated for the chosen tool. All those variables can be found in the user's manual or in the man page. The building and running of the test suite is made by giving an extra variable:

    $ TEST_CMD="LD_LIBRARY_PATH=mbdyn-src ./%s" scons -Q run-tests
    g++ -o tests/run_all_tests.o -c -Imbdyn-src tests/run_all_tests.cxx
    g++ -o tests/test_solver.o -c -Imbdyn-src tests/test_solver.cxx
    g++ -o tests/all-tests tests/run_all_tests.o tests/test_solver.o -Lmbdyn-src -lmbdyn
    run_test(["tests/run-tests"], ["tests/all-tests"])
    OK
    

    Conclusion

    That is rather convenient to build softwares by manipulating Python objects, moreover custom actions can be added in the process. SCons has also a configuration mechanism working like autotools macros that can be discovered in the user's manual.


  • Extended 256 colors in bash prompt

    2010/02/07 by Nicolas Chauvat

    The Mercurial 1.5 sprint is taking place in our offices this week-end and pair-programming with Steve made me want a better looking terminal. Have you seen his extravagant zsh prompt ? I used to have only 8 colors to decorate my shell prompt, but thanks to some time spent playing around, I now have 256.

    Here is what I used to have in my bashrc for 8 colors:

    NO_COLOUR="\[\033[0m\]"
    LIGHT_WHITE="\[\033[1;37m\]"
    WHITE="\[\033[0;37m\]"
    GRAY="\[\033[1;30m\]"
    BLACK="\[\033[0;30m\]"
    
    RED="\[\033[0;31m\]"
    LIGHT_RED="\[\033[1;31m\]"
    GREEN="\[\033[0;32m\]"
    LIGHT_GREEN="\[\033[1;32m\]"
    YELLOW="\[\033[0;33m\]"
    LIGHT_YELLOW="\[\033[1;33m\]"
    BLUE="\[\033[0;34m\]"
    LIGHT_BLUE="\[\033[1;34m\]"
    MAGENTA="\[\033[0;35m\]"
    LIGHT_MAGENTA="\[\033[1;35m\]"
    CYAN="\[\033[0;36m\]"
    LIGHT_CYAN="\[\033[1;36m\]"
    
    # set a fancy prompt
    export PS1="${RED}[\u@\h \W]\$${NO_COLOUR} "
    

    Just put the following lines in your bashrc to get the 256 colors:

    function EXT_COLOR () { echo -ne "\[\033[38;5;$1m\]"; }
    
    # set a fancy prompt
    export PS1="`EXT_COLOR 172`[\u@\h \W]\$${NO_COLOUR} "
    

    Yay, I now have an orange prompt! I now need to write a script that will display useful information depending on the context. Displaying the status of the mercurial repository I am in might be my next step.


  • We're happy to host the mercurial Sprint

    2010/02/02 by Arthur Lutz
    http://farm1.static.flickr.com/183/419945378_4ead41a76d_m.jpg

    We're very happy to be hosting the next mercurial sprint in our brand new offices in central Paris. It is quite an honor to be chosen when the other contender was Google.

    So a bunch of mercurial developers are heading out to our offices this coming Friday to sprint for three days on mercurial. We use mercurial a lot here over at Logilab and we also contribute a tool to visualize and manipulate a mercurial repository : hgview.

    To check out the things that we will be working on with the mercurial crew, check out the program of the sprint on their wiki.

    What is a sprint? "A sprint (sometimes called a Code Jam or hack-a-thon) is a short time period (three to five days) during which software developers work on a particular chunk of functionality. "The whole idea is to have a focused group of people make progress by the end of the week," explains Jeff Whatcott" [source]. For geographically distributed open source communities, it is also a way of physically meeting and working in the same room for a period of time.

    Sprinting is a practice that we encourage at Logilab, with CubicWeb we organize as often as possible open sprints, which is an opportunity for users and developers to come and code with us. We even use the sprint format for some internal stuff.

    photo by Sebastian Mary under creative commons licence.


show 207 results