Blog entries

Logilab at Debconf 2014 - Debian annual conference

2014/08/21 by Arthur Lutz

Logilab is proud to contribute to the annual debian conference which will take place in Portland (USA) from the 23rd to the 31st of august.

Julien Cristau (debian page) will be giving two talks at the conference :

Logilab is also contributing to the conference as a sponsor for the event.

Here is what we previously blogged about salt and the previous debconf . Stay tuned for a blog post about what we saw and heard at the conference.

Pylint 1.3 / Astroid 1.2 released

2014/07/28 by Sylvain Thenault

The EP14 Pylint sprint team (more on this here and there) is proud to announce they just released Pylint 1.3 together with its companion Astroid 1.2. As usual, this includes several new features as well and bug fixes. You'll find below some structured list of the changes.

Packages are uploaded to pypi, debian/ubuntu packages should be soon provided by Logilab, until they get into the standard packaging system of your favorite distribution.

Please notice Pylint 1.3 will be the last release branch support python 2.5 and 2.6. Starting from 1.4, we will only support python greater or equal to 2.7. This will be the occasion to do some great cleanup in the code base. Notice this is only about the Pylint's runtime, you should still be able to run Pylint on your Python 2.5 code, through using Python 2.7 at least.

New checks

  • Add multiple checks for PEP 3101 advanced string formatting: 'bad-format-string', 'missing-format-argument-key', 'unused-format-string-argument', 'format-combined-specification', 'missing-format-attribute' and 'invalid-format-index'
  • New 'invalid-slice-index' and 'invalid-sequence-index' for invalid sequence and slice indices
  • New 'assigning-non-slot' warning, which detects assignments to attributes not defined in slots

Improved checkers

  • Fixed 'fixme' false positive (#149)
  • Fixed 'unbalanced-iterable-unpacking' false positive when encountering starred nodes (#273)
  • Fixed 'bad-format-character' false positive when encountering the 'a' format on Python 3
  • Fixed 'unused-variable' false positive when the variable is assigned through an import (#196)
  • Fixed 'unused-variable' false positive when assigning to a nonlocal (#275)
  • Fixed 'pointless-string-statement' false positive for attribute docstrings (#193)
  • Emit 'undefined-variable' when using the Python 3 metaclass= argument. Also fix 'unused-import' false for that construction (#143)
  • Emit 'broad-except' and 'bare-except' even if the number of except handlers is different than 1. Fixes issue (#113)
  • Emit 'attribute-defined-outside-init' for all statements in the same module as the offended class, not just for the last assignment (#262, as well as a long standing output mangling problem in some edge cases)
  • Emit 'not-callable' when calling properties (#268)
  • Don't let ImportError propagate from the imports checker, leading to crash in some namespace package related cases (#203)
  • Don't emit 'no-name-in-module' for ignored modules (#223)
  • Don't emit 'unnecessary-lambda' if the body of the lambda call contains call chaining (#243)
  • Definition order is considered for classes, function arguments and annotations (#257)
  • Only emit 'attribute-defined-outside-init' for definition within the same module as the offended class, avoiding to mangle the output in some cases
  • Don't emit 'hidden-method' message when the attribute has been monkey-patched, you're on your own when you do that.

Others changes

  • Checkers are now properly ordered to respect priority(#229)
  • Use the proper mode for pickle when opening and writing the stats file (#148)

Astroid changes

  • Function nodes can detect decorator call chain and see if they are decorated with builtin descriptors (classmethod and staticmethod).
  • infer_call_result called on a subtype of the builtin type will now return a new Class rather than an Instance.
  • Class.metaclass() now handles module-level __metaclass__ declaration on python 2, and no longer looks at the __metaclass__ class attribute on python 3.
  • Add slots method to Class nodes, for retrieving the list of valid slots it defines.
  • Expose function annotation to astroid: Arguments node exposes 'varargannotation', 'kwargannotation' and 'annotations' attributes, while Function node has the 'returns' attribute.
  • Backported most of the logilab.common.modutils module there, as most things there are for pylint/astroid only and we want to be able to fix them without requiring a new logilab.common release
  • Fix names grabed using wildcard import in "absolute import mode" (i.e. with absolute_import activated from the __future__ or with python 3) (pylint issue #58)
  • Add support in brain for understanding enum classes.

EP14 Pylint sprint Day 2 and 3 reports

2014/07/28 by Sylvain Thenault

Here are the list of things we managed to achieve during those last two days at EuroPython.

After several attempts, Michal managed to have pylint running analysis on several files in parallel. This is still in a pull request ( because of some limitations, so we decided it won't be part of the 1.3 release.

Claudiu killed maybe 10 bugs or so and did some heavy issues cleanup in the trackers. He also demonstrated some experimental support of python 3 style annotation to drive a better inference. Pretty exciting! Torsten also killed several bugs, restored python 2.5 compat (though that will need a logilab-common release as well), introduced a new functional test framework that will replace the old one once all the existing tests will be backported. On wednesday, he did show us a near future feature they already have at Google: some kind of confidence level associated to messages so that you can filter out based on that. Sylvain fixed a couple of bugs (including which was annoying all the numpy community), started some refactoring of the PyLinter class so it does a little bit fewer things (still way too many though) and attempted to improve the pylint note on both pylint and astroid, which went down recently "thanks" to the new checks like 'bad-continuation'.

Also, we merged the pylint-brain project into astroid to simplify things, so you should now submit your brain plugins directly to the astroid project. Hopefuly you'll be redirected there on attempt to use the old (removed) pylint-brain project on bitbucket.

And, the good news is that now both Torsten and Claudiu have new powers: they should be able to do some releases of pylint and astroid. To celebrate that and the end of the sprint, we published Pylint 1.3 together with Astroid 1.2. More on this here.

EP14 Pylint sprint Day 1 report

2014/07/24 by Sylvain Thenault

We've had a fairly enjoyable and productive first day in our little hidden room at EuroPython in Berlin ! Below are some noticeable things we've worked on and discussed about.

First, we discussed and agreed that while we should at some point cut the cord to the logilab.common package, it will take some time notably because of the usage logilab.common.configuration which would be somewhat costly to replace (and is working pretty well). There are some small steps we should do but basically we should mostly get back some pylint/astroid specific things from logilab.common to astroid or pylint. This should be partly done during the sprint, and remaining work will go to tickets in the tracker.

We also discussed about release management. The point is that we should release more often, so every pylint maintainers should be able to do that easily. Sylvain will write some document about the release procedure and ensure access are granted to the pylint and astroid projects on pypi. We shall release pylint 1.3 / astroid 1.2 soon, and those releases branches will be the last one supporting python < 2.7.

During this first day, we also had the opportunity to meet Carl Crowder, the guy behind, as well as David Halter which is building the Jedi completion library ( runs pylint on thousands of projects, and it would be nice if we could test beta release on some part of this panel. On the other hand, there are probably many code to share with the Jedi library like the parser and ast generation, as well as a static inference engine. That deserves a sprint on his own though, so we agreed that a nice first step would be to build a common library for import resolution without relying on the python interpreter for that, while handling most of the python dark import features like zip/egg import, .pth files and so one. Indeed that may be two nice future collaborations!

Last but not least, we got some actual work done:

  • Michal Nowikowski from Intel in Poland joined us to work on the ability to run pylint in different processes so it may drastically improve performance on multiple cores box.
  • Torsten did continue some work on various improvements of the functionnal test framework.
  • Sylvain did merge logilab.common.modutils module into astroid as it's mostly driven by astroid and pylint needs. Also fixed the annoying namespace package crash.
  • Claudiu keep up the good work he does daily at improving and fixing pylint :)

PyLint sprint during EuroPython in Berlin

2014/07/11 by Sylvain Thenault

The three main maintainers/developpers of Pylint/astroid (Claudiu, Torsten and I) will meet together in Berlin during EuroPython 2014. While this is not an "official" EuroPython sprint but it's still worth announcing it since it's a good opportunity to meet and enhance Pylint. We should find place and time to work on Pylint between wednesday 23 and friday 25.

If you're interested, don't hesitate to contact me ( / @sythenault).

Tester MPI avec CMake et un framework de test comme Boost

2014/06/20 by Damien Garaud


Compiler et exécuter un fichier de test unitaire avec MPI.

Je suppose que :

  • une implémentation de MPI est installée (nous prendrons OpenMPI)
  • les bibliothèques Boost MPI et Boost Unit Test Framework sont présentes
  • vous connaissez quelques rudiments de CMake


On utilise le bien connu find_package pour Boost et MPI afin de récupérer tout ce qu'il nous faut pour les headers et les futurs links.

find_package (MPI REQUIRED)
find_package (Boost COMPONENTS mpi REQUIRED)

# Boost dirs for headers and libs.
include_directories (SYSTEM ${Boost_INCLUDE_DIR})
link_directories (${Boost_LIBRARY_DIRS})

Par la suite, on a essentiellement besoin des variables CMake :

  • Boost_MPI_LIBRARY pour le link avec Boost::MPI
  • MPI_CXX_LIBRARIES pour le link avec la bibliothèque OpenMPI
  • MPIEXEC qui nous donne la commande pour lancer un exécutable via MPI

On prend un fichier example_mpi.cpp (des exemples simples sont faciles à trouver sur la Toile). Pour le compiler, on fait :


# MPI example.
add_executable(example_mpi example_mpi.cpp)
target_link_libraries(example_mpi ${MPI_CXX_LIBRARIES})

voire juste

target_link_libraries(example_mpi ${Boost_MPI_LIBRARY})

si on décide d'utiliser la bibliothèque Boost MPI.


mpicxx est une commande qui enrobe le compilateur (g++ par exemple). On dit à CMake d'utiliser mpicxx au lieu du compilo par défaut.


Boost::MPI n'est pas une implémentation de MPI. C'est une bibliothèque plus haut niveau qui s'abstrait de l'implémentation de MPI. Il faut nécessairement en installer une (OpenMPI, LAM/MPI, MPICH2, ...).

L'exemple peut très simplement ressembler à :

#include <boost/mpi/environment.hpp>
#include <boost/mpi/communicator.hpp>
#include <iostream>

namespace mpi = boost::mpi;

int main()
  mpi::environment env;
  mpi::communicator world;
  std::cout << "I am process " << world.rank() << " of " << world.size()
            << "." << std::endl;
  return 0;


Une fois la compilation effectuée, faire simplement :

> mpiexec -np 4 ./example_mpi

pour lancer l'exécutable sur 4 cœurs.

Problème : mais pourquoi tu testes ?

On veut pouvoir faire des exécutables qui soient de vrais tests unitaires et non pas un exemple avec juste une fonction main. De plus, comme j'utilise CMake, je veux pouvoir automatiser le lancement de tous mes exécutables via CTest.

Problème : il faut bien initialiser et bien dire à MPI que j'en ai fini avec toutes mes MPI-series.

En "vrai" MPI, on a :

int main(int argc, char* argv[])
  MPI_Init(&argc, &argv);

  int rank;
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  // Code here..
  // ... and here

  return 0;


C'est ce que fait le constructeur/destructeur de boost::mpi::environment.

En d'autres termes, je veux me faire ma propre fonction main pour l'initialisation de tous mes cas tests Boost (ou avec n'importe quel autre framework de test unitaire C/C++).

La documentation de Boost Unit Test, que je trouve parfois très peu claire avec un manque cruel d'exemple, m'a fait galérer quelques heures avant de trouver quelque chose de simple qui fonctionne.

Conseil : aller regarder les exemples des sources en faisant quelques grep est parfois plus efficace que de trouver la bonne info dans la doc en ligne. On peut aussi en lire sur

Deux solutions :

  1. la première que j'ai trouvée dans les tests de Boost::MPI lui-même. Ils utilisent le minimal testing facility. Mais seule la macro BOOST_CHECK est utilisable. Et oubliez les BOOST_CHECK_EQUAL ainsi que l'enregistrement automatique de vos tests dans la suite de tests.
  2. la deuxième redéfinit la fonction main et appelle boost::unit_test::unit_test_main sans définir ni la macro BOOST_TEST_MODULE ni BOOST_TEST_MAIN qui impliquent la génération automatique de la fonction main par le framework de test (que l'on compile en statique ou dynamique). Pour plus de détails, lire

J'utiliserai la deuxième solution.


Ne pensez même pas faire une fixture Boost pour y mettre votre boost::mpi::environment puisque cette dernière sera construite/détruite pour chaque cas test (équivalent du setUp/tearDown). Et il est fort probable que vous ayez ce genre d'erreur :

*** The MPI_Errhandler_set() function was called after MPI_FINALIZE was invoked.
*** This is disallowed by the MPI standard.
*** Your MPI job will now abort.
[hostname:11843] Abort after MPI_FINALIZE completed successfully; not able to guarantee that all other processes were killed!

Un exemple qui marche

On souhaite ici tester que le nombre de procs passés en argument de mpiexec est au moins 2.

Le BOOST_TEST_DYN_LINK dit juste que je vais me lier dynamiquement à Boost::Test.

#include <boost/test/unit_test.hpp>

#include <boost/mpi/environment.hpp>
#include <boost/mpi/communicator.hpp>

namespace mpi = boost::mpi;

    mpi::communicator world;
    BOOST_CHECK(world.size() > 1);

// (empty) Initialization function. Can't use testing tools here.
bool init_function()
    return true;

int main(int argc, char* argv[])
    mpi::environment env(argc, argv);
    return ::boost::unit_test::unit_test_main( &init_function, argc, argv );

On lance tout ça avec un joli mpiexec -np 2 ./test_mpi et on est (presque) content.

Et un peu de CTest pour finir

Une dernière chose : on aimerait dire à CMake via CTest de lancer cet exécutable, mais avec mpiexec et les arguments qui vont bien.

Un cmake --help-command add_test nous indique qu'il est possible de lancer n'importe quel exécutable avec un nombre variable d'arguments. On va donc lui passer : /usr/bin/mpiexec -np NB_PROC ./test_mpi.

Un œil au module FindMPI.cmake nous dit aussi qu'on peut utiliser d'autres variables CMake MPI pour ce genre de chose.

Reprenons donc notre fichier CMake et ajoutons :

add_executable(test_mpi test_mpi.cpp)
target_link_libraries(test_mpi ${Boost_MPI_LIBRARY})
# Number of procs for MPI.
set (PROCS 2)
# Command to launch by CTest. Need the MPI executable and the number of procs.
add_test (test_mpi ${MPIEXEC} ${MPIEXEC_NUMPROC_FLAG} ${PROCS}

où mon exécutable et mon test porte le même nom : test_mpi. On lance le tout avec la commande ctest dans notre dossier de build et hop, le tour est joué !

Open Legislative Data Conference 2014

2014/06/10 by Nicolas Chauvat

I was at the Open Legislative Data Conference on may 28 2014 in Paris, to present a simple demo I worked on since the same event that happened two years ago.

The demo was called "Law is Code Rebooted with CubicWeb". It featured the use of the cubicweb-vcreview component to display the amendments of the hospital law ("loi hospitalière") gathered into a version control system (namely Mercurial).

The basic idea is to compare writing code and writing law, for both are collaborative and distributed writing processes. Could we reuse for the second one the tools developed for the first?

Here are the slides and a few screenshots.

Statistics with queries embedded in report page.

List of amendments.

User comment on an amendment.

While attending the conference, I enjoyed several interesting talks and chats with other participants, including:

  1. the study of co-sponsorship of proposals in the french parliament
  2. announcing their use of PostgreSQL and JSON.
  3. and last but not least, the great work done by RegardsCitoyens and SciencesPo MediaLab on visualizing the law making process.

Thanks to the organisation team and the other speakers. Hope to see you again!

SaltStack Meetup with Thomas Hatch in Paris France

2014/05/22 by Arthur Lutz

This monday (19th of may 2014), Thomas Hatch was in Paris for dotScale 2014. After presenting SaltStack there (videos will be published at some point), he spent the evening with members of the French SaltStack community during a meetup set up by Logilab at IRILL.

Here is a list of what we talked about :

  • Since Salt seems to have pushed ZMQ to its limits, SaltStack has been working on RAET (Reliable Asynchronous Event Transport Protocol ), a transport layer based on UDP and elliptic curve cryptography (Dan Berstein's CURVE-255-19) that works more like a stack than a socket and has reliability built in. RAET will be released as an optionnal beta feature in the next Salt release.
  • Folks from Dailymotion bumped into a bug that seems related to high latency networks and the auth_timeout. Updating to the very latest release should fix the issue.
  • Thomas told us about how a dedicated team at SaltStack handles pull requests and another team works on triaging github issues to input them into their internal SCRUM process. There are a lot of duplicate issues and old inactive issues that need attention and clutter the issue tracker. Help will be welcome.
  • Continuous integration is based on Jenkins and spins up VMs to test pull request. There is work in progress to test multiple clouds, various latencies and loads.
  • For the Docker integration, salt now keeps track of forwarded ports and relevant information about the containers.
  • salt-virt bumped into problems with chroots and timeouts due to ZMQ.
  • Multi-master: the problem lies with syncronisation of data which is sent to minions but also the data that is sent to the masters. Possible solutions to be explored are : the use of gitfs, there is no built-in solution for keys (salt-key has to be run on all masters), mine.send should send the data at both masters, for the jobs cache: one could use an external returner.
  • Thomas talked briefly about ioflo which should bring queuing, data hierarchy and data pub-sub to Salt.
  • About the rolling release question: versions in Salt are definitely not git snapshots, things get backported into previous versions. No clear definition yet of length of LTS versions.
  • salt-cloud and libcloud : in the next release, libcloud will not be a hard dependency. Some clouds didn't work in libcloud (for example AWS), so these providers got implemented directly in salt-cloud or by using third-party libraries (eg. python-boto).
  • Documentation: a sprint is planned next week. Reference documentation will not be completly revamped, but tutorial content will be added.

Boris Feld showed a demo of vagrant images orchestrated by salt and a web UI to monitor a salt install.

Thanks again to Thomas Hatch for coming and meeting up with (part of) the community here in France.

Compte rendu présentation Salt à Solution Linux

2014/05/21 by Arthur Lutz

Logilab était à l'édition 2014 de Solutions Linux qui se déroulait au CNIT à Paris. David Douard participait à la table ronde sur les outils libres pour la supervision lors de la session Administration Système, Devops, au cours de laquelle un certain nombre de projets libres ont été mentionnés : nagios, shinken, graphite, ElasticSearch, logstash, munin, saltstack, kibana, centreon, rsyslog.

Suite à des présentations sur OpenLDAP, LXC, btrfs et ElasticSearch David Douard a présenté notre approche agile de l'administration système articulée autour de Salt et en particulier le principe de l'administration système pilotée par les tests (diapos) (Test-Driven Infrastructure).

Merci aux organisateurs de Solutions Linux pour cette édition 2014.