Blog entries

EP14 Pylint sprint Day 1 report

2014/07/24 by Sylvain Thenault
https://ep2014.europython.eu/static_media/assets/images/logo.png

We've had a fairly enjoyable and productive first day in our little hidden room at EuroPython in Berlin ! Below are some noticeable things we've worked on and discussed about.

First, we discussed and agreed that while we should at some point cut the cord to the logilab.common package, it will take some time notably because of the usage logilab.common.configuration which would be somewhat costly to replace (and is working pretty well). There are some small steps we should do but basically we should mostly get back some pylint/astroid specific things from logilab.common to astroid or pylint. This should be partly done during the sprint, and remaining work will go to tickets in the tracker.

We also discussed about release management. The point is that we should release more often, so every pylint maintainers should be able to do that easily. Sylvain will write some document about the release procedure and ensure access are granted to the pylint and astroid projects on pypi. We shall release pylint 1.3 / astroid 1.2 soon, and those releases branches will be the last one supporting python < 2.7.

During this first day, we also had the opportunity to meet Carl Crowder, the guy behind http://landscape.io, as well as David Halter which is building the Jedi completion library (https://github.com/davidhalter/jedi). Landscape.io runs pylint on thousands of projects, and it would be nice if we could test beta release on some part of this panel. On the other hand, there are probably many code to share with the Jedi library like the parser and ast generation, as well as a static inference engine. That deserves a sprint on his own though, so we agreed that a nice first step would be to build a common library for import resolution without relying on the python interpreter for that, while handling most of the python dark import features like zip/egg import, .pth files and so one. Indeed that may be two nice future collaborations!

Last but not least, we got some actual work done:

  • Michal Novikowsky from Intel in Poland joined us to work on the ability to run pylint in different processes so it may drastically improve performance on multiple cores box.
  • Torsten did continue some work on various improvements of the functionnal test framework.
  • Sylvain did merge logilab.common.modutils module into astroid as it's mostly driven by astroid and pylint needs. Also fixed the annoying namespace package crash.
  • Claudiu keep up the good work he does daily at improving and fixing pylint :)

PyLint sprint during EuroPython in Berlin

2014/07/11 by Sylvain Thenault

The three main maintainers/developpers of Pylint/astroid (Claudiu, Torsten and I) will meet together in Berlin during EuroPython 2014. While this is not an "official" EuroPython sprint but it's still worth announcing it since it's a good opportunity to meet and enhance Pylint. We should find place and time to work on Pylint between wednesday 23 and friday 25.

If you're interested, don't hesitate to contact me (sylvain.thenault@logilab.fr / @sythenault).


Tester MPI avec CMake et un framework de test comme Boost

2014/06/20 by Damien Garaud

Objectif

Compiler et exécuter un fichier de test unitaire avec MPI.

Je suppose que :

  • une implémentation de MPI est installée (nous prendrons OpenMPI)
  • les bibliothèques Boost MPI et Boost Unit Test Framework sont présentes
  • vous connaissez quelques rudiments de CMake

CMake

On utilise le bien connu find_package pour Boost et MPI afin de récupérer tout ce qu'il nous faut pour les headers et les futurs links.

find_package (MPI REQUIRED)
find_package (Boost COMPONENTS mpi REQUIRED)

# Boost dirs for headers and libs.
include_directories (SYSTEM ${Boost_INCLUDE_DIR})
link_directories (${Boost_LIBRARY_DIRS})

Par la suite, on a essentiellement besoin des variables CMake :

  • Boost_MPI_LIBRARY pour le link avec Boost::MPI
  • MPI_CXX_LIBRARIES pour le link avec la bibliothèque OpenMPI
  • MPIEXEC qui nous donne la commande pour lancer un exécutable via MPI

On prend un fichier example_mpi.cpp (des exemples simples sont faciles à trouver sur la Toile). Pour le compiler, on fait :

set(CMAKE_CXX_COMPILER mpicxx)

# MPI example.
add_executable(example_mpi example_mpi.cpp)
target_link_libraries(example_mpi ${MPI_CXX_LIBRARIES})

voire juste

target_link_libraries(example_mpi ${Boost_MPI_LIBRARY})

si on décide d'utiliser la bibliothèque Boost MPI.

Note

mpicxx est une commande qui enrobe le compilateur (g++ par exemple). On dit à CMake d'utiliser mpicxx au lieu du compilo par défaut.

Note

Boost::MPI n'est pas une implémentation de MPI. C'est une bibliothèque plus haut niveau qui s'abstrait de l'implémentation de MPI. Il faut nécessairement en installer une (OpenMPI, LAM/MPI, MPICH2, ...).

L'exemple peut très simplement ressembler à :

#include <boost/mpi/environment.hpp>
#include <boost/mpi/communicator.hpp>
#include <iostream>

namespace mpi = boost::mpi;

int main()
{
  mpi::environment env;
  mpi::communicator world;
  std::cout << "I am process " << world.rank() << " of " << world.size()
            << "." << std::endl;
  return 0;
}

Exécuter

Une fois la compilation effectuée, faire simplement :

> mpiexec -np 4 ./example_mpi

pour lancer l'exécutable sur 4 cœurs.

Problème : mais pourquoi tu testes ?

On veut pouvoir faire des exécutables qui soient de vrais tests unitaires et non pas un exemple avec juste une fonction main. De plus, comme j'utilise CMake, je veux pouvoir automatiser le lancement de tous mes exécutables via CTest.

Problème : il faut bien initialiser et bien dire à MPI que j'en ai fini avec toutes mes MPI-series.

En "vrai" MPI, on a :

int main(int argc, char* argv[])
{
  MPI_Init(&argc, &argv);

  int rank;
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  // Code here..
  // ... and here
  MPI_Finalize();

  return 0;
}

Note

C'est ce que fait le constructeur/destructeur de boost::mpi::environment.

En d'autres termes, je veux me faire ma propre fonction main pour l'initialisation de tous mes cas tests Boost (ou avec n'importe quel autre framework de test unitaire C/C++).

La documentation de Boost Unit Test, que je trouve parfois très peu claire avec un manque cruel d'exemple, m'a fait galérer quelques heures avant de trouver quelque chose de simple qui fonctionne.

Conseil : aller regarder les exemples des sources en faisant quelques grep est parfois plus efficace que de trouver la bonne info dans la doc en ligne. On peut aussi en lire sur https://github.com/boostorg/test/tree/master/example

Deux solutions :

  1. la première que j'ai trouvée dans les tests de Boost::MPI lui-même. Ils utilisent le minimal testing facility. Mais seule la macro BOOST_CHECK est utilisable. Et oubliez les BOOST_CHECK_EQUAL ainsi que l'enregistrement automatique de vos tests dans la suite de tests.
  2. la deuxième redéfinit la fonction main et appelle boost::unit_test::unit_test_main sans définir ni la macro BOOST_TEST_MODULE ni BOOST_TEST_MAIN qui impliquent la génération automatique de la fonction main par le framework de test (que l'on compile en statique ou dynamique). Pour plus de détails, lire http://www.boost.org/doc/libs/release/libs/test/doc/html/utf/user-guide/test-runners.html

J'utiliserai la deuxième solution.

Note

Ne pensez même pas faire une fixture Boost pour y mettre votre boost::mpi::environment puisque cette dernière sera construite/détruite pour chaque cas test (équivalent du setUp/tearDown). Et il est fort probable que vous ayez ce genre d'erreur :

*** The MPI_Errhandler_set() function was called after MPI_FINALIZE was invoked.
*** This is disallowed by the MPI standard.
*** Your MPI job will now abort.
[hostname:11843] Abort after MPI_FINALIZE completed successfully; not able to guarantee that all other processes were killed!

Un exemple qui marche

On souhaite ici tester que le nombre de procs passés en argument de mpiexec est au moins 2.

Le BOOST_TEST_DYN_LINK dit juste que je vais me lier dynamiquement à Boost::Test.

#define BOOST_TEST_DYN_LINK
#include <boost/test/unit_test.hpp>

#include <boost/mpi/environment.hpp>
#include <boost/mpi/communicator.hpp>

namespace mpi = boost::mpi;

BOOST_AUTO_TEST_CASE(test_check_world_size)
{
    mpi::communicator world;
    BOOST_CHECK(world.size() > 1);
}


// (empty) Initialization function. Can't use testing tools here.
bool init_function()
{
    return true;
}

int main(int argc, char* argv[])
{
    mpi::environment env(argc, argv);
    return ::boost::unit_test::unit_test_main( &init_function, argc, argv );
}

On lance tout ça avec un joli mpiexec -np 2 ./test_mpi et on est (presque) content.

Et un peu de CTest pour finir

Une dernière chose : on aimerait dire à CMake via CTest de lancer cet exécutable, mais avec mpiexec et les arguments qui vont bien.

Un cmake --help-command add_test nous indique qu'il est possible de lancer n'importe quel exécutable avec un nombre variable d'arguments. On va donc lui passer : /usr/bin/mpiexec -np NB_PROC ./test_mpi.

Un œil au module FindMPI.cmake nous dit aussi qu'on peut utiliser d'autres variables CMake MPI pour ce genre de chose.

Reprenons donc notre fichier CMake et ajoutons :

add_executable(test_mpi test_mpi.cpp)
target_link_libraries(test_mpi ${Boost_MPI_LIBRARY})
# Number of procs for MPI.
set (PROCS 2)
# Command to launch by CTest. Need the MPI executable and the number of procs.
add_test (test_mpi ${MPIEXEC} ${MPIEXEC_NUMPROC_FLAG} ${PROCS}
   ${MPIEXEC_PREFLAGS}
   test_mpi
   ${MPIEXEC_POSTFLAGS})

où mon exécutable et mon test porte le même nom : test_mpi. On lance le tout avec la commande ctest dans notre dossier de build et hop, le tour est joué !


Open Legislative Data Conference 2014

2014/06/10 by Nicolas Chauvat

I was at the Open Legislative Data Conference on may 28 2014 in Paris, to present a simple demo I worked on since the same event that happened two years ago.

The demo was called "Law is Code Rebooted with CubicWeb". It featured the use of the cubicweb-vcreview component to display the amendments of the hospital law ("loi hospitalière") gathered into a version control system (namely Mercurial).

The basic idea is to compare writing code and writing law, for both are collaborative and distributed writing processes. Could we reuse for the second one the tools developed for the first?

Here are the slides and a few screenshots.

http://www.logilab.org/file/253394/raw/lawiscode1.png

Statistics with queries embedded in report page.

http://www.logilab.org/file/253400/raw/lawiscode2.png

List of amendments.

http://www.logilab.org/file/253396/raw/lawiscode3.png

User comment on an amendment.

While attending the conference, I enjoyed several interesting talks and chats with other participants, including:

  1. the study of co-sponsorship of proposals in the french parliament
  2. data.senat.fr announcing their use of PostgreSQL and JSON.
  3. and last but not least, the great work done by RegardsCitoyens and SciencesPo MediaLab on visualizing the law making process.

Thanks to the organisation team and the other speakers. Hope to see you again!


SaltStack Meetup with Thomas Hatch in Paris France

2014/05/22 by Arthur Lutz

This monday (19th of may 2014), Thomas Hatch was in Paris for dotScale 2014. After presenting SaltStack there (videos will be published at some point), he spent the evening with members of the French SaltStack community during a meetup set up by Logilab at IRILL.

http://www.logilab.org/file/248338/raw/thomas-hatch.png

Here is a list of what we talked about :

  • Since Salt seems to have pushed ZMQ to its limits, SaltStack has been working on RAET (Reliable Asynchronous Event Transport Protocol ), a transport layer based on UDP and elliptic curve cryptography (Dan Berstein's CURVE-255-19) that works more like a stack than a socket and has reliability built in. RAET will be released as an optionnal beta feature in the next Salt release.
  • Folks from Dailymotion bumped into a bug that seems related to high latency networks and the auth_timeout. Updating to the very latest release should fix the issue.
  • Thomas told us about how a dedicated team at SaltStack handles pull requests and another team works on triaging github issues to input them into their internal SCRUM process. There are a lot of duplicate issues and old inactive issues that need attention and clutter the issue tracker. Help will be welcome.
http://www.logilab.org/file/248336/raw/Salt-Logo.png
  • Continuous integration is based on Jenkins and spins up VMs to test pull request. There is work in progress to test multiple clouds, various latencies and loads.
  • For the Docker integration, salt now keeps track of forwarded ports and relevant information about the containers.
  • salt-virt bumped into problems with chroots and timeouts due to ZMQ.
  • Multi-master: the problem lies with syncronisation of data which is sent to minions but also the data that is sent to the masters. Possible solutions to be explored are : the use of gitfs, there is no built-in solution for keys (salt-key has to be run on all masters), mine.send should send the data at both masters, for the jobs cache: one could use an external returner.
  • Thomas talked briefly about ioflo which should bring queuing, data hierarchy and data pub-sub to Salt.
http://www.logilab.org/file/248335/raw/ioflo.png
  • About the rolling release question: versions in Salt are definitely not git snapshots, things get backported into previous versions. No clear definition yet of length of LTS versions.
  • salt-cloud and libcloud : in the next release, libcloud will not be a hard dependency. Some clouds didn't work in libcloud (for example AWS), so these providers got implemented directly in salt-cloud or by using third-party libraries (eg. python-boto).
  • Documentation: a sprint is planned next week. Reference documentation will not be completly revamped, but tutorial content will be added.

Boris Feld showed a demo of vagrant images orchestrated by salt and a web UI to monitor a salt install.

http://www.vagrantup.com/images/logo_vagrant-81478652.png

Thanks again to Thomas Hatch for coming and meeting up with (part of) the community here in France.


Compte rendu présentation Salt à Solution Linux

2014/05/21 by Arthur Lutz

Logilab était à l'édition 2014 de Solutions Linux qui se déroulait au CNIT à Paris. David Douard participait à la table ronde sur les outils libres pour la supervision lors de la session Administration Système, Devops, au cours de laquelle un certain nombre de projets libres ont été mentionnés : nagios, shinken, graphite, ElasticSearch, logstash, munin, saltstack, kibana, centreon, rsyslog.

http://www.logilab.org/file/248048/raw/solutionlinux.png

Suite à des présentations sur OpenLDAP, LXC, btrfs et ElasticSearch David Douard a présenté notre approche agile de l'administration système articulée autour de Salt et en particulier le principe de l'administration système pilotée par les tests (diapos) (Test-Driven Infrastructure).

https://www.logilab.org/file/248098/raw/Screenshot%20from%202014-05-21%2017%3A55%3A35.png

Merci aux organisateurs de Solutions Linux pour cette édition 2014.


Salt April Meetup in Paris (France)

2014/05/14 by Arthur Lutz

On the 15th of april, in Paris (France), we took part in yet another Salt meetup. The community is now meeting up once every two months.

We had two presentations:

  • Arthur Lutz made an introduction to returners and the scheduler using the SalMon monitoring system as an example. Salt is not only about configuration management Indeed!
  • The folks from Is Cool Entertainment did a presentation about how they are using salt-cloud to deploy and orchestrate clusters of EC2 machines (islands in their jargon) to reproduce parts of their production environment for testing and developement.

More discussions about various salty subjects followed and were pursued in an Italian restaurant (photos here).

In case it is not already in your diary : Thomas Hatch is coming to Paris next week, on Monday the 19th of May, and will be speaking at dotscale during the day and at a Salt meetup in the evening. The Salt Meetup will take place at IRILL (like the previous meetups, thanks again to them) and should start at 19h. The meetup is free and open to the public, but registering on this framadate would be appreciated.


Quelques pointeurs présentés lors d'un atelier sur le web sémantique à Nantes

2014/05/14 by Arthur Lutz

À l'appel du DataLab Pays de la Loire, nous avons co-animé (avec Hala Skaf-Molli) un atelier sur le web sémantique à la Cantine Numérique de Nantes.

Voici quelques diapos avec essentiellement des pointeurs pour donner des exemples de réalisations web sémantique mais aussi pour appuyer les concepts présentés. Vous trouverez les diapos de Hala Skaf sur sa page web (dans les prochains jours).

Si vous avez raté cette session et êtes intéressé par le sujet, n'hésitez pas à le faire savoir au DataLab.

http://www.datalab-paysdelaloire.org/auth/public/images/datalab.png

Pylint 1.2 released!

2014/04/22 by Sylvain Thenault

Once again, a lot of work has been achieved since the latest 1.1 release. Claudiu, who joined the maintainer team (Torsten and me) did a great work in the past few months. Also lately Torsten has backported a lot of things from their internal G[oogle]Pylint. Last but not least, various people contributed by reporting issues and proposing pull requests. So thanks to everybody!

Notice Pylint 1.2 depends on astroid 1.1 which has been released at the same time. Currently, code is available on Pypi, and Debian/Ubuntu packages should be ready shortly on Logilab's acceptance repositories.

Below is the changes summary, check the changelog for more info.

New and improved checks:

  • New message 'eval-used' checking that the builtin function eval was used.
  • New message 'bad-reversed-sequence' checking that the reversed builtin receive a sequence (i.e. something that implements __getitem__ and __len__, without being a dict or a dict subclass) or an instance which implements __reversed__.
  • New message 'bad-exception-context' checking that raise ... from ... uses a proper exception context (None or an exception).
  • New message 'abstract-class-instantiated' warning when abstract classes created with abc module and with abstract methods are instantied.
  • New messages checking for proper class __slots__: 'invalid-slots-object' and 'invalid-slots'.
  • New message 'undefined-all-variable' if a package's __all__ variable contains a missing submodule (#126).
  • New option logging-modules giving the list of module names that can be checked for 'logging-not-lazy'.
  • New option include-naming-hint to show a naming hint for invalid name (#138).
  • Mark file as a bad function when using python2 (#8).
  • Add support for enforcing multiple, but consistent name styles for different name types inside a single module.
  • Warn about empty docstrings on overridden methods.
  • Inspect arguments given to constructor calls, and emit relevant warnings.
  • Extend the number of cases in which logging calls are detected (#182).
  • Enhance the check for 'used-before-assignment' to look for nonlocal uses.
  • Improve cyclic import detection in the case of packages.

Bug fixes:

  • Do not warn about 'return-arg-in-generator' in Python 3.3+.
  • Do not warn about 'abstract-method' when the abstract method is implemented through assignment (#155).
  • Do not register most of the 'newstyle' checker warnings with python >= 3.
  • Fix 'unused-import' false positive with augment assignment (#78).
  • Fix 'access-member-before-definition' false negative with augment assign (#164).
  • Do not crash when looking for 'used-before-assignment' in context manager assignments (#128).
  • Do not attempt to analyze non python file, eg '.so' file (#122).
  • Pass the current python path to pylint process when invoked via epylint (#133).

Command line:

  • Add -i / --include-ids and -s / --symbols back as completely ignored options (#180).
  • Ensure init-hooks is evaluated before other options, notably load-plugins (#166).

Other:

  • Improve pragma handling to not detect 'pylint:*' strings in non-comments (#79).
  • Do not crash with UnknownMessage if an unknown message identifier/name appears in disable or enable in the configuration (#170).
  • Search for rc file in ~/.config/pylintrc if ~/.pylintrc doesn't exists (#121).
  • Python 2.5 support restored (#50 and #62).

Astroid:

  • Python 3.4 support
  • Enhanced support for metaclass
  • Enhanced namedtuple support

Nice easter egg, no?