Blog entries

LAX - Logilab Google AppEngin Sprint at Pycon-FR

2008/05/20 by Arthur Lutz

Here are a few pictures from the sprint we organized at Pycon-FR

We got a few people to install Google AppEngine and LAX on their machines, and explained the concepts at hand to a bunch of other people.

http://www.logilab.org/image/5002?vid=download

http://www.logilab.org/image/5003?vid=download

http://www.logilab.org/image/5005?vid=download

Update: LAX is now included in the CubicWeb semantic web framework.


New pylint/astng release, but... pylint needs you !

2009/08/27 by Sylvain Thenault

After several months with no time to fix/enhance pylint beside answering email and filing tickets, I've finally tackled some tasks yesterday night to publish bug fixes releases ([1] and [2]).

The problem is that we don't have enough free time at Logilab to lower the number of tickets in pylint tracker page . If you take a look at the ticket tab, you'll see a lot of pendings bug and must-have features (well, and some other less necessary...). You can already easily contribute thanks to the great mercurial dvcs, and some of you do, either by providing patches or by reporting bugs (more tickets, iiirk ! ;) Thank you all btw !!

Now I was wondering what could be done to make pylint going further, and the first ideas which came to my mind was :

  • do ~3 days sprint
  • do some 'tickets killing' days, as done in some popular oss projects

But for this to be useful, we need your support, so here are some questions for you:

  • would you come to a sprint at Logilab (in Paris, France), so you can meet us, learn a lot about pylint, and work on tickets you wish to have in pylint?
  • if France is too far away for most people, would you have another location to propose?
  • would you be on jabber for a tickets killing day, providing it's ok with your agenda? if so, what's your knowledge of pylint/astng internals?

you may answer by adding a comment to this blog (please register first by using the link at the top right of this page) or by mail to sylvain.thenault@logilab.fr. If we've enough positive answers, we'll take the time to organize such a thing.


Pylint a besoin de vous

2009/09/17

Après plusieurs mois au point mort ou presque, Sylvain a pu hier soir publier des versions corrigeant un certain nombre de bogues dans pylint et astng ([1] et [2]).

Il n'en demeure pas moins qu'à Logilab, nous manquons de temps pour faire baisser la pile de tickets ouverts dans le tracker de pylint. Si vous jetez un œuil dans l'onglet Tickets, vous y trouverez un grand nombre de bogues en souffrance et de fonctionalités indispensables (certaines peut-être un peu moins que d'autres...) Il est déjà possible de contribuer en utilisant mercurial pour fournir des patches, ou en signalant des bogues (aaaaaaaaaarg ! encore des tickets !) et certains s'y sont mis, qu'ils en soient remerciés.

Maintenant, nous nous demandions ce que nous pourrions faire pour faire avance Pylint, et nos premières idées sont :

  • organiser un petit sprint de 3 jours environ
  • organiser des jours de "tuage de ticket", comme ça se pratique sur différents projets OSS

Mais pour que ça soit utile, nous avons besoin de votre aide. Voici donc quelques questions :

  • est-ce que vous participeriez à un sprint à Logilab (à Paris, France), ce qui nous permettrait de nous rencontrer, de vous apprendre plein de choses sur le fonctionnement de Pylint et de travailler ensemble sur des tickets qui vous aideraient dans votre travail ?
  • si la France c'est trop loin, où est-ce que ça vous arrangerait ?
  • seriez-vous prêt à vous joindre à nous sur le serveur jabber de Logilab ou sur IRC, pour participer à une chasse au ticket (à une date à déterminer). Si oui, quel est votre degré de connaissance du fonctionnement interne de Pylint et astng ?

Vous pouvez répondre en commentant sur ce blog (pensez à vous enregistrer en utilisant le lien en haut à droite sur cette page) ou en écrivant à sylvain.thenault@logilab.fr. Si nous avons suffisamment de réponses positives nous organiserons quelque chose.


pylint bug day report

2009/12/04 by Pierre-Yves David
http://farm1.static.flickr.com/85/243306920_6a12bb48c7.jpg

The first pylint bug day took place on wednesday 25th. Four members of the Logilab crew and two other people spent the day working on pylint.

Several patches submitted before the bug day were processed and some tickets were closed.

Charles Hébert added James Lingard's patches for string formatting and is working on several improvements. Vincent Férotin submitted a patch for simple message listings. Sylvain Thenault fixed significant inference bugs in astng (an underlying module of pylint managing the syntax tree). Émile Anclin began a major astng refactoring to take advantage of new python2.6 functionality. For my part, I made several improvements to the test suite. I applied James Lingard patches for ++ operator and generalised it to -- too. I also added a new checker for function call arguments submitted by James Lingard once again. Finally I improved the message filtering of the --errors-only options.

We thank Maarten ter Huurne, Vincent Férotin for their participation and of course James Lingard for submitting numerous patches.

Another pylint bug day will be held in a few months.

image under creative commons by smccann


We're happy to host the mercurial Sprint

2010/02/02 by Arthur Lutz
http://farm1.static.flickr.com/183/419945378_4ead41a76d_m.jpg

We're very happy to be hosting the next mercurial sprint in our brand new offices in central Paris. It is quite an honor to be chosen when the other contender was Google.

So a bunch of mercurial developers are heading out to our offices this coming Friday to sprint for three days on mercurial. We use mercurial a lot here over at Logilab and we also contribute a tool to visualize and manipulate a mercurial repository : hgview.

To check out the things that we will be working on with the mercurial crew, check out the program of the sprint on their wiki.

What is a sprint? "A sprint (sometimes called a Code Jam or hack-a-thon) is a short time period (three to five days) during which software developers work on a particular chunk of functionality. "The whole idea is to have a focused group of people make progress by the end of the week," explains Jeff Whatcott" [source]. For geographically distributed open source communities, it is also a way of physically meeting and working in the same room for a period of time.

Sprinting is a practice that we encourage at Logilab, with CubicWeb we organize as often as possible open sprints, which is an opportunity for users and developers to come and code with us. We even use the sprint format for some internal stuff.

photo by Sebastian Mary under creative commons licence.


pylint bugs day #2 on april 16, 2010

2010/03/22 by Sylvain Thenault

Hey guys,

we'll hold the next pylint bugs day on april 16th 2010 (friday). If some of you want to come and work with us in our Paris office, you'll be much welcome.

Else you can still join us on jabber / irc:

See you then!


pylint bug days #2 report

2010/04/19 by Sylvain Thenault

First of all, I've to say that pylint bugs day wasn't that successful in term of 'community event': I've been sprinting almost alone. My Logilab's felows were tied to customer projects, and no outside people shown up on jabber. Fortunatly Tarek Ziade came to visit us, and that was a nice opportunity to talk about pylint, distribute, etc ... Thank you Tarek, you saved my day ;)

As I felt a bit alone, I decided to work on somethings funnier than bug fixing: refactoring!

First, I've greatly simplified the command line: enable-msg/enable-msg-cat/enable-checker/enable-report and their disable-* counterparts were all merged into single --enable/--disable options.

I've also simplified "pylint --help" output, providing a --long-help option to get what we had before. Generic support in `logilab.common.configuration of course.

And last but not least, I refactored pylint so we can have multiple checkers with the same name. The idea behind this is that we can split checker into smaller chunks, basically only responsible for one or a few related messages. When pylint runs, it only uses necessary checkers according to activated messages and reports. When all checkers will be splitted, it should improve performance of "pylint --error-only".

So, I can say I'm finally happy with the results of that pylint bugs day! And hopefuly we will be more people for the next edition...


Sprint CubicWeb chez Logilab - annonce de dernière minute

2010/04/29 by Arthur Lutz

Logilab est en ce moment en train d'acceuillir un sprint autour de la plateforme CubicWeb. L'objectif principal de ce sprint de 5 jours est d'améliorer l'usage de javascript et des css dans CubicWeb :

http://www.logilab.org/image/28586?vid=downloadhttp://codesnip.net/wp-content/uploads/javascript.png
  • avoir une API javascript propre, testée et documentée
  • pouvoir facilement changer le style d'une application cubicweb
  • gestion de bundle pour javascript et CSS
  • une documentation sur les standards d'écriture des fichiers JS et CSS pour cubicweb

Ce sprint aura lieu du jeudi 29 avril 2010 au 5 mai 2010 (weekend exlus - les bureaux seront fermés). Vous êtes les bienvenus pour contribuer, filer un coup de main, faire du pair programming... ou simplement rencontrer des développeurs cubicweb. Vous pouvez même venir une après-midi ou une seule journée. Pour ceux avec des portables, il y aura du réseau disponible pour être connecté.

Adresse : 104 Boulevard Auguste-Blanqui, Paris. Sonnez à "Logilab".

Métro : St Jacques or Corvisart (Glacière est la station la plus proche mais sera fermée à partir de lundi)

Contact : http://www.logilab.fr/contact

Dates : du 29/04/2010 au 30/04/2010 et du 03/05/2010 au 05/05/2010


Distutils2 January Sprint in Paris

2011/01/07 by Pierre-Yves David

At Logilab, we have the pleasure to host a distutils2 sprint in January. Sprinters are welcome in our Paris office from 9h on the 27th of January to 19h the 30th of January. This sprint will focus on polishing distutils2 for the next alpha release and on the install/remove scripts.

Distutils2 is an important project for Python. Every contribution will help to improve the current state of packaging in Python. See the wiki page on python.org for details about participation. If you can't attend or join us in Paris, you can participate on the #distutils channel of the freenode irc network

http://guide.python-distribute.org/_images/state_of_packaging.jpg

For additional details, see Tarek Ziadé's original announce, read the wiki page on python.org or contact us


Distutils2 Sprint at Logilab (first day)

2011/01/28 by Alain Leufroy

We're very happy to host the Distutils2 sprint this week in Paris.

The sprint has started yesterday with some of Logilab's developers and others contributors. We'll sprint during 4 days, trying to pull up the new python package manager.

Let's sumarize this first day:

  • Boris Feld and Pierre-Yves David worked on the new system for detecting and dispatching data-files.
  • Julien Miotte worked on
    • moving qGitFilterBranch from setuptools to distutils2
    • testing distutils2 installation and register (see the tutorial)
    • the backward compatibility to distutils in setup.py, using setup.cfg to fill the setup arguments of setup for helping users to switch to distutils2.
  • André Espaze and Alain Leufroy worked on the python script that help developers build a setup.cfg by recycling their existing setup.py (track).

Join us on IRC at #distutils on irc.freenode.net !


pylint bug day #3 on july 8, 2011

2011/07/04 by Sylvain Thenault

Hey guys,

we'll hold the next pylint bug day on july 8th 2011 (friday). If some of you want to come and work with us in our Paris office, you'll be welcome.

You can also join us on jabber / irc:

I know the announce is a bit late, but I hope some of you will be able to come or be online anyway!

Regarding the program, the goal is to decrease the number of tickets in the tracker. I'll try to do some triage earlier this week so you'll get a chance to talk about your super-important ticket that has not been selected. Of course, if you intend to work on it, there is a bigger chance of it being fixed next week-end ;)


Mercurial 2.3 day 0

2012/05/10 by Pierre-Yves David

I'm now at Copenhagen to attend the mercurial "2.3" sprint.

About twenty people are attending, including staff from Atlassian, Facebook, Google and Mozilla.

I expect code and discussion about various topic among:

  • the development process of mercurial itself,
  • performance improvement on huge repository,
  • integration of Obsolete Markers into mercurial core,
  • improvement on various aspect (merge diff, moving some extension in core, ...)

I'm of course very interested in the Obsolete Markers topic. I've been working on an experimental implementation for several months. An handful of people are using them at Logilab for two months and feedback are very promising.


Mercurial 2.3 sprint, Day 1-2-3

2012/05/15 by Pierre-Yves David

I'm now back from Copenhagen were I attended the mercurial 2.3 sprint with twenty other people. A huge amount of work was done in a very friendly atmosphere.

Regarding mercurial's core:

  • Bookmark behaviour was improved to get closer to named branch's behaviour.
  • Several performance improvements regarding branches and heads caches. The heads cache refactoring improves rebase performance on huge repository (thanks to Facebook and Atlassian).
  • The concept I'm working on, Obsolete markers, was a highly discussed subject and is expected to get partly into the core in the near future. Thanks to my employer Logilab for paying me to work on this topic.
  • General code cleanup and lock validation.
http://www.logilab.org/file/92956?vid=download

Regarding the bundled extension :

  • Some fixes where made to progress which is now closer to getting into mercurial's core.
  • Histedit and keyring extensions are scheduled to be shipped with mercurial.
  • Some old and unmaintained extensions (children, hgtk) are now deprecated.
  • The LargeFile extension got some new features (thanks to the folks from Unity3D)
  • Rebase will use the --detach flag by default in the next release.
http://www.logilab.org/file/92958?vid=download

Regarding the project itself:

http://www.logilab.org/file/92955?vid=download

Regarding other extensions:

http://www.logilab.org/file/92959?vid=download

And I'm probably forgetting some stuff. Special thanks to Unity3D for hosting the sprint and providing power, network and food during these 3 days.


Debian science sprint and workshop at ESRF

2012/06/22 by Julien Cristau

esrfdebian

From June 24th to June 26th, the European Synchrotron organises a workshop centered around Debian. On Monday, a number of talks about the use of Debian in scientific facilities will be featured. On Sunday and Tuesday, members of the Debian Science group will meet for a sprint focusing on the upcoming Debian 7.0 release.

Among the speakers will be Stefano Zacchiroli, the current Debian project leader. Logilab will be present with Nicolas Chauvat at Monday's conference, and Julien Cristau at both the sprint and the conference.

At the sprint we'll be discussing packaging of scientific libraries such as blas or MPI implementations, and working on polishing other scientific packages, such as python-related ones (including Salome on which we are currently working).


Sprint PyLint @ PyConFr 2012

2012/08/20 by Sylvain Thenault

Un sprint PyLint est organisé dans le cadre de la conférence PyConFR, les 13 et 14 septembre à Paris. Si vous voulez améliorer PyLint, c'est l'occasion : venez avec vos bugs et repartez sans !

Les débutants sont bienvenus, une introduction au code de Pylint sera réalisée en début de sprint. Une expérience avec le module ast de la librairie standard est un plus.

Croissants et café offerts par l'organisation, merci de vous inscrire pour faciliter la logistique. Voir avec Boris pour plus d'informations (merci à lui !)


Febuary 2013: Mercurial channel "tour"

2013/01/22 by Pierre-Yves David

The Release candidate version of Mercurial 2.5 was released last sunday.

http://mercurial.selenic.com/images/mercurial-logo.png

This new version makes a major change in the way "hidden" changesets are handled. In 2.4 only hg log (and a few others) would support effectively hiding "hidden" changesets. Now all hg commands are transparently compatible with the hidden revision concept. This is a considerable step towards changeset evolution, the next-generation collaboration technology that I'm developing for Mercurial.

https://fosdem.org/2013/assets/flyer-thumb-0505d19dbf3cf6139bc7490525310f8e253e60448a29ed4313801b723d5b2ef1.png

The 2.5 cycle is almost over, but there is no time to rest yet, Saturday the 2th of February, I will give a talk about changeset evolution concept at FOSDEM in the Mozilla Room. This talk in an updated version of the one I gave at OSDC.fr 2012 (video in french).

The week after, I'm crossing the channel to attend the Mercurial 2.6 Sprint hosted by Facebook London. I expect a lot of discussion about the user interface and network access of changeset evolution.

The HG 2.3 sprint

PyLint 10th years anniversary, 1.0 sprint

2013/03/29 by Sylvain Thenault

In a few week, pylint will be 10 years old (0.1 released on may 19 2003!). At this occasion, I would like to release a 1.0. Well, not exactly at that date, but not too long after would be great. Also, I think it would be a good time to have a few days sprint to work a bit on this 1.0 but also to meet all together and talk about pylint status and future, as more and more contributions come from outside Logilab (actually mostly Google, which employs Torsten and Martin, the most active contributors recently).

The first thing to do is to decide a date and place. Having discussed a bit with Torsten about that, it seems reasonable to target a sprint during june or july. Due to personal constraints, I would like to host this sprint in Logilab's Toulouse office.

So, who would like to jump in and sprint to make pylint even better? I've created a doodle so every one interested may tell his preferences: http://doodle.com/4uhk26zryis5x7as

Regarding the location, is everybody ok with Toulouse? Other ideas are Paris, or Florence around EuroPython, or... <add your proposition here>.

We'll talk about the sprint topics later, but there are plenty of exciting ideas around there.

Please, answer quickly so we can move on. And I hope to see you all there!


LMGC90 Sprint at Logilab in March 2013

2013/03/28 by Vladimir Popescu

LMGC90 Sprint at Logilab

At the end of March 2013, Logilab hosted a sprint on the LMGC90 simulation code in Paris.

LMGC90 is an open-source software developed at the LMGC ("Laboratoire de Mécanique et Génie Civil" -- "Mechanics and Civil Engineering Laboratory") of the CNRS, in Montpellier, France. LMGC90 is devoted to contact mechanics and is, thus, able to model large collections of deformable or undeformable physical objects of various shapes, with numerous interaction laws. LMGC90 also allows for multiphysics coupling.

Sprint Participants

https://www.logilab.org/file/143585/raw/logo_LMGC.jpghttps://www.logilab.org/file/143749/raw/logo_SNCF.jpghttps://www.logilab.org/file/143750/raw/logo_LaMSID.jpghttps://www.logilab.org/file/143751/raw/logo_LOGILAB.jpg

More than ten hackers joined in from:

  • the LMGC, which leads LMCG90 development and aims at constantly improving its architecture and usability;
  • the Innovation and Research Department of the SNCF (the French state-owned railway company), which uses LMGC90 to study railway mechanics, and more specifically, the ballast;
  • the LaMSID ("Laboratoire de Mécanique des Structures Industrielles Durables", "Laboratory for the Mechanics of Ageing Industrial Structures") laboratory of the EDF / CNRS / CEA , which has an strong expertise on Code_ASTER and LMGC90;
  • Logilab, as the developer, for the SNCF, of a CubicWeb-based platform dedicated to the simulation data and knowledge management.

After a great introduction to LMGC90 by Frédéric Dubois and some preliminary discussions, teams were quickly constituted around the common areas of interest.

Enhancing LMGC90's Python API to build core objects

As of the sprint date, LMGC90 is mainly developed in Fortran, but also contains Python code for two purposes:

  • Exposing the Fortran functions and subroutines in the LMGC90 core to Python; this is achieved using Fortran 2003's ISO_C_BINDING module and Swig. These Python bindings are grouped in a module called ChiPy.
  • Making it easy to generate input data (so called "DATBOX" files) using Python. This is done through a module called Pre_LMGC.

The main drawback of this approach is the double modelling of data that this architecture implies: once in the core and once in Pre_LMGC.

It was decided to build a unique user-level Python layer on top of ChiPy, that would be able to build the computational problem description and write the DATBOX input files (currently achieved by using Pre_LMGC), as well as to drive the simulation and read the OUTBOX result files (currently by using direct ChiPy calls).

This task has been met with success, since, in the short time span available (half a day, basically), the team managed to build some object types using ChiPy calls and save them into a DATBOX.

Using the Python API to feed a computation data store

This topic involved importing LMGC90 DATBOX data into the numerical platform developed by Logilab for the SNCF.

This was achieved using ChiPy as a Python API to the Fortran core to get:

  • the bodies involved in the computation, along with their materials, behaviour laws (with their associated parameters), geometries (expressed in terms of zones);
  • the interactions between these bodies, along with their interaction laws (and associated parameters, e.g. friction coefficient) and body pair (each interaction is defined between two bodies);
  • the interaction groups, which contain interactions that have the same interaction law.

There is still a lot of work to be done (notably regarding the charges applied to the bodies), but this is already a great achievement. This could only have occured in a sprint, were every needed expertise is available:

  • the SNCF experts were there to clarify the import needs and check the overall direction;

  • Logilab implemented a data model based on CubicWeb, and imported the data using the ChiPy bindings developed on-demand by the LMGC core developer team, using the usual-for-them ISO_C_BINDING/ Swig Fortran wrapping dance.

    https://www.logilab.org/file/143753/raw/logo_CubicWeb.jpg
  • Logilab undertook the data import; to this end, it asked the LMGC how the relevant information from LMGC90 can be exposed to Python via the ChiPy API.

Using HDF5 as a data storage backend for LMGC90

The main point of this topic was to replace the in-house DATBOX/OUTBOX textual format used by LMGC90 to store input and output data, with an open, standard and efficient format.

Several formats have been considered, like HDF5, MED and NetCDF4.

MED has been ruled out for the moment, because it lacks the support for storing body contact information. HDF5 was chosen at last because of the quality of its Python libraries, h5py and pytables, and the ease of use tools like h5fs provide.

https://www.logilab.org/file/143754/raw/logo_HDF.jpg

Alain Leufroy from Logilab quickly presented h5py and h5fs usage, and the team started its work, measuring the performance impact of the storage pattern of LMGC90 data. This was quickly achieved, as the LMGC experts made it easy to setup tests of various sizes, and as the Logilab developers managed to understand the concepts and implement the required code in a fast and agile way.

Debian / Ubuntu Packaging of LMGC90

This topic turned out to be more difficult than initially assessed, mainly because LMGC90 has dependencies to non-packaged external libraries, which thus had to be packaged first:

  • the Matlib linear algebra library, written in C,
  • the Lapack95 library, which is a Fortran95 interface to the Lapack library.

Logilab kept working on this after the sprint and produced packages that are currently being tested by the LMGC team. Some changes are expected (for instance, Python modules should be prefixed with a proper namespace) before the packages can be submitted for inclusion into Debian. The expertise of Logilab regarding Debian packaging was of great help for this task. This will hopefully help to spread the use of LMGC90.

https://www.logilab.org/file/143755/raw/logo_Debian.jpg

Distributed Version Control System for LMGC90

As you may know, Logilab is really fond of Mercurial as a DVCS. Our company invested a lot into the development of the great evolve extension, which makes Mercurial a very powerful tool to efficiently manage the team development of software in a clean fashion.

This is why Logilab presented Mercurial's features and advantages over the current VCS used to manage LMGC90 sources, namely svn, to the other participants of the Sprint. This was appreciated and will hopefully benefit to LMGC90 ease of development and spread among the Open Source community.

https://www.logilab.org/file/143756/raw/logo_HG.jpg

Conclusions

All in all, this two-day sprint on LMGC90, involving participants from several industrial and academic institutions has been a great success. A lot of code has been written but, more importantly, several stepping stones have been laid, such as:

  • the general LMGC90 data access architecture, with the Python layer on top of the LMGC90 core;
  • the data storage format, namely HDF5.

Colaterally somehow, several other results have also been achieved:

  • partial LMGC90 data import into the SNCF CubicWeb-based numerical platform,
  • Debian / Ubuntu packaging of LMGC90 and dependencies.

On a final note, one would say that we greatly appreciated the cooperation between the participants, which we found pleasant and efficient. We look forward to finding more occasions to work together.


Pylint 10th years anniversary from June 17 to 19 in Toulouse

2013/04/18 by Sylvain Thenault

After a quick survey, we're officially scheduling Pylint 10th years anniversary sprint from monday, June 17 to wednesday, June 19 in Logilab's Toulouse office.

There is still some room available if more people want to come, drop me a note (sylvain dot thenault at logilab dot fr).


PyLint 10th anniversary 1.0 sprint: day 1

2013/06/17 by Sylvain Thenault

Today was the first day of the Pylint sprint we organized using Pylint's 10th years anniversary as an excuse.

So I (Sylvain) have welcome my fellow Logilab friends David, Anthony and Julien as well as Torsten from Google into Logilab's new Toulouse office.

After a bit of presentation and talk about Pylint development, we decided to keep discussion for lunch and dinner and to setup priorities. We ended with the following tasks (picks from the pad at http://piratepad.net/oAvsUoGCAC):

  • rename astng to move it outside the logilab package,
  • Torsten gpylint (Google Pylint) patches review, as much as possible (but not all of them, starting by a review of the numberous internal checks Google has, seeing one by one which one should be backported upstream),
  • setuptools namespace package support (https://www.logilab.org/8796),
  • python 3.3 support,
  • enhance astroid (former astng) API to allow more ad-hoc customization for a better grasp of magic occuring in e.g. web frameworks (protocol buffer or SQLAlchemy may also be an application of this).

Regarding the astng renaming, we decided to move on with astroid as pointed out by the survey on StellarSurvey.com

In the afternoon, David and Julien tackled this, while Torsten was extracting patches from Google code and sending them to bitbucket as pulll request, Sylvain embrassing setuptools namespaces packages and Anthony discovering the code to spread the @check_message decorator usage.

By the end of the day:

  • David and Julien submitted patches to rename logilab.astng which were quickly integrated and now https://bitbucket.org/logilab/astroid should be used instead of https://bitbucket.org/logilab/astng
  • Torsten submitted 5 pull-requests with code extracted from gpylint, we reviewed them together and then Torsten used evolve to properly insert those in the pylint history once review comments were integrated
  • Sylvain submitted 2 patches on logilab-common to support both setuptools namespace packages and pkgutil.extend_path (but not bare __path__ manipulation
  • Anthony discovered various checkers and started adding proper @check_messages on visit methods

After doing some review all together, we even had some time to take a look at Python 3.3 support while writing this summary.

Hopefuly, our work on forthcoming days will be as efficient as on this first day!


PyLint 10th anniversary 1.0 sprint: day 2

2013/06/18 by Sylvain Thenault

Today was the second day of the 10th anniversary Pylint sprint in Logilab's Toulouse office.

This morning, we started with a presentation by myself about how the inference engine works in astroid (former astng). Then we started thinking all together about how we should change its API to be able to plug more information during the inference process. The first use-case we wanted to assert was namedtuple, as explained in http://www.logilab.org/ticket/8796.

We ended up by addressing it by:

  • enhancing the existing transformation feature so one may register a transformation function on any node rather than on a module node only;
  • being able to specify, on a node instance, a custom inference function to use instead of the default (class) implementation.

We would then be able to customize both the tree structure and the inference process and so to resolve the cases we were targeting.

Once this was sufficiently sketched out, everyone got his own tasks to do. Here is a quick summary of what has been achieved today:

  • Anthony resumed the check_messages thing and finished it for the simple cases, then he started on having a template for text reported,
  • Julien and David made a lot of progress on the Python 3.3 compatibility, though not enough to get the full green test suite,
  • Torsten continued backporting stuff from gpylint, all of them having been integrated by the end of the day,
  • Sylvain implemented the new transformation API and had the namedtuple proof of concept working, and even some documentation! Now this have to be tested for more real-world uses.

So things are going really well, and see you tomorrow for even more improvements to pylint!


PyLint 10th anniversary 1.0 sprint: day 3 - Sprint summary

2013/06/20 by Sylvain Thenault

Yesterday was the third and last day of the 10th anniversary Pylint sprint in Logilab's Toulouse office.

Design

To get started, we took advantage of this last day to have a few discussions about:

  • A "mode" feature gpylint has. It turns out that behind perhaps a few implementation details, this is something we definitly want into pylint (mode are specific configurations defined in the pylintrc and easilly recallable, they may even be specified per file).

  • How to avoid conflicts in the ChangeLog by using specific instruction in the commit message. We decided that a commit message should look like

    [my checker] do this and that. Closes #1234
    
    bla bla bla
    
    :release note: this will be a new item in the ChangeLog
    
    as well as anything until the end of the message
    

    now someone has to write the ChangeLog generation script so we may use this for post-1.0 releases

  • The roadmap. More on this later in this post.

Code

When we were not discussing, we were coding!

  • Anthony worked on having a template for the text reporter. His patch is available on Bitbucket but not yet integrated.
  • Julien and David pushed a bunch of patches on logilab-common, astroid and pylint for the Python 3.3 support. Not all tests are green on the pylint side, but much progress was done.
  • A couple other things were fixed, like a better "invalid name" message, stop complaining about string module being deprecated, etc.
  • A lot of patches have been integrated, from gpylint and others (e.g python 3 related)

All in all, an impressive amount of work was achieved during this sprint:

  • A lot of new checks or enhanced behaviour backported from gpylint (Take a look at Pylint's ChangeLog for more details on this, the list is impressively long).
  • The transformation API of astroid now allows to customize the tree structure as well as the inference process, hence to make pylint smarter than ever.
  • Better python 3 support.
  • A few bugs fixed and some enhancements added.
  • The templating stuff should land with the CLI cleanup (some output-formats will be removed as well as the --include-ids and --symbols option).
  • A lot of discussions, especially regarding the future community development of pylint/astroid on Bitbucket. Short summary being: more contributors and integrators are welcome! We should drop some note somewhere to describe how we are using bitbucket's pull requests and tracker.

Plan

Now here is the 1.O roadmap, which is expected by the begining of July:

  • Green tests under Python 3, including specification of Python version in message description (Julien).
  • Finish template for text reporters (Anthony).
  • Update web site (David).

And for later releases:

  • Backport mode from gpylint (Torsten).
  • Write ChangeLog update script (Sylvain).

So many thanks to everyone for this very successful sprint. I'm excited about this forthcoming 1.0 release!


The Great Salt Sprint Paris Location is Logilab

2013/07/12 by Nicolas Chauvat
http://farm1.static.flickr.com/183/419945378_4ead41a76d_m.jpg

We're happy to be part of the second Great Salt Sprint that will be held at the end of July 2013. We will be hosting the french sprinters on friday 26th in our offices in the center of Paris.

The focus of our Logilab team will probably be Test-Driven System Administration with Salt, but the more participants and the topics, the merrier the event.

Please register if you plan on joining us. We will be happy to meet with fellow hackers.

photo by Sebastian Mary under creative commons licence.


We hosted the Salt Sprint in Paris

2013/07/30 by Arthur Lutz

Last Friday, we hosted the French event for the international Great Salt Sprint. Here is a report on what was done and discussed on this occasion.

http://saltstack.com/images/SaltStack-Logo.png

We started off by discussing various points that were of interest to the participants :

  • automatically write documentation from salt sls files (for Sphinx)
  • salt-mine add security layer with restricted access (bug #5467 and #6437)
  • test compatibility of salt-cloud with openstack
  • module bridge bug correction : traceback on KeyError
  • setting up the network in debian (equivalent of rh_ip)
  • configure existing monitoring solution through salt (add machines, add checks, etc) on various backends with a common syntax

We then split up into pairs to tackle issues in small groups, with some general discussions from time to time.

6 people participated, 5 from Logilab, 1 from nbs-system. We were expecting more participants but some couldn't make it at the last minute, or though the sprint was taking place at some other time.

Unfortunately we had a major electricity black out all afternoon, some of us switched to battery and 3G tethering to carry on, but that couldn't last all afternoon. We ended up talking about design and use cases. ERDF (French electricity distribution company) ended up bringing generator trucks for the neighborhood !

Arthur & Benoit : monitoring adding machines or checks

http://www.logilab.org/file/157971/raw/salt-centreon-shinken.png

Some unfinished draft code for supervision backends was written and pushed on github. We explored how a common "interface" could be done in salt (using a combination of states and __virtual___). The official documentation was often very useful, reading code was also always a good resource (and the code is really readable).

While we were fixing stuff because of the power black out, Benoit submitted a bug fix.

David & Alain : generate documentation from salt state & salt master

The idea is to couple the SLS description and the current state of the salt master to generate documentation about one's infrastructure using Sphinx. This was transmitted to the mailing-list.

http://www.logilab.org/file/157976/raw/salt-sphinx.png

Design was done around which information should be extracted and display and how to configure access control to the salt-master, taking a further look to external_auth and salt-api will probably be the way forward.

General discussions

We had general discussions around concepts of access control to a salt master, on how to define this access. One of the things we believe to be missing (but haven't checked thoroughly) is the ability to separate the "read-only" operations to the "read-write" operations in states and modules, if this was done (through decorators?) we could easily tell salt-api to only give access to data collection. Complex scenarios of access were discussed. Having a configuration or external_auth based on ssh public keys (similar to mercurial-server would be nice, and would provide a "limited" shell to a mercurial server.

Conclusion

The power black out didn't help us get things done, but nevertheless, some sharing was done around our uses cases around SaltStack and features that we'd want to get out of it (or from third party applications). We hope to convert all the discussions into bug reports or further discussion on the mailing-lists and (obviously) into code and pull-requests. Check out the scoreboard for an overview of how the other cities contributed.

to comment this post you need to login or create an account


Going to EuroScipy2013

2013/09/04 by Alain Leufroy

The EuroScipy2013 conference was held in Bruxelles at the Université libre de Bruxelles.

http://www.logilab.org/file/175984/raw/logo-807286783.png

As usual the first two days were dedicated to tutorials while the last two ones were dedicated to scientific presentations and general python related talks. The meeting was extended by one more day for sprint sessions during which enthusiasts were able to help free software projects, namely sage, vispy and scipy.

Jérôme and I had the great opportunity to represent Logilab during the scientific tracks and the sprint day. We enjoyed many talks about scientific applications using python. We're not going to describe the whole conference. Visit the conference website if you want the complete list of talks. In this article we will try to focus on the ones we found the most interesting.

First of all the keynote by Cameron Neylon about Network ready research was very interesting. He presented some graphs about the impact of a group job on resolving complex problems. They revealed that there is a critical network size for which the effectiveness for solving a problem drastically increase. He pointed that the source code accessibility "friction" limits the "getting help" variable. Open sourcing software could be the best way to reduce this "friction" while unit testing and ongoing integration are facilitators. And, in general, process reproducibility is very important, not only in computing research. Retrieving experimental settings, metadata, and process environment is vital. We agree with this as we are experimenting it everyday in our work. That is why we encourage open source licenses and develop a collaborative platform that provides the distributed simulation traceability and reproducibility platform Simulagora (in french).

Ian Ozsvald's talk dealt with key points and tips from his own experience to grow a business based on open source and python, as well as mistakes to avoid (e.g. not checking beforehand there are paying customers interested by what you want to develop). His talk was comprehensive and mentioned a wide panel of situations.

http://vispy.org/_static/img/logo.png

We got a very nice presentation of a young but interesting visualization tools: Vispy. It is 6 months old and the first public release was early August. It is the result of the merge of 4 separated libraries, oriented toward interactive visualisation (vs. static figure generation for Matplotlib) and using OpenGL on GPUs to avoid CPU overload. A demonstration with large datasets showed vispy displaying millions of points in real time at 40 frames per second. During the talk we got interesting information about OpenGL features like anti-grain compared to Matplotlib Agg using CPU.

We also got to learn about cartopy which is an open source Python library originally written for weather and climate science. It provides useful and simple API to manipulate cartographic mapping.

Distributed computing systems was a hot topic and many talks were related to this theme.

https://www.openstack.org/themes/openstack/images/openstack-logo-preview-full-color.png

Gael Varoquaux reminded us what are the keys problems with "biggish data" and the key points to successfully process them. I think that some of his recommendations are generally useful like "choose simple solutions", "fail gracefully", "make it easy to debug". For big data processing when I/O limit is the constraint, first try to split the problem into random fractions of the data, then run algorithms and aggregate the results to circumvent this limit. He also presented mini-batch that takes a bunch of observations (trade-off memory usage/vectorization) and joblib.parallel that makes I/O faster using compression (CPUs are faster than disk access).

Benoit Da Mota talked about shared memory in parallel computing and Antonio Messina gave us a quick overview on how to build a computing cluster with Elasticluster, using OpenStack/Slurm/ansible. He demonstrated starting and stopping a cluster on OpenStack: once all VMs are started, ansible configures them as hosts to the cluster and new VMs can be created and added to the cluster on the fly thanks to a command line interface.

We also got a keynote by Peter Wang (from Continuum Analytics) about the future of data analysis with Python. As a PhD in physics I loved his metaphor of giving mass to data. He tried to explain the pain that scientists have when using databases.

https://scikits.appspot.com/static/images/scipyshiny_small.png

After the conference we participated to the numpy/scipy sprint. It was organized by Ralph Gommers and Pauli Virtanen. There were 18 people trying to close issues from different difficulty levels and had a quick tutorial on how easy it is to contribute: the easiest is to fork from the github project page on your own github account (you can create one for free), so that later your patch submission will be a simple "Pull Request" (PR). Clone locally your scipy fork repository, and make a new branch (git checkout -b <newbranch>) to tackle one specific issue. Once your patch is ready, commit it locally, push it on your github repository and from the github interface choose "Push request". You will be able to add something to your commit message before your PR is sent and looked at by the project lead developers. For example using "gh-XXXX" in your commit message will automatically add a link to the issue no. XXXX. Here is the list of open issues for scipy; you can filter them, e.g. displaying only the ones considered easy to fix :D

For more information: Contributing to SciPy.


PyLint sprint during EuroPython in Berlin

2014/07/11 by Sylvain Thenault

The three main maintainers/developpers of Pylint/astroid (Claudiu, Torsten and I) will meet together in Berlin during EuroPython 2014. While this is not an "official" EuroPython sprint but it's still worth announcing it since it's a good opportunity to meet and enhance Pylint. We should find place and time to work on Pylint between wednesday 23 and friday 25.

If you're interested, don't hesitate to contact me (sylvain.thenault@logilab.fr / @sythenault).


EP14 Pylint sprint Day 1 report

2014/07/24 by Sylvain Thenault
https://ep2014.europython.eu/static_media/assets/images/logo.png

We've had a fairly enjoyable and productive first day in our little hidden room at EuroPython in Berlin ! Below are some noticeable things we've worked on and discussed about.

First, we discussed and agreed that while we should at some point cut the cord to the logilab.common package, it will take some time notably because of the usage logilab.common.configuration which would be somewhat costly to replace (and is working pretty well). There are some small steps we should do but basically we should mostly get back some pylint/astroid specific things from logilab.common to astroid or pylint. This should be partly done during the sprint, and remaining work will go to tickets in the tracker.

We also discussed about release management. The point is that we should release more often, so every pylint maintainers should be able to do that easily. Sylvain will write some document about the release procedure and ensure access are granted to the pylint and astroid projects on pypi. We shall release pylint 1.3 / astroid 1.2 soon, and those releases branches will be the last one supporting python < 2.7.

During this first day, we also had the opportunity to meet Carl Crowder, the guy behind http://landscape.io, as well as David Halter which is building the Jedi completion library (https://github.com/davidhalter/jedi). Landscape.io runs pylint on thousands of projects, and it would be nice if we could test beta release on some part of this panel. On the other hand, there are probably many code to share with the Jedi library like the parser and ast generation, as well as a static inference engine. That deserves a sprint on his own though, so we agreed that a nice first step would be to build a common library for import resolution without relying on the python interpreter for that, while handling most of the python dark import features like zip/egg import, .pth files and so one. Indeed that may be two nice future collaborations!

Last but not least, we got some actual work done:

  • Michal Novikowsky from Intel in Poland joined us to work on the ability to run pylint in different processes so it may drastically improve performance on multiple cores box.
  • Torsten did continue some work on various improvements of the functionnal test framework.
  • Sylvain did merge logilab.common.modutils module into astroid as it's mostly driven by astroid and pylint needs. Also fixed the annoying namespace package crash.
  • Claudiu keep up the good work he does daily at improving and fixing pylint :)

EP14 Pylint sprint Day 2 and 3 reports

2014/07/28 by Sylvain Thenault
https://ep2014.europython.eu/static_media/assets/images/logo.png

Here are the list of things we managed to achieve during those last two days at EuroPython.

After several attempts, Michal managed to have pylint running analysis on several files in parallel. This is still in a pull request (https://bitbucket.org/logilab/pylint/pull-request/82/added-support-for-checking-files-in) because of some limitations, so we decided it won't be part of the 1.3 release.

Claudiu killed maybe 10 bugs or so and did some heavy issues cleanup in the trackers. He also demonstrated some experimental support of python 3 style annotation to drive a better inference. Pretty exciting! Torsten also killed several bugs, restored python 2.5 compat (though that will need a logilab-common release as well), introduced a new functional test framework that will replace the old one once all the existing tests will be backported. On wednesday, he did show us a near future feature they already have at Google: some kind of confidence level associated to messages so that you can filter out based on that. Sylvain fixed a couple of bugs (including https://bitbucket.org/logilab/pylint/issue/58/ which was annoying all the numpy community), started some refactoring of the PyLinter class so it does a little bit fewer things (still way too many though) and attempted to improve the pylint note on both pylint and astroid, which went down recently "thanks" to the new checks like 'bad-continuation'.

Also, we merged the pylint-brain project into astroid to simplify things, so you should now submit your brain plugins directly to the astroid project. Hopefuly you'll be redirected there on attempt to use the old (removed) pylint-brain project on bitbucket.

And, the good news is that now both Torsten and Claudiu have new powers: they should be able to do some releases of pylint and astroid. To celebrate that and the end of the sprint, we published Pylint 1.3 together with Astroid 1.2. More on this here.