Blog entries

new pylint / astng / common releases

2009/03/25 by Sylvain Thenault
http://janckos.net/blog/wp-content/uploads/2008/08/python.png

I'm pleased to announce releases of pylint 0.18, logilab-astng 0.19 and logilab-common 0.39. All these packages should now be cleanly available through easy install.

Also, happy pylint users will get:

  • fixed python 2.6 support (pylint/astng tested from 2.4 to 2.6)
  • get source code (and so astng) for zip/egg imports
  • some understanding of the property decorator and of unbound methods
  • some false positives fixed and others minor improvments

See projects home page and ChangeLog for more information:

http://www.logilab.org/project/pylint http://www.logilab.org/project/logilab-astng http://www.logilab.org/project/logilab-common

Please report any problem / question to the python-projects@lists.logilab.org mailing-list.

Enjoy!


Pylint at BayPIGgies

2009/03/31 by Sandrine Ribeau
http://farm4.static.flickr.com/3232/3140527012_23d9d97f69_m_d.jpg

I am pleased to announce that Pylint was presented during a Tools night meeting organized by BayPIGgies on thursday march 26th. This meeting has been recorded and you can enjoy the video.

One point was missing from the presentation and I'll take the opportunity now to mention it. Flymake, an on-the-fly syntax checker for GNU Emacs which has been discussed, does work in combination with Pylint (please see EmacsWiki for more informations).

photo by ten safe frogs under creative commons


New pylint/astng release, but... pylint needs you !

2009/08/27 by Sylvain Thenault

After several months with no time to fix/enhance pylint beside answering email and filing tickets, I've finally tackled some tasks yesterday night to publish bug fixes releases ([1] and [2]).

The problem is that we don't have enough free time at Logilab to lower the number of tickets in pylint tracker page . If you take a look at the ticket tab, you'll see a lot of pendings bug and must-have features (well, and some other less necessary...). You can already easily contribute thanks to the great mercurial dvcs, and some of you do, either by providing patches or by reporting bugs (more tickets, iiirk ! ;) Thank you all btw !!

Now I was wondering what could be done to make pylint going further, and the first ideas which came to my mind was :

  • do ~3 days sprint
  • do some 'tickets killing' days, as done in some popular oss projects

But for this to be useful, we need your support, so here are some questions for you:

  • would you come to a sprint at Logilab (in Paris, France), so you can meet us, learn a lot about pylint, and work on tickets you wish to have in pylint?
  • if France is too far away for most people, would you have another location to propose?
  • would you be on jabber for a tickets killing day, providing it's ok with your agenda? if so, what's your knowledge of pylint/astng internals?

you may answer by adding a comment to this blog (please register first by using the link at the top right of this page) or by mail to sylvain.thenault@logilab.fr. If we've enough positive answers, we'll take the time to organize such a thing.


Pylint a besoin de vous

2009/09/17

Après plusieurs mois au point mort ou presque, Sylvain a pu hier soir publier des versions corrigeant un certain nombre de bogues dans pylint et astng ([1] et [2]).

Il n'en demeure pas moins qu'à Logilab, nous manquons de temps pour faire baisser la pile de tickets ouverts dans le tracker de pylint. Si vous jetez un œuil dans l'onglet Tickets, vous y trouverez un grand nombre de bogues en souffrance et de fonctionalités indispensables (certaines peut-être un peu moins que d'autres...) Il est déjà possible de contribuer en utilisant mercurial pour fournir des patches, ou en signalant des bogues (aaaaaaaaaarg ! encore des tickets !) et certains s'y sont mis, qu'ils en soient remerciés.

Maintenant, nous nous demandions ce que nous pourrions faire pour faire avance Pylint, et nos premières idées sont :

  • organiser un petit sprint de 3 jours environ
  • organiser des jours de "tuage de ticket", comme ça se pratique sur différents projets OSS

Mais pour que ça soit utile, nous avons besoin de votre aide. Voici donc quelques questions :

  • est-ce que vous participeriez à un sprint à Logilab (à Paris, France), ce qui nous permettrait de nous rencontrer, de vous apprendre plein de choses sur le fonctionnement de Pylint et de travailler ensemble sur des tickets qui vous aideraient dans votre travail ?
  • si la France c'est trop loin, où est-ce que ça vous arrangerait ?
  • seriez-vous prêt à vous joindre à nous sur le serveur jabber de Logilab ou sur IRC, pour participer à une chasse au ticket (à une date à déterminer). Si oui, quel est votre degré de connaissance du fonctionnement interne de Pylint et astng ?

Vous pouvez répondre en commentant sur ce blog (pensez à vous enregistrer en utilisant le lien en haut à droite sur cette page) ou en écrivant à sylvain.thenault@logilab.fr. Si nous avons suffisamment de réponses positives nous organiserons quelque chose.


First Pylint Bug Day on Nov 25th, 2009 !

2009/10/21 by Sylvain Thenault
http://www.logilab.org/image/18785?vid=download

Since we don't stop being overloaded here at Logilab, and we've got some encouraging feedback after the "Pylint needs you" post, we decided to take some time to introduce more "community" in pylint.

And the easiest thing to do, rather sooner than later, is a irc/jabber synchronized bug day, which will be held on Wednesday november 25. We're based in France, so main developpers will be there between around 8am and 19pm UTC+1. If a few of you guys are around Paris at this time and wish to come at Logilab to sprint with us, contact us and we'll try to make this possible.

The focus for this bug killing day could be:

  • using logilab.org tracker : getting an account, submitting tickets, triaging existing tickets...
  • using mercurial to develop pylint / astng
  • guide people in the code so they're able to fix simple bugs

We will of course also try to kill a hella-lotta bugs, but the main idea is to help whoever wants to contribute to pylint... and plan for the next bug-killing day !

As we are in the process of moving to another place, we can't organize a sprint yet, but we should have some room available for the next time, so stay tuned :)


pylint bug day next wednesday!

2009/11/23 by Sylvain Thenault

Remember that the first pylint bug day will be held on wednesday, november 25, from around 8am to 8pm in the Paris (France) time zone.

We'll be a few people at Logilab and hopefuly a lot of other guys all around the world, trying to make pylint better.

Join us on the #public conference room of conference.jabber.logilab.org, or if you prefer using an IRC client, join #public on irc.logilab.org which is a gateway to the jabber forum. And if you're in Paris, come to work with us in our office.

People willing to help but without knowledge of pylint internals are welcome, it's the perfect occasion to learn a lot about it, and to be able to hack on pylint in the future!


pylint bug day report

2009/12/04 by Pierre-Yves David
http://farm1.static.flickr.com/85/243306920_6a12bb48c7.jpg

The first pylint bug day took place on wednesday 25th. Four members of the Logilab crew and two other people spent the day working on pylint.

Several patches submitted before the bug day were processed and some tickets were closed.

Charles Hébert added James Lingard's patches for string formatting and is working on several improvements. Vincent Férotin submitted a patch for simple message listings. Sylvain Thenault fixed significant inference bugs in astng (an underlying module of pylint managing the syntax tree). Émile Anclin began a major astng refactoring to take advantage of new python2.6 functionality. For my part, I made several improvements to the test suite. I applied James Lingard patches for ++ operator and generalised it to -- too. I also added a new checker for function call arguments submitted by James Lingard once again. Finally I improved the message filtering of the --errors-only options.

We thank Maarten ter Huurne, Vincent Férotin for their participation and of course James Lingard for submitting numerous patches.

Another pylint bug day will be held in a few months.

image under creative commons by smccann


pylint bugs day #2 on april 16, 2010

2010/03/22 by Sylvain Thenault

Hey guys,

we'll hold the next pylint bugs day on april 16th 2010 (friday). If some of you want to come and work with us in our Paris office, you'll be much welcome.

Else you can still join us on jabber / irc:

See you then!


Astng 0.20.0 and Pylint 0.20.0 releases

2010/03/24 by Emile Anclin

We are happy to announce the Astng 0.20.0 and Pylint 0.20.0 releases.

Pylint is a static code checker based on Astng, both depending on logilab-common 0.49.

Astng

Astng 0.20.0 is a major refactoring: instead of parsing and modifying the syntax tree generated from python's _ast or compiler.ast modules, the syntax tree is rebuilt. Thus the code becomes much clearer, and all monkey patching will eventually disappear from this module.

Speed improvement is achieved by caching the parsed modules earlier to avoid double parsing, and avoiding some repeated inferences, all along fixing a lot of important bugs.

Pylint

Pylint 0.20.0 uses the new Astng, and fixes a lot of bugs too, adding some new functionality:

  • parameters with leading "_" shouldn't count as "local" variables
  • warn on assert( a, b )
  • warning if return or break inside a finally
  • specific message for NotImplemented exception

We would like to thank Chmouel Boudjnah, Johnson Fletcher, Daniel Harding, Jonathan Hartley, Colin Moris, Winfried Plapper, Edward K. Ream and Pierre Rouleau for their contributions, and all other people helping the project to progress.


pylint bug days #2 report

2010/04/19 by Sylvain Thenault

First of all, I've to say that pylint bugs day wasn't that successful in term of 'community event': I've been sprinting almost alone. My Logilab's felows were tied to customer projects, and no outside people shown up on jabber. Fortunatly Tarek Ziade came to visit us, and that was a nice opportunity to talk about pylint, distribute, etc ... Thank you Tarek, you saved my day ;)

As I felt a bit alone, I decided to work on somethings funnier than bug fixing: refactoring!

First, I've greatly simplified the command line: enable-msg/enable-msg-cat/enable-checker/enable-report and their disable-* counterparts were all merged into single --enable/--disable options.

I've also simplified "pylint --help" output, providing a --long-help option to get what we had before. Generic support in `logilab.common.configuration of course.

And last but not least, I refactored pylint so we can have multiple checkers with the same name. The idea behind this is that we can split checker into smaller chunks, basically only responsible for one or a few related messages. When pylint runs, it only uses necessary checkers according to activated messages and reports. When all checkers will be splitted, it should improve performance of "pylint --error-only".

So, I can say I'm finally happy with the results of that pylint bugs day! And hopefuly we will be more people for the next edition...


Thoughts on the python3 conversion workflow

2010/11/30 by Emile Anclin

Python3

The 2to3 script is a very useful tool. We can just use it to run over all code base, and end up with a python3 compatible code whilst keeping a python2 code base. To make our code python3 compatible, we do (or did) two things:

  • small python2 compatible modifications of our source code
  • run 2to3 over our code base to generate a python3 compatible version

However, we not only want to have one python3 compatible version, but also keep developping our software. Hence, we want to be able to easily test it for both python2 and python3. Furthermore if we use patches to get nice commits, this is starting to be quite messy. Let's consider this in the case of Pylint. Indeed, the workflow described before proved to be unsatisfying.

  • I have two repositories, one for python2, one for python3. On the python3 side, I run 2to3 and store the modifications in a patch or a commit.

  • Whenever I implement a fix or a functionality on either side, I have to test if it still works on the other side; but as the 2to3 modifications are often quite heavy, directly creating patches on one side and applying them on the other side won't work most of the time.

  • Now say, I implement something in my python2 base and hold it in a patch or commit it. I can then pull it to my python3 repo:

    • running 2to3 on all Pylint is quite slow: around 30 sec for Pylint without the tests, and around 2 min with the tests. (I'd rather not imagine how long it would take for say CubicWeb).

    • even if I have all my 2to3 modifications on a patch, it takes 5-6 sec to "qpush" or "qpop" them all. Commiting the 2to3 changes instead and using:

      hg pull -u --rebase
      

      is not much faster. If I don't use --rebase, I will have merges on each pull up. Furthermore, we often have either a patch application failure, merge conflict or end up with something which is not python3 compatible (like a newly introduced "except Error, exc").

  • So quite often, I will have to fix it with:

    hg revert -r REV <broken_files>
    2to3 -nw <broken_files>
    hg qref # or hg resolve -m; hg rebase -c
    
  • Suppose that 2to3 transition worked fine, or that we fixed it. I run my tests with python3 and see it does not work; so I modify the patch: it all starts again; and the new patch or the patch modification will create a new head in my python3 repo...

2to3 Fixers

Considering all that, let's investigate 2to3: it comes with a lot of fixers that can be activated or desactived. Now, a lot of them fix just very seldom use cases or stuff deprecated since years. On the other hand, the 2to3 fixers work with regular expressions, so the more we remove, the faster 2to3 should be. Depending on the project, most cases will just not appear, and for the others, we should be able to find other means of disabling them. The lists proposed here after are just suggestions, it will depend on the source base and other overall considerations which and how fixers could actually be disabled.

python2 compatible

Following fixers are 2.x compatible and should be run once and for all (and can then be disabled on daily conversion usage):

  • apply
  • execfile (?)
  • exitfunc
  • getcwdu
  • has_key
  • idioms
  • ne
  • nonzero
  • paren
  • repr
  • standarderror
  • sys_exec
  • tuple_params
  • ws_comma

compat

This can be fixed using imports from a "compat" module like the logilab.common.compat module which holds convenient compatible objects.

  • callable
  • exec
  • filter (Wraps filter() usage in a list call)
  • input
  • intern
  • itertools_imports
  • itertools
  • map (Wraps map() in a list call)
  • raw_input
  • reduce
  • zip (Wraps zip() usage in a list call)

strings and bytes

Maybe they could also be handled by compat:

  • basestring
  • unicode
  • print

For print for example, we could think of a once-and-for-all custom fixer, that would replace it by a convenient echo function (or whatever name you like) defined in compat.

manually

Following issues could probably be fixed manually:

  • dict (it fixes dict iterator methods; it should be possible to have code where we can disable this fixer)
  • import (Detects sibling imports; we could convert them to absolute import)
  • imports, imports2 (renamed modules)

necessary

These changes seem to be necessary:

  • except
  • long
  • funcattrs
  • future
  • isinstance (Fixes duplicate types in the second argument of isinstance(). For example, isinstance(x, (int, int)) is converted to isinstance(x, (int)))
  • metaclass
  • methodattrs
  • numliterals
  • next
  • raise

Consider however that a lot of them might never be used in some projects, like long, funcattrs, methodattrs and numliterals or even metaclass. Also, isinstance is probably motivated by long to int and unicode to str conversions and hence might also be somehow avoided.

don't know

Can we fix these one also with compat ?

  • renames
  • throw
  • types
  • urllib
  • xrange
  • xreadlines

2to3 and Pylint

Pylint is a special case since its test suite has a lot of bad and deprecated code which should stay there. However, in order to have a reasonable work flow, it seems that something must be done to reduce the 1:30 minutes of 2to3 parsing of the tests. Probably nothing could be gained from the above considerations since most cases just should be in the tests, and actually are. Realise that We can expect to be supporting python2 and python3 for several years in parallel.

After a quick look, we see that 90 % of the refactorings of test/input files are just concerning the print statements; more over most of them have nothing to do with the tested functionality. Hence a solution might be to avoid to run 2to3 on the test/input directory, since we already have a mechanism to select depending on python version whether a test file should be tested or not. To some extend, astng is a similar case, but the test suite and the whole project is much smaller.


pylint bug day #3 on july 8, 2011

2011/07/04 by Sylvain Thenault

Hey guys,

we'll hold the next pylint bug day on july 8th 2011 (friday). If some of you want to come and work with us in our Paris office, you'll be welcome.

You can also join us on jabber / irc:

I know the announce is a bit late, but I hope some of you will be able to come or be online anyway!

Regarding the program, the goal is to decrease the number of tickets in the tracker. I'll try to do some triage earlier this week so you'll get a chance to talk about your super-important ticket that has not been selected. Of course, if you intend to work on it, there is a bigger chance of it being fixed next week-end ;)


Pylint 0.24 / logilab-astng 0.22

2011/07/21 by Sylvain Thenault

Hi there!

I'm pleased to announce new releases of pylint and its underlying library logilab-astng. See http://www.logilab.org/project/pylint/0.24.0 and http://www.logilab.org/project/logilab-astng/0.22.0 for more info.

Those releases include mostly fixes and a few enhancements. Python 2.6 relative / absolute imports should now work fine and Python 3 support has been enhanced. There are still two remaining failures in astng test suite when using python 3, but we're unfortunatly missing resources to fix them yet.

Many thanks to everyone who contributed to this release by submitting patches or by participating to the latest bugs day.


Helping pylint to understand things it doesn't

2011/10/10 by Sylvain Thenault

The latest release of logilab-astng (0.23), the underlying source code representation library used by PyLint, provides a new API that may change pylint users' life in the near future...

It aims to allow registration of functions that will be called after a module has been parsed. While this sounds dumb, it gives a chance to fix/enhance the understanding PyLint has about your code.

I see this as a major step towards greatly enhanced code analysis, improving the situation where PyLint users know that when running it against code using their favorite framework (who said CubicWeb? :p ), they should expect a bunch of false positives because of black magic in the ORM or in decorators or whatever else. There are also places in the Python standard library where dynamic code can cause false positives in PyLint.

The problem

Let's take a simple example, and see how we can improve things using the new API. The following code:

import hashlib

def hexmd5(value):
    """"return md5 checksum hexadecimal digest of the given value"""
    return hashlib.md5(value).hexdigest()

def hexsha1(value):
    """"return sha1 checksum hexadecimal digest of the given value"""
    return hashlib.sha1(value).hexdigest()

gives the following output when analyzed through pylint:

[syt@somewhere ~]$ pylint -E example.py
No config file found, using default configuration
************* Module smarter_astng
E:  5,11:hexmd5: Module 'hashlib' has no 'md5' member
E:  9,11:hexsha1: Module 'hashlib' has no 'sha1' member

However:

[syt@somewhere ~]$ python
Python 2.7.1+ (r271:86832, Apr 11 2011, 18:13:53)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import smarter_astng
>>> smarter_astng.hexmd5('hop')
'5f67b2845b51a17a7751f0d7fd460e70'
>>> smarter_astng.hexsha1('hop')
'cffb6b20e0eef296772f6c1457cdde0049bdfb56'

The code runs fine... Why does pylint bother me then? If we take a look at the hashlib module, we see that there are no sha1 or md5 defined in there. They are defined dynamically according to Openssl library availability in order to use the fastest available implementation, using code like:

for __func_name in __always_supported:
    # try them all, some may not work due to the OpenSSL
    # version not supporting that algorithm.
    try:
        globals()[__func_name] = __get_hash(__func_name)
    except ValueError:
        import logging
        logging.exception('code for hash %s was not found.', __func_name)

Honestly I don't blame PyLint for not understanding this kind of magic. The situation on this particular case could be improved, but that's some tedious work, and there will always be "similar but different" case that won't be understood.

The solution

The good news is that thanks to the new astng callback, I can help it be smarter! See the code below:

from logilab.astng import MANAGER, scoped_nodes

def hashlib_transform(module):
    if module.name == 'hashlib':
        for hashfunc in ('sha1', 'md5'):
            module.locals[hashfunc] = [scoped_nodes.Class(hashfunc, None)]

def register(linter):
    """called when loaded by pylint --load-plugins, register our tranformation
    function here
    """
    MANAGER.register_transformer(hashlib_transform)

What's in there?

  • A function that will be called with each astng module built during a pylint execution, i.e. not only the one that you analyses, but also those accessed for type inference.
  • This transformation function is fairly simple: if the module is the 'hashlib' module, it will insert into its locals dictionary a fake class node for each desired name.
  • It is registered using the register_transformer method of astng's MANAGER (the central access point to built syntax tree). This is done in the pylint plugin API register callback function (called when module is imported using 'pylint --load-plugins'.

Now let's try it! Suppose I stored the above code in a 'astng_hashlib.py' module in my PYTHONPATH, I can now run pylint with the plugin activated:

[syt@somewhere ~]$ pylint -E --load-plugins astng_hashlib example.py
No config file found, using default configuration
************* Module smarter_astng
E:  5,11:hexmd5: Instance of 'md5' has no 'hexdigest' member
E:  9,11:hexsha1: Instance of 'sha1' has no 'hexdigest' member

Huum. We have now a different error :( Pylint grasp there are some md5 and sha1 classes but it complains they don't have a hexdigest method. Indeed, we didn't give a clue about that.

We could continue on and on to give it a full representation of hashlib public API using the astng nodes API. But that would be painful, trust me. Or we could do something clever using some higher level astng API:

from logilab.astng import MANAGER
from logilab.astng.builder import ASTNGBuilder

def hashlib_transform(module):
    if module.name == 'hashlib':
    fake = ASTNGBuilder(MANAGER).string_build('''

class md5(object):
  def __init__(self, value): pass
  def hexdigest(self):
    return u''

class sha1(object):
  def __init__(self, value): pass
  def hexdigest(self):
    return u''

''')
    for hashfunc in ('sha1', 'md5'):
        module.locals[hashfunc] = fake.locals[hashfunc]

def register(linter):
    """called when loaded by pylint --load-plugins, register our tranformation
    function here
    """
    MANAGER.register_transformer(hashlib_transform)

The idea is to write a fake python implementation only documenting the prototype of the desired class, and to get an astng from it, using the string_build method of the astng builder. This method will return a Module node containing the astng for the given string. It's then easy to replace or insert additional information into the original module, as you can see in the above example.

Now if I run pylint using the updated plugin:

[syt@somewhere ~]$ pylint -E --load-plugins astng_hashlib example.py
No config file found, using default configuration

No error anymore, great!

What's next?

This fairly simple change could quickly provide great enhancements. We should probably improve the astng manipulation API now that it's exposed like that. But we can also easily imagine a code base of such pylint plugins maintained by each community around a python library or framework. One could then use a plugins stack matching stuff used by its software, and have a greatly enhanced experience of using pylint.

For a start, it would be great if pylint could be shipped with a plugin that explains all the magic found in the standard library, wouldn't it? Left as an exercice to the reader!


PyLint 0.25.2 and related projects released

2012/07/18 by Sylvain Thenault

I'm pleased to announce the new release of Pylint and related projects (i.e. logilab-astng and logilab-common)!

By installing PyLint 0.25.2, ASTNG 0.24 and logilab-common 0.58.1, you'll get a bunch of bug fixes and a few new features. Among the hot stuff:

  • PyLint should now work with alternative python implementations such as Jython, and at least go further with PyPy and IronPython (but those have not really been tested, please try it and provide feedback so we can improve their support)
  • the new ASTNG includes a description of dynamic code it is not able to understand. This is handled by a bitbucket hosted project described in another post.

Many thanks to everyone who contributed to these releases, Torsten Marek / Boris Feld in particular (both sponsored by Google by the way, Torsten as an employee and Boris as a GSoC student).

Enjoy!


Introducing the pylint-brain project

2012/07/18 by Sylvain Thenault

Huum, along with the new PyLint release, it's time to introduce the PyLint-Brain project I've recently started.

Despite its name, PyLint-Brain is actually a collection of extensions for ASTNG, with the goal of making ASTNG smarter (and this directly benefits PyLint) by describing stuff that is too dynamic to be understood automatically (such as functions in the hashlib module, defaultdict, etc.).

The PyLint-Brain collection of extensions is developped outside of ASTNG itself and hosted on a bitbucket project to ease community involvement and to allow distinct development cycles. Basically, ASTNG will include the PyLint-Brain extensions, but you may use earlier/custom versions by tweaking your PYTHONPATH.

Take a look at the code, it's fairly easy to contribute new descriptions, and help us make pylint smarter!


Sprint PyLint @ PyConFr 2012

2012/08/20 by Sylvain Thenault

Un sprint PyLint est organisé dans le cadre de la conférence PyConFR, les 13 et 14 septembre à Paris. Si vous voulez améliorer PyLint, c'est l'occasion : venez avec vos bugs et repartez sans !

Les débutants sont bienvenus, une introduction au code de Pylint sera réalisée en début de sprint. Une expérience avec le module ast de la librairie standard est un plus.

Croissants et café offerts par l'organisation, merci de vous inscrire pour faciliter la logistique. Voir avec Boris pour plus d'informations (merci à lui !)


PyLint 0.26 is out

2012/10/08 by Sylvain Thenault

I'm very pleased to announce new releases of Pylint and underlying ASTNG library, respectivly 0.26 and 0.24.1. The great news is that both bring a lot of new features and some bug fixes, mostly provided by the community effort.

We're still trying to make it easier to contribute on our free software project at Logilab, so I hope this will continue and we'll get even more contritions in a near future, and an even smarter/faster/whatever pylint!

For more details, see ChangeLog files or http://www.logilab.org/project/pylint/0.26.0 and http://www.logilab.org/project/logilab-astng/0.24.1

So many thanks to all those who made that release, and enjoy!


Logilab à PyConFR 2012 - compte rendu

2012/10/09 by Alain Leufroy
http://awesomeness.openphoto.me/custom/201209/4ed140-pycon3--1-of-37-_870x550.jpg

Logilab était à la conférence PyConFR qui a pris place à Paris il y a deux semaines.

Nous avons commencé par un sprint pylint, coordonné par Boris Feld, où pas mal de volontaires sont passés pour traquer des bogues ou ajouter des nouvelles fonctionnalités. Merci à tous!

Pour ceux qui ne connaissent pas encore, pylint est un utilitaire pratique que nous avons dans notre forge. C'est un outil très puissant d'analyse statique de scripts python qui aide à améliorer/maintenir la qualité du code.

Par la suite, après les "talks" des sponsors¸ où vous auriez pu voir Olivier, vous avons pu participer à quelques tutoriels et présentations vraiment excellentes. Il y avait des présentations pratiques avec, entre autres, les tests, scikit-learn ou les outils pour gérer des services (Cornice, Circus). Il y avait aussi des retours d'information sur le processus de développement de CPython, le développement communautaire ou un supercalculateur. Nous avons même pu faire de la musique avec python et un peu d'"embarqué" avec le Raspberry Pi et Arduino !

Nous avons, avec Pierre-Yves, proposé deux tutoriels d'introduction au gestionnaire de versions décentralisé Mercurial. Le premier tutoriel abordait les bases avec des cas pratiques. Lors du second tutoriel, que l'on avait prévu initialement dans la continuité du premier, nous avons finalement abordé des utilisations plus avancées permettant de résoudre avec énormément d'efficacité des problématiques quotidiennes, comme les requêtes sur les dépôts, ou la recherche automatique de régression par bissection. Vous pouvez retrouver le support avec les exercices .

Pierre-Yves a présenté une nouvelle propriété importante de Mercurial: l'obsolescence. Elle permet de mettre en place des outils d'édition d'historique en toute sécurité ! Parmi ces outils, Pierre-Yves a écrit une extension mutable-history qui vous offre une multitude de commandes très pratiques.

La présentation est disponible en PDF et en consultation en ligne sur slideshare. Nous mettrons bientôt la vidéo en ligne.

http://www.logilab.org/file/107770?vid=download

Si le sujet vous intéresse et que vous avez raté cette présentation, Pierre-Yves reparlera de ce sujet à l'OSDC.

Pour ceux qui en veulent plus, Tarek Ziadé à mis à disposition des photos de la conférence ici.


Announcing pylint.org

2012/12/04 by Arthur Lutz

Pylint - the world renowned Python code static checker - now has a landing page : http://www.pylint.org

http://www.python.org/images/python-logo.gif

We've tried to summarize all the things a newcomer should know about pylint. We hope it reflects the diversity of uses and support canals for pylint.

Open and decentralized Web

Note that pylint is not hosted on github or another well-known forge, since we firmly believe in a decentralized architecture for the web.

This applies especially to open source software development. Pylint's development is self-hosted on a forge and its code is version-controlled with mercurial, a distributed version control system (DVCS). Both tools are free software written in python.

http://www.zjulian.com/wp-content/uploads/2012/05/Centralized-Decentralized-And-Distributed-System.jpg

We know centralized (and closed source) platforms for managing software projects can make things easier for contributors. We have enabled a mirror on bitbucket (and pylint-brain) so as to ease forks and pull requests. Pull requests can be made there and even from a self-hosted mercurial (with a quick email on the mailing-list).

Feel free to add your comments or feedback below.


Release of PyLint 0.27 / logilab-astng 0.24.2

2013/02/28 by Sylvain Thenault

Hi there,

I'm very pleased to announce the release of pylint 0.27 and logilab-astng 0.24.2. There has been a lot of enhancements and bug fixes since the latest release, so you're strongly encouraged to upgrade. Here is a detailed list of changes:

  • #20693: replace pylint.el by Ian Eure version (patch by J.Kotta)
  • #105327: add support for --disable=all option and deprecate the 'disable-all' inline directive in favour of 'skip-file' (patch by A.Fayolle)
  • #110840: add messages I0020 and I0021 for reporting of suppressed messages and useless suppression pragmas. (patch by Torsten Marek)
  • #112728: add warning E0604 for non-string objects in __all__ (patch by Torsten Marek)
  • #120657: add warning W0110/deprecated-lambda when a map/filter of a lambda could be a comprehension (patch by Martin Pool)
  • #113231: logging checker now looks at instances of Logger classes in addition to the base logging module. (patch by Mike Bryant)
  • #111799: don't warn about octal escape sequence, but warn about o which is not octal in Python (patch by Martin Pool)
  • #110839: bind <F5> to Run button in pylint-gui
  • #115580: fix erroneous W0212 (access to protected member) on super call (patch by Martin Pool)
  • #110853: fix a crash when an __init__ method in a base class has been created by assignment rather than direct function definition (patch by Torsten Marek)
  • #110838: fix pylint-gui crash when include-ids is activated (patch by Omega Weapon)
  • #112667: fix emission of reimport warnings for mixed imports and extend the testcase (patch by Torsten Marek)
  • #112698: fix crash related to non-inferable __all__ attributes and invalid __all__ contents (patch by Torsten Marek)
  • Python 3 related fixes:
    • #110213: fix import of checkers broken with python 3.3, causing "No such message id W0704" breakage
    • #120635: redefine cmp function used in pylint.reporters
  • Include full warning id for I0020 and I0021 and make sure to flush warnings after each module, not at the end of the pylint run. (patch by Torsten Marek)
  • Changed the regular expression for inline options so that it must be preceeded by a # (patch by Torsten Marek)
  • Make dot output for import graph predictable and not depend on ordering of strings in hashes. (patch by Torsten Marek)
  • Add hooks for import path setup and move pylint's sys.path modifications into them. (patch by Torsten Marek)
  • pylint-brain: more subprocess.Popen faking (see #46273)
  • #109562 [jython]: java modules have no __doc__, causing crash
  • #120646 [py3]: fix for python3.3 _ast changes which may cause crash
  • #109988 [py3]: test fixes

Many thanks to all the people who contributed to this release!

Enjoy!


PyLint 10th years anniversary, 1.0 sprint

2013/03/29 by Sylvain Thenault

In a few week, pylint will be 10 years old (0.1 released on may 19 2003!). At this occasion, I would like to release a 1.0. Well, not exactly at that date, but not too long after would be great. Also, I think it would be a good time to have a few days sprint to work a bit on this 1.0 but also to meet all together and talk about pylint status and future, as more and more contributions come from outside Logilab (actually mostly Google, which employs Torsten and Martin, the most active contributors recently).

The first thing to do is to decide a date and place. Having discussed a bit with Torsten about that, it seems reasonable to target a sprint during june or july. Due to personal constraints, I would like to host this sprint in Logilab's Toulouse office.

So, who would like to jump in and sprint to make pylint even better? I've created a doodle so every one interested may tell his preferences: http://doodle.com/4uhk26zryis5x7as

Regarding the location, is everybody ok with Toulouse? Other ideas are Paris, or Florence around EuroPython, or... <add your proposition here>.

We'll talk about the sprint topics later, but there are plenty of exciting ideas around there.

Please, answer quickly so we can move on. And I hope to see you all there!


Pylint development moving to BitBucket

2013/04/12 by Sylvain Thenault

Hi everyone,

After 10 years of hosting Pylint on our own forge at logilab.org, we've decided to publish version 1.0 and move Pylint and astng development to BitBucket. There has been repository mirrors there for some time, but we intend now to use all BitBucket features, notably Pull Request, to handle various development tasks.

There are several reasons behind this. First, using both BitBucket and our own forge is rather cumbersome, for integrators at least. This is mainly because BitBucket doesn't provide support for Mercurial's changeset evolution feature while our forge relies on it. Second, our forge has several usability drawbacks that make it hard to use for newcomers, and we lack the time to be responsive on this. Finally, we think that our quality-control process, as exposed by our forge, is a bit heavy for such community projects and may keep potential contributors away.

All in all, we hope this will help to have a wider contributor audience as well as more regular maintainers / integrators which are not Logilab employees. And so, bring the best Pylint possible to the Python community!

Logilab.org web pages will be updated to mention this, but kept as there is still valuable information there (eg tickets). We may also keep automatic tests and package building services there.

So, please use https://bitbucket.org/logilab/pylint as main web site regarding pylint development. Bug reports, feature requests as well as contributions should be done there. The same move will be done for Pylint's underlying library, logilab-astng (https://bitbucket.org/logilab/astng). We also wish in this process to move it out of the 'logilab' python package. It may be a good time to give it another name, if you have any idea don't hesitate to express yourself.

Last but not least, remember that Pylint home page may be edited using Mercurial, and that the new http://docs.pylint.org is generated using the content found in Pylint source doc subdirectory.

Pylint turning 10 and moving out of its parents is probably a good time to thank Logilab for paying me and some colleagues to create and maintain this project!

https://bitbucket-assetroot.s3.amazonaws.com/c/photos/2013/Apr/05/pylint-logo-1661676867-0_avatar.png

Pylint 10th years anniversary from June 17 to 19 in Toulouse

2013/04/18 by Sylvain Thenault

After a quick survey, we're officially scheduling Pylint 10th years anniversary sprint from monday, June 17 to wednesday, June 19 in Logilab's Toulouse office.

There is still some room available if more people want to come, drop me a note (sylvain dot thenault at logilab dot fr).


PyLint 10th anniversary 1.0 sprint: day 1

2013/06/17 by Sylvain Thenault

Today was the first day of the Pylint sprint we organized using Pylint's 10th years anniversary as an excuse.

So I (Sylvain) have welcome my fellow Logilab friends David, Anthony and Julien as well as Torsten from Google into Logilab's new Toulouse office.

After a bit of presentation and talk about Pylint development, we decided to keep discussion for lunch and dinner and to setup priorities. We ended with the following tasks (picks from the pad at http://piratepad.net/oAvsUoGCAC):

  • rename astng to move it outside the logilab package,
  • Torsten gpylint (Google Pylint) patches review, as much as possible (but not all of them, starting by a review of the numberous internal checks Google has, seeing one by one which one should be backported upstream),
  • setuptools namespace package support (https://www.logilab.org/8796),
  • python 3.3 support,
  • enhance astroid (former astng) API to allow more ad-hoc customization for a better grasp of magic occuring in e.g. web frameworks (protocol buffer or SQLAlchemy may also be an application of this).

Regarding the astng renaming, we decided to move on with astroid as pointed out by the survey on StellarSurvey.com

In the afternoon, David and Julien tackled this, while Torsten was extracting patches from Google code and sending them to bitbucket as pulll request, Sylvain embrassing setuptools namespaces packages and Anthony discovering the code to spread the @check_message decorator usage.

By the end of the day:

  • David and Julien submitted patches to rename logilab.astng which were quickly integrated and now https://bitbucket.org/logilab/astroid should be used instead of https://bitbucket.org/logilab/astng
  • Torsten submitted 5 pull-requests with code extracted from gpylint, we reviewed them together and then Torsten used evolve to properly insert those in the pylint history once review comments were integrated
  • Sylvain submitted 2 patches on logilab-common to support both setuptools namespace packages and pkgutil.extend_path (but not bare __path__ manipulation
  • Anthony discovered various checkers and started adding proper @check_messages on visit methods

After doing some review all together, we even had some time to take a look at Python 3.3 support while writing this summary.

Hopefuly, our work on forthcoming days will be as efficient as on this first day!


PyLint 10th anniversary 1.0 sprint: day 2

2013/06/18 by Sylvain Thenault

Today was the second day of the 10th anniversary Pylint sprint in Logilab's Toulouse office.

This morning, we started with a presentation by myself about how the inference engine works in astroid (former astng). Then we started thinking all together about how we should change its API to be able to plug more information during the inference process. The first use-case we wanted to assert was namedtuple, as explained in http://www.logilab.org/ticket/8796.

We ended up by addressing it by:

  • enhancing the existing transformation feature so one may register a transformation function on any node rather than on a module node only;
  • being able to specify, on a node instance, a custom inference function to use instead of the default (class) implementation.

We would then be able to customize both the tree structure and the inference process and so to resolve the cases we were targeting.

Once this was sufficiently sketched out, everyone got his own tasks to do. Here is a quick summary of what has been achieved today:

  • Anthony resumed the check_messages thing and finished it for the simple cases, then he started on having a template for text reported,
  • Julien and David made a lot of progress on the Python 3.3 compatibility, though not enough to get the full green test suite,
  • Torsten continued backporting stuff from gpylint, all of them having been integrated by the end of the day,
  • Sylvain implemented the new transformation API and had the namedtuple proof of concept working, and even some documentation! Now this have to be tested for more real-world uses.

So things are going really well, and see you tomorrow for even more improvements to pylint!


PyLint 10th anniversary 1.0 sprint: day 3 - Sprint summary

2013/06/20 by Sylvain Thenault

Yesterday was the third and last day of the 10th anniversary Pylint sprint in Logilab's Toulouse office.

Design

To get started, we took advantage of this last day to have a few discussions about:

  • A "mode" feature gpylint has. It turns out that behind perhaps a few implementation details, this is something we definitly want into pylint (mode are specific configurations defined in the pylintrc and easilly recallable, they may even be specified per file).

  • How to avoid conflicts in the ChangeLog by using specific instruction in the commit message. We decided that a commit message should look like

    [my checker] do this and that. Closes #1234
    
    bla bla bla
    
    :release note: this will be a new item in the ChangeLog
    
    as well as anything until the end of the message
    

    now someone has to write the ChangeLog generation script so we may use this for post-1.0 releases

  • The roadmap. More on this later in this post.

Code

When we were not discussing, we were coding!

  • Anthony worked on having a template for the text reporter. His patch is available on Bitbucket but not yet integrated.
  • Julien and David pushed a bunch of patches on logilab-common, astroid and pylint for the Python 3.3 support. Not all tests are green on the pylint side, but much progress was done.
  • A couple other things were fixed, like a better "invalid name" message, stop complaining about string module being deprecated, etc.
  • A lot of patches have been integrated, from gpylint and others (e.g python 3 related)

All in all, an impressive amount of work was achieved during this sprint:

  • A lot of new checks or enhanced behaviour backported from gpylint (Take a look at Pylint's ChangeLog for more details on this, the list is impressively long).
  • The transformation API of astroid now allows to customize the tree structure as well as the inference process, hence to make pylint smarter than ever.
  • Better python 3 support.
  • A few bugs fixed and some enhancements added.
  • The templating stuff should land with the CLI cleanup (some output-formats will be removed as well as the --include-ids and --symbols option).
  • A lot of discussions, especially regarding the future community development of pylint/astroid on Bitbucket. Short summary being: more contributors and integrators are welcome! We should drop some note somewhere to describe how we are using bitbucket's pull requests and tracker.

Plan

Now here is the 1.O roadmap, which is expected by the begining of July:

  • Green tests under Python 3, including specification of Python version in message description (Julien).
  • Finish template for text reporters (Anthony).
  • Update web site (David).

And for later releases:

  • Backport mode from gpylint (Torsten).
  • Write ChangeLog update script (Sylvain).

So many thanks to everyone for this very successful sprint. I'm excited about this forthcoming 1.0 release!


Astroid 1.0 released!

2013/08/02 by Sylvain Thenault

Astroid is the new name of former logilab-astng library. It's an AST library, used as the basis of Pylint and including Python 2.5 -> 3.3 compatible tree representation, statical type inference and other features useful for advanced Python code analysis, such as an API to provide extra information when statistical inference can't overcome Python dynamic nature (see the pylint-brain project for instance).

It has been renamed and hosted to bitbucket to make clear that this is not a Logilab dedicated project but a community project that could benefit to any people manipulating Python code (statistical analysis tools, IDE, browser, etc).

Documentation is a bit rough but should quickly improve. Also a dedicated web-site is now online, visit www.astroid.org (or https://bitbucket.org/logilab/astroid for development).

You may download and install it from Pypi or from Logilab's debian repositories.


Pylint 1.0 released!

2013/08/06 by Sylvain Thenault

Hi there,

I'm very pleased to announce, after 10 years of existence, the 1.0 release of Pylint.

This release has a hell long ChangeLog, thanks to many contributions and to the 10th anniversary sprint we hosted during june. More details about changes below.

Chances are high that your Pylint score will go down with this new release that includes a lot of new checks :) Also, there are a lot of improvments on the Python 3 side (notably 3.3 support which was somewhat broken).

You may download and install it from Pypi or from Logilab's debian repositories. Notice Pylint has been updated to use the new Astroid library (formerly known as logilab-astng) and that the logilab-common 0.60 library includes some fixes necessary for using Pylint with Python3 as well as long-awaited support for namespace packages.

For those interested, below is a comprehensive list of what changed:

Command line and output formating

  • A new --msg-template option to control output, deprecating "msvc" and "parseable" output formats as well as killing --include-ids and --symbols options.
  • Fix spelling of max-branchs option, now max-branches.
  • Start promoting usage of symbolic name instead of numerical ids.

New checks

  • "missing-final-newline" (C0304) for files missing the final newline.
  • "invalid-encoded-data" (W0512) for files that contain data that cannot be decoded with the specified or default encoding.
  • "bad-open-mode" (W1501) for calls to open (or file) that specify invalid open modes (Original implementation by Sasha Issayev).
  • "old-style-class" (C1001) for classes that do not have any base class.
  • "trailing-whitespace" (C0303) that warns about trailing whitespace.
  • "unpacking-in-except" (W0712) about unpacking exceptions in handlers, which is unsupported in Python 3.
  • "old-raise-syntax" (W0121) for the deprecated syntax raise Exception, args.
  • "unbalanced-tuple-unpacking" (W0632) for unbalanced unpacking in assignments (bitbucket #37).

Enhanced behaviours

  • Do not emit [fixme] for every line if the config value 'notes' is empty
  • Emit warnings about lines exceeding the column limit when those lines are inside multiline docstrings.
  • Name check enhancement:
    • simplified message,
    • don't double-check parameter names with the regex for parameters and inline variables,
    • don't check names of derived instance class members,
    • methods that are decorated as properties are now treated as attributes,
    • names in global statements are now checked against the regular expression for constants,
    • for toplevel name assignment, the class name regex will be used if pylint can detect that value on the right-hand side is a class (like collections.namedtuple()),
    • add new name type 'class_attribute' for attributes defined in class scope. By default, allow both const and variable names.
  • Add a configuration option for missing-docstring to optionally exempt short functions/methods/classes from the check.
  • Add the type of the offending node to missing-docstring and empty-docstring.
  • Do not warn about redefinitions of variables that match the dummy regex.
  • Do not treat all variables starting with "_" as dummy variables, only "_" itself.
  • Make the line-too-long warning configurable by adding a regex for lines for with the length limit should not be enforced.
  • Do not warn about a long line if a pylint disable option brings it above the length limit.
  • Do not flag names in nested with statements as undefined.
  • Remove string module from the default list of deprecated modules (bitbucket #3).
  • Fix incomplete-protocol false positive for read-only containers like tuple (bitbucket #25).

Other changes

  • Support for pkgutil.extend_path and setuptools pkg_resources (logilab-common #8796).
  • New utility classes for per-checker unittests in testutils.py
  • Added a new base class and interface for checkers that work on the tokens rather than the syntax, and only tokenize the input file once.
  • epylint shouldn't hang anymore when there is a large output on pylint'stderr (bitbucket #15).
  • Put back documentation in source distribution (bitbucket #6).

Astroid

  • New API to make it smarter by allowing transformation functions on any node, providing a register_transform function on the manager instead of the register_transformer to make it more flexible wrt node selection
  • Use this new transformation API to provide support for namedtuple (actually in pylint-brain, logilab-astng #8766)
  • Better description of hashlib
  • Properly recognize methods annotated with abc.abstract{property,method} as abstract.
  • Added the test_utils module for building ASTs and extracting deeply nested nodes for easier testing.

A retrospective of 10 years animating the pylint free software projet

2013/11/25 by Sylvain Thenault

was the topic of the talk I gave last saturday at the Capitol du Libre in Toulouse.

Here are the slides (pdf) for those interested (in french). A video of the talk should be available soon on the Capitol du Libre web site. The slides are mirrored on slideshare (see below):


A quick take on continuous integration services for Bitbucket

2013/12/19 by Sylvain Thenault

Some time ago, we moved Pylint from this forge to Bitbucket (more on this here).

https://bitbucket-assetroot.s3.amazonaws.com/c/photos/2012/Oct/11/master-logo-2562750429-5_avatar.png

Since then, I somewhat continued to use the continuous integration (CI) service we provide on logilab.org to run tests on new commits, and to do the release job (publish a tarball on pypi, on our web site, build Debian and Ubuntu packages, etc.). This is fine, but not really handy since the logilab.org's CI service is not designed to be used for projects hosted elsewhere. Also I wanted to see what others have to offer, so I decided to find a public CI service to host Pylint and Astroid automatic tests at least.

Here are the results of my first swing at it. If you have others suggestions, some configuration proposal or whatever, please comment.

First, here are the ones I didn't test along with why:

The first one I actually tested, also the first one to show up when looking for "bitbucket continuous integration" on Google is https://drone.io. The UI is really simple, I was able to set up tests for Pylint in a matter of minutes: https://drone.io/bitbucket.org/logilab/pylint. Tests are automatically launched when a new commit is pushed to Pylint's Bitbucket repository and that setup was done automatically.

Trying to push Drone.io further, one missing feature is the ability to have different settings for my project, e.g. to launch tests on all the python flavor officially supported by Pylint (2.5, 2.6, 2.7, 3.2, 3.3, pypy, jython, etc.). Last but not least, the missing killer feature I want is the ability to launch tests on top of Pull Requests, which travis-ci supports.

Then I gave http://wercker.com a shot, but got stuck at the Bitbucket repository selection screen: none were displayed. Maybe because I don't own Pylint's repository, I'm only part of the admin/dev team? Anyway, wercker seems appealing too, though the configuration using yaml looks a bit more complicated than drone.io's, but as I was not able to test it further, there's not much else to say.

http://wercker.com/images/logo_header.png

So for now the winner is https://drone.io, but the first one allowing me to test on several Python versions and to launch tests on pull requests will be the definitive winner! Bonus points for automating the release process and checking test coverage on pull requests as well.

https://drone.io/drone3000/images/alien-zap-header.png

Pylint 1.1 christmas release

2013/12/24 by Sylvain Thenault

Pylint 1.1 eventually got released on pypi!

A lot of work has been achieved since the latest 1.0 release. Various people have contributed to add several new checks as well as various bug fixes and other enhancement.

Here is the changes summary, check the changelog for more info.

New checks:

  • 'deprecated-pragma', for use of deprecated pragma directives "pylint:disable-msg" or "pylint:enable-msg" (was previously emmited as a regular warn().
  • 'superfluous-parens' for unnecessary parentheses after certain keywords.
  • 'bad-context-manager' checking that '__exit__' special method accepts the right number of arguments.
  • 'raising-non-exception' / 'catching-non-exception' when raising/catching class non inheriting from BaseException
  • 'non-iterator-returned' for non-iterators returned by '__iter__'.
  • 'unpacking-non-sequence' for unpacking non-sequences in assignments and 'unbalanced-tuple-unpacking' when left-hand-side size doesn't match right-hand-side.

Command line:

  • New option for the multi-statement warning to allow single-line if statements.
  • Allow to run pylint as a python module 'python -m pylint' (anatoly techtonik).
  • Various fixes to epylint

Bug fixes:

  • Avoid false used-before-assignment for except handler defined identifier used on the same line (#111).
  • 'useless-else-on-loop' not emited if there is a break in the else clause of inner loop (#117).
  • Drop 'badly-implemented-container' which caused several problems in its current implementation.
  • Don't mark input as a bad function when using python3 (#110).
  • Use attribute regexp for properties in python3, as in python2
  • Fix false-positive 'trailing-whitespace' on Windows (#55)

Other:

  • Replaced regexp based format checker by a more powerful (and nit-picky) parser, combining 'no-space-after-operator', 'no-space-after-comma' and 'no-space-before-operator' into a new warning 'bad-whitespace'.
  • Create the PYLINTHOME directory when needed, it might fail and lead to spurious warnings on import of pylint.config.
  • Fix setup.py so that pylint properly install on Windows when using python3.
  • Various documentation fixes and enhancements

Packages will be available in Logilab's Debian and Ubuntu repository in the next few weeks.

Happy christmas!


Deuxième hackathon codes libres de mécanique

2014/04/07 by Nicolas Chauvat

Organisation

Le 27 mars 2014, Logilab a accueilli un hackathon consacré aux codes libres de simulation des phénomènes mécaniques. Etaient présents:

  • Patrick Pizette, Sébastien Rémond (Ecole des Mines de Douai / DemGCE)
  • Frédéric Dubois, Rémy Mozul (LMGC Montpellier / LMGC90)
  • Mickaël Abbas, Mathieu Courtois (EDF R&D / Code_Aster)
  • Alexandre Martin (LAMSID / Code_Aster)
  • Luca Dall'Olio, Maximilien Siavelis (Alneos)
  • Florent Cayré, Nicolas Chauvat, Denis Laxalde, Alain Leufroy (Logilab)

DemGCE et LMGC90

Patrick Pizette et Sébastien Rémond des Mines de Douai sont venus parler de leur code de modélisation DemGCE de "sphères molles" (aussi appelé smooth DEM), des potentialités d'intégration de leurs algorithmes dans LMGC90 avec Frédéric Dubois du LMGC et de l'interface Simulagora développée par Logilab. DemGCE est un code DEM en 3D développé en C par le laboratoire des Mines de Douai. Il effectuera bientôt des calculs parallèles en mémoire partagée grâce à OpenMP. Après une présentation générale de LMGC90, de son écosystème et de ses applications, ils ont pu lancer leurs premiers calculs en mode dynamique des contacts en appelant via l'interface Python leurs propres configurations d'empilements granulaires.

Ils ont grandement apprécié l'architecture logicielle de LMGC90, et en particulier son utilisation comme une bibliothèque de calcul via Python, la prise en compte de particules de forme polyhédrique et les aspects visualisations avec Paraview. Il a été discuté de la réutilisation de la partie post/traitement visualisation via un fichier standard ou une bibliothèque dédiée visu DEM.

Frédéric Dubois semblait intéressé par l'élargissement de la communauté et du spectre des cas d'utilisation, ainsi que par certains algorithmes mis au point par les Mines de Douai sur la génération géométrique d'empilements. Il serait envisageable d'ajouter à LMGC90 les lois d'interaction de la "smooth DEM" en 3D, car elles ne sont aujourd'hui implémentées dans LMGC90 que pour les cas 2D. Cela permettrait de tester en mode "utilisateur" le code LMGC90 et de faire une comparaison avec le code des Mines de Douai (efficacité parallélisation, etc.).

Florent Cayré a fait une démonstration du potentiel de Simulagora.

LMGC90 et Code_Aster dans Debian

Denis Laxalde de Logilab a travaillé d'une part avec Rémy Mozul du LMGC sur l'empaquetage Debian de LMGC90 (pour intégrer en amont les modifications nécessaires), et d'autre part avec Mathieu Courtois d'EDF R&D, pour finaliser l'empaquetage de Code_Aster et notamment discuter de la problématique du lien avec la bibliothèque Metis: la version actuellement utilisée dans Code_Aster (Metis 4), n'est pas publiée dans une licence compatible avec la section principale de Debian. Pour cette raison, Code_Aster n'est pas compilé avec le support MED dans Debian actuellement. En revanche la version 5 de Metis a une licence compatible et se trouve déjà dans Debian. Utiliser cette version permettrait d'avoir Code_Aster avec le support Metis dans Debian. Cependant, le passage de la version 4 à la version 5 de Metis ne semble pas trivial.

Voir les tickets:

Replier LibAster dans Code_Aster

Alain Leufroy et Nicolas Chauvat de Logilab ont travaillé à transformer LibAster en une liste de pull request sur la forge bitbucket de Code_Aster. Ils ont présenté leurs modifications à Mathieu Courtois d'EDF R&D ce qui facilitera leur intégration.

Voir les tickets:

Suppression du superviseur dans Code_Aster

En fin de journée, Alain Leufroy, Nicolas Chauvat et Mathieu Courtois ont échangé leurs idées sur la simplification/suppression du superviseur de commandes actuel de Code_Aster. Il est souhaitable que la vérification de la syntaxe (choix des mots-clés) soit dissociée de l'étape d'exécution.

La vérification pourrait s'appuyer sur un outil comme pylint, la description de la syntaxe des commandes de Code_Aster pour pylint pourrait également permettre de produire un catalogue compréhensible par Eficas.

L'avantage d'utiliser pylint serait de vérifier le fichier de commandes avant l'exécution même si celui-ci contient d'autres instructions Python.

Allocation mémoire dans Code_Aster

Mickaël Abbas d'EDF R&D s'est intéressé à la modernisation de l'allocation mémoire dans Code_Aster et a listé les difficultés techniques à surmonter ; l'objectif visé est un accès facilité aux données numériques du Fortran depuis l'interface Python. Une des difficultés est le partage des types dérivés Fortran en Python. Rémy Mozul du LMGC et Denis Laxalde de Logilab ont exploré une solution technique basée sur Cython et ISO-C-Bindings. De son côté Mickaël Abbas a contribué à l'avancement de cette tâche directement dans Code_Aster.

Doxygen pour documentation des sources de Code_Aster

Luca Dall'Olio d'Alneos et Mathieu Courtois ont testé la mise en place de Doxygen pour documenter Code_Aster. Le fichier de configuration pour doxygen a été modifié pour extraire les commentaires à partir de code Fortran (les commentaires doivent se trouver au dessus de la déclaration de la fonction, par exemple). La configuration doxygen a été restituée dans le depôt Bitbucket. Reste à évaluer s'il y aura besoin de plusieurs configurations (pour la partie C, Python et Fortran) ou si une seule suffira. Une configuration particulière permet d'extraire, pour chaque fonction, les points où elle est appelée et les autres fonctions utilisées. Un exemple a été produit pour montrer comment écrire des équations en syntaxe Latex, la génération de la documentation nécessite plus d'une heure (seule la partie graphique peut être parallélisée). La documentation produite devrait être publiée sur le site de Code_Aster.

La suite envisagée est de coupler Doxygen avec Breathe et Sphinx pour compléter la documentation extraite du code source de textes plus détaillés.

La génération de cette documentation devrait être une cible de waf, par exemple waf doc. Un aperçu rapide du rendu de la documentation d'un module serait possible par waf doc file1.F90 [file2.c [...]].

Voir Code Aster #18 configure doxygen to comment the source files

Catalogue d'éléments finis

Maximilien Siavelis d'Alneos et Alexandre Martin du LAMSID, rejoints en fin de journée par Frédéric Dubois du LMGC ainsi que Nicolas Chauvat et Florent Cayré de Logilab, ont travaillé à faciliter la description des catalogues d'éléments finis dans Code_Aster. La définition de ce qui caractérise un élément fini a fait l'objet de débats passionnés. Les points discutés nourriront le travail d'Alexandre Martin sur ce sujet dans Code_Aster. Alexandre Martin a déjà renvoyé aux participants un article qu'il a écrit pour résumer les débats.

Remontée d'erreurs de fortran vers Python

Mathieu Courtois d'EDF R&D a montré à Rémy Mozul du LMGC un mécanisme de remontée d'exception du Fortran vers le Python, qui permettra d'améliorer la gestion des erreurs dans LMGC90, qui a posé problème dans un projet réalisé par Denis Laxalde de Logilab pour la SNCF.

Voir aster_exceptions.c

Conclusion

Tous les participants semblaient contents de ce deuxième hackathon, qui faisait suite à la première édition de mars 2013 . La prochaine édition aura lieu à l'automne 2014 ou au printemps 2015, ne la manquez pas !