|
Blog entriesToday was the second day of the 10th anniversary Pylint sprint in Logilab's Toulouse office.
This morning, we started with a presentation by myself about how the inference engine works in astroid (former astng).
Then we started thinking all together about how we should change its API to be able to plug more information during the inference process. The first use-case we wanted to assert was namedtuple, as explained in http://www.logilab.org/ticket/8796.
We ended up by addressing it by:
- enhancing the existing transformation feature so one may register a transformation function on any node rather than on a module node only;
- being able to specify, on a node instance, a custom inference function to use instead of the default (class) implementation.
We would then be able to customize both the tree structure and the inference process and so to resolve the cases we were targeting.
Once this was sufficiently sketched out, everyone got his own tasks to do. Here is a quick summary of what has been achieved today:
- Anthony resumed the check_messages thing and finished it for the simple cases, then he started on having a template for text reported,
- Julien and David made a lot of progress on the Python 3.3 compatibility, though not enough to get the full green test suite,
- Torsten continued backporting stuff from gpylint, all of them having been integrated by the end of the day,
- Sylvain implemented the new transformation API and had the namedtuple proof of concept working, and even some documentation! Now this have to be tested for more real-world uses.
So things are going really well, and see you tomorrow for even more improvements to pylint!
Today was the first day of the Pylint sprint we organized using
Pylint's 10th years anniversary as an excuse.
So I (Sylvain) have welcome my fellow Logilab friends David, Anthony
and Julien as well as Torsten from Google into Logilab's new Toulouse
office.
After a bit of presentation and talk about Pylint development, we
decided to keep discussion for lunch and dinner and to setup
priorities. We ended with the following tasks (picks from the pad at
http://piratepad.net/oAvsUoGCAC):
- rename astng to move it outside the logilab package,
- Torsten gpylint (Google Pylint) patches review, as much as
possible (but not all of them, starting by a review of the numberous
internal checks Google has, seeing one by one which one should be
backported upstream),
- setuptools namespace package support
(https://www.logilab.org/8796),
- python 3.3 support,
- enhance astroid (former astng) API to allow more ad-hoc
customization for a better grasp of magic occuring in e.g. web
frameworks (protocol buffer or SQLAlchemy may also be an
application of this).
Regarding the astng renaming, we decided to move on with
astroid as pointed out by the survey on StellarSurvey.com
In the afternoon, David and Julien tackled this, while Torsten was
extracting patches from Google code and sending them to bitbucket as
pulll request, Sylvain embrassing setuptools namespaces packages and
Anthony discovering the code to spread the @check_message decorator
usage.
By the end of the day:
- David and Julien submitted patches to rename logilab.astng which
were quickly integrated and now https://bitbucket.org/logilab/astroid
should be used instead of https://bitbucket.org/logilab/astng
- Torsten submitted 5 pull-requests with code extracted from gpylint,
we reviewed them together and then Torsten used evolve to properly
insert those in the pylint history once review comments were
integrated
- Sylvain submitted 2 patches on logilab-common to support both
setuptools namespace packages and pkgutil.extend_path (but
not bare __path__ manipulation
- Anthony discovered various checkers and started adding proper
@check_messages on visit methods
After doing some review all together, we even had some time to take a
look at Python 3.3 support while writing this summary.
Hopefuly, our work on forthcoming days will be as efficient as on this first day!
Ce matin, j'ai assisté aux Rencontres INRIA Industries qui portaient sur le thème "Modélisation, simulation et calcul intensif".
Y était notamment présenté l'initiative HPC-PME dont l'objet est de faciliter l'accès des PME au calcul hautes performances.
Cette conférence a été pour moi l'occasion de formaliser mes réflexions sur les PME et leur recours à la simulation numérique.
Lors des différentes conversations que j'ai pu avoir avec des dirigeants de PME, il m'est apparu que leur principale demande est
de pouvoir effectuer très simplement des simulations en fournissant un jeu de données puis en cliquant sur un bouton. L'initiative
HPC-PME a pour objectif de démontrer l'intérêt du calcul hautes performances et d'aider les PME à acquérir les compétences leur
permettant de mettre en œuvre des solutions de HPC. Or, la plupart des PME :
- ne souhaitent pas mettre en place en interne une solution de calcul hautes performances (achat, configuration, maintenance),
- ne désirent pas acquérir en interne les compétences leur permettant d'utiliser une solution de calcul hautes performances,
- n'ont pas réellement besoin des performances extrêmes apportées par l'initiative HPC-PME (la Formule 1
de la simulation numérique), mais plutôt d'une solution solide, simple, et à coût d'entrée raisonnable (une bonne voiture
haut de gamme suffit),
- n'ont pas une demande continue leur permettant d'amortir un investissement en calcul numérique.
Les PME ont généralement une expertise métier forte, c'est là leur facteur différenciant. Mais elles n'ont, pour la majorité d'entre
elles, aucune compétence en analyse numérique ou en informatique. Attendu qu'elles n'ont que peu de calculs à effectuer, il leur
est surérogatoire d'internaliser ces compétences.
Partant de ce constat, je pense sincèrement que la meilleure façon pour une PME d'avoir accès au calcul numérique serait de disposer
d'une plateforme dont l'accès est facturé à l'utilisation et qui lui permet de :
- définir un cas de calcul en termes métier,
- avoir accès à des ressources de calcul sur lesquelles les codes de calcul sont installés et configurés,
- lancer en un clic le cas de calcul sur une ressource de calcul offrant des performances suffisantes,
- pouvoir obtenir de l'aide de la part d'experts numériciens connaissant les codes de calcul (en cas de problème numérique ou de modélisation),
- offrir des fonctionnalités de partage des cas de calcul selon des règles de sécurité strictes et entièrement contrôlables
(typiquement pour faire appel à un expert numéricien, montrer ses résultats au client final ou faire travailler un fournisseur).
After a quick survey, we're officially scheduling Pylint 10th years anniversary sprint from monday, June 17 to wednesday, June 19 in Logilab's Toulouse office.
There is still some room available if more people want to come, drop me a note (sylvain dot thenault at logilab dot fr).
Hi everyone,
After 10 years of hosting Pylint on our own forge at logilab.org, we've decided to publish version 1.0 and move Pylint and astng development to BitBucket. There has been repository mirrors there for some time, but we intend now to use all BitBucket features, notably Pull Request, to handle various development tasks.
There are several reasons behind this. First, using both BitBucket and our own forge is rather cumbersome, for integrators at least. This is mainly because BitBucket doesn't provide support for Mercurial's changeset evolution feature while our forge relies on it. Second, our forge has several usability drawbacks that make it hard to use for newcomers, and we lack the time to be responsive on this. Finally, we think that our quality-control process, as exposed by our forge, is a bit heavy for such community projects and may keep potential contributors away.
All in all, we hope this will help to have a wider contributor audience as well as more regular maintainers / integrators which are not Logilab employees. And so, bring the best Pylint possible to the Python community!
Logilab.org web pages will be updated to mention this, but kept as there is still valuable information there (eg tickets). We may also keep automatic tests and package building services there.
So, please use https://bitbucket.org/logilab/pylint as main web site regarding pylint development. Bug reports, feature requests as well as contributions should be done there. The same move will be done for Pylint's underlying library, logilab-astng (https://bitbucket.org/logilab/astng). We also wish in this process to move it out of the 'logilab' python package. It may be a good time to give it another name, if you have any idea don't hesitate to express yourself.
Last but not least, remember that Pylint home page may be edited using Mercurial, and that the new http://docs.pylint.org is generated using the content found in Pylint source doc subdirectory.
Pylint turning 10 and moving out of its parents is probably a good time to thank Logilab for paying me and some colleagues to create and maintain this project!
In a few week, pylint will be 10 years old (0.1 released on may 19 2003!).
At this occasion, I would like to release a 1.0. Well, not exactly at that date,
but not too long after would be great. Also, I think it would be a good time
to have a few days sprint to work a bit on this 1.0 but also to meet all together
and talk about pylint status and future, as more and more contributions come from
outside Logilab (actually mostly Google, which employs Torsten and Martin, the most
active contributors recently).
The first thing to do is to decide a date and place. Having discussed a bit with
Torsten about that, it seems reasonable to target a sprint during june or july.
Due to personal constraints, I would like to host this sprint in Logilab's
Toulouse office.
So, who would like to jump in and sprint to make pylint even better? I've created
a doodle so every one interested may tell his preferences:
http://doodle.com/4uhk26zryis5x7as
Regarding the location, is everybody ok with Toulouse? Other ideas are Paris, or
Florence around EuroPython, or... <add your proposition here>.
We'll talk about the sprint topics later, but there are plenty of exciting ideas
around there.
Please, answer quickly so we can move on. And I hope to see you all there!
LMGC90 Sprint at Logilab
At the end of March 2013, Logilab hosted a sprint on the LMGC90 simulation code
in Paris.
LMGC90 is an open-source software developed at the LMGC ("Laboratoire de
Mécanique et Génie Civil" -- "Mechanics and Civil Engineering Laboratory") of
the CNRS, in Montpellier, France. LMGC90 is devoted to contact mechanics and is,
thus, able to model large collections of deformable or undeformable physical
objects of various shapes, with numerous interaction laws. LMGC90 also allows
for multiphysics coupling.
More than ten hackers joined in from:
- the LMGC, which leads LMCG90 development and aims at constantly improving its
architecture and usability;
- the Innovation and Research Department of the SNCF (the French state-owned
railway company), which uses LMGC90 to study railway mechanics, and more
specifically, the ballast;
- the LaMSID ("Laboratoire de Mécanique des Structures Industrielles Durables",
"Laboratory for the Mechanics of Ageing Industrial Structures") laboratory
of the EDF / CNRS / CEA , which has an strong expertise on Code_ASTER
and LMGC90;
- Logilab, as the developer, for the SNCF, of a CubicWeb-based platform
dedicated to the simulation data and knowledge management.
After a great introduction to LMGC90 by Frédéric Dubois and some preliminary
discussions, teams were quickly constituted around the common areas of interest.
As of the sprint date, LMGC90 is mainly developed in Fortran, but also contains
Python code for two purposes:
- Exposing the Fortran functions and subroutines in the LMGC90 core to Python;
this is achieved using Fortran 2003's ISO_C_BINDING module and Swig.
These Python bindings are grouped in a module called ChiPy.
- Making it easy to generate input data (so called "DATBOX" files) using Python.
This is done through a module called Pre_LMGC.
The main drawback of this approach is the double modelling of data that this
architecture implies: once in the core and once in Pre_LMGC.
It was decided to build a unique user-level Python layer on top of ChiPy,
that would be able to build the computational problem description and write the
DATBOX input files (currently achieved by using Pre_LMGC), as well as
to drive the simulation and read the OUTBOX result files (currently by using
direct ChiPy calls).
This task has been met with success, since, in the short time span available
(half a day, basically), the team managed to build some object types using
ChiPy calls and save them into a DATBOX.
This topic involved importing LMGC90 DATBOX data into the numerical platform
developed by Logilab for the SNCF.
This was achieved using ChiPy as a Python API to the Fortran core to get:
- the bodies involved in the computation, along with their materials, behaviour
laws (with their associated parameters), geometries (expressed in terms of
zones);
- the interactions between these bodies, along with their interaction laws (and
associated parameters, e.g. friction coefficient) and body pair (each
interaction is defined between two bodies);
- the interaction groups, which contain interactions that have the same
interaction law.
There is still a lot of work to be done (notably regarding the charges applied
to the bodies), but this is already a great achievement. This could only have
occured in a sprint, were every needed expertise is available:
the SNCF experts were there to clarify the import needs and check the overall
direction;
Logilab implemented a data model based on CubicWeb, and imported the data
using the ChiPy bindings developed on-demand by the LMGC core developer team,
using the usual-for-them ISO_C_BINDING/ Swig Fortran wrapping dance.
Logilab undertook the data import; to this end, it asked the LMGC how the
relevant information from LMGC90 can be exposed to Python via the ChiPy API.
The main point of this topic was to replace the in-house DATBOX/OUTBOX textual
format used by LMGC90 to store input and output data, with an open, standard and
efficient format.
Several formats have been considered, like HDF5, MED and NetCDF4.
MED has been ruled out for the moment, because it lacks the support for storing
body contact information. HDF5 was chosen at last because of the quality of its
Python libraries, h5py and pytables, and the ease of use tools like h5fs provide.
Alain Leufroy from Logilab quickly presented h5py and h5fs usage, and the team
started its work, measuring the performance impact of the storage pattern of
LMGC90 data. This was quickly achieved, as the LMGC experts made it easy to
setup tests of various sizes, and as the Logilab developers managed to
understand the concepts and implement the required code in a fast and agile way.
This topic turned out to be more difficult than initially assessed, mainly
because LMGC90 has dependencies to non-packaged external libraries, which thus had
to be packaged first:
- the Matlib linear algebra library, written in C,
- the Lapack95 library, which is a Fortran95 interface to the Lapack library.
Logilab kept working on this after the sprint and produced packages that are
currently being tested by the LMGC team. Some changes are expected (for instance,
Python modules should be prefixed with a proper namespace) before the packages can be
submitted for inclusion into Debian. The expertise of Logilab regarding
Debian packaging was of great help for this task. This will hopefully help to
spread the use of LMGC90.
As you may know, Logilab is really fond of Mercurial as a DVCS. Our company
invested a lot into the development of the great evolve extension, which makes
Mercurial a very powerful tool to efficiently manage the team development of
software in a clean fashion.
This is why Logilab presented Mercurial's features and advantages over the
current VCS used to manage LMGC90 sources, namely svn, to the other
participants of the Sprint. This was appreciated and will hopefully benefit to
LMGC90 ease of development and spread among the Open Source community.
All in all, this two-day sprint on LMGC90, involving participants from several
industrial and academic institutions has been a great success. A lot of code has
been written but, more importantly, several stepping stones have been laid, such
as:
- the general LMGC90 data access architecture, with the Python layer on top of
the LMGC90 core;
- the data storage format, namely HDF5.
Colaterally somehow, several other results have also been achieved:
- partial LMGC90 data import into the SNCF CubicWeb-based numerical platform,
- Debian / Ubuntu packaging of LMGC90 and dependencies.
On a final note, one would say that we greatly appreciated the cooperation
between the participants, which we found pleasant and efficient. We look forward
to finding more occasions to work together.
0 Comment | tagged by contact mechanics, cubicweb, data model, debian, fortran, h5fs, h5py, hdf5, lamsid, lmgc, lmgc90, mercurial, python, sncf, sprint, swig | blogged in Logilab.org - enHi there,
I'm very pleased to announce the release of pylint 0.27 and
logilab-astng 0.24.2. There has been a lot of enhancements and
bug fixes since the latest release, so you're strongly encouraged
to upgrade. Here is a detailed list of changes:
- #20693: replace pylint.el by Ian Eure version (patch by J.Kotta)
- #105327: add support for --disable=all option and deprecate the
'disable-all' inline directive in favour of 'skip-file' (patch by
A.Fayolle)
- #110840: add messages I0020 and I0021 for reporting of suppressed
messages and useless suppression pragmas. (patch by Torsten Marek)
- #112728: add warning E0604 for non-string objects in __all__
(patch by Torsten Marek)
- #120657: add warning W0110/deprecated-lambda when a map/filter
of a lambda could be a comprehension (patch by Martin Pool)
- #113231: logging checker now looks at instances of Logger classes
in addition to the base logging module. (patch by Mike Bryant)
- #111799: don't warn about octal escape sequence, but warn about o
which is not octal in Python (patch by Martin Pool)
- #110839: bind <F5> to Run button in pylint-gui
- #115580: fix erroneous W0212 (access to protected member) on super call
(patch by Martin Pool)
- #110853: fix a crash when an __init__ method in a base class has been
created by assignment rather than direct function definition (patch by
Torsten Marek)
- #110838: fix pylint-gui crash when include-ids is activated (patch by
Omega Weapon)
- #112667: fix emission of reimport warnings for mixed imports and extend
the testcase (patch by Torsten Marek)
- #112698: fix crash related to non-inferable __all__ attributes and
invalid __all__ contents (patch by Torsten Marek)
- Python 3 related fixes:
- #110213: fix import of checkers broken with python 3.3, causing
"No such message id W0704" breakage
- #120635: redefine cmp function used in pylint.reporters
- Include full warning id for I0020 and I0021 and make sure to flush
warnings after each module, not at the end of the pylint run.
(patch by Torsten Marek)
- Changed the regular expression for inline options so that it must be
preceeded by a # (patch by Torsten Marek)
- Make dot output for import graph predictable and not depend
on ordering of strings in hashes. (patch by Torsten Marek)
- Add hooks for import path setup and move pylint's sys.path
modifications into them. (patch by Torsten Marek)
- pylint-brain: more subprocess.Popen faking (see #46273)
- #109562 [jython]: java modules have no __doc__, causing crash
- #120646 [py3]: fix for python3.3 _ast changes which may cause crash
- #109988 [py3]: test fixes
Many thanks to all the people who contributed to this release!
Enjoy!
I was in Bruxelles for FOSDEM 2013. As with previous FOSDEM there were too many
interesting talks and people to see. Here is a summary of what I saw:
In the Mozilla's room:
- The html5 pdf viewer pdfjs is impressive. The PDF specification is really
scary but this full featured "native" viewer is able to renders most of it
with very good performance. Have a look at the pdfjs demo!
- Firefox debug tools overview with a specific focus of Firefox OS emulator in
your browser.
- Introduction to webl10n: an internationalization format and library used in
Firefox OS. A successful mix that results in a format that is idiot-proof
enough for a duck to use, that relies on Unicode specifications to handle
complex pluralization rules and that allows cascading translation
definitions.
- Status of html5 video and audio support in Firefox. The topic looks like a
real headache but the team seems to be doing really well. Special mention
for the reverse demo effect: The speaker expected some format to be still
unsupported but someone else apparently implemented them over night.
- Last but not least I gave a talk about the changeset evolution concept that
I'm putting in Mercurial. Thanks goes to Feth for asking me his
not-scripted-at-all-questions during this talk. (slides)
In the postgresql room:
- Insightful talk about more event trigger in postgresql engine and how this may
becomes the perfect way to break your system.
- Full update of the capability of postgis 2.0. The postgis suite was already
impressive for storing and querying 2D data, but it now have impressive
capability regarding 3D data.
On python related topic:
- Victor Stinner has started an interesting project to improve CPython
performance. The first one: astoptimizer breaks some of the language
semantics to apply optimisation on compiling to byte code (lookup caching,
constant folding,…). The other, registervm is a full redefinition of how the interpreter
handles reference in byte code.
After the FOSDEM, I crossed the channel to attend a Mercurial sprint in London.
Expect more on this topic soon.
The Release candidate version of Mercurial 2.5 was released last sunday.
This new version makes a major change in the way "hidden" changesets are
handled. In 2.4 only hg log (and a few others) would support effectively
hiding "hidden" changesets. Now all hg commands are transparently compatible
with the hidden revision concept. This is a considerable step towards
changeset evolution, the next-generation collaboration technology that I'm
developing for Mercurial.
The 2.5 cycle is almost over, but there is no time to rest yet, Saturday the 2th of
February, I will give a talk about changeset evolution concept at FOSDEM in the
Mozilla Room. This talk in an updated version of the one I gave at
OSDC.fr 2012 (video in french).
The week after, I'm crossing the channel to attend the Mercurial 2.6 Sprint
hosted by Facebook London. I expect a lot of discussion about the user
interface and network access of changeset evolution.
|
[ 
