Update: LAX is now included in the CubicWeb semantic web framework.
Update: LAX is now included in the CubicWeb semantic web framework.
The problem is that we don't have enough free time at Logilab to lower the number of tickets in pylint tracker page . If you take a look at the ticket tab, you'll see a lot of pendings bug and must-have features (well, and some other less necessary...). You can already easily contribute thanks to the great mercurial dvcs, and some of you do, either by providing patches or by reporting bugs (more tickets, iiirk ! ;) Thank you all btw !!
Now I was wondering what could be done to make pylint going further, and the first ideas which came to my mind was :
But for this to be useful, we need your support, so here are some questions for you:
you may answer by adding a comment to this blog (please register first by using the link at the top right of this page) or by mail to email@example.com. If we've enough positive answers, we'll take the time to organize such a thing.
Il n'en demeure pas moins qu'à Logilab, nous manquons de temps pour faire baisser la pile de tickets ouverts dans le tracker de pylint. Si vous jetez un œuil dans l'onglet Tickets, vous y trouverez un grand nombre de bogues en souffrance et de fonctionalités indispensables (certaines peut-être un peu moins que d'autres...) Il est déjà possible de contribuer en utilisant mercurial pour fournir des patches, ou en signalant des bogues (aaaaaaaaaarg ! encore des tickets !) et certains s'y sont mis, qu'ils en soient remerciés.
Maintenant, nous nous demandions ce que nous pourrions faire pour faire avance Pylint, et nos premières idées sont :
Mais pour que ça soit utile, nous avons besoin de votre aide. Voici donc quelques questions :
Vous pouvez répondre en commentant sur ce blog (pensez à vous enregistrer en utilisant le lien en haut à droite sur cette page) ou en écrivant à firstname.lastname@example.org. Si nous avons suffisamment de réponses positives nous organiserons quelque chose.
The first pylint bug day took place on wednesday 25th. Four members of the Logilab crew and two other people spent the day working on pylint.
Several patches submitted before the bug day were processed and some tickets were closed.
Charles Hébert added James Lingard's patches for string formatting and is working on several improvements. Vincent Férotin submitted a patch for simple message listings. Sylvain Thenault fixed significant inference bugs in astng (an underlying module of pylint managing the syntax tree). Émile Anclin began a major astng refactoring to take advantage of new python2.6 functionality. For my part, I made several improvements to the test suite. I applied James Lingard patches for ++ operator and generalised it to -- too. I also added a new checker for function call arguments submitted by James Lingard once again. Finally I improved the message filtering of the --errors-only options.
We thank Maarten ter Huurne, Vincent Férotin for their participation and of course James Lingard for submitting numerous patches.
Another pylint bug day will be held in a few months.
We're very happy to be hosting the next mercurial sprint in our brand new offices in central Paris. It is quite an honor to be chosen when the other contender was Google.
So a bunch of mercurial developers are heading out to our offices this coming Friday to sprint for three days on mercurial. We use mercurial a lot here over at Logilab and we also contribute a tool to visualize and manipulate a mercurial repository : hgview.
To check out the things that we will be working on with the mercurial crew, check out the program of the sprint on their wiki.
What is a sprint? "A sprint (sometimes called a Code Jam or hack-a-thon) is a short time period (three to five days) during which software developers work on a particular chunk of functionality. "The whole idea is to have a focused group of people make progress by the end of the week," explains Jeff Whatcott" [source]. For geographically distributed open source communities, it is also a way of physically meeting and working in the same room for a period of time.
Sprinting is a practice that we encourage at Logilab, with CubicWeb we organize as often as possible open sprints, which is an opportunity for users and developers to come and code with us. We even use the sprint format for some internal stuff.
Else you can still join us on jabber / irc:
See you then!
First of all, I've to say that pylint bugs day wasn't that successful in term of 'community event': I've been sprinting almost alone. My Logilab's felows were tied to customer projects, and no outside people shown up on jabber. Fortunatly Tarek Ziade came to visit us, and that was a nice opportunity to talk about pylint, distribute, etc ... Thank you Tarek, you saved my day ;)
As I felt a bit alone, I decided to work on somethings funnier than bug fixing: refactoring!
First, I've greatly simplified the command line: enable-msg/enable-msg-cat/enable-checker/enable-report and their disable-* counterparts were all merged into single --enable/--disable options.
I've also simplified "pylint --help" output, providing a --long-help option to get what we had before. Generic support in `logilab.common.configuration of course.
And last but not least, I refactored pylint so we can have multiple checkers with the same name. The idea behind this is that we can split checker into smaller chunks, basically only responsible for one or a few related messages. When pylint runs, it only uses necessary checkers according to activated messages and reports. When all checkers will be splitted, it should improve performance of "pylint --error-only".
So, I can say I'm finally happy with the results of that pylint bugs day! And hopefuly we will be more people for the next edition...
Ce sprint aura lieu du jeudi 29 avril 2010 au 5 mai 2010 (weekend exlus - les bureaux seront fermés). Vous êtes les bienvenus pour contribuer, filer un coup de main, faire du pair programming... ou simplement rencontrer des développeurs cubicweb. Vous pouvez même venir une après-midi ou une seule journée. Pour ceux avec des portables, il y aura du réseau disponible pour être connecté.
Adresse : 104 Boulevard Auguste-Blanqui, Paris. Sonnez à "Logilab".
Métro : St Jacques or Corvisart (Glacière est la station la plus proche mais sera fermée à partir de lundi)
Contact : http://www.logilab.fr/contact
Dates : du 29/04/2010 au 30/04/2010 et du 03/05/2010 au 05/05/2010
At Logilab, we have the pleasure to host a distutils2 sprint in January. Sprinters are welcome in our Paris office from 9h on the 27th of January to 19h the 30th of January. This sprint will focus on polishing distutils2 for the next alpha release and on the install/remove scripts.
Distutils2 is an important project for Python. Every contribution will help to improve the current state of packaging in Python. See the wiki page on python.org for details about participation. If you can't attend or join us in Paris, you can participate on the #distutils channel of the freenode irc network
We're very happy to host the Distutils2 sprint this week in Paris.
The sprint has started yesterday with some of Logilab's developers and others contributors. We'll sprint during 4 days, trying to pull up the new python package manager.
Let's sumarize this first day:
Join us on IRC at #distutils on irc.freenode.net !
we'll hold the next pylint bug day on july 8th 2011 (friday). If some of you want to come and work with us in our Paris office, you'll be welcome.
You can also join us on jabber / irc:
I know the announce is a bit late, but I hope some of you will be able to come or be online anyway!
Regarding the program, the goal is to decrease the number of tickets in the tracker. I'll try to do some triage earlier this week so you'll get a chance to talk about your super-important ticket that has not been selected. Of course, if you intend to work on it, there is a bigger chance of it being fixed next week-end ;)
I'm now at Copenhagen to attend the mercurial "2.3" sprint.
About twenty people are attending, including staff from Atlassian, Facebook, Google and Mozilla.
I expect code and discussion about various topic among:
I'm of course very interested in the Obsolete Markers topic. I've been working on an experimental implementation for several months. An handful of people are using them at Logilab for two months and feedback are very promising.
I'm now back from Copenhagen were I attended the mercurial 2.3 sprint with twenty other people. A huge amount of work was done in a very friendly atmosphere.
Regarding mercurial's core:
Regarding the bundled extension :
Regarding the project itself:
Regarding other extensions:
And I'm probably forgetting some stuff. Special thanks to Unity3D for hosting the sprint and providing power, network and food during these 3 days.
From June 24th to June 26th, the European Synchrotron organises a workshop centered around Debian. On Monday, a number of talks about the use of Debian in scientific facilities will be featured. On Sunday and Tuesday, members of the Debian Science group will meet for a sprint focusing on the upcoming Debian 7.0 release.
Among the speakers will be Stefano Zacchiroli, the current Debian project leader. Logilab will be present with Nicolas Chauvat at Monday's conference, and Julien Cristau at both the sprint and the conference.
At the sprint we'll be discussing packaging of scientific libraries such as blas or MPI implementations, and working on polishing other scientific packages, such as python-related ones (including Salome on which we are currently working).
Les débutants sont bienvenus, une introduction au code de Pylint sera réalisée en début de sprint. Une expérience avec le module ast de la librairie standard est un plus.
The Release candidate version of Mercurial 2.5 was released last sunday.
This new version makes a major change in the way "hidden" changesets are handled. In 2.4 only hg log (and a few others) would support effectively hiding "hidden" changesets. Now all hg commands are transparently compatible with the hidden revision concept. This is a considerable step towards changeset evolution, the next-generation collaboration technology that I'm developing for Mercurial.
The 2.5 cycle is almost over, but there is no time to rest yet, Saturday the 2th of February, I will give a talk about changeset evolution concept at FOSDEM in the Mozilla Room. This talk in an updated version of the one I gave at OSDC.fr 2012 (video in french).
The week after, I'm crossing the channel to attend the Mercurial 2.6 Sprint hosted by Facebook London. I expect a lot of discussion about the user interface and network access of changeset evolution.
In a few week, pylint will be 10 years old (0.1 released on may 19 2003!). At this occasion, I would like to release a 1.0. Well, not exactly at that date, but not too long after would be great. Also, I think it would be a good time to have a few days sprint to work a bit on this 1.0 but also to meet all together and talk about pylint status and future, as more and more contributions come from outside Logilab (actually mostly Google, which employs Torsten and Martin, the most active contributors recently).
The first thing to do is to decide a date and place. Having discussed a bit with Torsten about that, it seems reasonable to target a sprint during june or july. Due to personal constraints, I would like to host this sprint in Logilab's Toulouse office.
So, who would like to jump in and sprint to make pylint even better? I've created a doodle so every one interested may tell his preferences: http://doodle.com/4uhk26zryis5x7as
Regarding the location, is everybody ok with Toulouse? Other ideas are Paris, or Florence around EuroPython, or... <add your proposition here>.
We'll talk about the sprint topics later, but there are plenty of exciting ideas around there.
Please, answer quickly so we can move on. And I hope to see you all there!
LMGC90 Sprint at Logilab
At the end of March 2013, Logilab hosted a sprint on the LMGC90 simulation code in Paris.
LMGC90 is an open-source software developed at the LMGC ("Laboratoire de Mécanique et Génie Civil" -- "Mechanics and Civil Engineering Laboratory") of the CNRS, in Montpellier, France. LMGC90 is devoted to contact mechanics and is, thus, able to model large collections of deformable or undeformable physical objects of various shapes, with numerous interaction laws. LMGC90 also allows for multiphysics coupling.
More than ten hackers joined in from:
After a great introduction to LMGC90 by Frédéric Dubois and some preliminary discussions, teams were quickly constituted around the common areas of interest.
As of the sprint date, LMGC90 is mainly developed in Fortran, but also contains Python code for two purposes:
The main drawback of this approach is the double modelling of data that this architecture implies: once in the core and once in Pre_LMGC.
It was decided to build a unique user-level Python layer on top of ChiPy, that would be able to build the computational problem description and write the DATBOX input files (currently achieved by using Pre_LMGC), as well as to drive the simulation and read the OUTBOX result files (currently by using direct ChiPy calls).
This task has been met with success, since, in the short time span available (half a day, basically), the team managed to build some object types using ChiPy calls and save them into a DATBOX.
This topic involved importing LMGC90 DATBOX data into the numerical platform developed by Logilab for the SNCF.
This was achieved using ChiPy as a Python API to the Fortran core to get:
There is still a lot of work to be done (notably regarding the charges applied to the bodies), but this is already a great achievement. This could only have occured in a sprint, were every needed expertise is available:
The main point of this topic was to replace the in-house DATBOX/OUTBOX textual format used by LMGC90 to store input and output data, with an open, standard and efficient format.
Several formats have been considered, like HDF5, MED and NetCDF4.
MED has been ruled out for the moment, because it lacks the support for storing body contact information. HDF5 was chosen at last because of the quality of its Python libraries, h5py and pytables, and the ease of use tools like h5fs provide.
Alain Leufroy from Logilab quickly presented h5py and h5fs usage, and the team started its work, measuring the performance impact of the storage pattern of LMGC90 data. This was quickly achieved, as the LMGC experts made it easy to setup tests of various sizes, and as the Logilab developers managed to understand the concepts and implement the required code in a fast and agile way.
This topic turned out to be more difficult than initially assessed, mainly because LMGC90 has dependencies to non-packaged external libraries, which thus had to be packaged first:
Logilab kept working on this after the sprint and produced packages that are currently being tested by the LMGC team. Some changes are expected (for instance, Python modules should be prefixed with a proper namespace) before the packages can be submitted for inclusion into Debian. The expertise of Logilab regarding Debian packaging was of great help for this task. This will hopefully help to spread the use of LMGC90.
As you may know, Logilab is really fond of Mercurial as a DVCS. Our company invested a lot into the development of the great evolve extension, which makes Mercurial a very powerful tool to efficiently manage the team development of software in a clean fashion.
This is why Logilab presented Mercurial's features and advantages over the current VCS used to manage LMGC90 sources, namely svn, to the other participants of the Sprint. This was appreciated and will hopefully benefit to LMGC90 ease of development and spread among the Open Source community.
All in all, this two-day sprint on LMGC90, involving participants from several industrial and academic institutions has been a great success. A lot of code has been written but, more importantly, several stepping stones have been laid, such as:
Colaterally somehow, several other results have also been achieved:
On a final note, one would say that we greatly appreciated the cooperation between the participants, which we found pleasant and efficient. We look forward to finding more occasions to work together.
After a quick survey, we're officially scheduling Pylint 10th years anniversary sprint from monday, June 17 to wednesday, June 19 in Logilab's Toulouse office.
There is still some room available if more people want to come, drop me a note (sylvain dot thenault at logilab dot fr).
Today was the first day of the Pylint sprint we organized using Pylint's 10th years anniversary as an excuse.
So I (Sylvain) have welcome my fellow Logilab friends David, Anthony and Julien as well as Torsten from Google into Logilab's new Toulouse office.
After a bit of presentation and talk about Pylint development, we decided to keep discussion for lunch and dinner and to setup priorities. We ended with the following tasks (picks from the pad at http://piratepad.net/oAvsUoGCAC):
Regarding the astng renaming, we decided to move on with astroid as pointed out by the survey on StellarSurvey.com
In the afternoon, David and Julien tackled this, while Torsten was extracting patches from Google code and sending them to bitbucket as pulll request, Sylvain embrassing setuptools namespaces packages and Anthony discovering the code to spread the @check_message decorator usage.
By the end of the day:
After doing some review all together, we even had some time to take a look at Python 3.3 support while writing this summary.
Hopefuly, our work on forthcoming days will be as efficient as on this first day!
Today was the second day of the 10th anniversary Pylint sprint in Logilab's Toulouse office.
This morning, we started with a presentation by myself about how the inference engine works in astroid (former astng). Then we started thinking all together about how we should change its API to be able to plug more information during the inference process. The first use-case we wanted to assert was namedtuple, as explained in http://www.logilab.org/ticket/8796.
We ended up by addressing it by:
We would then be able to customize both the tree structure and the inference process and so to resolve the cases we were targeting.
Once this was sufficiently sketched out, everyone got his own tasks to do. Here is a quick summary of what has been achieved today:
So things are going really well, and see you tomorrow for even more improvements to pylint!
Yesterday was the third and last day of the 10th anniversary Pylint sprint in Logilab's Toulouse office.
To get started, we took advantage of this last day to have a few discussions about:
When we were not discussing, we were coding!
All in all, an impressive amount of work was achieved during this sprint:
Now here is the 1.O roadmap, which is expected by the begining of July:
And for later releases:
So many thanks to everyone for this very successful sprint. I'm excited about this forthcoming 1.0 release!
The focus of our Logilab team will probably be Test-Driven System Administration with Salt, but the more participants and the topics, the merrier the event.
Please register if you plan on joining us. We will be happy to meet with fellow hackers.
Last Friday, we hosted the French event for the international Great Salt Sprint. Here is a report on what was done and discussed on this occasion.
We started off by discussing various points that were of interest to the participants :
We then split up into pairs to tackle issues in small groups, with some general discussions from time to time.
Unfortunately we had a major electricity black out all afternoon, some of us switched to battery and 3G tethering to carry on, but that couldn't last all afternoon. We ended up talking about design and use cases. ERDF (French electricity distribution company) ended up bringing generator trucks for the neighborhood !
Some unfinished draft code for supervision backends was written and pushed on github. We explored how a common "interface" could be done in salt (using a combination of states and __virtual___). The official documentation was often very useful, reading code was also always a good resource (and the code is really readable).
While we were fixing stuff because of the power black out, Benoit submitted a bug fix.
Design was done around which information should be extracted and display and how to configure access control to the salt-master, taking a further look to external_auth and salt-api will probably be the way forward.
We had general discussions around concepts of access control to a salt master, on how to define this access. One of the things we believe to be missing (but haven't checked thoroughly) is the ability to separate the "read-only" operations to the "read-write" operations in states and modules, if this was done (through decorators?) we could easily tell salt-api to only give access to data collection. Complex scenarios of access were discussed. Having a configuration or external_auth based on ssh public keys (similar to mercurial-server would be nice, and would provide a "limited" shell to a mercurial server.
The power black out didn't help us get things done, but nevertheless, some sharing was done around our uses cases around SaltStack and features that we'd want to get out of it (or from third party applications). We hope to convert all the discussions into bug reports or further discussion on the mailing-lists and (obviously) into code and pull-requests. Check out the scoreboard for an overview of how the other cities contributed.
to comment this post you need to login or create an account
As usual the first two days were dedicated to tutorials while the last two ones were dedicated to scientific presentations and general python related talks. The meeting was extended by one more day for sprint sessions during which enthusiasts were able to help free software projects, namely sage, vispy and scipy.
Jérôme and I had the great opportunity to represent Logilab during the scientific tracks and the sprint day. We enjoyed many talks about scientific applications using python. We're not going to describe the whole conference. Visit the conference website if you want the complete list of talks. In this article we will try to focus on the ones we found the most interesting.
First of all the keynote by Cameron Neylon about Network ready research was very interesting. He presented some graphs about the impact of a group job on resolving complex problems. They revealed that there is a critical network size for which the effectiveness for solving a problem drastically increase. He pointed that the source code accessibility "friction" limits the "getting help" variable. Open sourcing software could be the best way to reduce this "friction" while unit testing and ongoing integration are facilitators. And, in general, process reproducibility is very important, not only in computing research. Retrieving experimental settings, metadata, and process environment is vital. We agree with this as we are experimenting it everyday in our work. That is why we encourage open source licenses and develop a collaborative platform that provides the distributed simulation traceability and reproducibility platform Simulagora (in french).
Ian Ozsvald's talk dealt with key points and tips from his own experience to grow a business based on open source and python, as well as mistakes to avoid (e.g. not checking beforehand there are paying customers interested by what you want to develop). His talk was comprehensive and mentioned a wide panel of situations.
We got a very nice presentation of a young but interesting visualization tools: Vispy. It is 6 months old and the first public release was early August. It is the result of the merge of 4 separated libraries, oriented toward interactive visualisation (vs. static figure generation for Matplotlib) and using OpenGL on GPUs to avoid CPU overload. A demonstration with large datasets showed vispy displaying millions of points in real time at 40 frames per second. During the talk we got interesting information about OpenGL features like anti-grain compared to Matplotlib Agg using CPU.
We also got to learn about cartopy which is an open source Python library originally written for weather and climate science. It provides useful and simple API to manipulate cartographic mapping.
Distributed computing systems was a hot topic and many talks were related to this theme.
Gael Varoquaux reminded us what are the keys problems with "biggish data" and the key points to successfully process them. I think that some of his recommendations are generally useful like "choose simple solutions", "fail gracefully", "make it easy to debug". For big data processing when I/O limit is the constraint, first try to split the problem into random fractions of the data, then run algorithms and aggregate the results to circumvent this limit. He also presented mini-batch that takes a bunch of observations (trade-off memory usage/vectorization) and joblib.parallel that makes I/O faster using compression (CPUs are faster than disk access).
Benoit Da Mota talked about shared memory in parallel computing and Antonio Messina gave us a quick overview on how to build a computing cluster with Elasticluster, using OpenStack/Slurm/ansible. He demonstrated starting and stopping a cluster on OpenStack: once all VMs are started, ansible configures them as hosts to the cluster and new VMs can be created and added to the cluster on the fly thanks to a command line interface.
We also got a keynote by Peter Wang (from Continuum Analytics) about the future of data analysis with Python. As a PhD in physics I loved his metaphor of giving mass to data. He tried to explain the pain that scientists have when using databases.
After the conference we participated to the numpy/scipy sprint. It was organized by Ralph Gommers and Pauli Virtanen. There were 18 people trying to close issues from different difficulty levels and had a quick tutorial on how easy it is to contribute: the easiest is to fork from the github project page on your own github account (you can create one for free), so that later your patch submission will be a simple "Pull Request" (PR). Clone locally your scipy fork repository, and make a new branch (git checkout -b <newbranch>) to tackle one specific issue. Once your patch is ready, commit it locally, push it on your github repository and from the github interface choose "Push request". You will be able to add something to your commit message before your PR is sent and looked at by the project lead developers. For example using "gh-XXXX" in your commit message will automatically add a link to the issue no. XXXX. Here is the list of open issues for scipy; you can filter them, e.g. displaying only the ones considered easy to fix :D
For more information: Contributing to SciPy.
The three main maintainers/developpers of Pylint/astroid (Claudiu, Torsten and I) will meet together in Berlin during EuroPython 2014. While this is not an "official" EuroPython sprint but it's still worth announcing it since it's a good opportunity to meet and enhance Pylint. We should find place and time to work on Pylint between wednesday 23 and friday 25.
If you're interested, don't hesitate to contact me (email@example.com / @sythenault).
We've had a fairly enjoyable and productive first day in our little hidden room at EuroPython in Berlin ! Below are some noticeable things we've worked on and discussed about.
First, we discussed and agreed that while we should at some point cut the cord to the logilab.common package, it will take some time notably because of the usage logilab.common.configuration which would be somewhat costly to replace (and is working pretty well). There are some small steps we should do but basically we should mostly get back some pylint/astroid specific things from logilab.common to astroid or pylint. This should be partly done during the sprint, and remaining work will go to tickets in the tracker.
We also discussed about release management. The point is that we should release more often, so every pylint maintainers should be able to do that easily. Sylvain will write some document about the release procedure and ensure access are granted to the pylint and astroid projects on pypi. We shall release pylint 1.3 / astroid 1.2 soon, and those releases branches will be the last one supporting python < 2.7.
During this first day, we also had the opportunity to meet Carl Crowder, the guy behind http://landscape.io, as well as David Halter which is building the Jedi completion library (https://github.com/davidhalter/jedi). Landscape.io runs pylint on thousands of projects, and it would be nice if we could test beta release on some part of this panel. On the other hand, there are probably many code to share with the Jedi library like the parser and ast generation, as well as a static inference engine. That deserves a sprint on his own though, so we agreed that a nice first step would be to build a common library for import resolution without relying on the python interpreter for that, while handling most of the python dark import features like zip/egg import, .pth files and so one. Indeed that may be two nice future collaborations!
Last but not least, we got some actual work done:
Here are the list of things we managed to achieve during those last two days at EuroPython.
After several attempts, Michal managed to have pylint running analysis on several files in parallel. This is still in a pull request (https://bitbucket.org/logilab/pylint/pull-request/82/added-support-for-checking-files-in) because of some limitations, so we decided it won't be part of the 1.3 release.
Claudiu killed maybe 10 bugs or so and did some heavy issues cleanup in the trackers. He also demonstrated some experimental support of python 3 style annotation to drive a better inference. Pretty exciting! Torsten also killed several bugs, restored python 2.5 compat (though that will need a logilab-common release as well), introduced a new functional test framework that will replace the old one once all the existing tests will be backported. On wednesday, he did show us a near future feature they already have at Google: some kind of confidence level associated to messages so that you can filter out based on that. Sylvain fixed a couple of bugs (including https://bitbucket.org/logilab/pylint/issue/58/ which was annoying all the numpy community), started some refactoring of the PyLinter class so it does a little bit fewer things (still way too many though) and attempted to improve the pylint note on both pylint and astroid, which went down recently "thanks" to the new checks like 'bad-continuation'.
Also, we merged the pylint-brain project into astroid to simplify things, so you should now submit your brain plugins directly to the astroid project. Hopefuly you'll be redirected there on attempt to use the old (removed) pylint-brain project on bitbucket.
And, the good news is that now both Torsten and Claudiu have new powers: they should be able to do some releases of pylint and astroid. To celebrate that and the end of the sprint, we published Pylint 1.3 together with Astroid 1.2. More on this here.