Logilab.org - en
News from Logilab and our Free Software projects, as well as on topics dear to our hearts (Python, Debian, Linux, the semantic web, scientific computing...)
Last week I attended DebConf14 in Portland, Oregon. As usual the conference was a blur, with lots of talks, lots of new people, and lots of old friends.
The organizers tried to do something different this year, with a longer conference (9 days instead of a week) and some dedicated hack time, instead of a pre-DebConf "DebCamp" week.
That worked quite well for me, as it meant the schedule was not quite so full with talks, and even though I didn't really get any hacking done, it felt a bit more relaxed and allowed some more hallway track discussions.
On the talks side, the keynotes from Zack and Biella provided some interesting thoughts. Some nice progress was made on making package builds reproducible.
I gave two talks: an introduction to salt (odp),
and a report on the Debian jessie release progress (pdf).
And as usual all talks were streamed live and recorded, and many are already available thanks to the awesome DebConf video team. Also for a change, and because I'm a sucker for punishment, I came back with more stuff to do.
Logilab is proud to contribute to the annual debian conference which will take place in Portland (USA) from the 23rd to the 31st of august.
Julien Cristau (debian page) will be giving two talks at the conference :
Logilab is also contributing to the conference as a sponsor for the event.
Here is what we previously blogged about salt and the previous debconf . Stay tuned for a blog post about what we saw and heard at the conference.
The EP14 Pylint sprint team (more on this here and there) is proud to announce they just released Pylint 1.3 together with its companion Astroid 1.2. As usual, this includes several new features as well and bug fixes. You'll find below some structured list of the changes.
Packages are uploaded to pypi, debian/ubuntu packages should be soon provided by Logilab, until they get into the standard packaging system of your favorite distribution.
Please notice Pylint 1.3 will be the last release branch support python 2.5 and 2.6. Starting from 1.4, we will only support python greater or equal to 2.7. This will be the occasion to do some great cleanup in the code base. Notice this is only about the Pylint's runtime, you should still be able to run Pylint on your Python 2.5 code, through using Python 2.7 at least.
- Add multiple checks for PEP 3101 advanced string formatting: 'bad-format-string', 'missing-format-argument-key', 'unused-format-string-argument', 'format-combined-specification', 'missing-format-attribute' and 'invalid-format-index'
- New 'invalid-slice-index' and 'invalid-sequence-index' for invalid sequence and slice indices
- New 'assigning-non-slot' warning, which detects assignments to attributes not defined in slots
- Fixed 'fixme' false positive (#149)
- Fixed 'unbalanced-iterable-unpacking' false positive when encountering starred nodes (#273)
- Fixed 'bad-format-character' false positive when encountering the 'a' format on Python 3
- Fixed 'unused-variable' false positive when the variable is assigned through an import (#196)
- Fixed 'unused-variable' false positive when assigning to a nonlocal (#275)
- Fixed 'pointless-string-statement' false positive for attribute docstrings (#193)
- Emit 'undefined-variable' when using the Python 3 metaclass= argument. Also fix 'unused-import' false for that construction (#143)
- Emit 'broad-except' and 'bare-except' even if the number of except handlers is different than 1. Fixes issue (#113)
- Emit 'attribute-defined-outside-init' for all statements in the same module as the offended class, not just for the last assignment (#262, as well as a long standing output mangling problem in some edge cases)
- Emit 'not-callable' when calling properties (#268)
- Don't let ImportError propagate from the imports checker, leading to crash in some namespace package related cases (#203)
- Don't emit 'no-name-in-module' for ignored modules (#223)
- Don't emit 'unnecessary-lambda' if the body of the lambda call contains call chaining (#243)
- Definition order is considered for classes, function arguments and annotations (#257)
- Only emit 'attribute-defined-outside-init' for definition within the same module as the offended class, avoiding to mangle the output in some cases
- Don't emit 'hidden-method' message when the attribute has been monkey-patched, you're on your own when you do that.
- Checkers are now properly ordered to respect priority(#229)
- Use the proper mode for pickle when opening and writing the stats file (#148)
- Function nodes can detect decorator call chain and see if they are decorated with builtin descriptors (classmethod and staticmethod).
- infer_call_result called on a subtype of the builtin type will now return a new Class rather than an Instance.
- Class.metaclass() now handles module-level __metaclass__ declaration on python 2, and no longer looks at the __metaclass__ class attribute on python 3.
- Add slots method to Class nodes, for retrieving the list of valid slots it defines.
- Expose function annotation to astroid: Arguments node exposes 'varargannotation', 'kwargannotation' and 'annotations' attributes, while Function node has the 'returns' attribute.
- Backported most of the logilab.common.modutils module there, as most things there are for pylint/astroid only and we want to be able to fix them without requiring a new logilab.common release
- Fix names grabed using wildcard import in "absolute import mode" (i.e. with absolute_import activated from the __future__ or with python 3) (pylint issue #58)
- Add support in brain for understanding enum classes.
Here are the list of things we managed to achieve during those last two days at EuroPython.
After several attempts, Michal managed to have pylint running analysis on several files in parallel. This is still in a pull request (https://bitbucket.org/logilab/pylint/pull-request/82/added-support-for-checking-files-in) because of some limitations, so we decided it won't be part of the 1.3 release.
Claudiu killed maybe 10 bugs or so and did some heavy issues cleanup in the trackers. He also demonstrated some experimental support of python 3 style annotation to drive a better inference. Pretty exciting!
Torsten also killed several bugs, restored python 2.5 compat (though that will need a logilab-common release as well), introduced a new functional test framework that will replace the old one once all the existing tests will be backported. On wednesday, he did show us a near future feature they already have at Google: some kind of confidence level associated to messages so that you can filter out based on that.
Sylvain fixed a couple of bugs (including https://bitbucket.org/logilab/pylint/issue/58/ which was annoying all the numpy community), started some refactoring of the PyLinter class so it does a little bit fewer things (still way too many though) and attempted to improve the pylint note on both pylint and astroid, which went down recently "thanks" to the new checks like 'bad-continuation'.
Also, we merged the pylint-brain project into astroid to simplify things, so you should now submit your brain plugins directly to the astroid project. Hopefuly you'll be redirected there on attempt to use the old (removed) pylint-brain project on bitbucket.
And, the good news is that now both Torsten and Claudiu have new powers: they should be able to do some releases of pylint and astroid. To celebrate that and the end of the sprint, we published Pylint 1.3 together with Astroid 1.2. More on this here.
We've had a fairly enjoyable and productive first day in our little hidden room at EuroPython in Berlin ! Below are some noticeable things we've worked on and discussed about.
First, we discussed and agreed that while we should at some point cut the cord to the logilab.common package, it will take some time notably because of the usage logilab.common.configuration which would be somewhat costly to replace (and is working pretty well). There are some small steps we should do but basically we should mostly get back some pylint/astroid specific things from logilab.common to astroid or pylint. This should be partly done during the sprint, and remaining work will go to tickets in the tracker.
We also discussed about release management. The point is that we should release more often, so every pylint maintainers should be able to do that easily. Sylvain will write some document about the release procedure and ensure access are granted to the pylint and astroid projects on pypi. We shall release pylint 1.3 / astroid 1.2 soon, and those releases branches will be the last one supporting python < 2.7.
During this first day, we also had the opportunity to meet Carl Crowder, the guy behind http://landscape.io, as well as David Halter which is building the Jedi completion library (https://github.com/davidhalter/jedi). Landscape.io runs pylint on thousands of projects, and it would be nice if we could test beta release on some part of this panel. On the other hand, there are probably many code to share with the Jedi library like the parser and ast generation, as well as a static inference engine. That deserves a sprint on his own though, so we agreed that a nice first step would be to build a common library for import resolution without relying on the python interpreter for that, while handling most of the python dark import features like zip/egg import, .pth files and so one. Indeed that may be two nice future collaborations!
Last but not least, we got some actual work done:
- Michal Nowikowski from Intel in Poland joined us to work on the ability to run pylint in different processes so it may drastically improve performance on multiple cores box.
- Torsten did continue some work on various improvements of the functionnal test framework.
- Sylvain did merge logilab.common.modutils module into astroid as it's mostly driven by astroid and pylint needs. Also fixed the annoying namespace package crash.
- Claudiu keep up the good work he does daily at improving and fixing pylint :)
We have just published the following ipython notebooks explaining how to perform record linkage and entities matching with Nazca:
I was at the Open Legislative Data Conference on may 28 2014 in Paris, to present a simple demo I worked on since the same event that happened two years ago.
The demo was called "Law is Code Rebooted with CubicWeb". It featured the use of the cubicweb-vcreview component to display the amendments of the hospital law ("loi hospitalière") gathered into a version control system (namely Mercurial).
The basic idea is to compare writing code and writing law, for both are collaborative and distributed writing processes. Could we reuse for the second one the tools developed for the first?
Here are the slides and a few screenshots.
Statistics with queries embedded in report page.
List of amendments.
User comment on an amendment.
While attending the conference, I enjoyed several interesting talks and chats with other participants, including:
- the study of co-sponsorship of proposals in the french parliament
- data.senat.fr announcing their use of PostgreSQL and JSON.
- and last but not least, the great work done by RegardsCitoyens and SciencesPo MediaLab on visualizing the law making process.
Thanks to the organisation team and the other speakers. Hope to see you again!
This monday (19th of may 2014), Thomas Hatch was in Paris for
dotScale 2014. After presenting SaltStack there (videos will be published at some
point), he spent the evening with members of the French SaltStack community
during a meetup set up by Logilab at IRILL.
Here is a list of what we talked about :
- Since Salt seems to have pushed ZMQ
to its limits, SaltStack has been working on RAET
(Reliable Asynchronous Event Transport Protocol ), a transport layer based on UDP and
elliptic curve cryptography (Dan Berstein's CURVE-255-19) that works more like a
stack than a socket and has reliability built in. RAET will be released as an
optionnal beta feature in the next Salt release.
- Folks from Dailymotion bumped into a bug that seems related to high
latency networks and the auth_timeout. Updating to the very latest
release should fix the issue.
- Thomas told us about how a dedicated team at SaltStack handles pull
requests and another team works on triaging github issues to input them
into their internal SCRUM process. There are a lot of duplicate issues and old inactive
issues that need attention and clutter the issue tracker. Help will be welcome.
- Continuous integration is based on Jenkins and spins up VMs to test pull request.
There is work in progress to test multiple clouds, various latencies and loads.
- For the Docker integration, salt now keeps track of forwarded ports
and relevant information about the containers.
- salt-virt bumped into problems with chroots and timeouts due to ZMQ.
- Multi-master: the problem lies with syncronisation of data which is
sent to minions but also the data that is sent to the
masters. Possible solutions to be explored are : the use of gitfs,
there is no built-in solution for keys (salt-key has to be run on
all masters), mine.send should send the data at both masters,
for the jobs cache: one could use an external returner.
- Thomas talked briefly about ioflo which should bring queuing,
data hierarchy and data pub-sub to Salt.
- About the rolling release question: versions in Salt are definitely
not git snapshots, things get backported into previous versions. No
clear definition yet of length of LTS versions.
- salt-cloud and libcloud : in the next release, libcloud will not be
a hard dependency. Some clouds didn't work in libcloud (for example
AWS), so these providers got implemented directly in salt-cloud or by using
third-party libraries (eg. python-boto).
- Documentation: a sprint is planned next week. Reference
documentation will not be completly revamped, but tutorial content
will be added.
Boris Feld showed a demo of vagrant images orchestrated by salt and a web UI
to monitor a salt install.
Thanks again to Thomas Hatch for coming and meeting up with (part of) the
community here in France.
On the 15th of april, in Paris (France), we took part in yet another Salt meetup. The community is now meeting up once every two months.
We had two presentations:
- Arthur Lutz made an introduction to returners and the scheduler using the SalMon monitoring system as an example. Salt is not only about configuration management Indeed!
- The folks from Is Cool Entertainment did a presentation about how they are using salt-cloud to deploy and orchestrate clusters of EC2 machines (islands in their jargon) to reproduce parts of their production environment for testing and developement.
More discussions about various salty subjects followed and were pursued in an Italian restaurant (photos here).
In case it is not already in your diary : Thomas Hatch is coming to Paris next week, on Monday the 19th of May, and will be speaking at dotscale during the day and at a Salt meetup in the evening. The Salt Meetup will take place at IRILL (like the previous meetups, thanks again to them) and should start at 19h. The meetup is free and open to the public, but registering on this framadate would be appreciated.
Once again, a lot of work has been achieved since the latest 1.1
release. Claudiu, who joined the maintainer team (Torsten and me) did a
great work in the past few months. Also lately Torsten has backported a lot of
things from their internal G[oogle]Pylint. Last but not least, various people contributed by
reporting issues and proposing pull requests. So thanks to everybody!
Notice Pylint 1.2 depends on astroid 1.1 which has been released at the same
time. Currently, code is available on Pypi, and Debian/Ubuntu packages should be ready shortly on Logilab's acceptance repositories.
Below is the changes summary, check the changelog for more info.
New and improved checks:
- New message 'eval-used' checking that the builtin function eval was used.
- New message 'bad-reversed-sequence' checking that the reversed
builtin receive a sequence (i.e. something that implements __getitem__ and __len__, without being
a dict or a dict subclass) or an instance which implements __reversed__.
- New message 'bad-exception-context' checking that raise ... from ... uses a proper exception context (None or an exception).
- New message 'abstract-class-instantiated' warning when abstract classes created with abc module and with abstract methods are instantied.
- New messages checking for proper class __slots__: 'invalid-slots-object' and 'invalid-slots'.
- New message 'undefined-all-variable' if a package's __all__ variable contains a missing submodule (#126).
- New option logging-modules giving the list of module names that can be checked for 'logging-not-lazy'.
- New option include-naming-hint to show a naming hint for invalid name (#138).
- Mark file as a bad function when using python2 (#8).
- Add support for enforcing multiple, but consistent name styles for different
name types inside a single module.
- Warn about empty docstrings on overridden methods.
- Inspect arguments given to constructor calls, and emit relevant warnings.
- Extend the number of cases in which logging calls are detected (#182).
- Enhance the check for 'used-before-assignment' to look for nonlocal uses.
- Improve cyclic import detection in the case of packages.
- Do not warn about 'return-arg-in-generator' in Python 3.3+.
- Do not warn about 'abstract-method' when the abstract method is implemented through assignment (#155).
- Do not register most of the 'newstyle' checker warnings with python >= 3.
- Fix 'unused-import' false positive with augment assignment (#78).
- Fix 'access-member-before-definition' false negative with augment assign (#164).
- Do not crash when looking for 'used-before-assignment' in context manager assignments (#128).
- Do not attempt to analyze non python file, eg '.so' file (#122).
- Pass the current python path to pylint process when invoked via epylint (#133).
- Add -i / --include-ids and -s / --symbols back as completely ignored options (#180).
- Ensure init-hooks is evaluated before other options, notably load-plugins (#166).
- Improve pragma handling to not detect 'pylint:*' strings in non-comments (#79).
- Do not crash with UnknownMessage if an unknown message identifier/name appears in
disable or enable in the configuration (#170).
- Search for rc file in ~/.config/pylintrc if ~/.pylintrc doesn't exists (#121).
- Python 2.5 support restored (#50 and #62).
- Python 3.4 support
- Enhanced support for metaclass
- Enhanced namedtuple support
Nice easter egg, no?