blog entries created by Sylvain Thenault

Pylint 1.2 released!

2014/04/22 by Sylvain Thenault

Once again, a lot of work has been achieved since the latest 1.1 release. Claudiu, who joined the maintainer team (Torsten and me) did a great work in the past few months. Also lately Torsten has backported a lot of things from their internal G[oogle]Pylint. Last but not least, various people contributed by reporting issues and proposing pull requests. So thanks to everybody!

Notice Pylint 1.2 depends on astroid 1.1 which has been released at the same time. Currently, code is available on Pypi, and Debian/Ubuntu packages should be ready shortly on Logilab's acceptance repositories.

Below is the changes summary, check the changelog for more info.

New and improved checks:

  • New message 'eval-used' checking that the builtin function eval was used.
  • New message 'bad-reversed-sequence' checking that the reversed builtin receive a sequence (i.e. something that implements __getitem__ and __len__, without being a dict or a dict subclass) or an instance which implements __reversed__.
  • New message 'bad-exception-context' checking that raise ... from ... uses a proper exception context (None or an exception).
  • New message 'abstract-class-instantiated' warning when abstract classes created with abc module and with abstract methods are instantied.
  • New messages checking for proper class __slots__: 'invalid-slots-object' and 'invalid-slots'.
  • New message 'undefined-all-variable' if a package's __all__ variable contains a missing submodule (#126).
  • New option logging-modules giving the list of module names that can be checked for 'logging-not-lazy'.
  • New option include-naming-hint to show a naming hint for invalid name (#138).
  • Mark file as a bad function when using python2 (#8).
  • Add support for enforcing multiple, but consistent name styles for different name types inside a single module.
  • Warn about empty docstrings on overridden methods.
  • Inspect arguments given to constructor calls, and emit relevant warnings.
  • Extend the number of cases in which logging calls are detected (#182).
  • Enhance the check for 'used-before-assignment' to look for nonlocal uses.
  • Improve cyclic import detection in the case of packages.

Bug fixes:

  • Do not warn about 'return-arg-in-generator' in Python 3.3+.
  • Do not warn about 'abstract-method' when the abstract method is implemented through assignment (#155).
  • Do not register most of the 'newstyle' checker warnings with python >= 3.
  • Fix 'unused-import' false positive with augment assignment (#78).
  • Fix 'access-member-before-definition' false negative with augment assign (#164).
  • Do not crash when looking for 'used-before-assignment' in context manager assignments (#128).
  • Do not attempt to analyze non python file, eg '.so' file (#122).
  • Pass the current python path to pylint process when invoked via epylint (#133).

Command line:

  • Add -i / --include-ids and -s / --symbols back as completely ignored options (#180).
  • Ensure init-hooks is evaluated before other options, notably load-plugins (#166).

Other:

  • Improve pragma handling to not detect 'pylint:*' strings in non-comments (#79).
  • Do not crash with UnknownMessage if an unknown message identifier/name appears in disable or enable in the configuration (#170).
  • Search for rc file in ~/.config/pylintrc if ~/.pylintrc doesn't exists (#121).
  • Python 2.5 support restored (#50 and #62).

Astroid:

  • Python 3.4 support
  • Enhanced support for metaclass
  • Enhanced namedtuple support

Nice easter egg, no?


Pylint 1.1 christmas release

2013/12/24 by Sylvain Thenault

Pylint 1.1 eventually got released on pypi!

A lot of work has been achieved since the latest 1.0 release. Various people have contributed to add several new checks as well as various bug fixes and other enhancement.

Here is the changes summary, check the changelog for more info.

New checks:

  • 'deprecated-pragma', for use of deprecated pragma directives "pylint:disable-msg" or "pylint:enable-msg" (was previously emmited as a regular warn().
  • 'superfluous-parens' for unnecessary parentheses after certain keywords.
  • 'bad-context-manager' checking that '__exit__' special method accepts the right number of arguments.
  • 'raising-non-exception' / 'catching-non-exception' when raising/catching class non inheriting from BaseException
  • 'non-iterator-returned' for non-iterators returned by '__iter__'.
  • 'unpacking-non-sequence' for unpacking non-sequences in assignments and 'unbalanced-tuple-unpacking' when left-hand-side size doesn't match right-hand-side.

Command line:

  • New option for the multi-statement warning to allow single-line if statements.
  • Allow to run pylint as a python module 'python -m pylint' (anatoly techtonik).
  • Various fixes to epylint

Bug fixes:

  • Avoid false used-before-assignment for except handler defined identifier used on the same line (#111).
  • 'useless-else-on-loop' not emited if there is a break in the else clause of inner loop (#117).
  • Drop 'badly-implemented-container' which caused several problems in its current implementation.
  • Don't mark input as a bad function when using python3 (#110).
  • Use attribute regexp for properties in python3, as in python2
  • Fix false-positive 'trailing-whitespace' on Windows (#55)

Other:

  • Replaced regexp based format checker by a more powerful (and nit-picky) parser, combining 'no-space-after-operator', 'no-space-after-comma' and 'no-space-before-operator' into a new warning 'bad-whitespace'.
  • Create the PYLINTHOME directory when needed, it might fail and lead to spurious warnings on import of pylint.config.
  • Fix setup.py so that pylint properly install on Windows when using python3.
  • Various documentation fixes and enhancements

Packages will be available in Logilab's Debian and Ubuntu repository in the next few weeks.

Happy christmas!


A quick take on continuous integration services for Bitbucket

2013/12/19 by Sylvain Thenault

Some time ago, we moved Pylint from this forge to Bitbucket (more on this here).

https://bitbucket-assetroot.s3.amazonaws.com/c/photos/2012/Oct/11/master-logo-2562750429-5_avatar.png

Since then, I somewhat continued to use the continuous integration (CI) service we provide on logilab.org to run tests on new commits, and to do the release job (publish a tarball on pypi, on our web site, build Debian and Ubuntu packages, etc.). This is fine, but not really handy since the logilab.org's CI service is not designed to be used for projects hosted elsewhere. Also I wanted to see what others have to offer, so I decided to find a public CI service to host Pylint and Astroid automatic tests at least.

Here are the results of my first swing at it. If you have others suggestions, some configuration proposal or whatever, please comment.

First, here are the ones I didn't test along with why:

The first one I actually tested, also the first one to show up when looking for "bitbucket continuous integration" on Google is https://drone.io. The UI is really simple, I was able to set up tests for Pylint in a matter of minutes: https://drone.io/bitbucket.org/logilab/pylint. Tests are automatically launched when a new commit is pushed to Pylint's Bitbucket repository and that setup was done automatically.

Trying to push Drone.io further, one missing feature is the ability to have different settings for my project, e.g. to launch tests on all the python flavor officially supported by Pylint (2.5, 2.6, 2.7, 3.2, 3.3, pypy, jython, etc.). Last but not least, the missing killer feature I want is the ability to launch tests on top of Pull Requests, which travis-ci supports.

Then I gave http://wercker.com a shot, but got stuck at the Bitbucket repository selection screen: none were displayed. Maybe because I don't own Pylint's repository, I'm only part of the admin/dev team? Anyway, wercker seems appealing too, though the configuration using yaml looks a bit more complicated than drone.io's, but as I was not able to test it further, there's not much else to say.

http://wercker.com/images/logo_header.png

So for now the winner is https://drone.io, but the first one allowing me to test on several Python versions and to launch tests on pull requests will be the definitive winner! Bonus points for automating the release process and checking test coverage on pull requests as well.

https://drone.io/drone3000/images/alien-zap-header.png

A retrospective of 10 years animating the pylint free software projet

2013/11/25 by Sylvain Thenault

was the topic of the talk I gave last saturday at the Capitol du Libre in Toulouse.

Here are the slides (pdf) for those interested (in french). A video of the talk should be available soon on the Capitol du Libre web site. The slides are mirrored on slideshare (see below):


Pylint 1.0 released!

2013/08/06 by Sylvain Thenault

Hi there,

I'm very pleased to announce, after 10 years of existence, the 1.0 release of Pylint.

This release has a hell long ChangeLog, thanks to many contributions and to the 10th anniversary sprint we hosted during june. More details about changes below.

Chances are high that your Pylint score will go down with this new release that includes a lot of new checks :) Also, there are a lot of improvments on the Python 3 side (notably 3.3 support which was somewhat broken).

You may download and install it from Pypi or from Logilab's debian repositories. Notice Pylint has been updated to use the new Astroid library (formerly known as logilab-astng) and that the logilab-common 0.60 library includes some fixes necessary for using Pylint with Python3 as well as long-awaited support for namespace packages.

For those interested, below is a comprehensive list of what changed:

Command line and output formating

  • A new --msg-template option to control output, deprecating "msvc" and "parseable" output formats as well as killing --include-ids and --symbols options.
  • Fix spelling of max-branchs option, now max-branches.
  • Start promoting usage of symbolic name instead of numerical ids.

New checks

  • "missing-final-newline" (C0304) for files missing the final newline.
  • "invalid-encoded-data" (W0512) for files that contain data that cannot be decoded with the specified or default encoding.
  • "bad-open-mode" (W1501) for calls to open (or file) that specify invalid open modes (Original implementation by Sasha Issayev).
  • "old-style-class" (C1001) for classes that do not have any base class.
  • "trailing-whitespace" (C0303) that warns about trailing whitespace.
  • "unpacking-in-except" (W0712) about unpacking exceptions in handlers, which is unsupported in Python 3.
  • "old-raise-syntax" (W0121) for the deprecated syntax raise Exception, args.
  • "unbalanced-tuple-unpacking" (W0632) for unbalanced unpacking in assignments (bitbucket #37).

Enhanced behaviours

  • Do not emit [fixme] for every line if the config value 'notes' is empty
  • Emit warnings about lines exceeding the column limit when those lines are inside multiline docstrings.
  • Name check enhancement:
    • simplified message,
    • don't double-check parameter names with the regex for parameters and inline variables,
    • don't check names of derived instance class members,
    • methods that are decorated as properties are now treated as attributes,
    • names in global statements are now checked against the regular expression for constants,
    • for toplevel name assignment, the class name regex will be used if pylint can detect that value on the right-hand side is a class (like collections.namedtuple()),
    • add new name type 'class_attribute' for attributes defined in class scope. By default, allow both const and variable names.
  • Add a configuration option for missing-docstring to optionally exempt short functions/methods/classes from the check.
  • Add the type of the offending node to missing-docstring and empty-docstring.
  • Do not warn about redefinitions of variables that match the dummy regex.
  • Do not treat all variables starting with "_" as dummy variables, only "_" itself.
  • Make the line-too-long warning configurable by adding a regex for lines for with the length limit should not be enforced.
  • Do not warn about a long line if a pylint disable option brings it above the length limit.
  • Do not flag names in nested with statements as undefined.
  • Remove string module from the default list of deprecated modules (bitbucket #3).
  • Fix incomplete-protocol false positive for read-only containers like tuple (bitbucket #25).

Other changes

  • Support for pkgutil.extend_path and setuptools pkg_resources (logilab-common #8796).
  • New utility classes for per-checker unittests in testutils.py
  • Added a new base class and interface for checkers that work on the tokens rather than the syntax, and only tokenize the input file once.
  • epylint shouldn't hang anymore when there is a large output on pylint'stderr (bitbucket #15).
  • Put back documentation in source distribution (bitbucket #6).

Astroid

  • New API to make it smarter by allowing transformation functions on any node, providing a register_transform function on the manager instead of the register_transformer to make it more flexible wrt node selection
  • Use this new transformation API to provide support for namedtuple (actually in pylint-brain, logilab-astng #8766)
  • Better description of hashlib
  • Properly recognize methods annotated with abc.abstract{property,method} as abstract.
  • Added the test_utils module for building ASTs and extracting deeply nested nodes for easier testing.

Astroid 1.0 released!

2013/08/02 by Sylvain Thenault

Astroid is the new name of former logilab-astng library. It's an AST library, used as the basis of Pylint and including Python 2.5 -> 3.3 compatible tree representation, statical type inference and other features useful for advanced Python code analysis, such as an API to provide extra information when statistical inference can't overcome Python dynamic nature (see the pylint-brain project for instance).

It has been renamed and hosted to bitbucket to make clear that this is not a Logilab dedicated project but a community project that could benefit to any people manipulating Python code (statistical analysis tools, IDE, browser, etc).

Documentation is a bit rough but should quickly improve. Also a dedicated web-site is now online, visit www.astroid.org (or https://bitbucket.org/logilab/astroid for development).

You may download and install it from Pypi or from Logilab's debian repositories.


PyLint 10th anniversary 1.0 sprint: day 3 - Sprint summary

2013/06/20 by Sylvain Thenault

Yesterday was the third and last day of the 10th anniversary Pylint sprint in Logilab's Toulouse office.

Design

To get started, we took advantage of this last day to have a few discussions about:

  • A "mode" feature gpylint has. It turns out that behind perhaps a few implementation details, this is something we definitly want into pylint (mode are specific configurations defined in the pylintrc and easilly recallable, they may even be specified per file).

  • How to avoid conflicts in the ChangeLog by using specific instruction in the commit message. We decided that a commit message should look like

    [my checker] do this and that. Closes #1234
    
    bla bla bla
    
    :release note: this will be a new item in the ChangeLog
    
    as well as anything until the end of the message
    

    now someone has to write the ChangeLog generation script so we may use this for post-1.0 releases

  • The roadmap. More on this later in this post.

Code

When we were not discussing, we were coding!

  • Anthony worked on having a template for the text reporter. His patch is available on Bitbucket but not yet integrated.
  • Julien and David pushed a bunch of patches on logilab-common, astroid and pylint for the Python 3.3 support. Not all tests are green on the pylint side, but much progress was done.
  • A couple other things were fixed, like a better "invalid name" message, stop complaining about string module being deprecated, etc.
  • A lot of patches have been integrated, from gpylint and others (e.g python 3 related)

All in all, an impressive amount of work was achieved during this sprint:

  • A lot of new checks or enhanced behaviour backported from gpylint (Take a look at Pylint's ChangeLog for more details on this, the list is impressively long).
  • The transformation API of astroid now allows to customize the tree structure as well as the inference process, hence to make pylint smarter than ever.
  • Better python 3 support.
  • A few bugs fixed and some enhancements added.
  • The templating stuff should land with the CLI cleanup (some output-formats will be removed as well as the --include-ids and --symbols option).
  • A lot of discussions, especially regarding the future community development of pylint/astroid on Bitbucket. Short summary being: more contributors and integrators are welcome! We should drop some note somewhere to describe how we are using bitbucket's pull requests and tracker.

Plan

Now here is the 1.O roadmap, which is expected by the begining of July:

  • Green tests under Python 3, including specification of Python version in message description (Julien).
  • Finish template for text reporters (Anthony).
  • Update web site (David).

And for later releases:

  • Backport mode from gpylint (Torsten).
  • Write ChangeLog update script (Sylvain).

So many thanks to everyone for this very successful sprint. I'm excited about this forthcoming 1.0 release!


PyLint 10th anniversary 1.0 sprint: day 2

2013/06/18 by Sylvain Thenault

Today was the second day of the 10th anniversary Pylint sprint in Logilab's Toulouse office.

This morning, we started with a presentation by myself about how the inference engine works in astroid (former astng). Then we started thinking all together about how we should change its API to be able to plug more information during the inference process. The first use-case we wanted to assert was namedtuple, as explained in http://www.logilab.org/ticket/8796.

We ended up by addressing it by:

  • enhancing the existing transformation feature so one may register a transformation function on any node rather than on a module node only;
  • being able to specify, on a node instance, a custom inference function to use instead of the default (class) implementation.

We would then be able to customize both the tree structure and the inference process and so to resolve the cases we were targeting.

Once this was sufficiently sketched out, everyone got his own tasks to do. Here is a quick summary of what has been achieved today:

  • Anthony resumed the check_messages thing and finished it for the simple cases, then he started on having a template for text reported,
  • Julien and David made a lot of progress on the Python 3.3 compatibility, though not enough to get the full green test suite,
  • Torsten continued backporting stuff from gpylint, all of them having been integrated by the end of the day,
  • Sylvain implemented the new transformation API and had the namedtuple proof of concept working, and even some documentation! Now this have to be tested for more real-world uses.

So things are going really well, and see you tomorrow for even more improvements to pylint!


PyLint 10th anniversary 1.0 sprint: day 1

2013/06/17 by Sylvain Thenault

Today was the first day of the Pylint sprint we organized using Pylint's 10th years anniversary as an excuse.

So I (Sylvain) have welcome my fellow Logilab friends David, Anthony and Julien as well as Torsten from Google into Logilab's new Toulouse office.

After a bit of presentation and talk about Pylint development, we decided to keep discussion for lunch and dinner and to setup priorities. We ended with the following tasks (picks from the pad at http://piratepad.net/oAvsUoGCAC):

  • rename astng to move it outside the logilab package,
  • Torsten gpylint (Google Pylint) patches review, as much as possible (but not all of them, starting by a review of the numberous internal checks Google has, seeing one by one which one should be backported upstream),
  • setuptools namespace package support (https://www.logilab.org/8796),
  • python 3.3 support,
  • enhance astroid (former astng) API to allow more ad-hoc customization for a better grasp of magic occuring in e.g. web frameworks (protocol buffer or SQLAlchemy may also be an application of this).

Regarding the astng renaming, we decided to move on with astroid as pointed out by the survey on StellarSurvey.com

In the afternoon, David and Julien tackled this, while Torsten was extracting patches from Google code and sending them to bitbucket as pulll request, Sylvain embrassing setuptools namespaces packages and Anthony discovering the code to spread the @check_message decorator usage.

By the end of the day:

  • David and Julien submitted patches to rename logilab.astng which were quickly integrated and now https://bitbucket.org/logilab/astroid should be used instead of https://bitbucket.org/logilab/astng
  • Torsten submitted 5 pull-requests with code extracted from gpylint, we reviewed them together and then Torsten used evolve to properly insert those in the pylint history once review comments were integrated
  • Sylvain submitted 2 patches on logilab-common to support both setuptools namespace packages and pkgutil.extend_path (but not bare __path__ manipulation
  • Anthony discovered various checkers and started adding proper @check_messages on visit methods

After doing some review all together, we even had some time to take a look at Python 3.3 support while writing this summary.

Hopefuly, our work on forthcoming days will be as efficient as on this first day!


Pylint 10th years anniversary from June 17 to 19 in Toulouse

2013/04/18 by Sylvain Thenault

After a quick survey, we're officially scheduling Pylint 10th years anniversary sprint from monday, June 17 to wednesday, June 19 in Logilab's Toulouse office.

There is still some room available if more people want to come, drop me a note (sylvain dot thenault at logilab dot fr).