Blog entries

  • Astng 0.20.0 and Pylint 0.20.0 releases

    2010/03/24 by Emile Anclin

    We are happy to announce the Astng 0.20.0 and Pylint 0.20.0 releases.

    Pylint is a static code checker based on Astng, both depending on logilab-common 0.49.

    Astng

    Astng 0.20.0 is a major refactoring: instead of parsing and modifying the syntax tree generated from python's _ast or compiler.ast modules, the syntax tree is rebuilt. Thus the code becomes much clearer, and all monkey patching will eventually disappear from this module.

    Speed improvement is achieved by caching the parsed modules earlier to avoid double parsing, and avoiding some repeated inferences, all along fixing a lot of important bugs.

    Pylint

    Pylint 0.20.0 uses the new Astng, and fixes a lot of bugs too, adding some new functionality:

    • parameters with leading "_" shouldn't count as "local" variables
    • warn on assert( a, b )
    • warning if return or break inside a finally
    • specific message for NotImplemented exception

    We would like to thank Chmouel Boudjnah, Johnson Fletcher, Daniel Harding, Jonathan Hartley, Colin Moris, Winfried Plapper, Edward K. Ream and Pierre Rouleau for their contributions, and all other people helping the project to progress.


  • Pylint 0.24 / logilab-astng 0.22

    2011/07/21 by Sylvain Thenault

    Hi there!

    I'm pleased to announce new releases of pylint and its underlying library logilab-astng. See http://www.logilab.org/project/pylint/0.24.0 and http://www.logilab.org/project/logilab-astng/0.22.0 for more info.

    Those releases include mostly fixes and a few enhancements. Python 2.6 relative / absolute imports should now work fine and Python 3 support has been enhanced. There are still two remaining failures in astng test suite when using python 3, but we're unfortunatly missing resources to fix them yet.

    Many thanks to everyone who contributed to this release by submitting patches or by participating to the latest bugs day.


  • Helping pylint to understand things it doesn't

    2011/10/10 by Sylvain Thenault

    The latest release of logilab-astng (0.23), the underlying source code representation library used by PyLint, provides a new API that may change pylint users' life in the near future...

    It aims to allow registration of functions that will be called after a module has been parsed. While this sounds dumb, it gives a chance to fix/enhance the understanding PyLint has about your code.

    I see this as a major step towards greatly enhanced code analysis, improving the situation where PyLint users know that when running it against code using their favorite framework (who said CubicWeb? :p ), they should expect a bunch of false positives because of black magic in the ORM or in decorators or whatever else. There are also places in the Python standard library where dynamic code can cause false positives in PyLint.

    The problem

    Let's take a simple example, and see how we can improve things using the new API. The following code:

    import hashlib
    
    def hexmd5(value):
        """"return md5 checksum hexadecimal digest of the given value"""
        return hashlib.md5(value).hexdigest()
    
    def hexsha1(value):
        """"return sha1 checksum hexadecimal digest of the given value"""
        return hashlib.sha1(value).hexdigest()
    

    gives the following output when analyzed through pylint:

    [syt@somewhere ~]$ pylint -E example.py
    No config file found, using default configuration
    ************* Module smarter_astng
    E:  5,11:hexmd5: Module 'hashlib' has no 'md5' member
    E:  9,11:hexsha1: Module 'hashlib' has no 'sha1' member
    

    However:

    [syt@somewhere ~]$ python
    Python 2.7.1+ (r271:86832, Apr 11 2011, 18:13:53)
    [GCC 4.5.2] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import smarter_astng
    >>> smarter_astng.hexmd5('hop')
    '5f67b2845b51a17a7751f0d7fd460e70'
    >>> smarter_astng.hexsha1('hop')
    'cffb6b20e0eef296772f6c1457cdde0049bdfb56'
    

    The code runs fine... Why does pylint bother me then? If we take a look at the hashlib module, we see that there are no sha1 or md5 defined in there. They are defined dynamically according to Openssl library availability in order to use the fastest available implementation, using code like:

    for __func_name in __always_supported:
        # try them all, some may not work due to the OpenSSL
        # version not supporting that algorithm.
        try:
            globals()[__func_name] = __get_hash(__func_name)
        except ValueError:
            import logging
            logging.exception('code for hash %s was not found.', __func_name)
    

    Honestly I don't blame PyLint for not understanding this kind of magic. The situation on this particular case could be improved, but that's some tedious work, and there will always be "similar but different" case that won't be understood.

    The solution

    The good news is that thanks to the new astng callback, I can help it be smarter! See the code below:

    from logilab.astng import MANAGER, scoped_nodes
    
    def hashlib_transform(module):
        if module.name == 'hashlib':
            for hashfunc in ('sha1', 'md5'):
                module.locals[hashfunc] = [scoped_nodes.Class(hashfunc, None)]
    
    def register(linter):
        """called when loaded by pylint --load-plugins, register our tranformation
        function here
        """
        MANAGER.register_transformer(hashlib_transform)
    

    What's in there?

    • A function that will be called with each astng module built during a pylint execution, i.e. not only the one that you analyses, but also those accessed for type inference.
    • This transformation function is fairly simple: if the module is the 'hashlib' module, it will insert into its locals dictionary a fake class node for each desired name.
    • It is registered using the register_transformer method of astng's MANAGER (the central access point to built syntax tree). This is done in the pylint plugin API register callback function (called when module is imported using 'pylint --load-plugins'.

    Now let's try it! Suppose I stored the above code in a 'astng_hashlib.py' module in my PYTHONPATH, I can now run pylint with the plugin activated:

    [syt@somewhere ~]$ pylint -E --load-plugins astng_hashlib example.py
    No config file found, using default configuration
    ************* Module smarter_astng
    E:  5,11:hexmd5: Instance of 'md5' has no 'hexdigest' member
    E:  9,11:hexsha1: Instance of 'sha1' has no 'hexdigest' member
    

    Huum. We have now a different error :( Pylint grasp there are some md5 and sha1 classes but it complains they don't have a hexdigest method. Indeed, we didn't give a clue about that.

    We could continue on and on to give it a full representation of hashlib public API using the astng nodes API. But that would be painful, trust me. Or we could do something clever using some higher level astng API:

    from logilab.astng import MANAGER
    from logilab.astng.builder import ASTNGBuilder
    
    def hashlib_transform(module):
        if module.name == 'hashlib':
        fake = ASTNGBuilder(MANAGER).string_build('''
    
    class md5(object):
      def __init__(self, value): pass
      def hexdigest(self):
        return u''
    
    class sha1(object):
      def __init__(self, value): pass
      def hexdigest(self):
        return u''
    
    ''')
        for hashfunc in ('sha1', 'md5'):
            module.locals[hashfunc] = fake.locals[hashfunc]
    
    def register(linter):
        """called when loaded by pylint --load-plugins, register our tranformation
        function here
        """
        MANAGER.register_transformer(hashlib_transform)
    

    The idea is to write a fake python implementation only documenting the prototype of the desired class, and to get an astng from it, using the string_build method of the astng builder. This method will return a Module node containing the astng for the given string. It's then easy to replace or insert additional information into the original module, as you can see in the above example.

    Now if I run pylint using the updated plugin:

    [syt@somewhere ~]$ pylint -E --load-plugins astng_hashlib example.py
    No config file found, using default configuration
    

    No error anymore, great!

    What's next?

    This fairly simple change could quickly provide great enhancements. We should probably improve the astng manipulation API now that it's exposed like that. But we can also easily imagine a code base of such pylint plugins maintained by each community around a python library or framework. One could then use a plugins stack matching stuff used by its software, and have a greatly enhanced experience of using pylint.

    For a start, it would be great if pylint could be shipped with a plugin that explains all the magic found in the standard library, wouldn't it? Left as an exercice to the reader!


  • PyLint 0.25.2 and related projects released

    2012/07/18 by Sylvain Thenault

    I'm pleased to announce the new release of Pylint and related projects (i.e. logilab-astng and logilab-common)!

    By installing PyLint 0.25.2, ASTNG 0.24 and logilab-common 0.58.1, you'll get a bunch of bug fixes and a few new features. Among the hot stuff:

    • PyLint should now work with alternative python implementations such as Jython, and at least go further with PyPy and IronPython (but those have not really been tested, please try it and provide feedback so we can improve their support)
    • the new ASTNG includes a description of dynamic code it is not able to understand. This is handled by a bitbucket hosted project described in another post.

    Many thanks to everyone who contributed to these releases, Torsten Marek / Boris Feld in particular (both sponsored by Google by the way, Torsten as an employee and Boris as a GSoC student).

    Enjoy!


  • Introducing the pylint-brain project

    2012/07/18 by Sylvain Thenault

    Huum, along with the new PyLint release, it's time to introduce the PyLint-Brain project I've recently started.

    Despite its name, PyLint-Brain is actually a collection of extensions for ASTNG, with the goal of making ASTNG smarter (and this directly benefits PyLint) by describing stuff that is too dynamic to be understood automatically (such as functions in the hashlib module, defaultdict, etc.).

    The PyLint-Brain collection of extensions is developped outside of ASTNG itself and hosted on a bitbucket project to ease community involvement and to allow distinct development cycles. Basically, ASTNG will include the PyLint-Brain extensions, but you may use earlier/custom versions by tweaking your PYTHONPATH.

    Take a look at the code, it's fairly easy to contribute new descriptions, and help us make pylint smarter!


  • PyLint 0.26 is out

    2012/10/08 by Sylvain Thenault

    I'm very pleased to announce new releases of Pylint and underlying ASTNG library, respectivly 0.26 and 0.24.1. The great news is that both bring a lot of new features and some bug fixes, mostly provided by the community effort.

    We're still trying to make it easier to contribute on our free software project at Logilab, so I hope this will continue and we'll get even more contritions in a near future, and an even smarter/faster/whatever pylint!

    For more details, see ChangeLog files or http://www.logilab.org/project/pylint/0.26.0 and http://www.logilab.org/project/logilab-astng/0.24.1

    So many thanks to all those who made that release, and enjoy!


  • Release of PyLint 0.27 / logilab-astng 0.24.2

    2013/02/28 by Sylvain Thenault

    Hi there,

    I'm very pleased to announce the release of pylint 0.27 and logilab-astng 0.24.2. There has been a lot of enhancements and bug fixes since the latest release, so you're strongly encouraged to upgrade. Here is a detailed list of changes:

    • #20693: replace pylint.el by Ian Eure version (patch by J.Kotta)
    • #105327: add support for --disable=all option and deprecate the 'disable-all' inline directive in favour of 'skip-file' (patch by A.Fayolle)
    • #110840: add messages I0020 and I0021 for reporting of suppressed messages and useless suppression pragmas. (patch by Torsten Marek)
    • #112728: add warning E0604 for non-string objects in __all__ (patch by Torsten Marek)
    • #120657: add warning W0110/deprecated-lambda when a map/filter of a lambda could be a comprehension (patch by Martin Pool)
    • #113231: logging checker now looks at instances of Logger classes in addition to the base logging module. (patch by Mike Bryant)
    • #111799: don't warn about octal escape sequence, but warn about o which is not octal in Python (patch by Martin Pool)
    • #110839: bind <F5> to Run button in pylint-gui
    • #115580: fix erroneous W0212 (access to protected member) on super call (patch by Martin Pool)
    • #110853: fix a crash when an __init__ method in a base class has been created by assignment rather than direct function definition (patch by Torsten Marek)
    • #110838: fix pylint-gui crash when include-ids is activated (patch by Omega Weapon)
    • #112667: fix emission of reimport warnings for mixed imports and extend the testcase (patch by Torsten Marek)
    • #112698: fix crash related to non-inferable __all__ attributes and invalid __all__ contents (patch by Torsten Marek)
    • Python 3 related fixes:
      • #110213: fix import of checkers broken with python 3.3, causing "No such message id W0704" breakage
      • #120635: redefine cmp function used in pylint.reporters
    • Include full warning id for I0020 and I0021 and make sure to flush warnings after each module, not at the end of the pylint run. (patch by Torsten Marek)
    • Changed the regular expression for inline options so that it must be preceeded by a # (patch by Torsten Marek)
    • Make dot output for import graph predictable and not depend on ordering of strings in hashes. (patch by Torsten Marek)
    • Add hooks for import path setup and move pylint's sys.path modifications into them. (patch by Torsten Marek)
    • pylint-brain: more subprocess.Popen faking (see #46273)
    • #109562 [jython]: java modules have no __doc__, causing crash
    • #120646 [py3]: fix for python3.3 _ast changes which may cause crash
    • #109988 [py3]: test fixes

    Many thanks to all the people who contributed to this release!

    Enjoy!


  • Astroid 1.0 released!

    2013/08/02 by Sylvain Thenault

    Astroid is the new name of former logilab-astng library. It's an AST library, used as the basis of Pylint and including Python 2.5 -> 3.3 compatible tree representation, statical type inference and other features useful for advanced Python code analysis, such as an API to provide extra information when statistical inference can't overcome Python dynamic nature (see the pylint-brain project for instance).

    It has been renamed and hosted to bitbucket to make clear that this is not a Logilab dedicated project but a community project that could benefit to any people manipulating Python code (statistical analysis tools, IDE, browser, etc).

    Documentation is a bit rough but should quickly improve. Also a dedicated web-site is now online, visit www.astroid.org (or https://bitbucket.org/logilab/astroid for development).

    You may download and install it from Pypi or from Logilab's debian repositories.