subscribe to this blog

Logilab.org - en

News from Logilab and our Free Software projects, as well as on topics dear to our hearts (Python, Debian, Linux, the semantic web, scientific computing...)

show 204 results
  • Helping pylint to understand things it doesn't

    2011/10/10 by Sylvain Thenault

    The latest release of logilab-astng (0.23), the underlying source code representation library used by PyLint, provides a new API that may change pylint users' life in the near future...

    It aims to allow registration of functions that will be called after a module has been parsed. While this sounds dumb, it gives a chance to fix/enhance the understanding PyLint has about your code.

    I see this as a major step towards greatly enhanced code analysis, improving the situation where PyLint users know that when running it against code using their favorite framework (who said CubicWeb? :p ), they should expect a bunch of false positives because of black magic in the ORM or in decorators or whatever else. There are also places in the Python standard library where dynamic code can cause false positives in PyLint.

    The problem

    Let's take a simple example, and see how we can improve things using the new API. The following code:

    import hashlib
    
    def hexmd5(value):
        """"return md5 checksum hexadecimal digest of the given value"""
        return hashlib.md5(value).hexdigest()
    
    def hexsha1(value):
        """"return sha1 checksum hexadecimal digest of the given value"""
        return hashlib.sha1(value).hexdigest()
    

    gives the following output when analyzed through pylint:

    [syt@somewhere ~]$ pylint -E example.py
    No config file found, using default configuration
    ************* Module smarter_astng
    E:  5,11:hexmd5: Module 'hashlib' has no 'md5' member
    E:  9,11:hexsha1: Module 'hashlib' has no 'sha1' member
    

    However:

    [syt@somewhere ~]$ python
    Python 2.7.1+ (r271:86832, Apr 11 2011, 18:13:53)
    [GCC 4.5.2] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import smarter_astng
    >>> smarter_astng.hexmd5('hop')
    '5f67b2845b51a17a7751f0d7fd460e70'
    >>> smarter_astng.hexsha1('hop')
    'cffb6b20e0eef296772f6c1457cdde0049bdfb56'
    

    The code runs fine... Why does pylint bother me then? If we take a look at the hashlib module, we see that there are no sha1 or md5 defined in there. They are defined dynamically according to Openssl library availability in order to use the fastest available implementation, using code like:

    for __func_name in __always_supported:
        # try them all, some may not work due to the OpenSSL
        # version not supporting that algorithm.
        try:
            globals()[__func_name] = __get_hash(__func_name)
        except ValueError:
            import logging
            logging.exception('code for hash %s was not found.', __func_name)
    

    Honestly I don't blame PyLint for not understanding this kind of magic. The situation on this particular case could be improved, but that's some tedious work, and there will always be "similar but different" case that won't be understood.

    The solution

    The good news is that thanks to the new astng callback, I can help it be smarter! See the code below:

    from logilab.astng import MANAGER, scoped_nodes
    
    def hashlib_transform(module):
        if module.name == 'hashlib':
            for hashfunc in ('sha1', 'md5'):
                module.locals[hashfunc] = [scoped_nodes.Class(hashfunc, None)]
    
    def register(linter):
        """called when loaded by pylint --load-plugins, register our tranformation
        function here
        """
        MANAGER.register_transformer(hashlib_transform)
    

    What's in there?

    • A function that will be called with each astng module built during a pylint execution, i.e. not only the one that you analyses, but also those accessed for type inference.
    • This transformation function is fairly simple: if the module is the 'hashlib' module, it will insert into its locals dictionary a fake class node for each desired name.
    • It is registered using the register_transformer method of astng's MANAGER (the central access point to built syntax tree). This is done in the pylint plugin API register callback function (called when module is imported using 'pylint --load-plugins'.

    Now let's try it! Suppose I stored the above code in a 'astng_hashlib.py' module in my PYTHONPATH, I can now run pylint with the plugin activated:

    [syt@somewhere ~]$ pylint -E --load-plugins astng_hashlib example.py
    No config file found, using default configuration
    ************* Module smarter_astng
    E:  5,11:hexmd5: Instance of 'md5' has no 'hexdigest' member
    E:  9,11:hexsha1: Instance of 'sha1' has no 'hexdigest' member
    

    Huum. We have now a different error :( Pylint grasp there are some md5 and sha1 classes but it complains they don't have a hexdigest method. Indeed, we didn't give a clue about that.

    We could continue on and on to give it a full representation of hashlib public API using the astng nodes API. But that would be painful, trust me. Or we could do something clever using some higher level astng API:

    from logilab.astng import MANAGER
    from logilab.astng.builder import ASTNGBuilder
    
    def hashlib_transform(module):
        if module.name == 'hashlib':
        fake = ASTNGBuilder(MANAGER).string_build('''
    
    class md5(object):
      def __init__(self, value): pass
      def hexdigest(self):
        return u''
    
    class sha1(object):
      def __init__(self, value): pass
      def hexdigest(self):
        return u''
    
    ''')
        for hashfunc in ('sha1', 'md5'):
            module.locals[hashfunc] = fake.locals[hashfunc]
    
    def register(linter):
        """called when loaded by pylint --load-plugins, register our tranformation
        function here
        """
        MANAGER.register_transformer(hashlib_transform)
    

    The idea is to write a fake python implementation only documenting the prototype of the desired class, and to get an astng from it, using the string_build method of the astng builder. This method will return a Module node containing the astng for the given string. It's then easy to replace or insert additional information into the original module, as you can see in the above example.

    Now if I run pylint using the updated plugin:

    [syt@somewhere ~]$ pylint -E --load-plugins astng_hashlib example.py
    No config file found, using default configuration
    

    No error anymore, great!

    What's next?

    This fairly simple change could quickly provide great enhancements. We should probably improve the astng manipulation API now that it's exposed like that. But we can also easily imagine a code base of such pylint plugins maintained by each community around a python library or framework. One could then use a plugins stack matching stuff used by its software, and have a greatly enhanced experience of using pylint.

    For a start, it would be great if pylint could be shipped with a plugin that explains all the magic found in the standard library, wouldn't it? Left as an exercice to the reader!


  • Text mode makes it into hgview 1.4.0

    2011/10/06 by Alain Leufroy

    Here is at last the release of the version 1.4.0 of hgview.

    http://www.logilab.org/image/77974?vid=download

    Small description

    Besides the classic bugfixes this release introduces a new text based user interface thanks to the urwid library.

    Running hgview in a shell, in a terminal, over a ssh session is now possible! If you are trying not to use X (or use it less), have a geek mouse-killer window manager such as wmii/dwm/ion/awesome/... this is for you!

    This TUI (Text User Interface!) adopts the principal features of the Qt4 based GUI. Although only the main view has been implemented for now.

    In a nutshell, this interface includes the following features :

    • display the revision graph (with working directory as a node, and basic support for the mq extension),
    • display the files affected by a selected changeset (with basic support for the bfiles extension)
    • display diffs (with syntax highlighting thanks to pygments),
    • automatically refresh the displayed revision graph when the repository is being modified (requires pyinotify),
    • easy key-based navigation in revisions' history of a repo (same as the GUI),
    • a command system for special actions (see help)

    Installation

    There are packages for debian and ubuntu in the logilab's debian repository.

    Note:you have to install the hgview-curses package to get the text based interface.

    Or you can simply clone our Mercurial repository:

    hg clone http://hg.logilab.org/hgview

    (more on the hgview home page)

    Running the text based interface

    A new --interface option is now available to choose the interface:

    hgview --interface curses

    Or you can fix it in the [hgview] section of your ~/.hgrc:

    [hgview]
    interface = curses # or qt or raw
    

    Then run:

    hgview

    What's next

    We'll be working on including other features from the Qt4 interface and making it fully configurable.

    We'll also work on bugfixes and new features, so stay tuned! And feel free to file bugs and feature requests.


  • Drawing UML diagrams with Python

    2011/09/26 by Nicolas Chauvat
    http://www.umlgraph.org/doc/seq-eg.gif?vid=download

    It started with a desire to draw diagrams of hierarchical systems with Python. Since this is similar to what we do in CubicWeb with schemas of the data model, I read the code and realized we had that graph submodule in the logilab.common library. This module uses dot from graphviz as a backend to draw the diagrams.

    Reading about UML diagrams drawn with GraphViz, I learned about UMLGraph, that uses GNU Pic to draw sequence diagrams. Pic is a language based on groff and the pic2plot tool is part of plotutils (apt-get install plotutils). Here is a tutorial. I have found some Python code wrapping pic2plot available as plugin to wikipad. It is worth noticing that TeX seems to have a nice package for UML sequence diagrams called pgf-umlsd.

    Since nowadays everything is moving into the web browser, I looked for a javascript library that does what graphviz does and I found canviz which looks nice.

    If (only) I had time, I would extend pyreverse to draw sequence diagrams and not only class diagrams...


  • EuroSciPy'11 - Annual European Conference for Scientists using Python.

    2011/08/24 by Alain Leufroy
    http://www.logilab.org/image/9852?vid=download

    The EuroScipy2011 conference will be held in Paris at the Ecole Normale Supérieure from August 25th to 28th and is co-organized and sponsored by INRIA, Logilab and other companies.

    The conference is dedicated to cross-disciplinary gathering focused on the use and development of the Python language in scientific research.

    August 25th and 26th are dedicated to tutorial tracks -- basic and advanced tutorials. August 27th and 28th are dedicated to talks, posters and demos sessions.

    Damien Garaud, Vincent Michel and Alain Leufroy (and others) from Logilab will be there. We will talk about a RSS feeds aggregator based on Scikits.learn and CubicWeb and we have a poster about LibAster (a python library for thermomechanical simulation based on Code_Aster).


  • Pylint 0.24 / logilab-astng 0.22

    2011/07/21 by Sylvain Thenault

    Hi there!

    I'm pleased to announce new releases of pylint and its underlying library logilab-astng. See http://www.logilab.org/project/pylint/0.24.0 and http://www.logilab.org/project/logilab-astng/0.22.0 for more info.

    Those releases include mostly fixes and a few enhancements. Python 2.6 relative / absolute imports should now work fine and Python 3 support has been enhanced. There are still two remaining failures in astng test suite when using python 3, but we're unfortunatly missing resources to fix them yet.

    Many thanks to everyone who contributed to this release by submitting patches or by participating to the latest bugs day.


  • pylint bug day #3 on july 8, 2011

    2011/07/04 by Sylvain Thenault

    Hey guys,

    we'll hold the next pylint bug day on july 8th 2011 (friday). If some of you want to come and work with us in our Paris office, you'll be welcome.

    You can also join us on jabber / irc:

    I know the announce is a bit late, but I hope some of you will be able to come or be online anyway!

    Regarding the program, the goal is to decrease the number of tickets in the tracker. I'll try to do some triage earlier this week so you'll get a chance to talk about your super-important ticket that has not been selected. Of course, if you intend to work on it, there is a bigger chance of it being fixed next week-end ;)


  • Setting up my Microsoft Natural Keyboard under Debian Squeeze

    2011/06/08 by Nicolas Chauvat

    I upgraded to Debian Squeeze over the week-end and it broke my custom Xmodmap. While I was fixing it, I realized that the special keys of my Microsoft Natural keyboard that were not working under Lenny were now functionnal. The only piece missing was the "zoom" key. Here is how I got it to work.

    I found on the askubuntu forum an solution to the same problem, that is missing the following details.

    To find which keysym to map, I listed input devices:

    $ ls /dev/input/by-id/
    usb-Logitech_USB-PS.2_Optical_Mouse-mouse        usb-Logitech_USB-PS_2_Optical_Mouse-mouse
    usb-Logitech_USB-PS_2_Optical_Mouse-event-mouse  usb-Microsoft_Natural??_Ergonomic_Keyboard_4000-event-kbd
    

    then used evtest to find the keysym:

    $ evtest /dev/input/by-id/usb-Microsoft*
    

    then used udevadm to find the identifiers:

    $ udevadm info --export-db | less
    

    then edited /lib/udev/rules.d/95-keymap.rules to add:

    ENV{ID_VENDOR}=="Microsoft", ENV{ID_MODEL_ID}=="00db", RUN+="keymap $name microsoft-natural-keyboard-4000"
    

    in the section keyboard_usbcheck

    and created the keymap file:

    $ cat /lib/udev/keymaps/microsoft-natural-keyboard-4000
    0xc022d pageup
    0xc022e pagedown
    

    then loaded the keymap:

    $ /lib/udev/keymap /dev/input/by-id/usb-Microsoft_Natural®_Ergonomic_Keyboard_4000-event-kbd /lib/udev/keymaps/microsoft-natural-keyboard-4000
    

    then used evtest again to check it was working.

    Of course, you do not have to map the events to pageup and pagedown, but I found it convenient to use that key to scroll up and down pages.

    Hope this helps :)


  • Coding sprint scikits.learn

    2011/03/22 by Vincent Michel

    We are planning a one day coding sprint on scikits.learn the 1st April.
    Venues, or remote participation on IRC are more than welcome !

    More information can be found on the wiki:
    https://github.com/scikit-learn/scikit-learn/wiki/Upcoming-events


  • Distutils2 Sprint at Logilab (first day)

    2011/01/28 by Alain Leufroy

    We're very happy to host the Distutils2 sprint this week in Paris.

    The sprint has started yesterday with some of Logilab's developers and others contributors. We'll sprint during 4 days, trying to pull up the new python package manager.

    Let's sumarize this first day:

    • Boris Feld and Pierre-Yves David worked on the new system for detecting and dispatching data-files.
    • Julien Miotte worked on
      • moving qGitFilterBranch from setuptools to distutils2
      • testing distutils2 installation and register (see the tutorial)
      • the backward compatibility to distutils in setup.py, using setup.cfg to fill the setup arguments of setup for helping users to switch to distutils2.
    • André Espaze and Alain Leufroy worked on the python script that help developers build a setup.cfg by recycling their existing setup.py (track).

    Join us on IRC at #distutils on irc.freenode.net !


  • The Python Package Index is not a "Software Distribution"

    2011/01/26 by Pierre-Yves David

    Recent discussions on the #disutils irc channel and with my logilab co-workers led me to the following conclusions:

    • The Python Package Index is not a software distribution
    • There is more than one way to distribute python software
    • Distribution packagers are power users and need super cow-powers
    • Users want it to "just works"
    • The Python Package Index is used by many as a software distribution
    • Pypi has a lot of contributions because requirements are low.

    The Python Package Index is not a software distribution

    I would define a software distribution as :

    • Organised group of people
    • Who apply a Unified Quality process
    • To a finite set of software
    • Which includes all its dependencies
    • With a consistent set of versions that work together
    • For a finite set of platforms
    • Managed and installed by dedicated tools.

    Pypi is a public index where:

    • Any python developer
    • Can upload any tarball containing something related
    • To any python package
    • Which might have external dependencies (outside Pypi)
    • The latest version of something is always available disregarding its compatibility with other packages.
    • Binary packages can be provided for any platform but are usually not.
    • There are several tools to install and manage python packages from pypi.

    Pypi is not a software distribution, it is a software index.

    Card File by Mr. Ducke / Matt

    There is more than one way to distribute python software

    There is a long way from the pure source used by the developer to the software installed on the system of the end user.

    First, the source must be extracted from a (D)VCS to make a version tarball, while executing several release specific actions (eg: changelog generation from a tracker) Second, the version tarball is used to generate a platform independent build, while executing several build steps (eg, Cython compilation into C files or documentation generation). Third, the platform independent build is used to generate a platform dependant build, while executing several platforms dependant build (eg, compilation of C extension). Finally, the platform dependant build is installed and each file gets dispatched to its proper location during the installation process.

    Pieces of software can be distributed as development snapshots taken from the (D)VCS, version tarballs, source packages, platform independent package or platform dependent package.

    package! by Beck Gusler

    Distribution packagers are power users and need super cow-powers

    Distribution packagers usually have the necessary infrastructure and skills to build packages from version tarballs. Moreover they might have specific needs that require as much control as possible over the various build steps. For example:

    • Specific help system requiring a custom version of sphinx.
    • Specific security or platform constraint that require a specific version of Cython
    Cheese Factory by James Yu

    Users want it to "just work"

    Standard users want it to "just work". They prefer simple and quick ways to install stuff. Build steps done on their machine increase the duration of the installation, add potential new dependencies and may trigger an error. Standard users are very disappointed when an installed failed because an error occurred while building the documentation. User give up when they have to download extra dependency and setup complicated compilation environment.

    Users want as many build steps as possible to be done by someone else. That's why many users usually choose a distribution that do the job for them (eg, ubuntu, red-hat, python xy)

    The Python Package Index is used by many as a software distribution

    But there are several situations where the user can't rely on his distribution to install python software:

    • There is no distribution available for the platform (Windows, Mac OS X)
    • They want to install a python package outside of their distribution system (to test or because they do not have the credentials to install it system-wide)
    • The software or version they need is not included in the finite set of software included in their distribution.

    When this happens, the user will use Pypi to fetch python packages. To help them, Pypi accepts binary packages of python modules and people have developed dedicated tools that ease installation of packages and their dependencies: pip, easy_install.

    Pip + Pypi provides the tools of a distribution without its consistency. This is better than nothing.

    Pypi has a lot of contributions because requirements are low

    Pypi should contain version tarballs of all known python modules. It is the first purpose of an index. Version tarball should let distribution and power user perform as many build steps as possible. Pypi will continue to be used as a distribution by people without a better option. Packages provided to these users should require as little as possible to be installed, meaning they either have no build step to perform or have only platforms dependent build step (that could not be executed by the developer).

    Thomas Fisher Rare Book Library by bookchen

    If the incoming distutils2 provides a way to differentiate platform dependent build steps from platform independent ones, python developers will be able to upload three different kind of package on Pypi.

    sdist:Pure source version released by upstream targeted at packagers and power users.
    idist:Platform-independent package with platform independent build steps done (Cython, docs). If there is no such build step, the package is the same as sdist.
    bdist:Platform-dependent package with all build steps performed. For package with no platform dependent build step this package is the same that idist.

    (Image under creative commons Card File by-nc-nd by Mr. Ducke / Matt, Thomas Fisher Rare Book Library by bookchen, package! by Beck Gusler, Cheese Factory by James Yu)


show 204 results