subscribe to this blog

Logilab.org - en

News from Logilab and our Free Software projects, as well as on topics dear to our hearts (Python, Debian, Linux, the semantic web, scientific computing...)

show 200 results
  • Astroid 1.0 released!

    2013/08/02 by Sylvain Thenault

    Astroid is the new name of former logilab-astng library. It's an AST library, used as the basis of Pylint and including Python 2.5 -> 3.3 compatible tree representation, statical type inference and other features useful for advanced Python code analysis, such as an API to provide extra information when statistical inference can't overcome Python dynamic nature (see the pylint-brain project for instance).

    It has been renamed and hosted to bitbucket to make clear that this is not a Logilab dedicated project but a community project that could benefit to any people manipulating Python code (statistical analysis tools, IDE, browser, etc).

    Documentation is a bit rough but should quickly improve. Also a dedicated web-site is now online, visit www.astroid.org (or https://bitbucket.org/logilab/astroid for development).

    You may download and install it from Pypi or from Logilab's debian repositories.


  • Going to DebConf13

    2013/08/01 by Julien Cristau

    The 14th Debian developers conference (DebConf13) will take place between August 11th and August 18th in Vaumarcus, Switzerland.

    Logilab is a DebConf13 sponsor, and I'll attend the conference. There are quite a lot of cloud-related events on the schedule this year, plus the usual impromptu discussions and hallway track. Looking forward to meeting the usual suspects there!

    https://www.logilab.org/file/158611/raw/dc13-btn0-going-bg.png

  • We hosted the Salt Sprint in Paris

    2013/07/30 by Arthur Lutz

    Last Friday, we hosted the French event for the international Great Salt Sprint. Here is a report on what was done and discussed on this occasion.

    http://www.logilab.org/file/228931/raw/saltstack_logo.jpg

    We started off by discussing various points that were of interest to the participants :

    • automatically write documentation from salt sls files (for Sphinx)
    • salt-mine add security layer with restricted access (bug #5467 and #6437)
    • test compatibility of salt-cloud with openstack
    • module bridge bug correction : traceback on KeyError
    • setting up the network in debian (equivalent of rh_ip)
    • configure existing monitoring solution through salt (add machines, add checks, etc) on various backends with a common syntax

    We then split up into pairs to tackle issues in small groups, with some general discussions from time to time.

    6 people participated, 5 from Logilab, 1 from nbs-system. We were expecting more participants but some couldn't make it at the last minute, or though the sprint was taking place at some other time.

    Unfortunately we had a major electricity black out all afternoon, some of us switched to battery and 3G tethering to carry on, but that couldn't last all afternoon. We ended up talking about design and use cases. ERDF (French electricity distribution company) ended up bringing generator trucks for the neighborhood !

    Arthur & Benoit : monitoring adding machines or checks

    http://www.logilab.org/file/157971/raw/salt-centreon-shinken.png

    Some unfinished draft code for supervision backends was written and pushed on github. We explored how a common "interface" could be done in salt (using a combination of states and __virtual___). The official documentation was often very useful, reading code was also always a good resource (and the code is really readable).

    While we were fixing stuff because of the power black out, Benoit submitted a bug fix.

    David & Alain : generate documentation from salt state & salt master

    The idea is to couple the SLS description and the current state of the salt master to generate documentation about one's infrastructure using Sphinx. This was transmitted to the mailing-list.

    http://www.logilab.org/file/157976/raw/salt-sphinx.png

    Design was done around which information should be extracted and display and how to configure access control to the salt-master, taking a further look to external_auth and salt-api will probably be the way forward.

    General discussions

    We had general discussions around concepts of access control to a salt master, on how to define this access. One of the things we believe to be missing (but haven't checked thoroughly) is the ability to separate the "read-only" operations to the "read-write" operations in states and modules, if this was done (through decorators?) we could easily tell salt-api to only give access to data collection. Complex scenarios of access were discussed. Having a configuration or external_auth based on ssh public keys (similar to mercurial-server would be nice, and would provide a "limited" shell to a mercurial server.

    Conclusion

    The power black out didn't help us get things done, but nevertheless, some sharing was done around our uses cases around SaltStack and features that we'd want to get out of it (or from third party applications). We hope to convert all the discussions into bug reports or further discussion on the mailing-lists and (obviously) into code and pull-requests. Check out the scoreboard for an overview of how the other cities contributed.

    to comment this post you need to login or create an account


  • The Great Salt Sprint Paris Location is Logilab

    2013/07/12 by Nicolas Chauvat
    http://farm1.static.flickr.com/183/419945378_4ead41a76d_m.jpg

    We're happy to be part of the second Great Salt Sprint that will be held at the end of July 2013. We will be hosting the french sprinters on friday 26th in our offices in the center of Paris.

    The focus of our Logilab team will probably be Test-Driven System Administration with Salt, but the more participants and the topics, the merrier the event.

    Please register if you plan on joining us. We will be happy to meet with fellow hackers.

    photo by Sebastian Mary under creative commons licence.


  • PyLint 10th anniversary 1.0 sprint: day 3 - Sprint summary

    2013/06/20 by Sylvain Thenault

    Yesterday was the third and last day of the 10th anniversary Pylint sprint in Logilab's Toulouse office.

    Design

    To get started, we took advantage of this last day to have a few discussions about:

    • A "mode" feature gpylint has. It turns out that behind perhaps a few implementation details, this is something we definitly want into pylint (mode are specific configurations defined in the pylintrc and easilly recallable, they may even be specified per file).

    • How to avoid conflicts in the ChangeLog by using specific instruction in the commit message. We decided that a commit message should look like

      [my checker] do this and that. Closes #1234
      
      bla bla bla
      
      :release note: this will be a new item in the ChangeLog
      
      as well as anything until the end of the message
      

      now someone has to write the ChangeLog generation script so we may use this for post-1.0 releases

    • The roadmap. More on this later in this post.

    Code

    When we were not discussing, we were coding!

    • Anthony worked on having a template for the text reporter. His patch is available on Bitbucket but not yet integrated.
    • Julien and David pushed a bunch of patches on logilab-common, astroid and pylint for the Python 3.3 support. Not all tests are green on the pylint side, but much progress was done.
    • A couple other things were fixed, like a better "invalid name" message, stop complaining about string module being deprecated, etc.
    • A lot of patches have been integrated, from gpylint and others (e.g python 3 related)

    All in all, an impressive amount of work was achieved during this sprint:

    • A lot of new checks or enhanced behaviour backported from gpylint (Take a look at Pylint's ChangeLog for more details on this, the list is impressively long).
    • The transformation API of astroid now allows to customize the tree structure as well as the inference process, hence to make pylint smarter than ever.
    • Better python 3 support.
    • A few bugs fixed and some enhancements added.
    • The templating stuff should land with the CLI cleanup (some output-formats will be removed as well as the --include-ids and --symbols option).
    • A lot of discussions, especially regarding the future community development of pylint/astroid on Bitbucket. Short summary being: more contributors and integrators are welcome! We should drop some note somewhere to describe how we are using bitbucket's pull requests and tracker.

    Plan

    Now here is the 1.O roadmap, which is expected by the begining of July:

    • Green tests under Python 3, including specification of Python version in message description (Julien).
    • Finish template for text reporters (Anthony).
    • Update web site (David).

    And for later releases:

    • Backport mode from gpylint (Torsten).
    • Write ChangeLog update script (Sylvain).

    So many thanks to everyone for this very successful sprint. I'm excited about this forthcoming 1.0 release!


  • PyLint 10th anniversary 1.0 sprint: day 2

    2013/06/18 by Sylvain Thenault

    Today was the second day of the 10th anniversary Pylint sprint in Logilab's Toulouse office.

    This morning, we started with a presentation by myself about how the inference engine works in astroid (former astng). Then we started thinking all together about how we should change its API to be able to plug more information during the inference process. The first use-case we wanted to assert was namedtuple, as explained in http://www.logilab.org/ticket/8796.

    We ended up by addressing it by:

    • enhancing the existing transformation feature so one may register a transformation function on any node rather than on a module node only;
    • being able to specify, on a node instance, a custom inference function to use instead of the default (class) implementation.

    We would then be able to customize both the tree structure and the inference process and so to resolve the cases we were targeting.

    Once this was sufficiently sketched out, everyone got his own tasks to do. Here is a quick summary of what has been achieved today:

    • Anthony resumed the check_messages thing and finished it for the simple cases, then he started on having a template for text reported,
    • Julien and David made a lot of progress on the Python 3.3 compatibility, though not enough to get the full green test suite,
    • Torsten continued backporting stuff from gpylint, all of them having been integrated by the end of the day,
    • Sylvain implemented the new transformation API and had the namedtuple proof of concept working, and even some documentation! Now this have to be tested for more real-world uses.

    So things are going really well, and see you tomorrow for even more improvements to pylint!


  • PyLint 10th anniversary 1.0 sprint: day 1

    2013/06/17 by Sylvain Thenault

    Today was the first day of the Pylint sprint we organized using Pylint's 10th years anniversary as an excuse.

    So I (Sylvain) have welcome my fellow Logilab friends David, Anthony and Julien as well as Torsten from Google into Logilab's new Toulouse office.

    After a bit of presentation and talk about Pylint development, we decided to keep discussion for lunch and dinner and to setup priorities. We ended with the following tasks (picks from the pad at http://piratepad.net/oAvsUoGCAC):

    • rename astng to move it outside the logilab package,
    • Torsten gpylint (Google Pylint) patches review, as much as possible (but not all of them, starting by a review of the numberous internal checks Google has, seeing one by one which one should be backported upstream),
    • setuptools namespace package support (https://www.logilab.org/8796),
    • python 3.3 support,
    • enhance astroid (former astng) API to allow more ad-hoc customization for a better grasp of magic occuring in e.g. web frameworks (protocol buffer or SQLAlchemy may also be an application of this).

    Regarding the astng renaming, we decided to move on with astroid as pointed out by the survey on StellarSurvey.com

    In the afternoon, David and Julien tackled this, while Torsten was extracting patches from Google code and sending them to bitbucket as pulll request, Sylvain embrassing setuptools namespaces packages and Anthony discovering the code to spread the @check_message decorator usage.

    By the end of the day:

    • David and Julien submitted patches to rename logilab.astng which were quickly integrated and now https://bitbucket.org/logilab/astroid should be used instead of https://bitbucket.org/logilab/astng
    • Torsten submitted 5 pull-requests with code extracted from gpylint, we reviewed them together and then Torsten used evolve to properly insert those in the pylint history once review comments were integrated
    • Sylvain submitted 2 patches on logilab-common to support both setuptools namespace packages and pkgutil.extend_path (but not bare __path__ manipulation
    • Anthony discovered various checkers and started adding proper @check_messages on visit methods

    After doing some review all together, we even had some time to take a look at Python 3.3 support while writing this summary.

    Hopefuly, our work on forthcoming days will be as efficient as on this first day!


  • About salt-ami-cloud-builder

    2013/06/07 by Paul Tonelli

    What

    At Logilab we are big fans of SaltStack, we use it quite extensivelly to centralize, configure and automate deployments.

    http://www.logilab.org/file/145398/raw/SaltStack-Logo.png

    We've talked on this blog about how to build a Debian AMI "by hand" and we wanted to automate this fully. Hence the salt way seemed to be the obvious way to go.

    So we wrote salt-ami-cloud-builder. It is mainly glue between existing pieces of software that we use and like. If you already have some definition of a type of host that you provision using salt-stack, salt-ami-cloud-builder should be able to generate the corresponding AMI.

    http://www.logilab.org/file/145397/raw/open-stack-cloud-computing-logo-2.png

    Why

    Building a Debian based OpenStack private cloud using salt made us realize that we needed a way to generate various flavours of AMIs for the following reasons:

    • Some of our openstack users need "preconfigured" AMIs (for example a Debian system with Postgres 9.1 and the appropriate Python bindings) without doing the modifications by hand or waiting for an automated script to do the job at AMI boot time.
    • Some cloud use cases require that you boot many (hundreds for instance) machines with the same configuration. While tools like salt automate the job, waiting while the same download and install takes place hundreds of times is a waste of resources. If the modifications have already been integrated into a specialized ami, you save a lot of computing time. And especially in the Amazon (or other pay-per-use cloud infrastructures), these resources are not free.
    • Sometimes one needs to repeat a computation on an instance with the very same packages and input files, possibly years after the first run. Freezing packages and files in one preconfigured AMI helps this a lot. When relying only on a salt configuration the installed packages may not be (exactly) the same from one run to the other.

    Relation to other projects

    While multiple tools like build-debian-cloud exist, their objective is to build a vanilla AMI from scratch. The salt-ami-cloud-builder starts from such vanilla AMIs to create variations. Other tools like salt-cloud focus instead on the boot phase of the deployment of (multiple) machines.

    Chef & Puppet do the same job as Salt, however Salt being already extensively deployed at Logilab, we continue to build on it.

    Get it now !

    Grab the code here: http://hg.logilab.org/master/salt-ami-cloud-builder

    The project page is http://www.logilab.org/project/salt-ami-cloud-builder

    The docs can be read here: http://docs.logilab.org/salt-ami-cloud-builder

    We hope you find it useful. Bug reports and contributions are welcome.

    The logilab-salt-ami-cloud-builder team :)


  • Pylint 10th years anniversary from June 17 to 19 in Toulouse

    2013/04/18 by Sylvain Thenault

    After a quick survey, we're officially scheduling Pylint 10th years anniversary sprint from monday, June 17 to wednesday, June 19 in Logilab's Toulouse office.

    There is still some room available if more people want to come, drop me a note (sylvain dot thenault at logilab dot fr).


  • Pylint development moving to BitBucket

    2013/04/12 by Sylvain Thenault

    Hi everyone,

    After 10 years of hosting Pylint on our own forge at logilab.org, we've decided to publish version 1.0 and move Pylint and astng development to BitBucket. There has been repository mirrors there for some time, but we intend now to use all BitBucket features, notably Pull Request, to handle various development tasks.

    There are several reasons behind this. First, using both BitBucket and our own forge is rather cumbersome, for integrators at least. This is mainly because BitBucket doesn't provide support for Mercurial's changeset evolution feature while our forge relies on it. Second, our forge has several usability drawbacks that make it hard to use for newcomers, and we lack the time to be responsive on this. Finally, we think that our quality-control process, as exposed by our forge, is a bit heavy for such community projects and may keep potential contributors away.

    All in all, we hope this will help to have a wider contributor audience as well as more regular maintainers / integrators which are not Logilab employees. And so, bring the best Pylint possible to the Python community!

    Logilab.org web pages will be updated to mention this, but kept as there is still valuable information there (eg tickets). We may also keep automatic tests and package building services there.

    So, please use https://bitbucket.org/logilab/pylint as main web site regarding pylint development. Bug reports, feature requests as well as contributions should be done there. The same move will be done for Pylint's underlying library, logilab-astng (https://bitbucket.org/logilab/astng). We also wish in this process to move it out of the 'logilab' python package. It may be a good time to give it another name, if you have any idea don't hesitate to express yourself.

    Last but not least, remember that Pylint home page may be edited using Mercurial, and that the new http://docs.pylint.org is generated using the content found in Pylint source doc subdirectory.

    Pylint turning 10 and moving out of its parents is probably a good time to thank Logilab for paying me and some colleagues to create and maintain this project!

    https://bitbucket-assetroot.s3.amazonaws.com/c/photos/2013/Apr/05/pylint-logo-1661676867-0_avatar.png

show 200 results