subscribe to this blog

Logilab.org - en

News from Logilab and our Free Software projects, as well as on topics dear to our hearts (Python, Debian, Linux, the semantic web, scientific computing...)

show 198 results
  • We hosted the Salt Sprint in Paris

    2013/07/30 by Arthur Lutz

    Last Friday, we hosted the French event for the international Great Salt Sprint. Here is a report on what was done and discussed on this occasion.

    http://www.logilab.org/file/228931/raw/saltstack_logo.jpg

    We started off by discussing various points that were of interest to the participants :

    • automatically write documentation from salt sls files (for Sphinx)
    • salt-mine add security layer with restricted access (bug #5467 and #6437)
    • test compatibility of salt-cloud with openstack
    • module bridge bug correction : traceback on KeyError
    • setting up the network in debian (equivalent of rh_ip)
    • configure existing monitoring solution through salt (add machines, add checks, etc) on various backends with a common syntax

    We then split up into pairs to tackle issues in small groups, with some general discussions from time to time.

    6 people participated, 5 from Logilab, 1 from nbs-system. We were expecting more participants but some couldn't make it at the last minute, or though the sprint was taking place at some other time.

    Unfortunately we had a major electricity black out all afternoon, some of us switched to battery and 3G tethering to carry on, but that couldn't last all afternoon. We ended up talking about design and use cases. ERDF (French electricity distribution company) ended up bringing generator trucks for the neighborhood !

    Arthur & Benoit : monitoring adding machines or checks

    http://www.logilab.org/file/157971/raw/salt-centreon-shinken.png

    Some unfinished draft code for supervision backends was written and pushed on github. We explored how a common "interface" could be done in salt (using a combination of states and __virtual___). The official documentation was often very useful, reading code was also always a good resource (and the code is really readable).

    While we were fixing stuff because of the power black out, Benoit submitted a bug fix.

    David & Alain : generate documentation from salt state & salt master

    The idea is to couple the SLS description and the current state of the salt master to generate documentation about one's infrastructure using Sphinx. This was transmitted to the mailing-list.

    http://www.logilab.org/file/157976/raw/salt-sphinx.png

    Design was done around which information should be extracted and display and how to configure access control to the salt-master, taking a further look to external_auth and salt-api will probably be the way forward.

    General discussions

    We had general discussions around concepts of access control to a salt master, on how to define this access. One of the things we believe to be missing (but haven't checked thoroughly) is the ability to separate the "read-only" operations to the "read-write" operations in states and modules, if this was done (through decorators?) we could easily tell salt-api to only give access to data collection. Complex scenarios of access were discussed. Having a configuration or external_auth based on ssh public keys (similar to mercurial-server would be nice, and would provide a "limited" shell to a mercurial server.

    Conclusion

    The power black out didn't help us get things done, but nevertheless, some sharing was done around our uses cases around SaltStack and features that we'd want to get out of it (or from third party applications). We hope to convert all the discussions into bug reports or further discussion on the mailing-lists and (obviously) into code and pull-requests. Check out the scoreboard for an overview of how the other cities contributed.

    to comment this post you need to login or create an account


  • The Great Salt Sprint Paris Location is Logilab

    2013/07/12 by Nicolas Chauvat
    http://farm1.static.flickr.com/183/419945378_4ead41a76d_m.jpg

    We're happy to be part of the second Great Salt Sprint that will be held at the end of July 2013. We will be hosting the french sprinters on friday 26th in our offices in the center of Paris.

    The focus of our Logilab team will probably be Test-Driven System Administration with Salt, but the more participants and the topics, the merrier the event.

    Please register if you plan on joining us. We will be happy to meet with fellow hackers.

    photo by Sebastian Mary under creative commons licence.


  • PyLint 10th anniversary 1.0 sprint: day 3 - Sprint summary

    2013/06/20 by Sylvain Thenault

    Yesterday was the third and last day of the 10th anniversary Pylint sprint in Logilab's Toulouse office.

    Design

    To get started, we took advantage of this last day to have a few discussions about:

    • A "mode" feature gpylint has. It turns out that behind perhaps a few implementation details, this is something we definitly want into pylint (mode are specific configurations defined in the pylintrc and easilly recallable, they may even be specified per file).

    • How to avoid conflicts in the ChangeLog by using specific instruction in the commit message. We decided that a commit message should look like

      [my checker] do this and that. Closes #1234
      
      bla bla bla
      
      :release note: this will be a new item in the ChangeLog
      
      as well as anything until the end of the message
      

      now someone has to write the ChangeLog generation script so we may use this for post-1.0 releases

    • The roadmap. More on this later in this post.

    Code

    When we were not discussing, we were coding!

    • Anthony worked on having a template for the text reporter. His patch is available on Bitbucket but not yet integrated.
    • Julien and David pushed a bunch of patches on logilab-common, astroid and pylint for the Python 3.3 support. Not all tests are green on the pylint side, but much progress was done.
    • A couple other things were fixed, like a better "invalid name" message, stop complaining about string module being deprecated, etc.
    • A lot of patches have been integrated, from gpylint and others (e.g python 3 related)

    All in all, an impressive amount of work was achieved during this sprint:

    • A lot of new checks or enhanced behaviour backported from gpylint (Take a look at Pylint's ChangeLog for more details on this, the list is impressively long).
    • The transformation API of astroid now allows to customize the tree structure as well as the inference process, hence to make pylint smarter than ever.
    • Better python 3 support.
    • A few bugs fixed and some enhancements added.
    • The templating stuff should land with the CLI cleanup (some output-formats will be removed as well as the --include-ids and --symbols option).
    • A lot of discussions, especially regarding the future community development of pylint/astroid on Bitbucket. Short summary being: more contributors and integrators are welcome! We should drop some note somewhere to describe how we are using bitbucket's pull requests and tracker.

    Plan

    Now here is the 1.O roadmap, which is expected by the begining of July:

    • Green tests under Python 3, including specification of Python version in message description (Julien).
    • Finish template for text reporters (Anthony).
    • Update web site (David).

    And for later releases:

    • Backport mode from gpylint (Torsten).
    • Write ChangeLog update script (Sylvain).

    So many thanks to everyone for this very successful sprint. I'm excited about this forthcoming 1.0 release!


  • PyLint 10th anniversary 1.0 sprint: day 2

    2013/06/18 by Sylvain Thenault

    Today was the second day of the 10th anniversary Pylint sprint in Logilab's Toulouse office.

    This morning, we started with a presentation by myself about how the inference engine works in astroid (former astng). Then we started thinking all together about how we should change its API to be able to plug more information during the inference process. The first use-case we wanted to assert was namedtuple, as explained in http://www.logilab.org/ticket/8796.

    We ended up by addressing it by:

    • enhancing the existing transformation feature so one may register a transformation function on any node rather than on a module node only;
    • being able to specify, on a node instance, a custom inference function to use instead of the default (class) implementation.

    We would then be able to customize both the tree structure and the inference process and so to resolve the cases we were targeting.

    Once this was sufficiently sketched out, everyone got his own tasks to do. Here is a quick summary of what has been achieved today:

    • Anthony resumed the check_messages thing and finished it for the simple cases, then he started on having a template for text reported,
    • Julien and David made a lot of progress on the Python 3.3 compatibility, though not enough to get the full green test suite,
    • Torsten continued backporting stuff from gpylint, all of them having been integrated by the end of the day,
    • Sylvain implemented the new transformation API and had the namedtuple proof of concept working, and even some documentation! Now this have to be tested for more real-world uses.

    So things are going really well, and see you tomorrow for even more improvements to pylint!


  • PyLint 10th anniversary 1.0 sprint: day 1

    2013/06/17 by Sylvain Thenault

    Today was the first day of the Pylint sprint we organized using Pylint's 10th years anniversary as an excuse.

    So I (Sylvain) have welcome my fellow Logilab friends David, Anthony and Julien as well as Torsten from Google into Logilab's new Toulouse office.

    After a bit of presentation and talk about Pylint development, we decided to keep discussion for lunch and dinner and to setup priorities. We ended with the following tasks (picks from the pad at http://piratepad.net/oAvsUoGCAC):

    • rename astng to move it outside the logilab package,
    • Torsten gpylint (Google Pylint) patches review, as much as possible (but not all of them, starting by a review of the numberous internal checks Google has, seeing one by one which one should be backported upstream),
    • setuptools namespace package support (https://www.logilab.org/8796),
    • python 3.3 support,
    • enhance astroid (former astng) API to allow more ad-hoc customization for a better grasp of magic occuring in e.g. web frameworks (protocol buffer or SQLAlchemy may also be an application of this).

    Regarding the astng renaming, we decided to move on with astroid as pointed out by the survey on StellarSurvey.com

    In the afternoon, David and Julien tackled this, while Torsten was extracting patches from Google code and sending them to bitbucket as pulll request, Sylvain embrassing setuptools namespaces packages and Anthony discovering the code to spread the @check_message decorator usage.

    By the end of the day:

    • David and Julien submitted patches to rename logilab.astng which were quickly integrated and now https://bitbucket.org/logilab/astroid should be used instead of https://bitbucket.org/logilab/astng
    • Torsten submitted 5 pull-requests with code extracted from gpylint, we reviewed them together and then Torsten used evolve to properly insert those in the pylint history once review comments were integrated
    • Sylvain submitted 2 patches on logilab-common to support both setuptools namespace packages and pkgutil.extend_path (but not bare __path__ manipulation
    • Anthony discovered various checkers and started adding proper @check_messages on visit methods

    After doing some review all together, we even had some time to take a look at Python 3.3 support while writing this summary.

    Hopefuly, our work on forthcoming days will be as efficient as on this first day!


  • About salt-ami-cloud-builder

    2013/06/07 by Paul Tonelli

    What

    At Logilab we are big fans of SaltStack, we use it quite extensivelly to centralize, configure and automate deployments.

    http://www.logilab.org/file/145398/raw/SaltStack-Logo.png

    We've talked on this blog about how to build a Debian AMI "by hand" and we wanted to automate this fully. Hence the salt way seemed to be the obvious way to go.

    So we wrote salt-ami-cloud-builder. It is mainly glue between existing pieces of software that we use and like. If you already have some definition of a type of host that you provision using salt-stack, salt-ami-cloud-builder should be able to generate the corresponding AMI.

    http://www.logilab.org/file/145397/raw/open-stack-cloud-computing-logo-2.png

    Why

    Building a Debian based OpenStack private cloud using salt made us realize that we needed a way to generate various flavours of AMIs for the following reasons:

    • Some of our openstack users need "preconfigured" AMIs (for example a Debian system with Postgres 9.1 and the appropriate Python bindings) without doing the modifications by hand or waiting for an automated script to do the job at AMI boot time.
    • Some cloud use cases require that you boot many (hundreds for instance) machines with the same configuration. While tools like salt automate the job, waiting while the same download and install takes place hundreds of times is a waste of resources. If the modifications have already been integrated into a specialized ami, you save a lot of computing time. And especially in the Amazon (or other pay-per-use cloud infrastructures), these resources are not free.
    • Sometimes one needs to repeat a computation on an instance with the very same packages and input files, possibly years after the first run. Freezing packages and files in one preconfigured AMI helps this a lot. When relying only on a salt configuration the installed packages may not be (exactly) the same from one run to the other.

    Relation to other projects

    While multiple tools like build-debian-cloud exist, their objective is to build a vanilla AMI from scratch. The salt-ami-cloud-builder starts from such vanilla AMIs to create variations. Other tools like salt-cloud focus instead on the boot phase of the deployment of (multiple) machines.

    Chef & Puppet do the same job as Salt, however Salt being already extensively deployed at Logilab, we continue to build on it.

    Get it now !

    Grab the code here: http://hg.logilab.org/master/salt-ami-cloud-builder

    The project page is http://www.logilab.org/project/salt-ami-cloud-builder

    The docs can be read here: http://docs.logilab.org/salt-ami-cloud-builder

    We hope you find it useful. Bug reports and contributions are welcome.

    The logilab-salt-ami-cloud-builder team :)


  • Pylint 10th years anniversary from June 17 to 19 in Toulouse

    2013/04/18 by Sylvain Thenault

    After a quick survey, we're officially scheduling Pylint 10th years anniversary sprint from monday, June 17 to wednesday, June 19 in Logilab's Toulouse office.

    There is still some room available if more people want to come, drop me a note (sylvain dot thenault at logilab dot fr).


  • Pylint development moving to BitBucket

    2013/04/12 by Sylvain Thenault

    Hi everyone,

    After 10 years of hosting Pylint on our own forge at logilab.org, we've decided to publish version 1.0 and move Pylint and astng development to BitBucket. There has been repository mirrors there for some time, but we intend now to use all BitBucket features, notably Pull Request, to handle various development tasks.

    There are several reasons behind this. First, using both BitBucket and our own forge is rather cumbersome, for integrators at least. This is mainly because BitBucket doesn't provide support for Mercurial's changeset evolution feature while our forge relies on it. Second, our forge has several usability drawbacks that make it hard to use for newcomers, and we lack the time to be responsive on this. Finally, we think that our quality-control process, as exposed by our forge, is a bit heavy for such community projects and may keep potential contributors away.

    All in all, we hope this will help to have a wider contributor audience as well as more regular maintainers / integrators which are not Logilab employees. And so, bring the best Pylint possible to the Python community!

    Logilab.org web pages will be updated to mention this, but kept as there is still valuable information there (eg tickets). We may also keep automatic tests and package building services there.

    So, please use https://bitbucket.org/logilab/pylint as main web site regarding pylint development. Bug reports, feature requests as well as contributions should be done there. The same move will be done for Pylint's underlying library, logilab-astng (https://bitbucket.org/logilab/astng). We also wish in this process to move it out of the 'logilab' python package. It may be a good time to give it another name, if you have any idea don't hesitate to express yourself.

    Last but not least, remember that Pylint home page may be edited using Mercurial, and that the new http://docs.pylint.org is generated using the content found in Pylint source doc subdirectory.

    Pylint turning 10 and moving out of its parents is probably a good time to thank Logilab for paying me and some colleagues to create and maintain this project!

    https://bitbucket-assetroot.s3.amazonaws.com/c/photos/2013/Apr/05/pylint-logo-1661676867-0_avatar.png

  • PyLint 10th years anniversary, 1.0 sprint

    2013/03/29 by Sylvain Thenault

    In a few week, pylint will be 10 years old (0.1 released on may 19 2003!). At this occasion, I would like to release a 1.0. Well, not exactly at that date, but not too long after would be great. Also, I think it would be a good time to have a few days sprint to work a bit on this 1.0 but also to meet all together and talk about pylint status and future, as more and more contributions come from outside Logilab (actually mostly Google, which employs Torsten and Martin, the most active contributors recently).

    The first thing to do is to decide a date and place. Having discussed a bit with Torsten about that, it seems reasonable to target a sprint during june or july. Due to personal constraints, I would like to host this sprint in Logilab's Toulouse office.

    So, who would like to jump in and sprint to make pylint even better? I've created a doodle so every one interested may tell his preferences: http://doodle.com/4uhk26zryis5x7as

    Regarding the location, is everybody ok with Toulouse? Other ideas are Paris, or Florence around EuroPython, or... <add your proposition here>.

    We'll talk about the sprint topics later, but there are plenty of exciting ideas around there.

    Please, answer quickly so we can move on. And I hope to see you all there!


  • LMGC90 Sprint at Logilab in March 2013

    2013/03/28 by Vladimir Popescu

    LMGC90 Sprint at Logilab

    At the end of March 2013, Logilab hosted a sprint on the LMGC90 simulation code in Paris.

    LMGC90 is an open-source software developed at the LMGC ("Laboratoire de Mécanique et Génie Civil" -- "Mechanics and Civil Engineering Laboratory") of the CNRS, in Montpellier, France. LMGC90 is devoted to contact mechanics and is, thus, able to model large collections of deformable or undeformable physical objects of various shapes, with numerous interaction laws. LMGC90 also allows for multiphysics coupling.

    Sprint Participants

    https://www.logilab.org/file/143585/raw/logo_LMGC.jpg https://www.logilab.org/file/143749/raw/logo_SNCF.jpg https://www.logilab.org/file/143750/raw/logo_LaMSID.jpg https://www.logilab.org/file/143751/raw/logo_LOGILAB.jpg

    More than ten hackers joined in from:

    • the LMGC, which leads LMCG90 development and aims at constantly improving its architecture and usability;
    • the Innovation and Research Department of the SNCF (the French state-owned railway company), which uses LMGC90 to study railway mechanics, and more specifically, the ballast;
    • the LaMSID ("Laboratoire de Mécanique des Structures Industrielles Durables", "Laboratory for the Mechanics of Ageing Industrial Structures") laboratory of the EDF / CNRS / CEA , which has an strong expertise on Code_ASTER and LMGC90;
    • Logilab, as the developer, for the SNCF, of a CubicWeb-based platform dedicated to the simulation data and knowledge management.

    After a great introduction to LMGC90 by Frédéric Dubois and some preliminary discussions, teams were quickly constituted around the common areas of interest.

    Enhancing LMGC90's Python API to build core objects

    As of the sprint date, LMGC90 is mainly developed in Fortran, but also contains Python code for two purposes:

    • Exposing the Fortran functions and subroutines in the LMGC90 core to Python; this is achieved using Fortran 2003's ISO_C_BINDING module and Swig. These Python bindings are grouped in a module called ChiPy.
    • Making it easy to generate input data (so called "DATBOX" files) using Python. This is done through a module called Pre_LMGC.

    The main drawback of this approach is the double modelling of data that this architecture implies: once in the core and once in Pre_LMGC.

    It was decided to build a unique user-level Python layer on top of ChiPy, that would be able to build the computational problem description and write the DATBOX input files (currently achieved by using Pre_LMGC), as well as to drive the simulation and read the OUTBOX result files (currently by using direct ChiPy calls).

    This task has been met with success, since, in the short time span available (half a day, basically), the team managed to build some object types using ChiPy calls and save them into a DATBOX.

    Using the Python API to feed a computation data store

    This topic involved importing LMGC90 DATBOX data into the numerical platform developed by Logilab for the SNCF.

    This was achieved using ChiPy as a Python API to the Fortran core to get:

    • the bodies involved in the computation, along with their materials, behaviour laws (with their associated parameters), geometries (expressed in terms of zones);
    • the interactions between these bodies, along with their interaction laws (and associated parameters, e.g. friction coefficient) and body pair (each interaction is defined between two bodies);
    • the interaction groups, which contain interactions that have the same interaction law.

    There is still a lot of work to be done (notably regarding the charges applied to the bodies), but this is already a great achievement. This could only have occured in a sprint, were every needed expertise is available:

    • the SNCF experts were there to clarify the import needs and check the overall direction;

    • Logilab implemented a data model based on CubicWeb, and imported the data using the ChiPy bindings developed on-demand by the LMGC core developer team, using the usual-for-them ISO_C_BINDING/ Swig Fortran wrapping dance.

      https://www.logilab.org/file/143753/raw/logo_CubicWeb.jpg
    • Logilab undertook the data import; to this end, it asked the LMGC how the relevant information from LMGC90 can be exposed to Python via the ChiPy API.

    Using HDF5 as a data storage backend for LMGC90

    The main point of this topic was to replace the in-house DATBOX/OUTBOX textual format used by LMGC90 to store input and output data, with an open, standard and efficient format.

    Several formats have been considered, like HDF5, MED and NetCDF4.

    MED has been ruled out for the moment, because it lacks the support for storing body contact information. HDF5 was chosen at last because of the quality of its Python libraries, h5py and pytables, and the ease of use tools like h5fs provide.

    https://www.logilab.org/file/143754/raw/logo_HDF.jpg

    Alain Leufroy from Logilab quickly presented h5py and h5fs usage, and the team started its work, measuring the performance impact of the storage pattern of LMGC90 data. This was quickly achieved, as the LMGC experts made it easy to setup tests of various sizes, and as the Logilab developers managed to understand the concepts and implement the required code in a fast and agile way.

    Debian / Ubuntu Packaging of LMGC90

    This topic turned out to be more difficult than initially assessed, mainly because LMGC90 has dependencies to non-packaged external libraries, which thus had to be packaged first:

    • the Matlib linear algebra library, written in C,
    • the Lapack95 library, which is a Fortran95 interface to the Lapack library.

    Logilab kept working on this after the sprint and produced packages that are currently being tested by the LMGC team. Some changes are expected (for instance, Python modules should be prefixed with a proper namespace) before the packages can be submitted for inclusion into Debian. The expertise of Logilab regarding Debian packaging was of great help for this task. This will hopefully help to spread the use of LMGC90.

    https://www.logilab.org/file/143755/raw/logo_Debian.jpg

    Distributed Version Control System for LMGC90

    As you may know, Logilab is really fond of Mercurial as a DVCS. Our company invested a lot into the development of the great evolve extension, which makes Mercurial a very powerful tool to efficiently manage the team development of software in a clean fashion.

    This is why Logilab presented Mercurial's features and advantages over the current VCS used to manage LMGC90 sources, namely svn, to the other participants of the Sprint. This was appreciated and will hopefully benefit to LMGC90 ease of development and spread among the Open Source community.

    https://www.logilab.org/file/143756/raw/logo_HG.jpg

    Conclusions

    All in all, this two-day sprint on LMGC90, involving participants from several industrial and academic institutions has been a great success. A lot of code has been written but, more importantly, several stepping stones have been laid, such as:

    • the general LMGC90 data access architecture, with the Python layer on top of the LMGC90 core;
    • the data storage format, namely HDF5.

    Colaterally somehow, several other results have also been achieved:

    • partial LMGC90 data import into the SNCF CubicWeb-based numerical platform,
    • Debian / Ubuntu packaging of LMGC90 and dependencies.

    On a final note, one would say that we greatly appreciated the cooperation between the participants, which we found pleasant and efficient. We look forward to finding more occasions to work together.


show 198 results