The view cw.archive.by_date can not be applied to this query
show 208 results

Blog entries

  • Typing Mercurial with pytype

    2019/11/14 by Denis Laxalde

    Following the recent introduction of Python type annotations (aka "type hints") in Mercurial (see, e.g. this changeset by Augie Fackler), I've been playing a bit with this and pytype.

    pytype is a static type analyzer for Python code. It compares with the more popular mypy but I don't have enough perspective to make a meaningful comparison at the moment. In this post, I'll illustrate how I worked with pytype to gradually add type hints in a Mercurial module and while doing so, fix bugs!

    The module I focused on is mercurial.mail, which contains mail utilities and that I know quite well. Other modules are also being worked on, this one is a good starting point because it has a limited number of "internal" dependencies, which both makes it faster to iterate with pytype and reduces side effects of other modules not being correctly typed already.

    shell $ pytype mercurial/ Computing dependencies Analyzing 1 sources with 36 local dependencies ninja: Entering directory `.pytype' [19/19] check mercurial.mail Success: no errors found

    The good news is that the module apparently already type-checks. Let's go deeper and merge the type annotations generated by pytype:

    $ merge-pyi -i mercurial/ out/mercurial/mail.pyi

    (In practice, we'd use --as-comments option to write type hints as comments, so that the module is still usable on Python 2.)

    Now we have all declarations annotated with types. Typically, we'd get many things like:

    sourceCode def codec2iana(cs) -> Any: cs = pycompat.sysbytes(email.charset.Charset(cs).input_charset.lower()) # "latin1" normalizes to "iso8859-1", standard calls for "iso-8859-1" if cs.startswith(b"iso") and not cs.startswith(b"iso-"): return b"iso-" + cs[3:] return cs

    The function signature has been annotated with Any (omitted for parameters, explicit for return value). This somehow means that type inference failed to find the type of that function. As it's (quite) obvious, let's change that into:

    sourceCode def codec2iana(cs: bytes) -> bytes: ...

    And re-run pytype:

    $ pytype mercurial/
    Computing dependencies
    Analyzing 1 sources with 36 local dependencies
    ninja: Entering directory `.pytype'
    [1/1] check mercurial.mail
    FAILED: .pytype/pyi/mercurial/mail.pyi 
    pytype-single --imports_info .pytype/imports/mercurial.mail.imports --module-name mercurial.mail -V 3.7 -o .pytype/pyi/mercurial/mail.pyi --analyze-annotated --nofail --quick mercurial/
    File "mercurial/", line 253, in codec2iana: Function Charset.__init__ was called with the wrong arguments [wrong-arg-types]
      Expected: (self, input_charset: str = ...)
      Actually passed: (self, input_charset: bytes)
    For more details, see
    ninja: build stopped: subcommand failed.

    Interesting! email.charset.Charset is apparently instantiated with the wrong argument type. While it's not exactly a bug, because Python will handle bytes instead of str well in general, we can again change the signature (and code) to:

    sourceCode def codec2iana(cs: str) -> str: cs = email.charset.Charset(cs).input_charset.lower() # "latin1" normalizes to "iso8859-1", standard calls for "iso-8859-1" if cs.startswith("iso") and not cs.startswith("iso-"): return "iso-" + cs[3:] return cs

    Obviously, this involves a larger refactoring in client code of this simple function, see respective changeset for details.

    Another example is this function:

    sourceCode def _encode(ui, s, charsets) -> Any: '''Returns (converted) string, charset tuple. Finds out best charset by cycling through sendcharsets in descending order. Tries both encoding and fallbackencoding for input. Only as last resort send as is in fake ascii. Caveat: Do not use for mail parts containing patches!''' sendcharsets = charsets or _charsets(ui) if not isinstance(s, bytes): # We have unicode data, which we need to try and encode to # some reasonable-ish encoding. Try the encodings the user # wants, and fall back to garbage-in-ascii. for ocs in sendcharsets: try: return s.encode(pycompat.sysstr(ocs)), ocs except UnicodeEncodeError: pass except LookupError: ui.warn(_(b'ignoring invalid sendcharset: %s\n') % ocs) else: # Everything failed, ascii-armor what we've got and send it. return s.encode('ascii', 'backslashreplace') # We have a bytes of unknown encoding. We'll try and guess a valid # encoding, falling back to pretending we had ascii even though we # know that's wrong. try: s.decode('ascii') except UnicodeDecodeError: for ics in (encoding.encoding, encoding.fallbackencoding): ics = pycompat.sysstr(ics) try: u = s.decode(ics) except UnicodeDecodeError: continue for ocs in sendcharsets: try: return u.encode(pycompat.sysstr(ocs)), ocs except UnicodeEncodeError: pass except LookupError: ui.warn(_(b'ignoring invalid sendcharset: %s\n') % ocs) # if ascii, or all conversion attempts fail, send (broken) ascii return s, b'us-ascii'

    It quite clear from the return value (last line) that we can change its type signature to:

    sourceCode def _encode(ui, s:Union[bytes, str], charsets: List[bytes]) -> Tuple[bytes, bytes] ...

    And re-run pytype:

    $ pytype mercurial/
    Computing dependencies
    Analyzing 1 sources with 36 local dependencies
    ninja: Entering directory `.pytype'
    [1/1] check mercurial.mail
    FAILED: .pytype/pyi/mercurial/mail.pyi 
    pytype-single --imports_info .pytype/imports/mercurial.mail.imports --module-name mercurial.mail -V 3.7 -o .pytype/pyi/mercurial/mail.pyi --analyze-annotated --nofail --quick mercurial/
    File "mercurial/", line 342, in _encode: bad option in return type [bad-return-type]
      Expected: Tuple[bytes, bytes]
      Actually returned: bytes
    For more details, see
    ninja: build stopped: subcommand failed.

    That's a real bug. Line 371 contains return s.encode('ascii', 'backslashreplace') in the middle of the function. We indeed return a bytes value instead of a Tuple[bytes, bytes] as elsewhere in the function. The following changes fixes the bug:

    sourceCode diff --git a/mercurial/ b/mercurial/ --- a/mercurial/ +++ b/mercurial/ @@ -339,7 +339,7 @@ def _encode(ui, s, charsets) -> Any: ui.warn(_(b'ignoring invalid sendcharset: %s\n') % ocs) else: # Everything failed, ascii-armor what we've got and send it. - return s.encode('ascii', 'backslashreplace') + return s.encode('ascii', 'backslashreplace'), b'us-ascii' # We have a bytes of unknown encoding. We'll try and guess a valid # encoding, falling back to pretending we had ascii even though we # know that's wrong.

    Steps by steps, by replacing Any with real types, adjusting "wrong" types like in the first example and fixing bugs, we finally get the whole module (almost) completely annotated.

  • Mercurial conference in Paris: May 28th, 2019

    2019/05/03 by Marla Da Silva

    Mercurial Paris conference will take place on May 28th at Mozilla's headquarters, in Paris.

    Mercurial is a free distributed Source Control Management system. It offers an intuitive interface to efficiently handle projects of any size. With its powerful extension system, Mercurial can easily adapt to any environment.

    This first edition targets organizations that are currently using Mercurial or considering switching from another Version Control System, such as Subversion.

    Attending the conference will allow users to share ideas and version control experiences in different industries and at a different scale. It is a great opportunity to connect with Mercurial core developers and get updates about modern workflow and features.

    You are welcome to register here and be part of this first edition with us!

    Mercurial conference is co-organized by Logilab, Octobus & Rhode-Code.

  • Mercurial mini-sprint: from April 4th to 7th in Paris

    2019/03/19 by Marla Da Silva

    Logilab co-organizes with Octobus, a mini-sprint Mercurial to be held from Thursday 4 to Sunday 7 April in Paris.

    Logilab will host mercurial mini-sprint in its Paris premises on Thursday 4 and Friday 5 April. Octobus will be communicating very soon the place chosen to sprint during the weekend.

    To participate to mercurial mini-sprint, please complete the survey by informing your name and which days you will be joining us.

    Some of the developers working on mercurial or associated tooling plan to focus on improving the workflow and tool and documentation used for online collaboration through Mercurial (Kalithea, RhodeCode, Heptapod, Phabricator, sh.rt, etc. You can also fill the pad below to indicate the themes that you want to tackle during this sprint:

    Let's code together!

  • Logilab trip report for FOSDEM 2019

    2019/02/13 by Nicolas Chauvat

    A very large conference

    This year I attended the FOSDEM in Brussels for the first time. I have been doing free software for more than 20 years, but for some reason, I had never been to FOSDEM. I was pleasantly surprised to see that it was much larger than I thought and that it gathered thousands of people. This is by far the largest free software event I have been to. My congratulations to the organizers and volunteers, since this must be a huge effort to pull off.

    My presentation about CubicWeb

    I went to FOSDEM to present Logilab's latest project, a reboot of CubicWeb to turn it into a web extension to browse the web of data. The recording of the talk, the slides and the video of the demo are online, I hope you enjoy them and get in touch with me if you are to comment or contribute.

    My highlights

    As usual, the "hallway track" was the most useful for me and there are more sets of slides I read than talks I could attend.

    I met with Bram, the author of redbaron and we had a long discussion about programming in different languages.

    I also met with Octobus. We discussed Heptapod, a project to add Mercurial support to Gitlab. Logilab would back such a project with money if it were to become usable and (please) move towards federation (with ActivityPub?) and user queries (with GraphQL?). We also discussed the so-called oxydation of Mercurial, which consists in rewriting some parts in Rust. After a quick search I saw that tools like PyO3 can help writing Python extensions in Rust.

    Some of the talks that interested me included:

    • Memex, that reuses the name of the very first hypertext system described in the litterature, and tries to implement a decentralized annotation system for the web. It reminded me of Web hypothesis and W3C's annotations recommendation which they say they will be compatible with.
    • GraphQL was presented both in GraphQL with Python and Testing GraphQL with Javascript. I have been following GraphQL about two years because it compares to the RQL language of CubicWeb. We have been testing it at Logilab with cubicweb-graphql.
    • Web Components are one of the options to develop user interfaces in the browser. I had a look at the Future of Web Components, which I relate to the work we are doing with the CubicWeb browser (see above) and the work the Virtual Assembly has been doing to implement Transiscope.
    • Pyodide, the scientific python stack compiled to Web Assembly, I try to compare it to using Jupyter notebooks.
    • Chat-over-IMAP another try to rule them all chat protocols. It is right that everyone has more than one email address, that email addresses are more and more used as logins in many web sites and that using these email addresses as instant-messaging / chat addresses would be nice. We will see if it takes off!

  • Experiment to safely refactor your code in Python

    2018/05/05 by Nicolas Chauvat

    "Will my refactoring break my code ?" is the question the developer asks himself because he is not sure the tests cover all the cases. He should wonder, because tests that cover all the cases would be costly to write, run and maintain. Hence, most of the time, small decisions are made day after day to test this and not that. After some time, you could consider that in a sense, the implementation has become the specification and the rest of the code expects it not to change.

    Enters Scientist, by GitHub, that inspired a Python port named Laboratory.

    Let us assume you want to add a cache to a function that reads data from a database. The function would be named read_from_db, it would take an int as parameter item_id and return a dict with attributes of the items and their values.

    You could experiment with the new version of this function like so:

    import laboratory
    def read_from_db_with_cache(item_id):
        data = {}
        # some implementation with a cache
        return data
    def read_from_db(item_id):
         data = {}
         #  fetch data from db
         return data

    When you run the above code, calling read_from_db returns its result as usual, but thanks to laboratory, a call to read_from_db_with_cache is made and its execution time and result are compared with the first one. These measurements are logged to a file or sent to your metrics solution for you to compare and study.

    In other words, things continue to work as usual as you keep the original function, but at the same time you experiment with its candidate replacement to make sure switching will not break or slow things down.

    I like the idea ! Thank you for Scientist and Laboratory that are both available under the MIT license.

  • Reduce docker image size with multi-stage builds

    2018/03/21 by Philippe Pepiot

    At logilab we use more and more docker both for test and production purposes.

    For a CubicWeb application I had to write a Dockerfile that does a pip install and compiles javascript code using npm.

    The first version of the Dockerfile was:

    FROM debian:stretch
    RUN apt-get update && apt-get -y install \
        wget gnupg ca-certificates apt-transport-https \
        python-pip python-all-dev libgecode-dev g++
    RUN echo "deb stretch main" > /etc/apt/sources.list.d/nodesource.list
    RUN wget -O - | apt-key add -
    RUN apt-get update && apt-get -y install nodejs
    COPY . /sources
    RUN pip install /sources
    RUN cd /sources/frontend && npm install && npm run build && \
        mv /sources/frontend/bundles /app/bundles/
    # ...

    The resulting image size was about 1.3GB which cause issues while uploading it to registries and with the required disk space on production servers.

    So I looked how to reduce this image size. What is important to know about Dockerfile is that each operation result in a new docker layer, so removing useless files at the end will not reduce the image size.

    The first change was to use debian:stretch-slim as base image instead of debian:stretch, this reduced the image size by 18MB.

    Also by default apt-get would pull a lot of extra optional packages which are "Suggestions" or "Recommends". We can simply disable it globally using:

    RUN echo 'APT::Install-Recommends "0";' >> /etc/apt/apt.conf && \
        echo 'APT::Install-Suggests "0";' >> /etc/apt/apt.conf

    This reduced the image size by 166MB.

    Then I looked at the content of the image and see a lot of space used in /root/.pip/cache. By default pip build and cache python packages (as wheels), this can be disabled by adding --no-cache to pip install calls. This reduced the image size by 26MB.

    In the image we also have a full nodejs build toolchain which is useless after the bundles are generated. The old workaround is to install nodejs, build files, remove useless build artifacts (node_modules) and uninstall nodejs in a single RUN operation, but this result in a ugly Dockerfile and will not be an optimal use of the layer cache. Instead we can setup a multi stage build

    The idea behind multi-stage builds is be able to build multiple images within a single Dockerfile (only the latest is tagged) and to copy files from one image to another within the same build using COPY --from= (also we can use a base image in the FROM clause).

    Let's extract the javascript building in a single image:

    FROM debian:stretch-slim as base
    RUN echo 'APT::Install-Recommends "0";' >> /etc/apt/apt.conf && \
        echo 'APT::Install-Suggests "0";' >> /etc/apt/apt.conf
    FROM base as node-builder
    RUN apt-get update && apt-get -y install \
        wget gnupg ca-certificates apt-transport-https \
    RUN echo "deb stretch main" > /etc/apt/sources.list.d/nodesource.list
    RUN wget -O - | apt-key add -
    RUN apt-get update && apt-get -y install nodejs
    COPY . /sources
    RUN cd /sources/frontend && npm install && npm run build
    FROM base
    RUN apt-get update && apt-get -y install python-pip python-all-dev libgecode-dev g++
    RUN pip install --no-cache /sources
    COPY --from=node-builder /sources/frontend/bundles /app/bundles

    This reduced the image size by 252MB

    The Gecode build toolchain is required to build rql with gecode extension (which is a lot faster that native python), my next idea was to build rql as a wheel inside a staging image, so the resulting image would only need the gecode library:

    FROM debian:stretch-slim as base
    RUN echo 'APT::Install-Recommends "0";' >> /etc/apt/apt.conf && \
        echo 'APT::Install-Suggests "0";' >> /etc/apt/apt.conf
    FROM base as node-builder
    RUN apt-get update && apt-get -y install \
        wget gnupg ca-certificates apt-transport-https \
    RUN echo "deb stretch main" > /etc/apt/sources.list.d/nodesource.list
    RUN wget -O - | apt-key add -
    RUN apt-get update && apt-get -y install nodejs
    COPY . /sources
    RUN cd /sources/frontend && npm install && npm run build
    FROM base as wheel-builder
    RUN apt-get update && apt-get -y install python-pip python-all-dev libgecode-dev g++ python-wheel
    RUN pip wheel --no-cache-dir --no-deps --wheel-dir /wheels rql
    FROM base
    RUN apt-get update && apt-get -y install python-pip
    COPY --from=wheel-builder /wheels /wheels
    RUN pip install --no-cache --find-links /wheels /sources
    # a test to be sure that rql is built with gecode extension
    RUN python -c 'import rql.rql_solve'
    COPY --from=node-builder /sources/frontend/bundles /app/bundles

    This reduced the image size by 297MB.

    So the final image size goes from 1.3GB to 300MB which is more suitable to use in a production environment.

    Unfortunately I didn't find a way to copy packages from a staging image, install it and remove the package in a single docker layer, a possible workaround is to use an intermediate http server.

    The next step could be to build a Debian package inside a staging image, so the build process would be separated from the Dockerfile and we could provide Debian packages along with the docker image.

    Another approach could be to use Alpine as base image instead of Debian.

  • Mercurial Sprint 4.4

    2017/10/10 by Denis Laxalde

    In late September, I participated on behalf of Logilab to the Mercurial 4.4 sprint that held at Facebook Dublin. It was the opportunity to meet developers of the project, follow active topics and forthcoming plans for Mercurial. Here, I'm essentially summarizing the main points that held my attention.

    Amongst three days of sprint, the first two mostly dedicated to discussions and the last one to writing code. Overall, the organization was pretty smooth thanks to Facebook guys doing a fair amount of preparation. Amongst an attendance of 25 community members, some companies had a significant presence: noticeably Facebook had 10 employees and Google 5, the rest consisting of either unaffiliated people and single-person representing their affiliation. No woman, unfortunately, and a vast majority of people with English as a native or dominant usage language.

    The sprint started by with short talks presenting the state of the Mercurial project. Noticeably, in the state of the community talk, it was recalled that the project governance now holds on the steering committee while some things are still not completely formalized (in particular, the project does not have a code of conduct yet). Quite surprisingly, the committee made no explicit mention of the recurring tensions in the project which recently lead to a banishment of a major contributor.

    Facebook, Google, Mozilla and Unity then presented by turns the state of Mercurial usages in their organization. Both Facebook and Google have a significant set of tools around hg, most of them either aiming at making the user experience more smooth with respect to performance problems coming from their monorepos or towards GUI tools. Other than that, it's interesting to note most of these corporate users have now integrated evolve on the user side, either as is or with a convenience wrapper layer.

    After that, followed several "vision statements" presentations combined with breakout sessions. (Just presenting a partial selection.)

    The first statement was about streamlining the style of the code base: that one was fairly consensual as most people agreed upon the fact that something has to be done in this respect; it was finally decided (on the second day) to adopt a PEP-like process. Let's see how things evolve!

    Second, Boris Feld and I talked about the development of the evolve extension and the ongoing task about moving it into Mercurial core (slides). We talked about new usages and workflows around the changeset evolution concept and the topics concept. Despite the recent tensions on this technical topic in the community, we tried to feel positive and reaffirmed that the whole community has interests in moving evolve into core. On the same track, in another vision statement, Facebook presented their restack workflow (sort of evolve for "simple" cases) and suggested to push this into core: this is encouraging as it means that evolve-like workflows tend to get mainstream.


    Another interesting vision statement was about using Rust in Mercurial. Most people agreed that Mercurial would benefit from porting its native C code in Rust, essentially for security reasons and hopefully to gain a bit of performance and maintainability. More radical ideas were also proposed such as making the hg executable a Rust program (thus embedding a Python interpreter along with its standard library) or reimplementing commands in Rust (which would pose problems with respect to the Python extension system). Later on, Facebook presented mononoke, a promising Mercurial server implemented in Rust that is expected to scale better with respect to high committing rates.

    Back to a community subject, we discussed about code review and related tooling. It was first recalled that the project would benefit from more reviewers, including people without a committer status. Then the discussion essentially focused on the Phabricator experiment that started in July. Introduction of a web-based review tool in Mercurial was arguably a surprise for the community at large since many reviewers and long-time contributors have always expressed a clear preference over email-based review. This experiment is apparently meant to lower the contribution barrier so it's nice to see the project moving forward on this topic and attempt to favor diversity by contribution. On the other hand, the choice of Phabricator was quite controversial. From the beginning (see replies to the announcement email linked above), several people expressed concerns (about notifications notably) and some reviewers also complained about the increase of review load and loss of efficiency induced by the new system. A survey recently addressed to the whole community apparently (no official report yet at the time of this writing) confirms that most employees from Facebook or Google seem pretty happy with the experiment while other community members generally dislike it. Despite that, it was decided to keep the "experiment" going while trying to improve the notification system (better threading support, more diff context, subscription-based notification, etc.). That may work, but there's a risk of community split as non-corporate members might feel left aside. Overall, adopting a consensus-based decision model on such important aspects would be beneficial.

    Yet another interesting discussion took place around branching models. Noticeably, Facebook and Google people presented their recommended (internal) workflow respectively based on remotenames and local bookmarks while Pulkit Goyal presented the current state to the topics extension and associated workflows. Bridging the gap between all these approaches would be nice and it seems that a first step would be to agree upon the definition of a stack of changesets to define a current line of work. In particular, both the topics extension and the show command have their own definition, which should be aligned.

    (Comments on reddit.)

  • Better code archaeology with Mercurial

    2017/09/21 by Denis Laxalde

    For about a year, Logilab has been involved in Mercurial development in the framework of a Mozilla Open Source Support (MOSS) program. Mercurial is a foundational technology for Mozilla, as the VCS used for the development of Firefox. As the main protagonist of the program, I'd first like to mention this has been a very nice experience for me, both from the social and technical perspectives. I've learned a lot and hope to continue working on this nice piece of software.

    The general objective of the program was to improve "code archaeology" workflows, especially when using hgweb (the web UI for Mercurial). Outcomes of program spread from versions 4.1 to 4.4 of Mercurial and I'm going to present the main (new) features in this post.

    Better navigation in "blame/annotate" view (hgweb)

    The first feature concerns the "annotate" view in hgweb; the idea was to improve navigation along "blamed" revisions, a process that is often tedious and involves a lot of clicks in the web UI (not easier from the CLI, for that matter). Basically, we added an hover box on the left side of file content displaying more context on blamed revision along with a couple of links to make navigation easier (links to parent changesets, diff and changeset views). See below for an example about mercurial.error module (to try it at

    The hover box in annotate view in hgweb.


    While this wasn't exactly in the initial scope of the program, the most interesting result of my work is arguably the introduction of the "followlines" feature set. The idea is to build upon the log command instead of annotate to make it easier to follow changes across revisions and the new thing is to make this possible by filtering changes only affecting a block of lines in a particular file.

    The first component introduced a revset, named followlines, which accepts at least two arguments a file path and a line range written as fromline:toline (following Python slice syntax). For instance, say we are interested the history of LookupError class in mercurial/ module which, at tag 4.3, lives between line 43 and 59, we'll use the revset as follows:

    $ hg log -r 'followlines(mercurial/, 43:59)'
    changeset:   7633:08cabecfa8a8
    user:        Matt Mackall <>
    date:        Sun Jan 11 22:48:28 2009 -0600
    summary:     errors: move revlog errors
    changeset:   24038:10d02cd18604
    user:        Martin von Zweigbergk <>
    date:        Wed Feb 04 13:57:35 2015 -0800
    summary:     error: store filename and message on LookupError for later
    changeset:   24137:dcfdfd63bde4
    user:        Siddharth Agarwal <>
    date:        Wed Feb 18 16:45:16 2015 -0800
    summary:     error.LookupError: rename 'message' property to something
    changeset:   25945:147bd9e238a1
    user:        Gregory Szorc <>
    date:        Sat Aug 08 19:09:09 2015 -0700
    summary:     error: use absolute_import
    changeset:   34016:6df193b5c437
    user:        Yuya Nishihara <>
    date:        Thu Jun 01 22:43:24 2017 +0900
    summary:     py3: implement __bytes__() on most of our exception classes

    This only yielded changesets touching this class.

    This is not an exact science (because the algorithm works on diff hunks), but in many situations it gives interesting results way faster that with an iterative "annotate" process (in which one has to step from revision to revision and run annotate every times).

    The followlines() predicate accepts other arguments, in particular, descend which, in combination with the startrev, lets you walk the history of a block of lines in the descending direction of history. Below the detailed help (hg help revset.followlines) of this revset:

    $ hg help revsets.followlines
        "followlines(file, fromline:toline[, startrev=., descend=False])"
          Changesets modifying 'file' in line range ('fromline', 'toline').
          Line range corresponds to 'file' content at 'startrev' and should hence
          be consistent with file size. If startrev is not specified, working
          directory's parent is used.
          By default, ancestors of 'startrev' are returned. If 'descend' is True,
          descendants of 'startrev' are returned though renames are (currently)
          not followed in this direction.

    In hgweb

    The second milestone of the program was to port this followlines filtering feature into hgweb. This has been implemented as a line selection mechanism (using mouse) reachable from both the file and annotate views. There, you'll see a small green ± icon close to the line number. By clicking on this icon, you start the line selection process which can be completed by clicking on a similar icon on another line of the file.

    Starting a line selection for followlines in hgweb.

    After that, you'll see a box inviting you to follow the history of lines <selected range> , either in the ascending (older) or descending (newer) direction.

    Line selection completed for followlines in hgweb.

    Here clicking on the "newer" link, we get:

    Followlines result in hgweb.

    As you can notice, this gives a similar result as with the command line but it also displays the patch of each changeset. In these patch blocks, only the diff hunks affecting selected line range are displayed (sometimes with some extra context).

    What's next?

    At this point, the "followlines" feature is probably complete as far as hgweb is concerned. In the remainder of the MOSS program, I'd like to focus on a command-line interface producing a similar output as the one above in hgweb, i.e. filtering patches to only show followed lines. That would take the form of a --followlines/-L option to hg log command, e.g.:

    $ hg log -L mercurial/,43:59 -p

    That's something I'd like to tackle at the upcoming Mercurial 4.4 sprint in Dublin!

  • A non-numeric Pandas example with Geonames

    2017/02/20 by Yann Voté

    The aim of this 2-parts blog post is to show some useful, yet not very complicated, features of the Pandas Python library that are not found in most (numeric oriented) tutorials.

    We will illustrate these techniques with Geonames data : extract useful data from the Geonames dump, transform it, and load it into another file. There is no numeric computation involved here, nor statictics. We will prove that pandas can be used in a wide range of cases beyond numerical analysis.

    While the first part was an introduction to Geonames data, this second part contains the real work with Pandas. Please read part 1 if you are not familiar with Geonames data.

    The project

    The goal is to read Geonames data, from allCountries.txt and alternateNames.txt files, combine them, and produce a new CSV file with the following columns:

    • gid, the Geonames id,
    • frname, French name when available,
    • gname, Geonames main name,
    • fclass, feature class,
    • fcode, feature code,
    • parent_gid, Geonames id of the parent location,
    • lat, latitude,
    • lng, longitude.

    Also we don't want to use too much RAM during the process. 1 GB is a maximum.

    You may think that this new CSV file is a low-level objective, not very interesting. But it can be a step into a larger process. For example one can build a local Geonames database that can then be used by tools like Elastic Search or Solr to provide easy searching and auto-completion in French.

    Another interesting feature is provided by the parent_gid column. One can use this column to build a tree of locations, or a SKOS thesaurus of concepts.

    The method

    Before diving into technical details, let us first look at the big picture, the overall process to achieve the aforementioned goal.

    Considering the wanted columns, we can see the following differences with allCountries.txt:

    • we are not interested in some original columns (population, elevation, ...),
    • the column frname is new and will come from alternateNames.txt, thanks to the Geonames id,
    • the column parent_gid is new too. We must derive this information from the adminX_code (X=1, 2, 3, 4) columns in allCountries.txt.

    This column deserves an example. Look at Arrondissement de toulouse.

    feature_code: ADM3, country_code: FR, admin1_code: 76, admin2_code: 31, admin3_code: 313

    To find its parent, we must look at a place with the following properties,

    feature_code: ADM2, country_code: FR, admin1_code: 76, admin2_code: 31

    and there is only one place with such properties, Geonames id 3,013,767, namely Département de la Haute-Garonne. Thus we must find a way to derive the Geonames id from the feature_code, country_code and adminX_code columns. Pandas will make this easy for us.

    Let's get to work. And of course, we must first import the Pandas library to make it available. We also import the csv module because we will need it to perform basic operations on CSV files.

    >>> import pandas as pd
    >>> import csv

    The French names step

    Loading file

    We begin by loading data from the alternateNames.txt file.

    And indeed, that's a big file. To save memory we won't load the whole file. Recall from previous part, that the alternateNames.txt file provides the following columns in order.

    alternateNameId   : the id of this alternate name, int
    geonameid         : geonameId referring to id in table 'geoname', int
    isolanguage       : iso 639 language code 2- or 3-characters; (...)
    alternate name    : alternate name or name variant, varchar(400)
    isPreferredName   : '1', if this alternate name is an official/preferred name
    isShortName       : '1', if this is a short name like 'California' for 'State of California'
    isColloquial      : '1', if this alternate name is a colloquial or slang term
    isHistoric        : '1', if this alternate name is historic and was used in the pastq

    For our purpose we are only interested in columns geonameid (so that we can find corresponding place in allCountries.txt file), isolanguage (so that we can keep only French names), alternate name (of course), and isPreferredName (because we want to keep preferred names when possible).

    Another way to save memory is to filter the file before loading it. Indeed, it's a better practice to load a smaller dataset (filter before loading) than to load a big one and then filter it after loading. It's important to keep those things in mind when you are working with large datasets. So in our case, it is cleaner (but slower) to prepare the CSV file before, keeping only French names.

    >>> # After restarting Python
    >>> with open('alternateNames.txt') as f:
    ...     reader = csv.reader(f, delimiter='\t', quoting=csv.QUOTE_NONE)
    ...     with open('frAltNames.txt', 'w') as g:
    ...         writer = csv.writer(g, delimiter='\t', quoting=csv.QUOTE_NONE,
    ...                             escapechar='\\')
    ...         for row in reader:
    ...             if row[2] == u'fr':
    ...                 writer.writerow(row)

    To load a CSV file, Pandas provides the read_csv function. It returns a dataframe populated with data from a CSV file.

    >>> with open('frAltNames.txt') as altf:
    ...     frnames = pd.read_csv(
    ...         altf, encoding='utf-8',
    ...         sep='\t', quoting=csv.QUOTE_NONE,
    ...         header=None, usecols=(1, 3, 4), index_col=0,
    ...         names=('gid', 'frname', 'pref'),
    ...         dtype={'gid': 'uint32', 'frname': str, 'pref': 'bool_'},
    ...         true_values=['1'], false_values=[u''], na_filter=False,
    ...         skipinitialspace=True, error_bad_lines=False
    ...     )

    The read_csv function is quite complex and it takes some time to use it rightly. In our cases, the first few parameters are self-explanatory: encoding for the file encoding, sep for the CSV separator, quoting for the quoting protocol (here there is none), and header for lines to be considered as label lines (here there is none).

    usecols tells pandas to load only the specified columns. Beware that indices start at 0, so column 1 is the second column in the file (geonameid in this case).

    index_col says to use one of the columns as a row index (instead of creating a new index from scratch). Note that the number for index_col is relative to usecols. In other words, 0 means the first column of usecols, not the first column of the file.

    names gives labels for columns (instead of using integers from 0). Hence we can extract the last column with frnames['pref'] (instead of frnames[3]). Please note, that this parameter is not compatible with headers=0 for example (in that case, the first line is used to label columns).

    The dtype parameter is interesting. It allows you to specify one type per column. When possible, prefer to use NumPy types to save memory (eg np.uint32, np.bool_).

    Since we want boolean values for the pref column, we can tell Pandas to convert '1' strings to True and empty strings ('') to False. That is the point of using parameters true_values and false_values.

    But, by default, Pandas detect empty strings and affect them np.nan (NaN means not a number). To prevent this behavior, setting na_filter to False will leave empty strings as empty strings. Thus, empty strings in the pref column will be converted to False (thanks to the false_values parameter and to the 'bool_' data type).

    Finally, skipinitialspace tells Pandas to left-strip strings to remove leading spaces. Without it, a line like a, b (commas-separated) would give values 'a' and ' b' (note leading space before b).

    And lastly, error_bad_lines make pandas ignore lines he cannot understand (eg. wrong number of columns). The default behavior is to raise an exception.

    There are many more parameters to this function, which is much more powerful than the simple reader object from the csv module. Please, refer to the documentation at for a full list of options. For example, the header parameter can accept a list of integers, like [1, 3, 4], saying that lines at position 1, 3 and 4 are label lines.

    <class 'pandas.core.frame.DataFrame'>
    Int64Index: 55684 entries, 18918 to 11441793
    Data columns (total 2 columns):
    frname     55684 non-null object
    pref       55684 non-null bool
    dtypes: bool(1), object(1)
    memory usage: 6.6 MB

    We can see here that our dataframe contains 55,684 entries and that it takes about 6.6 MB in memory.

    Let's look at entry number 3,013,767 for the Haute-Garonne department.

    >>> frnames.loc[3013767]
                                     frname    pref
    3013767  Département de la Haute-Garonne  False
    3013767                    Haute-Garonne   True

    Removing duplicated indices

    There is one last thing to do with the frnames dataframe: make its row indices uniques. Indeed, we can see from the previous example that there are two entries for the Haute-Garonne department. We need to keep only one, and, when possible, the preferred form (when the pref column is True).

    This is not always possible: we can see at the beginning of the dataframe (use the head method) that there is no preferred French name for Rwanda.

    >>> frnames.head()
                         frname   pref
    18918              Protaras   True
    49518  République du Rwanda  False
    49518                Rwanda  False
    50360       Woqooyi Galbeed  False
    51230              Togdheer  False

    In such a case, we will take one of the two lines at random. Maybe a clever rule to decide which one to keep would be useful (like involving other columns), but for the purpose of this tutorial it does not matter which one is kept.

    Back to our problem, how to make indices uniques ? Pandas provides the duplicated method on Index objects. This method returns a boolean NumPy 1d-array (a vector), the size of which is the number of entries. So, since our dataframe has 55,684 entries, the length of the returned vector is 55,684.

    >>> dups_idx = frnames.index.duplicated()
    >>> dups_idx
    array([False, False,  True, ..., False, False, False], dtype=bool)
    >>> len(dups_idx)

    The meaning is simple: when you encounter True, it means that the index at this position is a duplicate of a previously encountered index. For example, we can see that the third value in dups_idx is True. And indeed, the third line of frnames has an index (49,518) which is a duplicate of the second line.

    So duplicated is meant to mark as True duplicated indices and to keep only the first one (there is an optional parameter to change this: read the doc). How do we make sure that the first entry is the preferred entry ? Sorting the dataframe of course! We can sort a table by a column (or by a list of columns), using the sort_values method. We give ascending=False because True > False (that's a philosophical question!), and inplace=True to sort the dataframe in place, thus we do not create a copy (still thinking about memory usage).

    >>> frnames.sort_values('pref', ascending=False, inplace=True)
    >>> frnames.head()
                                   frname  pref
    18918                        Protaras  True
    8029850  Hôtel de ville de Copenhague  True
    8127417                        Kalmar  True
    8127361                     Sundsvall  True
    8127291                        Örebro  True
    >>> frnames.loc[3013767]
                                      frname   pref
    3013767                    Haute-Garonne   True
    3013767  Département de la Haute-Garonne  False

    Great! All preferred names are now first in the dataframe. We can then use the duplicated index method to filter out duplicated entries.

    >>> frnames = frnames[~(frnames.index.duplicated())]
    >>> frnames.loc[3013767]
    frname     Haute-Garonne
    pref                True
    Name: 3013767, dtype: object
    >>> frnames.loc[49518]
    frname     Rwanda
    pref        False
    Name: 49518, dtype: object
    <class 'pandas.core.frame.DataFrame'>
    Int64Index: 49047 entries, 18918 to 11441793
    Data columns (total 2 columns):
    frname     49047 non-null object
    pref       49047 non-null bool
    dtypes: bool(1), object(1)
    memory usage: 5.7 MB

    We end up with 49,047 French names. Notice the ~ in the filter expression ? That's because we want to keep the first entry for each duplicated index, and the first entry is False in the vector returned by duplicated.

    Summary for the french names step

    One last thing to do is to remove the pref column which won't be used anymore. We already know how to do it.

    >>> frnames.drop('pref', axis=1, inplace=True)

    There has been a lot of talking until now. But if we summarize, very few commands where needed to obtain this table with only two columns (gid and frname):

    1. Prepare a smaller file so there is less data to load (keep only french names).

      with open('alternateNames.txt') as f:
          reader = csv.reader(f, delimiter='\t', quoting=csv.QUOTE_NONE)
          with open('frAltNames.txt', 'w') as g:
              writer = csv.writer(g, delimiter='\t', quoting=csv.QUOTE_NONE,
              for row in reader:
                  if row[2] == u'fr':
    2. Load the file into Pandas.

      with open('frAltNames.txt') as altf:
          frnames = pd.read_csv(
              altf, encoding='utf-8',
              sep='\t', quoting=csv.QUOTE_NONE,
              header=None, usecols=(1, 3, 4), index_col=0,
              names=('gid', 'frname', 'pref'),
              dtype={'gid': 'uint32', 'frname': str, 'pref': 'bool_'},
              true_values=['1'], false_values=[u''], na_filter=False,
              skipinitialspace=True, error_bad_lines=False
    3. Sort on the pref column and remove duplicated indices.

      frnames.sort_values('pref', ascending=False, inplace=True)
      frnames = frnames[~(frnames.index.duplicated())]
    4. Remove the pref column.

      frnames.drop('pref', axis=1, inplace=True)

    Simple, isn't it ? We'll keep this dataframe for later use. Pandas will make it a breeze to merge it with the main dataframe coming from allCountries.txt to create the new frname column.

    The administrative tree step

    But for now, let's look at the second problem: how to derive a gid from a feature_code, a country_code, and a bunch of adminX_code (X=1, 2, 3, 4).

    Loading file

    First we need the administrative part of the file allCountries.txt, that is all places with the A feature class.

    Of course we can load the whole file into Pandas and then filter to keep only A-class entries, but now you know that this is memory intensive (and this file is much bigger than alternateNames.txt). So we'll be more clever and first prepare a smaller file.

    >>> with open('allCountries.txt') as f:
    ...     reader = csv.reader(f, delimiter='\t', quoting=csv.QUOTE_NONE)
    ...     with open('adm_geonames.txt', 'w') as g:
    ...         writer = csv.writer(g, delimiter='\t', quoting=csv.QUOTE_NONE, escapechar='\\')
    ...         for row in reader:
    ...             if row[6] == 'A':
    ...                 writer.writerow(row)

    Now we load this file into pandas. Remembering our goal, the resulting dataframe will only be used to compute gid from the code columns. So all columns we need are geonameid, feature_code, country_code, admin1_code, admin2_code, admin3_code, and admin4_code.

    What about data types. All these columns are strings except for geonameid which is an integer. Since it will be painful to type <colname>: str for the dtype parameter dictionary, let's use the fromkeys constructor instead.

    >>> d_types = dict.fromkeys(['fcode', 'code0', 'code1', 'code2', 'code3',
    ...                          'code4'], str)
    >>> d_types['gid'] = 'uint32'  # Shorter geonameid in gid

    We can now load the file.

    >>> with open('adm_geonames.txt') as admf:
    ...     admgids = pd.read_csv(
    ...         admf, encoding='utf-8',
    ...         sep='\t', quoting=csv.QUOTE_NONE,
    ...         header=None, usecols=(0, 7, 8, 10, 11, 12, 13),
    ...         names=('gid', 'fcode', 'code0', 'code1', 'code2', 'code3', 'code4'),
    ...         dtype=d_types, na_values='', keep_default_na=False,
    ...         error_bad_lines=False
    ...     )

    We recognize most parameters in this instruction. Notice that we didn't use index_col=0: gid is now a normal column, not an index, and Pandas will generate automatically a row index with integers starting at 0.

    Two new parameters are na_values and keep_default_na. The first one gives Pandas additional strings to be considerer as NaN (Not a Number). The astute reader would say that empty strings ('') are already considered by Pandas as NaN, and he would be right.

    But here comes the second parameter which, if set to False, tells Pandas to forget about its default list of strings recognized as NaN. The default list contains a bunch of strings like 'N/A' or '#NA' or, this is interesting, simply 'NA'. But 'NA' is used in allCountries.txt as country code for Namibia. If we keep the default list, this whole country will be ignored. So the combination of these two parameters tells Pandas to:

    • reset its default list of NaN strings,
    • use '' as a NaN string, which, consequently, will be the only one.

    That's it for NaN. What can Pandas tells us about our dataframe ?

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 369277 entries, 0 to 369276
    Data columns (total 7 columns):
    gid      369277 non-null uint32
    fcode    369273 non-null object
    code0    369272 non-null object
    code1    369197 non-null object
    code2    288494 non-null object
    code3    215908 non-null object
    code4    164899 non-null object
    dtypes: object(6), uint32(1)
    memory usage: 128.8 MB

    Note the RangeIndex which Pandas has created for us. The dataframe takes about 130 MB of memory because it has a lots of object columns. Apart from that, everything looks good.

    Replacing values

    One thing we want to do before going on, is to replace all 'PCL<X>' values in the fcode column with just PCL. This will make our life easier later when searching in this dataframe.

    >>> pd.unique(admgids.fcode)
    array([u'ADMD', u'ADM1', u'PCLI', u'ADM2', u'ADM3', u'ADM2H', u'PCLD',
           u'ADM4', u'ZN', u'ADM1H', u'PCLH', u'TERR', u'ADM3H', u'PRSH',
           u'PCLIX', u'ADM5', u'ADMDH', nan, u'PCLS', u'LTER', u'ADM4H',
           u'ZNB', u'PCLF', u'PCL'], dtype=object)

    Pandas provides the replace method for that.

    >>> admgids.replace({'fcode': {r'PCL[A-Z]{1,2}': 'PCL'}}, regex=True, inplace=True)
    >>> pd.unique(admgids.fcode)
    array([u'ADMD', u'ADM1', u'PCL', u'ADM2', u'ADM3', u'ADM2H', u'ADM4',
           u'ZN', u'ADM1H', u'TERR', u'ADM3H', u'PRSH', u'ADM5', u'ADMDH', nan,
           u'LTER', u'ADM4H', u'ZNB'], dtype=object)

    The replace method has lot of different signatures, refer to the Pandas documentation for a comprehensive description. Here, with the dictionary, we hare saying to look only in column fcode, and in this column, replace strings matching the regular expression with the given value. Since we are using regular expressions, the regex parameter must be set to True.

    And as usual, the inplace parameter avoids creation of a copy.


    Remember the goal: we want to be able to get the gid from the others columns. Well, dear reader, you'll be happy to know that Pandas allows an index to be composite, to be composed of multiple columns, what Pandas calls a MultiIndex.

    To put it simply, a multi-index is useful when you have hierarchical indices Consider for example the following table.

    lvl1 lvl2 N S
    A AA 11 1x1
    A AB 12 1x2
    B BA 21 2x1
    B BB 22 2x2
    B BC 33 3x3

    If we were to load such a table in a Pandas dataframe df (exercise: do it), we would be able to use it as follows.

    >>> df.loc['A']
         N    S
    AA  11  1x1
    AB  12  1x2
    >>> df.loc['B']
         N    S
    BA  21  2x1
    BB  22  2x2
    BC  23  2x3
    >>> df.loc[('A',), 'S']
    AA    1x1
    AB    1x2
    Name: S, dtype: object
    >>> df.loc[('A', 'AB'), 'S']
    >>> df.loc['B', 'BB']
    N     22
    S    2x2
    Name: (B, BB), dtype: object
    >>> df.loc[('B', 'BB')]
    N     22
    S    2x2
    Name: (B, BB), dtype: object

    So, basically, we can query the multi-index using tuples (and we can omit tuples if column indexing is not involved). But must importantly we can query a multi-index partially: df.loc['A'] returns a sub-dataframe with one level of index gone.

    Back to our subject, we clearly have hierarchical information in our code columns: the country code, then the admin1 code, then the admin2 code, and so on. Moreover, we can put the feature code at the top. But how to do that ?

    It can't be more simpler. The main issue is to find the correct method: set_index.

    >>> admgids.set_index(['fcode', 'code0', 'code1', 'code2', 'code3', 'code4'],
    ...                    inplace=True)
    <class 'pandas.core.frame.DataFrame'>
    MultiIndex: 369277 entries, (ADMD, AD, 00, nan, nan, nan) to (PCLH, nan, nan, nan, nan, nan)
    Data columns (total 1 columns):
    gid      369277 non-null uint32
    dtypes: uint32(1)
    memory usage: 21.9 MB
    >>> admgids.head()
    fcode code0 code1 code2 code3 code4
    ADMD  AD    00    NaN   NaN   NaN    3038817
                                  NaN    3039039
    ADM1  AD    06    NaN   NaN   NaN    3039162
                05    NaN   NaN   NaN    3039676
                04    NaN   NaN   NaN    3040131

    That's not really it, because Pandas has kept the original dataframe order, so multi-index is all messed up. No problem, there is the sort_index method.

    >>> admgids.sort_index(inplace=True)
    <class 'pandas.core.frame.DataFrame'>
    MultiIndex: 369277 entries, (nan, CA, 10, 02, 94235, nan) to (ZNB, SY, 00, nan, nan, nan)
    Data columns (total 1 columns):
    gid      369277 non-null uint32
    dtypes: uint32(1)
    memory usage: 21.9 MB
    >>> admgids.head()
    fcode code0 code1 code2 code3 code4
    NaN   CA    10    02    94235 NaN    6544163
          CN    NaN   NaN   NaN   NaN    6255000
          CY    04    NaN   NaN   NaN    6640324
          MX    16    NaN   NaN   NaN    6618819
    ADM1  AD    02    NaN   NaN   NaN    3041203

    Much better. We can even see that there are three entries without a feature code.

    Let's see if all this effort was worth it. Can we fulfill our goal ? Can we get a gid from a bunch of codes ? Can we get the Haute-Garonne gid ?

    >>> admgids.loc['ADM2', 'FR', '76', '31']
    code3 code4
    NaN   NaN    3013767

    And for the Toulouse ADM4 ?

    >>> admgids.loc['ADM4', 'FR', '76', '31', '313', '31555']
    fcode code0 code1 code2 code3 code4
    ADM4  FR    76    31    313   31555  6453974

    Cheers! You've deserved a glass of wine!

    Summary for the administrative data step

    Before moving on four our grand finale, let's summarize what have been done to prepare administrative data.

    1. We've prepared a smaller file with only administrative data.

      with open('allCountries.txt') as f:
          reader = csv.reader(f, delimiter='\t', quoting=csv.QUOTE_NONE)
          with open('adm_geonames.txt', 'w') as g:
              writer = csv.writer(g, delimiter='\t', quoting=csv.QUOTE_NONE,
              for row in reader:
                  if row[6] == 'A':
    2. We've loaded the file into Pandas.

      d_types = dict.fromkeys(['fcode', 'code0', 'code1', 'code2', 'code3',
                               'code4'], str)
      d_types['gid'] = 'uint32'
      with open('adm_geonames.txt') as admf:
          admgids = pd.read_csv(
              admf, encoding='utf-8',
              sep='\t', quoting=csv.QUOTE_NONE,
              header=None, usecols=(0, 7, 8, 10, 11, 12, 13),
              names=('gid', 'fcode', 'code0', 'code1', 'code2', 'code3', 'code4'),
              dtype=d_types, na_values='', keep_default_na=False,
    3. We've replaced all 'PCL<XX>' values with just 'PCL' in the fcode column.

      admgids.replace({'fcode': {r'PCL[A-Z]{1,2}': 'PCL'}}, regex=True,
    4. Then we've created a multi-index with columns fcode, code0, code1, code2, code3, code4.

      admgids.set_index(['fcode', 'code0', 'code1', 'code2', 'code3', 'code4'],

    Putting it all together

    Time has finally come to load the main file now: allCountries.txt. On one hand, we will then be able to use the frnames dataframe to get French name for each entry and populate the frname column, and on the other hand we will use the admgids dataframe to compute parent gid for each line too.

    Loading file

    On my computer, loading the whole allCountries.txt at once takes 5.5 GB of memory, clearly too much! And in this case, there is no trick to reduce the size of the file first: we want all the data.

    Pandas can help us with the chunk_size parameter to the read_csv function. It allows us to read the file chunk by chunk (it returns an iterator). The idea is to first create an empty CSV file for our final data, then read each chunk, perform data manipulation (that is add frname and parent_gid) of this chunk, and append the data to the file.

    So we load data into Pandas the usual way. The only difference is that we add a new parameter chunksize with value 1,000,000. You can choose a smaller or a larger number of rows depending of your memory limit.

    >>> d_types = dict.fromkeys(['fclass', 'fcode', 'code0', 'code1',
    ...                          'code2', 'code3', 'code4'], str)
    >>> d_types['gid'] = 'uint32'
    >>> d_types['lat'] = 'float16'
    >>> d_types['lng'] = 'float16'
    >>> with open('allCountries.txt') as geof:
    ...     reader = pd.read_csv(
    ...         geof, encoding='utf-8',
    ...         sep='\t', quoting=csv.QUOTE_NONE,
    ...         header=None, usecols=(0, 1, 4, 5, 6, 7, 8, 10, 11, 12, 13),
    ...         names=('gid', 'name', 'lat', 'lng', 'fclass', 'fcode',
    ...                'code0', 'code1', 'code2', 'code3', 'code4'),
    ...         dtype=d_types, index_col=0,
    ...         na_values='', keep_default_na=False,
    ...         chunksize=1000000, error_bad_lines=False
    ...     )
    ...     for chunk in reader:
    ...         pass  # We will put here code to work on each chunk

    Joining tables

    Pandas can perform a JOIN on two dataframes, much like in SQL. The function to do so is merge.

    ...     for chunk in reader:
    ...         chunk = pd.merge(chunk, frnames, how='left',
    ...                          left_index=True, right_index=True)

    merge expects first the two dataframes to be joined. The how parameter tells what type of JOIN to perform (it can be left, right, inner, ...). Here we wants to keep all lines in chunk, which is the first parameter, so it is 'left' (if chunk was the second parameter, it would have been 'right').

    left_index=True and right_index=True tell Pandas that the pivot column is the index on each table. Indeed, in our case the gid index will be use in both tables to compute the merge. If in one table, for example the right one, the pivot column is not the index, one can set right_index=False and add parameter right_on='<column_name>' (same parameter exists for left table).

    Additionally, If there are name clashes (same column name in both tables), one can also use the suffixes parameter. For example suffixes=('_first', '_second').

    And that's all we need to add the frname column. Simple isn't it ?

    Computing parent gid

    The parent_gid column is trickier. We'll delegate computation of the gid from administrative codes to a separate function. But first let's define two reverse dictionaries linking the fcode column to a level number

    >>> level_fcode = {0: 'PCL', 1: 'ADM1', 2: 'ADM2', 3: 'ADM3', 4: 'ADM4'}
    >>> fcode_level = dict((v, k) for k, v in level_fcode.items())

    And here is the function computing the Geonames id from administrative codes.

    >>> def geonameid_from_codes(level, **codes):
    ...     """Return the Geoname id of the *administrative* place with the
    ...     given information.
    ...     Return ``None`` if there is no match.
    ...     ``level`` is an integer from 0 to 4. 0 means we are looking for a
    ...     political entity (``PCL<X>`` feature code), 1 for a first-level
    ...     administrative division (``ADM1``), and so on until fourth-level
    ...     (``ADM4``).
    ...     Then user must provide at least one of the ``code0``, ...,
    ...     ``code4`` keyword parameters, depending on ``level``.
    ...     Examples::
    ...         >>> geonameid_from_codes(level=0, code0='RE')
    ...         935317
    ...         >>> geonameid_from_codes(level=3, code0='RE', code1='RE',
    ...         ...                      code2='974', code3='9742')
    ...         935213
    ...         >>> geonameid_from_codes(level=0, code0='AB')  # None
    ...     """
    ...     try:
    ...         idx = tuple(codes['code{0}'.format(i)] for i in range(level+1))
    ...     except KeyError:
    ...         raise ValueError('Not enough codeX parameters for level {0}'.format(level))
    ...     idx = (level_fcode[level],) + idx
    ...     try:
    ...         return admgids.loc[idx, 'gid'].values[0]
    ...     except (KeyError, TypeError):
    ...         return None

    A few comments about this function: the first part builds a tuple idx with the given codes. Then this idx is used as an index value in the admgids dataframe to find the matching gid.

    We also need another function which will compute the parent gid for a Pandas row.

    >>> from six import string_types
    >>> def parent_geonameid(row):
    ...     """Return the Geoname id of the parent of the given Pandas row.
    ...     Return ``None`` if we can't find the parent's gid.
    ...     """
    ...     # Get the parent's administrative level (PCL or ADM1, ..., ADM4)
    ...     level = fcode_level.get(row.fcode)
    ...     if (level is None and isinstance(fcode, string_types)
    ...             and len(fcode) >= 3):
    ...         fcode_level.get(row.fcode[:3], 5)
    ...     level = level or 5
    ...     level -= 1
    ...     if level < 0:  # We were on a country, no parent
    ...         return None
    ...     # Compute available codes
    ...     l = list(range(5))
    ...     while l and pd.isnull(row['code{0}'.format(l[-1])]):
    ...         l.pop()  # Remove NaN values backwards from code4
    ...     codes = {}
    ...     for i in l:
    ...         if i > level:
    ...             break
    ...         code_label = 'code{0}'.format(i)
    ...         codes[code_label] = row[code_label]
    ...     try:
    ...         return geonameid_from_codes(level, **codes)
    ...     except ValueError:
    ...         return None

    In this function, we first look at the row fcode to get the row administrative level. If the fcode is ADMX we get the level directly. If it is PCL<X>, we get level 0 from PCL. Else we set it to level 5 to say that it is below level 4. So the parent's level is the found level minus one. And if it is -1, we know that we were on a country and there is no parent.

    Then we compute all available administrative codes, removing codes with NaN values from the end.

    With the level and the codes, we can search for the parent's gid using the previous function.

    Now, how do we use this function. No need for a for loop, Pandas gives us the apply method.

    ...     for chunk in reader:
    ...         # (...) pd.merge as before
    ...         parent_gids = chunk.apply(parent_geonameid, axis=1)

    This will apply the parent_geonameid to each row and return a new Pandas series whose head looks like this.

    2986043     nan
    2993838     nan
    2994701     nan
    3007683     nan
    3017832     nan
    3039162     3041565.0

    None values have been converted to NaN. Thus, integer values have been converted to float (you cannot have NaN within an integer column), and this is not what we want. As a compromise, we are going to convert this into str and suppress the decimal part.

    ...         parent_gids = parent_gid.astype(str)  # no inplace=True here
    ...         parent_gids.replace(r'\.0', '', regex=True, inplace=True)

    We also add a label to the column. That's the name the column in our future dataframe.

    ...         parent_gids = parent_gids.rename('parent_gid')

    And we can now append this new column to our chunk dataframe.

    ...         chunk = pd.concat([chunk, parent_gids], axis=1)

    We're almost there. Before we can save the chunk in a CSV file, we must reorganize its columns to the expected order. For now, the frname and parent_gid columns have been appended at the end of the dataframe.

    ...         chunk = chunk.reindex(columns=['frname', 'name', 'fclass',
    ...                                        'fcode', 'parent_gid', 'lat',
    ...                                        'lng'])

    At last, we save the chunk to the file opened in append mode.

    ...         chunk.to_csv('final.txt', mode='a', encoding='utf-8',
    ...                      quoting=csv.QUOTE_NONE, sep='\t', header=None)

    Caching to speed up

    Currently, creating the new CSV file from Geonames takes hours, and this is not acceptable. There are multiple ways to make thing go faster. One of the most significant change is to cache results of the parent_geonameid function. Indeed, many places in Geonames have the same parent ; computing the parent gid once and caching it sounds like a good idea.

    If you are using Python3, you can simply use the @functools.lru_cache decorator on the parent_geonameid function. But let us try to define our own custom cache.

    >>> from six import string_types
    >>> gid_cache = {}
    >>> def parent_geonameid(row):
    ...     """Return the Geoname id of the parent of the given Pandas row.
    ...     Return ``None`` if we can't find the parent's gid.
    ...     """
    ...     # Get the parent's administrative level (PCL or ADM1, ..., ADM4)
    ...     level = fcode_level.get(row.fcode)
    ...     if (level is None and isinstance(fcode, string_types)
    ...             and len(fcode) >= 3):
    ...         fcode_level.get(row.fcode[:3], 5)
    ...     level = level or 5
    ...     level -= 1
    ...     if level < 0:  # We were on a country, no parent
    ...         return None
    ...     # Compute available codes
    ...     l = list(range(5))
    ...     while l and pd.isnull(row['code{0}'.format(l[-1])]):
    ...         l.pop()  # Remove NaN values backwards from code4
    ...     codes = {}
    ...     code_tuple = [level]
    ...     for i in l:
    ...         if i > level:
    ...             break
    ...         code_label = 'code{0}'.format(i)
    ...         code = row[code_label]
    ...         codes[code_label] = code
    ...         code_tuple.append(code)
    ...     code_tuple = tuple(code_tuple)
    ...     try:
    ...         parent_gid = (gid_cache.get(code_tuple)
    ...                       or geonameid_from_codes(level, **codes))
    ...     except ValueError:
    ...         parent_gid = None
    ...     # Put value in cache if not already to speed up future lookup
    ...     if code_tuple not in gid_cache:
    ...         gid_cache[code_tuple] = parent_gid
    ...     return parent_gid

    The only difference with the previous version is the use of a gid_cache dictionary. Keys for this dictionary are tuples (<level>, <code0>, [[<code1>], ..., <code4>]) (stored in the code_tuple variable), and the corresponding value is the parent gid for this combination of level and codes. Then the returned parent_gid is first looked in this dictionary for a previous cached result, else is computed from the geonameid_from_codes function like before, and the result is cached.

    Summary for the final step

    Let's review what we have done.

    1. We have defined three useful dictionaries. Two to get a level number from a feature code and conversely, and one to cache computation results.

      level_fcode = {0: 'PCL', 1: 'ADM1', 2: 'ADM2', 3: 'ADM3', 4: 'ADM4'}
      fcode_level = dict((v, k) for k, v in level_fcode.items())
      gid_cache = {}
    2. We have defined a function computing a Geoname id from administrative codes.

      def geonameid_from_codes(level, **codes):
          """Return the Geoname id of the *administrative* place with the
          given information.
          Return ``None`` if there is no match.
          ``level`` is an integer from 0 to 4. 0 means we are looking for a
          political entity (``PCL<X>`` feature code), 1 for a first-level
          administrative division (``ADM1``), and so on until fourth-level
          Then user must provide at least one of the ``code0``, ...,
          ``code4`` keyword parameters, depending on ``level``.
              >>> geonameid_from_codes(level=0, code0='RE')
              >>> geonameid_from_codes(level=3, code0='RE', code1='RE',
              ...                      code2='974', code3='9742')
              >>> geonameid_from_codes(level=0, code0='AB')  # None
              idx = tuple(codes['code{0}'.format(i)] for i in range(level+1))
          except KeyError:
              raise ValueError('Not enough codeX parameters for level {0}'.format(level))
          idx = (level_fcode[level],) + idx
              return admgids.loc[idx, 'gid'].values[0]
          except (KeyError, TypeError):
              return None
    3. We have defined a function computing the parent's gid of a Pandas row.

      def parent_geonameid(row):
          """Return the Geoname id of the parent of the given Pandas row.
          Return ``None`` if we can't find the parent's gid.
          # Get the parent's administrative level (PCL or ADM1, ..., ADM4)
          level = fcode_level.get(row.fcode)
          if (level is None and isinstance(fcode, string_types)
                  and len(fcode) >= 3):
              fcode_level.get(row.fcode[:3], 5)
          level = level or 5
          level -= 1
          if level < 0:  # We were on a country, no parent
              return None
          # Compute available codes
          l = list(range(5))
          while l and pd.isnull(row['code{0}'.format(l[-1])]):
              l.pop()  # Remove NaN values backwards from code4
          codes = {}
          code_tuple = [level]
          for i in l:
              if i > level:
              code_label = 'code{0}'.format(i)
              code = row[code_label]
              codes[code_label] = code
          code_tuple = tuple(code_tuple)
              parent_gid = (gid_cache.get(code_tuple)
                            or geonameid_from_codes(level, **codes))
          except ValueError:
              parent_gid = None
          # Put value in cache if not already to speed up future lookup
          if code_tuple not in gid_cache:
              gid_cache[code_tuple] = parent_gid
          return parent_gid
    4. And finally we have loaded the file allCountries.txt into Pandas using chunks of 1,000,000 rows to save memory. For each chunk, we have merged it with the frnames table to add the frname column, and we applied the parent_geonameid function to add the parent_gid column. We then reordered the columns and append the chunk to the final CSV file.

      d_types = dict.fromkeys(['fclass', 'fcode', 'code0', 'code1',
                               'code2', 'code3', 'code4'], str)
      d_types['gid'] = 'uint32'
      d_types['lat'] = 'float16'
      d_types['lng'] = 'float16'
      with open('allCountries.txt') as geof:
          reader = pd.read_csv(
              geof, encoding='utf-8',
              sep='\t', quoting=csv.QUOTE_NONE,
              header=None, usecols=(0, 1, 4, 5, 6, 7, 8, 10, 11, 12, 13),
              names=('gid', 'name', 'lat', 'lng', 'fclass', 'fcode',
                     'code0', 'code1', 'code2', 'code3', 'code4'),
              dtype=d_types, index_col=0,
              na_values='', keep_default_na=False,
              chunksize=1000000, error_bad_lines=False
          for chunk in reader:
              chunk = pd.merge(chunk, frnames, how='left',
                               left_index=True, right_index=True)
              parent_gids = chunk.apply(parent_geonameid, axis=1)
              parent_gids = parent_gids.astype(str)  # no inplace=True here
              parent_gids.replace(r'\.0', '', regex=True, inplace=True)
              parent_gids = parent_gids.rename('parent_gid')
              chunk = pd.concat([chunk, parent_gids], axis=1)
              chunk = chunk.reindex(columns=['frname', 'name', 'fclass',
                                             'fcode', 'parent_gid', 'lat',
              chunk.to_csv('final.txt', mode='a', encoding='utf-8',
                           quoting=csv.QUOTE_NONE, sep='\t', header=None)

    This final part is the longest, because the parent_geonameid function takes some time on each chunk to compute all parents gids. But at the end of the process we'll proudly see a final.txt file with data the way we want it, and without using too much memory... High five!

    What can be improved

    Congratulations! This ends our journey into the Pandas world.

    Regarding Geonames, to be honest, we've only scratch the surface of its complexity. There's so much to be done.

    If you look at the file we've just produced, you'll see plenty of empty values in the parent_gid column. May be our method to get the Geonames id of the parent needs to be improved. May be all those orphan places should be moved inside their countries.

    Another problem lies within Geonames data. France has overseas territories, for example Reunion Island, Geoname id 935,317. This place has feature code PCLD, which means "dependent political entity". And indeed, Reunion Island is not a country and should not appear at the top level of the tree, at the same level as France. So some work should be done here to have Reunion Island linked to France in some way, may be using the until now ignored cc2 column (for "alternate country codes").

    Still another improvement, easier this one, is to have yet another parent level for continents. For this, one can use the file countryInfo.txt, downloadable from the same page.

    Considering speed this time, there is also room for improvements. First, the code itself might be better designed to avoid some tests and for loops. Another possibility is tu use multiprocessing, since each chunk in allCountries.txt isindependent. Processes can put their finished chunk on a queue that a writer process will read to write data in the output file. Another way to go is Cython (see:

  • Understanding Geonames dump

    2017/02/20 by Yann Voté

    The aim of this 2-parts blog post is to show some useful, yet not very complicated, features of the Pandas Python library that are not found in most (numeric oriented) tutorials.

    We will illustrate these techniques with Geonames data : extract useful data from the Geonames dump, transform it, and load it into another file. There is no numeric computation involved here, nor statictics. We will prove that pandas can be used in a wide range of cases beyond numerical analysis.

    This first part is an introduction to Geonames data. The real work with Pandas will be shown on the second part You can skip this part and go directly to part 2 if you are already familiar with Geonames.

    Main data

    Geonames data can be downloaded from The main file to download is Once extracted, you'll get a CSV file named allCountries.txt which contains nearly all Geonames data. In this file, CSV data are separated by tabulation.

    A sample Geonames entry

    Let's look in Geonames data for the city of Toulouse in France.

    $ grep -P '\tToulouse\t' allCountries.txt

    You'll find multiple results (including one in the United States of America), and among them the following line.

    2972315     Toulouse        Toulouse        Gorad Tuluza,Lapangan Terbang Blagnac,TLS,Tolosa,(...)  43.60426        1.44367 P       PPLA    FR              76      31      313     31555   433055          150     Europe/Paris    2016-02-18

    What is the meaning of each column ? There is no header line at the top of the file... Go back to the web page from where you downloaded the file. Below download links, you'll find some documentation. In particular, consider the following excerpt.

    The main 'geoname' table has the following fields :
    geonameid         : integer id of record in geonames database
    name              : name of geographical point (utf8) varchar(200)
    asciiname         : name of geographical point in plain ascii characters, varchar(200)
    alternatenames    : alternatenames, comma separated, ascii names automatically transliterated, convenience attribute from alternatename table, varchar(10000)
    latitude          : latitude in decimal degrees (wgs84)
    longitude         : longitude in decimal degrees (wgs84)
    feature class     : see, char(1)
    feature code      : see, varchar(10)
    country code      : ISO-3166 2-letter country code, 2 characters
    cc2               : alternate country codes, comma separated, ISO-3166 2-letter country code, 200 characters
    admin1 code       : fipscode (subject to change to iso code), see exceptions below, see file admin1Codes.txt for display names of this code; varchar(20)
    admin2 code       : code for the second administrative division, a county in the US, see file admin2Codes.txt; varchar(80)
    admin3 code       : code for third level administrative division, varchar(20)
    admin4 code       : code for fourth level administrative division, varchar(20)
    population        : bigint (8 byte int)
    elevation         : in meters, integer
    dem               : digital elevation model, srtm3 or gtopo30, average elevation of 3''x3'' (ca 90mx90m) or 30''x30'' (ca 900mx900m) area in meters, integer. srtm processed by cgiar/ciat.
    timezone          : the iana timezone id (see file timeZone.txt) varchar(40)
    modification date : date of last modification in yyyy-MM-dd format

    So we see that entry 2,972,315 represents a place named Toulouse, for which latitude is 43.60 and longitude is 1.44. The place is located in France (FR country code), its population is estimated to 433,055, elevation is 150 m, and timezone is the same than Paris.

    To understand the meaning of P and PPLA, as feature class and feature code respectively, the indicated web page at provides us with the following information.

    P city, village,...
    PPLA        seat of a first-order administrative division

    Thus line 2,972,315 is about a city which is the seat of a first-order administrative division in France. And indeed, Toulouse is the seat of the Occitanie region in France.

    Geonames administrative divisions

    So let's look for Occitanie.

    $ grep -P '\tOccitanie\t' allCountries.txt

    You'll end up with the following line.

    11071623    Occitanie       Occitanie       Languedoc-Roussillon-Midi-Pyrenees,(...)        44.02722        1.63559 A       ADM1    FR              76              5626858         188     Europe/Paris    2016-11-16

    This is entry number 11,071,623, and we have latitude, longitude, population, elevation and timezone as usual. The place's class is A and its code is ADM1. Information about these codes can be found in the same previous page.

    A country, state, region,...
    ADM1        first-order administrative division

    This confirms that Occitanie is a first-order administrative division of France.

    Now look at the adminX_code part of the Occitanie line (X=1, 2, 3, 4).


    Only column admin1_code is filled with value 76. Compare this with the same part from the Toulouse line.

    76  31      313     31555

    Here all columns admin1_code, admin2_code, admin3_code and admin4_code are given a value, respectively 76, 31 313, and 31555.

    Furthermore, we see that column admin1_code matches for Occitanie and Toulouse (value 76). This let us deduce that Toulouse is actually a city located in Occitanie.

    So, following the same logic, we can infer that Toulouse is also located in a second-order administrative division of France (feature code ADM2) with admin1_code of 76 and admin2_code equals to 31. Let's look for such a line in allCountries.txt.

    $ grep -P '\tADM2\t' allCountries.txt | grep -P '\tFR\t' | grep -P '\t76\t' \
    > | grep -P '\t31\t'

    We get the following line.

    3013767     Département de la Haute-Garonne Departement de la Haute-Garonne Alta Garonna,Alto Garona,(...)  43.41667        1.5     A       ADM2    FR              76      31                      1254347         181     Europe/Paris    2016-02-18

    Success! Toulouse is actually in the Haute-Garonne department, which is a department in Occitanie region.

    Is this going to work for third-level administrative division ? Let's look for an ADM3 line, with admin1_code=76, admin2_code=31, and admin3_code=313.

    $ grep -P '\tADM3\t' allCountries.txt | grep -P '\tFR\t' | grep -P '\t76\t' \
    > | grep -P '\t31\t' | grep -P '\t313\t'

    Here is the result.

    2972314     Arrondissement de Toulouse      Arrondissement de Toulouse      Arrondissement de Toulouse      43.58333        1.5     A       ADM3    FR              76      31      313             972098          139     Europe/Paris    2016-12-05

    Still works! Finally, let's find out the fourth-order administrative division which contains the city of Toulouse.

    $ grep -P '\tADM4\t' allCountries.txt | grep -P '\tFR\t' | grep -P '\t76\t' \
    > | grep -P '\t31\t' | grep -P '\t313\t' | grep -P '\t31555\t'

    We get a place also named Toulouse.

    6453974     Toulouse        Toulouse        Toulouse        43.60444        1.44194 A       ADM4    FR              76      31      313     31555   440204          153     Europe/Paris    2016-02-18

    That's a surprise. That's because in France, an arrondissement is the smallest administrative division above a city. So for Geonames, city of Toulouse is both an administrative place (feature class A with feature code ADM4) and a populated place (feature class P with feature code PPLA), so there is two different entries with the same name (beware that this may be different for other countries).

    And that's it! We have found the whole hierarchy of administrative divisions above city of Toulouse, thanks to the adminX_code columns: Occitanie, Département de la Haute-Garonne, Arrondissement de Toulouse, Toulouse (ADM4), and Toulouse (PPL).

    That's it, really ? What about the top-most administrative level, the country ? This may not be intuitive: feature codes for countries start with PCL (for political entity).

    $ grep -P '\tPCL' allCountries.txt | grep -P '\tFR\t'

    Among the result, there's only one PCLI, which means independant political entity.

    3017382     Republic of France      Republic of France      An Fhrainc,An Fhraing,(...)     46      2       A       PCLI    FR              00      64768389                543     Europe/Paris    2015-01-08

    Now we have the whole hierarchy, summarized in the following table.

    level name id fclass fcode ccode adm1_code adm2_code adm3_code adm4_code
    country Republic of France 3017382 A PCLI FR        
    level 1 Occitanie 11071623 A ADM1 FR 76      
    level 2 Département de la Haute-Garonne 3013767 A ADM2 FR 76 31    
    level 3 Arrondissement de Toulouse 2972314 A ADM3 FR 76 31 313  
    level 4 Toulouse 6453974 A ADM4 FR 76 31 313 31555
    city Toulouse 2972315 P PPL FR 76 31 313 31555

    Alternate names

    In the previous example, "Republic of France" is the main name for Geonames. But this is not the name in French ("République française"), and even the name in French is not the most commonly used name which is "France".

    In the same way, "Département de la Haute-Garonne" is not the most commonly used name for the department, which is just "Haute-Garonne".

    The fourth column in allCountries.txt provides a comma-separated list of alternate names for a place, in other languages and in other forms. But this is not very useful because we can't decide which form in the list is in which language.

    For this, the Geonames project provides another file to download: Go back to the download page, download it, and extract it. You'll get another tabulation-separeted CSV file named alternateNames.txt.

    Let's look at alternate names for the Haute-Garonne department.

    $ grep -P '\t3013767\t' alternateNames.txt

    Multiple names are printed.

    2080345     3013767 fr      Département de la Haute-Garonne
    2080346     3013767 es      Alto Garona     1       1
    2187116     3013767 fr      Haute-Garonne   1       1
    2431178     3013767 it      Alta Garonna    1       1
    2703076     3013767 en      Upper Garonne   1       1
    2703077     3013767 de      Haute-Garonne   1       1
    3047130     3013767 link
    3074362     3013767 en      Haute Garonne
    4288095     3013767         Département de la Haute-Garonne
    10273497    3013767 link

    To understand these columns, let's look again at the documentation.

    The table 'alternate names' :
    alternateNameId   : the id of this alternate name, int
    geonameid         : geonameId referring to id in table 'geoname', int
    isolanguage       : iso 639 language code 2- or 3-characters; (...)
    alternate name    : alternate name or name variant, varchar(400)
    isPreferredName   : '1', if this alternate name is an official/preferred name
    isShortName       : '1', if this is a short name like 'California' for 'State of California'
    isColloquial      : '1', if this alternate name is a colloquial or slang term
    isHistoric        : '1', if this alternate name is historic and was used in the pastq

    We can see that "Haute-Garonne" is a short version of "Département de la Haute-Garonne" and is the preferred form in French. As an exercise, the reader can confirm in the same way that "France" is the preferred shorter form for "Republic of France" in French.

    And that's it for our introductory journey into Geonames. You are now familiar enough with this data to begin working with it using Pandas in Python. In fact, what we have done until now, namely working with grep commands, is not very useful... See you in part 2!

  • SciviJS

    2016/10/10 by Martin Renou


    The goal of my work at Logilab is to create tools to visualize scientific 3D volumic-mesh-based data (mechanical data, electromagnetic...) in a standard web browser. It's a part of the european OpenDreamKit project. Franck Wang has been working on this subject last year. I based my work on his results and tried to improve them.

    Our goal is to create widgets to be used in Jupyter Notebook (formerly IPython) for easy 3D visualization and analysis. We also want to create a graphical user interface in order to enable users to intuitively compute multiple effects on their meshes.

    As Franck Wang worked with X3DOM, which is an open source JavaScript framework that makes it possible to display 3D scenes using HTML nodes, we first thought it was a good idea to keep on working with this framework. But X3DOM is not very well maintained these days, as can be seen on their GitHub repository.

    As a consequence, we decided to take a look at another 3D framework. Our best candidates were:

    • ThreeJS
    • BabylonJS

    ThreeJS and BabylonJS are two well-known Open Source frameworks for 3D web visualization. They are well maintained by hundreds of contributors since several years. Even if BabylonJS was first thought for video games, these two engines are interesting for our project. Some advantages of ThreeJS are:

    Finally, the choice of using ThreeJS was quite obvious because of its Nodes feature, contributed by Sunag Entertainment. It allows users to compose multiple effects like isocolor, threshold, clip plane, etc. As ThreeJS is an Open Source framework, it is quite easy to propose new features and contributors are very helpful.


    As we want to compose multiple effects like isocolor and threshold (the pixel color correspond to a pressure but if this pressure is under a certain threshold we don't want to display it), it seems a good idea to compose shaders instead of creating a big shader with all the features we want to implement. The problem is that WebGL is still limited (as of the 1.x version) and it's not possible for shaders to exchange data with other shaders. Only the vertex shader can send data to the fragment shader through varyings.

    So it's not really possible to compose shaders, but the good news is we can use the new node system of ThreeJS to easily compute and compose a complex material for a mesh.

    alternate text

    It's the graphical view of what you can do in your code, but you can see that it's really simple to implement effects in order to visualize your data.


    With this great tools as a solid basis, I designed a first version of a javascript library, SciviJS, that aims at loading, displaying and analyzing mesh data in a standard web browser (i.e. without any plugin).

    You can define your visualization in a .yml file containing urls to your mesh and data and a hierarchy of effects (called block structures).

    See for an online demo.

    You can see the block structure like following:

    Data blocks are instantiated to load the mesh and define basic parameters like color, position etc. Blocks are connected together to form a tree that helps building a visual analysis of your mesh data. Each block receives data (like mesh variables, color and position) from its parent and can modify them independently.

    Following parameters must be set on dataBlocks:

    • coordURL: URL to the binary file containing coordinate values of vertices.
    • facesURL: URL to the binary file containing indices of faces defining the skin of the mesh.
    • tetrasURL: URL to the binary file containing indices of tetrahedrons. Default is ''.
    • dataURL: URL to the binary file containing data that you want to visualize for each vertices.

    Following parameters can be set on dataBlocks or plugInBlocks:

    • type: type of the block, which is dataBlock or the name of the plugInBlock that you want.
    • colored: define whether or not the 3D object is colored. Default is false, object is rendered gray.
    • colorMap: color map used for coloration, available values are rainbow and gray. Default is rainbow.
    • colorMapMin and colorMapMax: bounds for coloration scaled in [0, 1]. Default is (0, 1).
    • visualizedData: data used as input for coloration. If data are 3D vectors available values are magnitude, X, Y, Z, and default is magnitude. If data are scalar values you don't need to set this parameter.
    • position, rotation, scale: 3D vectors representing position, rotation and scale of the object. Default are [0., 0., 0.], [0., 0., 0.] and [1., 1., 1.].
    • visible: define whether or not the object is visible. Default is true if there's no childrenBlock, false otherwise.
    • childrenBlocks: array of children blocks. Default is empty.

    As of today, there are 6 types of plug-in blocks:

    • Threshold: hide areas of your mesh based on a variable's value and bound parameters

      • lowerBound: lower bound used for threshold. Default is 0 (representing dataMin). If inputData is under lowerBound, then it's not displayed.
      • upperBound: upper bound used for threshold. Default is 1 (representing dataMax). If inputData is above upperBound, then it's not displayed.
      • inputData: data used for threshold effect. Default is visualizedData, but you can set it to magnitude, X, Y or Z.
    • ClipPlane: hide a part of the mesh by cutting it with a plane

      • planeNormal: 3D array representing the normal of the plane used for section. Default is [1., 0., 0.].
      • planePosition: position of the plane for the section. It's a scalar scaled bewteen -1 and 1. Default is 0.
    • Slice: make a slice of your mesh

      • sliceNormal
      • slicePosition
    • Warp: deform the mesh along the direction of an input vector data

      • warpFactor: deformation factor. Default is 1, can be negative.
      • inputData: vector data used for warp effect. Default is data, but you can set it to X, Y or Z to use only one vector component.
    • VectorField: represent the input vector data with arrow glyphs

      • lengthFactor: factor of length of vectors. Default is 1, can be negative.
      • inputData
      • nbVectors: max number of vectors. Default is the number of vertices of the mesh (which is the maximum value).
      • mode: mode of distribution. Default is volume, you can set it to surface.
      • distribution: type of distribution. Default is regular, you can set it to random.
    • Points: represent the data with points

      • pointsSize: size of points in pixels. Default is 3.
      • nbPoints
      • mode
      • distribution

    Using those blocks you can easily render interesting 3D scenes like this:

    Future works

    • Integration to Jupyter Notebook
    • As of today you only can define a .yml file defining the tree of blocks, we plan to develop a Graphical User Interface to enable users to define this tree interactively with drag and drop
    • Support of most file types (for now it only supports binary files)

  • ngReact: getting angular and react to work together

    2016/08/03 by Nicolas Chauvat

    ngReact is an Angular module that allows React components to be used in AngularJS applications.

    I had to work on enhancing an Angular-based application and wanted to provide the additionnal functionnality as an isolated component that I could develop and test without messing with a large Angular controller that several other people were working on.

    Here is my Angular+React "Hello World", with a couple gotchas that were not underlined in the documentation and took me some time to figure out.

    To set things up, just run:

    $ mkdir angulareacthello && cd angulareacthello
    $ npm init && npm install --save angular ngreact react react-dom

    Then write into index.html:

    <!doctype html>
                 <title>my angular react demo</title>
         <body ng-app="app" ng-controller="helloController">
                         <input type="text" ng-model="" placeholder="Enter a name here">
                         <h1><react-component name="HelloComponent" props="person" /></h1>
         <script src="node_modules/angular/angular.js"></script>
         <script src="node_modules/react/dist/react.js"></script>
         <script src="node_modules/react-dom/dist/react-dom.js"></script>
         <script src="node_modules/ngreact/ngReact.js"></script>
         // include the ngReact module as a dependency for this Angular app
         var app = angular.module('app', ['react']);
         // define a controller that has the name attribute
         app.controller('helloController', function($scope) {
                 $scope.person = { name: 'you' };
         // define a React component that displays "Hello {name}"
         var HelloComponent = React.createClass({
                 render: function() {
                         return React.DOM.span(null, "Hello ";
         // tell Angular about this React component
         app.value('HelloComponent', HelloComponent);

    I took me time to get a couple things clear in my mind.

    <react-component> is not a React component, but an Angular directive that delegates to a React component. Therefore, you should not expect the interface of this tag to be the same as the one of a React component. More precisely, you can only use the props attribute and can not set your react properties by adding more attributes to this tag. If you want to be able to write something like <react-component firstname="person.firstname" lastname="person.lastname"> you will have to use reactDirective to create a specific Angular directive.

    You have to set an object as the props attribute of the react-component tag, because it will be used as the value of this.props in the code of your React class. For example if you set the props attribute to a string ( instead of person in the above example) , you will have trouble using it on the React side because you will get an object built from the enumeration of the string. Therefore, the above example can not be made simpler. If we had written $ = 'you' we could not have passed it correctly to the react component.

    The above was tested with angular@1.5.8, ngreact@0.3.0, react@15.3.0 and react-dom@15.3.0.

    All in all, it worked well. Thank you to all the developers and contributors of these projects.

  • Testing salt formulas with testinfra

    2016/07/21 by Philippe Pepiot

    In a previous post we talked about an environment to develop salt formulas. To add some spicy requirements, the formula must now handle multiple target OS (Debian and Centos), have tests and a continuous integration (CI) server setup.

    I started a year ago to write a framework to this purpose, it's called testinfra and is used to execute commands on remote systems and make assertions on the state and the behavior of the system. The modules API provides a pythonic way to inspect the system. It has a smooth integration with pytest that adds some useful features out of the box like parametrization to run tests against multiple systems.

    Writing useful tests is not an easy task, my advice is to test code that triggers implicit actions, code that has caused issues in the past or simply test the application is working correctly like you would do in the shell.

    For instance this is one of the tests I wrote for the saemref formula

    def test_saemref_running(Process, Service, Socket, Command):
        assert Service("supervisord").is_enabled
        supervisord = Process.get(comm="supervisord")
        # Supervisor run as root
        assert supervisord.user == "root"
        assert == "root"
        cubicweb = Process.get(
        # Cubicweb should run as saemref user
        assert cubicweb.user == "saemref"
        assert == "saemref"
        assert cubicweb.comm == "uwsgi"
        # Should have 2 worker process with 8 thread each and 1 http proccess with one thread
        child_threads = sorted([c.nlwp for c in Process.filter(])
        assert child_threads == [1, 8, 8]
        # uwsgi should bind on all ipv4 adresses
        assert Socket("tcp://").is_listening
        html = Command.check_output("curl http://localhost:8080")
        assert "<title>accueil (Référentiel SAEM)</title>" in html

    Now we can run tests against a running container by giving its name or docker id to testinfra:

    % testinfra --hosts=docker://1a8ddedf8164
    test/[docker:/1a8ddedf8164] PASSED

    The immediate advantage of writing such test is that you can reuse it for monitoring purpose, testinfra can behave like a nagios plugin:

    % testinfra -qq --nagios --hosts=ssh://prod
    TESTINFRA OK - 1 passed, 0 failed, 0 skipped in 2.31 seconds

    We can now integrate the test suite in our by adding some code to build and run a provisioned docker image and add a test command that runs testinfra tests against it.

    provision_option = click.option('--provision', is_flag=True, help="Provision the container")
    @cli.command(help="Build an image")
    def build(image, provision=False):
        dockerfile = "test/{0}.Dockerfile".format(image)
        tag = "{0}-formula:{1}".format(formula, image)
        if provision:
            dockerfile_content = open(dockerfile).read()
            dockerfile_content += "\n" + "\n".join([
                "ADD test/minion.conf /etc/salt/minion.d/minion.conf",
                "ADD {0} /srv/formula/{0}".format(formula),
                "RUN salt-call --retcode-passthrough state.sls {0}".format(formula),
            ]) + "\n"
            dockerfile = "test/{0}_provisioned.Dockerfile".format(image)
            with open(dockerfile, "w") as f:
            tag += "-provisioned"
        subprocess.check_call(["docker", "build", "-t", tag, "-f", dockerfile, "."])
        return tag
    @cli.command(help="Spawn an interactive shell in a new container")
    def dev(ctx, image, provision=False):
        tag = ctx.invoke(build, image=image, provision=provision)[
            "docker", "run", "-i", "-t", "--rm", "--hostname", image,
            "-v", "{0}/test/minion.conf:/etc/salt/minion.d/minion.conf".format(BASEDIR),
            "-v", "{0}/{1}:/srv/formula/{1}".format(BASEDIR, formula),
            tag, "/bin/bash",
    @cli.command(help="Run tests against a provisioned container",
                 context_settings={"allow_extra_args": True})
    def test(ctx, image):
        import pytest
        tag = ctx.invoke(build, image=image, provision=True)
        docker_id = subprocess.check_output([
            "docker", "run", "-d", "--hostname", image,
            "-v", "{0}/test/minion.conf:/etc/salt/minion.d/minion.conf".format(BASEDIR),
            "-v", "{0}/{1}:/srv/formula/{1}".format(BASEDIR, formula),
            tag, "tail", "-f", "/dev/null",
            ctx.exit(pytest.main(["--hosts=docker://" + docker_id] + ctx.args))
            subprocess.check_call(["docker", "rm", "-f", docker_id])

    Tests can be run on a local CI server or on travis, they "just" require a docker server, here is an example of .travis.yml

    sudo: required
      - docker
    language: python
      - "2.7"
        - IMAGE=centos7
        - IMAGE=jessie
      - pip install testinfra
      - python test $IMAGE -- -v

    I wrote a dummy formula with the above code, feel free to use it as a template for your own formula or open pull requests and break some tests.

    There is a highly enhanced version of this code in the saemref formula repository, including:

    • Building a provisioned docker image with custom pillars, we use it to run an online demo
    • Destructive tests where each test is run in a dedicated "fresh" container
    • Run Systemd in the containers to get a system close to the production one (this enables the use of Salt service module)
    • Run a postgresql container linked to the tested container for specific tests like upgrading a Cubicweb instance.

    Destructive tests rely on advanced pytest features that may produce weird bugs when mixed together, too much magic involved here. Also, handling Systemd in docker is really painful and adds a lot of complexity, for instance some systemctl commands require a running systemd as PID 1 and this is not the case during the docker build phase. So the trade-off between complexity and these features may not be worth.

    There is also a lot of quite new tools to develop and test infrastructure code that you could include in your stack like test-kitchen, serverspec, and goss. Choose your weapon and go test your infrastructure code.

  • Developing salt formulas with docker

    2016/07/21 by Philippe Pepiot

    While developing salt formulas I was looking for a simple and reproducible environment to allow faster development, less bugs and more fun. The formula must handle multiple target OS (Debian and Centos).

    The first barrier is the master/minion installation of Salt, but fortunately Salt has a masterless mode. The idea is quite simple, bring up a virtual machine, install a Salt minion on it, expose the code inside the VM and call Salt states.

    At Logilab we like to work with docker, a lightweight OS-level virtualization solution. One of the key features is docker volumes to share local files inside the container. So I started to write a simple Python script to build a container with a Salt minion installed and run it with formula states and a few config files shared inside the VM.

    The formula I was working on is used to deploy the saemref project, which is a Cubicweb based application:

    % cat test/centos7.Dockerfile
    FROM centos:7
    RUN yum -y install epel-release && \
        yum -y install && \
        yum clean expire-cache && \
        yum -y install salt-minion
    % cat test/jessie.Dockerfile
    FROM debian:jessie
    RUN apt-get update && apt-get -y install wget
    RUN wget -O - | apt-key add -
    RUN echo "deb jessie main" > /etc/apt/sources.list.d/saltstack.list
    RUN apt-get update && apt-get -y install salt-minion
    % cat test/minion.conf
    file_client: local
        - /srv/salt
        - /srv/formula

    And finally the file, using the beautiful click module

    #!/usr/bin/env python
    import os
    import subprocess
    import click
    def cli():
    formula = "saemref"
    BASEDIR = os.path.abspath(os.path.dirname(__file__))
    image_choice = click.argument("image", type=click.Choice(["centos7", "jessie"]))
    @cli.command(help="Build an image")
    def build(image):
        dockerfile = "test/{0}.Dockerfile".format(image)
        tag = "{0}-formula:{1}".format(formula, image)
        subprocess.check_call(["docker", "build", "-t", tag, "-f", dockerfile, "."])
        return tag
    @cli.command(help="Spawn an interactive shell in a new container")
    def dev(ctx, image):
        tag = ctx.invoke(build, image=image)[
            "docker", "run", "-i", "-t", "--rm", "--hostname", image,
            "-v", "{0}/test/minion.conf:/etc/salt/minion.d/minion.conf".format(BASEDIR),
            "-v", "{0}/{1}:/srv/formula/{1}".format(BASEDIR, formula),
            tag, "/bin/bash",
    if __name__ == "__main__":

    Now I can run quickly multiple containers and test my Salt states inside the containers while editing the code locally:

    % ./ dev centos7
    [root@centos7 /]# salt-call state.sls saemref
    [ ... ]
    [root@centos7 /]# ^D
    % # The container is destroyed when it exits

    Notice that we could add some custom pillars and state files simply by adding specific docker shared volumes.

    With a few lines we created a lightweight vagrant like, but faster, with docker instead of virtualbox and it remain fully customizable for future needs.

  • Introduction to thesauri and SKOS

    2016/06/27 by Yann Voté

    Recently, I've faced the problem to import the European Union thesaurus, Eurovoc, into cubicweb using the SKOS cube. Eurovoc doesn't follow the SKOS data model and I'll show here how I managed to adapt Eurovoc to fit in SKOS.

    This article is in two parts:

    • this is the first part where I introduce what a thesaurus is and what SKOS is,
    • the second part will show how to convert Eurovoc to plain SKOS.

    The whole text assumes familiarity with RDF, as describing RDF would require more than a blog entry and is out of scope.

    What is a thesaurus ?

    A common need in our digital lives is to attach keywords to documents, web pages, pictures, and so on, so that search is easier. For example, you may want to add two keywords:

    • lily,
    • lilium

    in a picture's metadata about this flower. If you have a large collection of flower pictures, this will make your life easier when you want to search for a particular species later on.

    free-text keywords on a picture

    In this example, keywords are free: you can choose whatever keyword you want, very general or very specific. For example you may just use the keyword:

    • flower

    if you don't care about species. You are also free to use lowercase or uppercase letters, and to make typos...

    free-text keyword on a picture

    On the other side, sometimes you have to select keywords from a list. Such a constrained list is called a controlled vocabulary. For instance, a very simple controlled vocabulary with only two keywords is the one about a person's gender:

    • male (or man),
    • female (or woman).
    a simple controlled vocabulary

    But there are more complex examples: think about how a library organizes books by themes: there are very general themes (eg. Science), then more and more specific ones (eg. Computer science -> Software -> Operating systems). There may also be synonyms (eg. Computing for Computer science) or referrals (eg. there may be a "see also" link between keywords Algebra and Geometry). Such a controlled vocabulary where keywords are organized in a tree structure, and with relations like synonym and referral, is called a thesaurus.

    an example thesaurus with a tree of keywords

    For the sake of simplicity, in the following we will call thesaurus any controlled vocabulary, even a simple one with two keywords like male/female.


    SKOS, from the World Wide Web Consortium (W3C), is an ontology for the semantic web describing thesauri. To make it simple, it is a common data model for thesauri that can be used on the web. If you have a thesaurus and publish it on the web using SKOS, then anyone can understand how your thesaurus is organized.

    SKOS is very versatile. You can use it to produce very simple thesauri (like male/female) and very complex ones, with a tree of keywords, even in multiple languages.

    To cope with this complexity, SKOS data model splits each keyword into two entities: a concept and its labels. For example, the concept of a male person have multiple labels: male and man in English, homme and masculin in French. The concept of a lily flower also has multiple labels: lily in English, lilium in Latin, lys in French.

    Among all labels for a given concept, some can be preferred, while others are alternative. There may be only one preferred label per language. In the person's gender example, man may be the preferred label in English and male an alternative one, while in French homme would be the preferred label and masculin and alternative one. In the flower example, lily (resp. lys) is the preferred label in English (resp. French), and lilium is an alternative label in Latin (no preferred label in Latin).

    SKOS concepts and labels

    And of course, in SKOS, it is possible to say that a concept is broader than another one (just like topic Science is broader than topic Computer science).

    So to summarize, in SKOS, a thesaurus is a tree of concepts, and each concept have one or more labels, preferred or alternative. A thesaurus is also called a concept scheme in SKOS.

    Also, please note that SKOS data model is slightly more complicated than what we've shown here, but this will be sufficient for our purpose.

    RDF URIs defined by SKOS

    In order to publish a thesaurus in RDF using SKOS ontology, SKOS introduces the "skos:" namespace associated to the following URI:

    Within that namespace, SKOS defines some classes and predicates corresponding to what has been described above. For example:

    • the triple (<uri>, rdf:type, skos:ConceptScheme) says that <uri> belongs to class skos:ConceptScheme (that is, is a concept scheme),
    • the triple (<uri>, rdf:type, skos:Concept) says that <uri> belongs to class skos:Concept (that is, is a concept),
    • the triple (<uri>, skos:prefLabel, <literal>) says that <literal> is a preferred label for concept <uri>,
    • the triple (<uri>, skos:altLabel, <literal>) says that <literal> is an alternative label for concept <uri>,
    • the triple (<uri1>, skos:broader, <uri2>) says that concept <uri2> is a broder concept of <uri1>.

  • One way to convert Eurovoc into plain SKOS

    2016/06/27 by Yann Voté

    This is the second part of an article where I show how to import the Eurovoc thesaurus from the European Union into an application using a plain SKOS data model. I've recently faced the problem of importing Eurovoc into CubicWeb using the SKOS cube, and the solution I've chose is discussed here.

    The first part was an introduction to thesauri and SKOS.

    The whole article assumes familiarity with RDF, as describing RDF would require more than a blog entry and is out of scope.

    Difficulties with Eurovoc and SKOS


    Eurovoc is the main thesaurus covering European Union business domains. It is published and maintained by the EU commission. It is quite complex and big, structured as a tree of keywords.

    You can see Eurovoc keywords and browse the tree from the Eurovoc homepage using the link Browse the subject-oriented version.

    For example, when publishing statistics about education in the EU, you can tag the published data with the broadest keyword Education and communications. Or you can be more precise and use the following narrower keywords, in increasing order of preference: Education, Education policy, Education statistics.

    Problem: hierarchy of thesauri

    The EU commission uses SKOS to publish its Eurovoc thesaurus, so it should be straightforward to import Eurovoc into our own application. But things are not that simple...

    For some reasons, Eurovoc uses a hierarchy of concept schemes. For example, Education and communications is a sub-concept scheme of Eurovoc (it is called a domain), and Education is a sub-concept scheme of Education and communications (it is called a micro-thesaurus). Education policy is (a label of) the first concept in this hierarchy.

    But with SKOS this is not possible: a concept scheme cannot be contained into another concept scheme.

    Possible solutions

    So to import Eurovoc into our SKOS application, and not loose data, one solution is to turn sub-concept schemes into concepts. We have two strategies:

    • keep only one concept scheme (Eurovoc) and turn domains and micro-thesauri into concepts,
    • keep domains as concept schemes, drop Eurovoc concept scheme, and only turn micro-thesauri into concepts.

    Here we will discuss the latter solution.

    Lets get to work

    Eurovoc thesaurus can be downloaded at the following URL:

    The ZIP archive contains only one XML file named eurovoc_skos.rdf. Put it somewhere where you can find it easily.

    To read this file easily, we will use the RDFLib Python library. This library makes it really convenient to work with RDF data. It has only one drawback: it is very slow. Reading the whole Eurovoc thesaurus with it takes a very long time. Make the process faster is the first thing to consider for later improvements.

    Reading the Eurovoc thesaurus is as simple as creating an empty RDF Graph and parsing the file. As said above, this takes a long long time (from half an hour to two hours).

    import rdflib
    eurovoc_graph = rdflib.Graph()
    eurovoc_graph.parse('<path/to/eurovoc_skos.rdf>', format='xml')
    <Graph identifier=N52834ca3766d4e71b5e08d50788c5a13 (<class 'rdflib.graph.Graph'>)>

    We can see that Eurovoc contains more than 2 million triples.


    Now, before actually converting Eurovoc to plain SKOS, lets introduce some helper functions:

    • the first one, uriref(), will allow us to build RDFLib URIRef objects from simple prefixed URIs like skos:prefLabel or dcterms:title,
    • the second one, capitalized_eurovoc_domains(), is used to convert Eurovoc domain names, all uppercase (eg. 32 EDUCATION ET COMMUNICATION) to a string where only first letter is uppercase (eg. 32 Education and communication)
    import re
    from rdflib import Literal, Namespace, RDF, URIRef
    from rdflib.namespace import DCTERMS, SKOS
    eu_ns = Namespace('')
    thes_ns = Namespace('')
    prefixes = {
        'dcterms': DCTERMS,
        'skos': SKOS,
        'eu': eu_ns,
        'thes': thes_ns,
    def uriref(prefixed_uri):
        prefix, value = prefixed_uri.split(':', 1)
        ns = prefixes[prefix]
        return ns[value]
    def capitalized_eurovoc_domain(domain):
        """Return the given Eurovoc domain name with only the first letter uppercase."""
        return re.sub(r'^(\d+\s)(.)(.+)$',
                      lambda m: u'{0}{1}{2}'.format(,,,
                      domain, re.UNICODE)

    Now the actual work. After using variables to reference URIs, the loop will parse each triple in original graph and:

    • discard it if it contains deprecated data,
    • if triple is like (<uri>, rdf:type, eu:Domain), replace it with (<uri>, rdf:type, skos:ConceptScheme),
    • if triple is like (<uri>, rdf:type, eu:MicroThesaurus), replace it with (<uri>, rdf:type, skos:Concept) and add triple (<uri>, skos:inScheme, <domain_uri>),
    • if triple is like (<uri>, rdf:type, eu:ThesaurusConcept), replace it with (<uri>, rdf:type, skos:Concept),
    • if triple is like (<uri>, skos:topConceptOf, <microthes_uri>), replace it with (<uri>, skos:broader, <microthes_uri>),
    • if triple is like (<uri>, skos:inScheme, <microthes_uri>), replace it with (<uri>, skos:inScheme, <domain_uri>),
    • keep triples like (<uri>, skos:prefLabel, <label_uri>), (<uri>, skos:altLabel, <label_uri>), and (<uri>, skos:broader, <concept_uri>),
    • discard all other non-deprecated triples.

    Note that, to replace a micro thesaurus with a domain, we have to build a mapping between each micro thesaurus and its containing domain (microthes2domain dict).

    This loop is also quite long.

    eurovoc_ref = URIRef(u'')
    deprecated_ref = URIRef(u'')
    title_ref = uriref('dcterms:title')
    status_ref = uriref('thes:status')
    class_domain_ref = uriref('eu:Domain')
    rel_domain_ref = uriref('eu:domain')
    microthes_ref = uriref('eu:MicroThesaurus')
    thesconcept_ref = uriref('eu:ThesaurusConcept')
    concept_scheme_ref = uriref('skos:ConceptScheme')
    concept_ref = uriref('skos:Concept')
    pref_label_ref = uriref('skos:prefLabel')
    alt_label_ref = uriref('skos:altLabel')
    in_scheme_ref = uriref('skos:inScheme')
    broader_ref = uriref('skos:broader')
    top_concept_ref = uriref('skos:topConceptOf')
    microthes2domain = dict((mt, next(eurovoc_graph.objects(mt, uriref('eu:domain'))))
                            for mt in eurovoc_graph.subjects(RDF.type, uriref('eu:MicroThesaurus')))
    new_graph = rdflib.ConjunctiveGraph()
    for subj_ref, pred_ref, obj_ref in eurovoc_graph:
        if deprecated_ref in list(eurovoc_graph.objects(subj_ref, status_ref)):
        # Convert eu:Domain into a skos:ConceptScheme
        if obj_ref == class_domain_ref:
            new_graph.add((subj_ref, RDF.type, concept_scheme_ref))
            for title in eurovoc_graph.objects(subj_ref, pref_label_ref):
                if title.language == u'en':
                    new_graph.add((subj_ref, title_ref,
        # Convert eu:MicroThesaurus into a skos:Concept
        elif obj_ref == microthes_ref:
            new_graph.add((subj_ref, RDF.type, concept_ref))
            scheme_ref = next(eurovoc_graph.objects(subj_ref, rel_domain_ref))
            new_graph.add((subj_ref, in_scheme_ref, scheme_ref))
        # Convert eu:ThesaurusConcept into a skos:Concept
        elif obj_ref == thesconcept_ref:
            new_graph.add((subj_ref, RDF.type, concept_ref))
        # Replace <concept> topConceptOf <MicroThesaurus> by <concept> broader <MicroThesaurus>
        elif pred_ref == top_concept_ref:
            new_graph.add((subj_ref, broader_ref, obj_ref))
        # Replace <concept> skos:inScheme <MicroThes> by <concept> skos:inScheme <Domain>
        elif pred_ref == in_scheme_ref and obj_ref in microthes2domain:
            new_graph.add((subj_ref, in_scheme_ref, microthes2domain[obj_ref]))
        # Keep label triples
        elif (subj_ref != eurovoc_ref and obj_ref != eurovoc_ref
              and pred_ref in (pref_label_ref, alt_label_ref)):
            new_graph.add((subj_ref, pred_ref, obj_ref))
        # Keep existing skos:broader relations and existing concepts
        elif pred_ref == broader_ref or obj_ref == concept_ref:
            new_graph.add((subj_ref, pred_ref, obj_ref))

    We can check that we now have far less triples than before.


    Now we dump this new graph to disk. We choose the Turtle format as it is far more readable than RDF/XML for humans, and slightly faster to parse for machines. This file will contain plain SKOS data that can be directly imported into any application able to read SKOS.

    with open('eurovoc.n3', 'w') as f:
        new_graph.serialize(f, format='n3')

    With CubicWeb using the SKOS cube, it is a one command step:

    cubicweb-ctl skos-import --cw-store=massive <instance_name> eurovoc.n3

  • Installing Debian Jessie on a "pure UEFI" system

    2016/06/13 by David Douard

    At the core of the Logilab infrastructure is a highly-available pair of small machines dedicated to our main directory and authentication services: LDAP, DNS, DHCP, Kerberos and Radius.

    The machines are small fanless boxes powered by a 1GHz Via Eden processor, 512Mb of RAM and 2Gb of storage on a CompactFlash module.

    They have served us well for many years, but now is the time for an improvement. We've bought a pair of Lanner FW-7543B that have the same form-factor. They are not fanless, but are much more powerful. They are pretty nice, but have one major drawback: their firmware does not boot on a legacy BIOS-mode device when set up in UEFI. Another hard point is that they do not have a video connector (there is a VGA output on the motherboard, but the connector is optional), so everything must be done via the serial console.

    I knew the Debian Jessie installer would provide everything that is required to handle an UEFI-based system, but it took me a few tries to get it to boot.

    First, I tried the standard netboot image, but the firmware did not want to boot from a USB stick, probably because the image requires a MBR-based bootloader.

    Then I tried to boot from the Refind bootable image and it worked! At least I had the proof this little beast could boot in UEFI. But, although it is probably possible, I could not figure out how to tweak the Refind config file to make it boot properly the Debian installer kernel and initrd.

    Finally I gave a try to something I know much better: Grub. Here is what I did to have a working UEFI Debian installer on a USB key.


    First, in the UEFI world, you need a GPT partition table with a FAT partition typed "EFI System":

    david@laptop:~$ sudo fdisk /dev/sdb
    Welcome to fdisk (util-linux 2.25.2).
    Changes will remain in memory only, until you decide to write them.
    Be careful before using the write command.
    Command (m for help): g
    Created a new GPT disklabel (GUID: 52FFD2F9-45D6-40A5-8E00-B35B28D6C33D).
    Command (m for help): n
    Partition number (1-128, default 1): 1
    First sector (2048-3915742, default 2048): 2048
    Last sector, +sectors or +size{K,M,G,T,P} (2048-3915742, default 3915742):  +100M
    Created a new partition 1 of type 'Linux filesystem' and of size 100 MiB.
    Command (m for help): t
    Selected partition 1
    Partition type (type L to list all types): 1
    Changed type of partition 'Linux filesystem' to 'EFI System'.
    Command (m for help): p
    Disk /dev/sdb: 1.9 GiB, 2004877312 bytes, 3915776 sectors
    Units: sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disklabel type: gpt
    Disk identifier: 52FFD2F9-45D6-40A5-8E00-B35B28D6C33D
    Device     Start    End Sectors  Size Type
    /dev/sdb1   2048 206847  204800  100M EFI System
    Command (m for help): w

    Install Grub

    Now we need to install a grub-efi bootloader in this partition:

    david@laptop:~$ pmount sdb1
    david@laptop:~$ sudo grub-install --target x86_64-efi --efi-directory /media/sdb1/ --removable --boot-directory=/media/sdb1/boot
    Installing for x86_64-efi platform.
    Installation finished. No error reported.

    Copy the Debian Installer

    Our next step is to copy the Debian's netboot kernel and initrd on the USB key:

    david@laptop:~$ mkdir /media/sdb1/EFI/debian
    david@laptop:~$ wget -O /media/sdb1/EFI/debian/linux
    --2016-06-13 18:40:02--  /images/netboot/debian-installer/amd64/linux
    Resolving (, 2a01:e0c:1:1598::2
    Connecting to (||:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 3120416 (3.0M) [text/plain]
    Saving to: ‘/media/sdb1/EFI/debian/linux’
    /media/sdb1/EFI/debian/linux      100%[========================================================>]   2.98M      464KB/s   in 6.6s
    2016-06-13 18:40:09 (459 KB/s) - ‘/media/sdb1/EFI/debian/linux’ saved [3120416/3120416]
    david@laptop:~$ wget -O /media/sdb1/EFI/debian/initrd.gz
    --2016-06-13 18:41:30--
    Resolving (, 2a01:e0c:1:1598::2
    Connecting to (||:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 15119287 (14M) [application/x-gzip]
    Saving to: ‘/media/sdb1/EFI/debian/initrd.gz’
    /media/sdb1/EFI/debian/initrd.g    100%[========================================================>]  14.42M    484KB/s   in 31s
    2016-06-13 18:42:02 (471 KB/s) - ‘/media/sdb1/EFI/debian/initrd.gz’ saved [15119287/15119287]

    Configure Grub

    Then, we must write a decent grub.cfg file to load these:

    david@laptop:~$ echo >/media/sdb1/boot/grub/grub.cfg <<EOF
    menuentry "Jessie Installer" {
      insmod part_msdos
      insmod ext2
      insmod part_gpt
      insmod fat
      insmod gzio
      echo  'Loading Linux kernel'
      linux /EFI/debian/linux --- console=ttyS0,115200
      echo 'Loading InitRD'
      initrd /EFI/debian/initrd.gz

    Et voilà, piece of cake!

  • Our work for the OpenDreamKit project during the 77th Sage days

    2016/04/18 by Florent Cayré

    Logilab is part of OpenDreamKit, a Horizon 2020 European Research Infrastructure project that will run until 2019 and provides substantial funding to the open source computational mathematics ecosystem.

    One of the goals of this project is improve the packaging and documentation of SageMath, the open source alternative to Maple and Mathematica.

    The core developers of SageMath organised the 77th Sage days last week and Logilab has taken part, with David Douard, Julien Cristau and I, Florent Cayre.

    David and Julien have been working on packaging SageMath for Debian. This is a huge task (several man-months of work), split into two sub-tasks for now:

    • building SageMath with Debian-packaged versions of its dependencies, if available;
    • packaging some of the missing dependencies, starting with the most expected ones, like the latest releases of Jupyter and IPython.

    As a first result, the following packages have been pushed into Debian experimental:

    There is still a lot of work to be done, and packaging the notebook is the next task on the list.

    One hiccup along the way was a python crash involving multiple inheritance from Cython extensions classes. Having people nearby who knew the SageMath codebase well (or even wrote the relevant parts) was invaluable for debugging, and allowed us to blame a recent CPython change.

    Julien also gave a hand to Florent Hivert and Robert Lehmann who were trying to understand why building SageMath's documentation needed this much memory.

    As far as I am concerned, I made a prototype of a structured HTML documentation produced with Sphinx and containing Python executable code ran on thanks to the Thebe javascript library that interfaces statically delivered HTML pages with a Jupyter notebook server.

    The Sage days have been an excellent opportunity to efficiently work on the technical tasks with skillfull and enthusiastic people. We would like to thank the OpenDreamKit core team for the organization and their hard work. We look forward to the next workshop.

  • 3D Visualization of simulation data with x3dom

    2016/02/16 by Yuanxiang Wang

    X3DOM Plugins

    As part of the Open Dream Kit project, we are working at Logilab on the creation of tools for mesh data visualization and analysis in a web application. Our goal was to create widgets to use in Jupyter notebook (formerly IPython) for 3D visualization and analysis.

    We found two interesting technologies for 3D rendering: ThreeJS and X3DOM. ThreeJS is a large JavaScript 3D library and X3DOM a HTML5 framework for 3D. After working with both, we chose to use X3DOM because of its high level architecture. With X3DOM the 3D model is defined in the DOM in HTML5, so the parameters of the nodes can be changed easily with the setAttribute DOM function. This makes the creation of user interfaces and widgets much easier.

    We worked to create new DOM nodes that integrate nicely in a standard X3DOM tree, namely IsoColor, Threshold and ClipPlane.

    We had two goals in mind:

    1. create an X3DOM plugin API that allows one to create new DOM nodes which extend X3DOM functionality;
    2. keep a simple X3DOM-like interface for the final users.

    Example of the plugins Threshold and IsoColor:


    The Threshold and IsoColor nodes work like any X3DOM node and react to attribute changes performed with the setAttribute method. This makes it easy to use HTML widgets like sliders / buttons to drive the plugin's parameters.

    X3Dom API

    The goal is to create custom nodes that affect the rendering based on data (positions, pressure, temperature...). The idea is to manipulate the shaders, since it gives low-level manipulation on the 3D rendering. Shaders give more freedom and efficiency compared to reusing other X3DOM nodes. (Reminder : Shaders are parts of GLSL, used to work with the GPU).

    X3DOM has a native node to all users to write shaders : the ComposedShader node. The problem of this node is it overwrites the shaders generated by X3DOM. For example, nodes like ClipPlane are disabled with a ComposedShader node in the DOM. Another example is image texturing, the computation of the color from texture coordinate should be written within the ComposedShader.

    In order to add parts of shader to the generated shader without overwriting it I created a new node: CustomAttributeNode. This node is a generic node to add uniforms, varying and shader parts into X3DOW. The data of the geometry (attributes) are set using the X3DOM node named FloatVertexAttribute.

    Example of CustomAttributeNode to create a threshold node:


    The CustomAttributeNode is the entry point in x3dom for the javascript API.

    JavaScript API

    The idea of the the API is to create a new node inherited from CustomAttributeNode. We wrote some functions to make the implementation of the node easier.

    Ideas for future improvement

    There are still some points that need improvement

    • Create a tree widget using the grouping nodes in X3DOM
    • Add high level functions to X3DGeometricPropertyNode to set the values. For instance the IsoColor node is only a node that set the values of the TextureCoordinate node from the FloatVertexAttribute node.
    • Add high level function to return the variable name needed to overwrite the basic attributes like positions in a Geometry. With my API, the IsoColor use a varying defined in X3DOM to overwrite the values of the texture coordinate. Because there are no documentation, it is hard for the users to find the varying names. On the other hand there are no specification on the varying names so it might need to be maintained.
    • Maybe the CustomAttributeNode should be a X3DChildNode instead of a X3DGeometricPropertyNode.


    This structure might allow the "use" attribute in X3DOM. Like that, X3DOM avoid data duplication and writing too much HTML. The following code illustrate what I expect.


  • We went to cfgmgmtcamp 2016 (after FOSDEM)

    2016/02/09 by Arthur Lutz

    Following a day at FOSDEM (another post about it), we spend two days at cfgmgmtcamp in Gent. At cfgmgmtcamp, we obviously spent some time in the Salt track since it's our tool of choice as you might have noticed. But checking out how some of the other tools and communities are finding solutions to similar problems is also great.

    cfgmgmtcamp logo

    I presented Roll out active Supervision with Salt, Graphite and Grafana (mirrored on slideshare), you can find the code on bitbucket.

    We saw :

    Day 1

    • Mark Shuttleworth from Canonical presenting Juju and its ecosystem, software modelling. MASS (Metal As A Service) was demoed on the nice "OrangeBox". It promises to spin up an OpenStack infrastructure in 15 minutes. One of the interesting things with charms and bundles of charms is the interfaces that need to be established between different service bricks. In the salt community we have salt-formulas but they lack maturity in the sense that there's no possibility to plug in multiple formulas that interact with each other... yet.
    juju deploy of openstack
    • Mitch Michell from Hashicorp presented vault. Vault stores your secrets (certificates, passwords, etc.) and we will probably be trying it out in the near future. A lot of concepts in vault are really well thought out and resonate with some of the things we want to do and automate in our infrastructure. The use of Shamir Secret Sharing technique (also used in the debian infrastructure team) for the N-man challenge to unvault the secrets is quite nice. David is already looking into automating it with Salt and having GSSAPI (kerberos) authentication. bikes!

    Day 2

    • Gareth Rushgrove from PuppetLabs talked about the importance of metadata in docker images and docker containers by explaining how these greatly benefit tools like dpkg and rpm and that the container community should be inspired by the amazing skills and experience that has been built by these package management communities (think of all the language-specific package managers that each reinvent the wheel one after the other).
    • Testing Immutable Infrastructure: we found some inspiration from test-kitchen and running the tests inside a docker container instead of vagrant virtual machine. We'll have to take a look at the SaltStack provisioner for test-kitchen. We already do some of that stuff in docker and OpenStack using salt-cloud. But maybe we can take it further with such tools (or testinfra whose author will be joining Logilab next month).
    coreos, rkt, kubernetes
    • How CoreOS is built, modified, and updated: From repo sync to Omaha by Brian "RedBeard" Harrington. Interesting presentation of the CoreOS system. Brian also revealed that CoreOS is now capable of using the TPM to enforce a signed OS, but also signed containers. Official CoreOS images shipped through Omaha are now signed with a root key that can be installed in the TPM of the host (ie. they didn't use a pre-installed Microsoft key), along with a modified TPM-aware version of GRUB. For now, the Omaha platform is not open source, so it may not be that easy to build one's own CoreOS images signed with a personal root key, but it is theoretically possible. Brian also said that he expect their Omaha server implementation to become open source some day.
    • The use of Salt in Foreman was presented and demoed by Stephen Benjamin. We'll have to retry using that tool with the newest features of the smart proxy.
    • Jonathan Boulle from CoreOS presented "rkt and Kubernetes: What’s new with Container Runtimes and Orchestration" In this last talk, Johnathan gave a tour of the rkt project and how it is used to build, coupled with kubernetes, a comprehensive, secure container running infrastructure (which uses saltstack!). He named the result "rktnetes". The idea is to use rkt as the kubelet's (primany node agent) container runtime of a kubernetes cluster powered by CoreOS. Along with the new CoreOS support for TPM-based trust chain, it allows to ensure completely secured executions, from the bootloader to the container! The possibility to run fully secured containers is one of the reasons why CoreOS developed the rkt project.

    We would like to thank the cfgmgmntcamp organisation team, it was a great conference, we highly recommend it. Thanks for the speaker event the night before the conference, and the social event on Monday evening. (and thanks for the chocolate!).

  • We went to FOSDEM 2016 (and cfgmgmtcamp)

    2016/02/09 by Arthur Lutz

    David & I went to FOSDEM and cfgmgmtcamp this year to see some conferences, do two presentations, and discuss with the members of the open source communities we contribute to.

    At FOSDEM, we started early by doing a presentation at 9.00 am in the "Configuration Management devroom", which to our surprise was a large room which was almost full. The presentation was streamed over the Internet and should be available to view shortly.

    I presented "Once you've configured your infrastructure using salt, monitor it by re-using that definition". (mirrored on slideshare. The main part was a demo, the code being published on bitbucket.

    The presentation was streamed live (I came across someone that watched it on the Internet to "sleep in"), and should be available to watch when it gets encoded on

    FOSDEM video box

    We then saw the following talks :

    • Unified Framework for Big Data Foreign Data Wrappers (FDW) by Shivram Mani in the Postgresql Track
    • Mainflux Open Source IoT Cloud
    • EzBench, a tool to help you benchmark and bisect the Graphics Stack's performance
    • The RTC components in the debian infrastructure
    • CoreOS: A Linux distribution designed for application containers that scale
    • Using PostgreSQL for Bibliographic Data (since we've worked on with and PostgreSQL)
    • The FOSDEM infrastructure review

    Congratulations to all the FOSDEM organisers, volunteers and speakers. We will hopefully be back for more.

    We then took the train to Gent where we spent two days learning and sharing about Configuration Management Systems and all the ecosystem around it (orchestration, containers, clouds, testing, etc.).

    More on our cfmgmtcamp experience in another blog post.

    Photos under creative commons CC-BY, by Ludovic Hirlimann and Deborah Bryant here and here

  • DebConf15 wrap-up

    2015/08/25 by Julien Cristau

    I just came back from two weeks in Heidelberg for DebCamp15 and DebConf15.

    In the first week, besides helping out DebConf's infrastructure team with network setup, I tried to make some progress on the library transitions triggered by libstdc++6's C++11 changes. At first, I spent many hours going through header files for a bunch of libraries trying to figure out if the public API involved std::string or std::list. It turns out that is time-consuming, error-prone, and pretty efficient at making me lose the will to live. So I ended up stealing a script from Steve Langasek to automatically rename library packages for this transition. This ended in 29 non-maintainer uploads to the NEW queue, quickly processed by the FTP team. Sadly the transition is not quite there yet, as making progress with the initial set of packages reveals more libraries that need renaming.

    Building on some earlier work from Laurent Bigonville, I've also moved the setuid root Xorg wrapper from the xserver-xorg package to xserver-xorg-legacy, which is now in experimental. Hopefully that will make its way to sid and stretch soon (need to figure out what to do with non-KMS drivers first).

    Finally, with the help of the security team, the security tracker was moved to a new VM that will hopefully not eat its root filesystem every week as the old one was doing the last few months. Of course, the evening we chose to do this was the night DebConf15's network was being overhauled, which made things more interesting.

    DebConf itself was the opportunity to meet a lot of people. I was particularly happy to meet Andreas Boll, who has been a member of pkg-xorg for two years now, working on our mesa package, among other things. I didn't get to see a lot of talks (too many other things going on), but did enjoy Enrico's stand up comedy, the CitizenFour screening, and Jake Applebaum's keynote. Thankfully, for the rest the video team has done a great job as usual.


    Above picture is by Aigars Mahinovs, licensed under CC-BY 2.0

  • Going to DebConf15

    2015/08/11 by Julien Cristau

    On Sunday I travelled to Heidelberg, Germany, to attend the 16th annual Debian developer's conference, DebConf15.

    The conference itself is not until next week, but this week is DebCamp, a hacking session. I've already met a few of my DSA colleagues, who've been working on setting up the network infrastructure. My other plans for this week involve helping the Big Transition of 2015 along, and trying to remove the setuid bit from /usr/bin/X in the default Debian install (bug #748203 in particular).

    As for next week, there's a rich schedule in which I'll need to pick a few things to go see.


  • Experiments on building a Jenkins CI service with Salt

    2015/06/17 by Denis Laxalde

    In this blog post, I'll talk about my recent experiments on building a continuous integration service with Jenkins that is, as much as possible, managed through Salt. We've been relying on a Jenkins platform for quite some time at Logilab (Tolosa team). The service was mostly managed by me with sporadic help from other team-mates but I've never been entirely satisfied about the way it was managed because it involved a lot of boilerplate configuration through Jenkins user interface and this does not scale very well nor does it make long term maintenance easy.

    So recently, I've taken a stance and decided to go through a Salt-based configuration and management of our Jenkins CI platform. There are actually two aspects here. The first concerns the setup of Jenkins itself (this includes installation, security configuration, plugins management amongst other things). The second concerns the management of client projects (or jobs in Jenkins jargon). For this second aspect, one of the design goals was to enable easy configuration of jobs by users not necessarily familiar with Jenkins setup and to make collaborative maintenance easy. To tackle these two aspects I've essentially been using (or developing) two distinct Salt formulas which I'll detail hereafter.

    Jenkins jobs salt

    Core setup: the jenkins formula

    The core setup of Jenkins is based on an existing Salt formula, the jenkins-formula which I extended a bit to support map.jinja and which was further improved to support installation of plugins by Yann and Laura (see 3b524d4).

    With that, deploying a Jenkins server is as simple as adding the following to your states and pillars top.sls files:

        - jenkins
        - jenkins.plugins

    Base pillar configuration is used to declare anything that differs from the default Jenkins settings in a jenkins section, e.g.:

        - home: /opt/jenkins

    Plugins configuration is declared in plugins subsection as follows:

            url: ''
            hash: 'md5=9574c07bf6bfd02a57b451145c870f0e'
            url: ''
            hash: 'md5=1b46e2732be31b078001bcc548149fe5'

    (Note that plugins dependency is not handled by Jenkins when installing from the command line, neither by this formula. So in the preceding example, just having an entry for the Mercurial plugin would have not been enough because this plugin depends on scm-api.)

    Other aspects (such as security setup) are not handled yet (neither by the original formula, nor by our extension), but I tend to believe that this is acceptable to manage this "by hand" for now.

    Jobs management : the jenkins_jobs formula

    For this task, I leveraged the excellent jenkins-job-builder tool which makes it possible to configure jobs using a declarative YAML syntax. The tool takes care of installing the job and also handles any housekeeping tasks such as checking configuration validity or deleting old configurations. With this tool, my goal was to let end-users of the Jenkins service add their own project by providing a minima a YAML job description file. So for instance, a simple Job description for a CubicWeb job could be:

    - scm:
        name: cubicweb
          - hg:
             clean: true
    - job:
        name: cubicweb
        display-name: CubicWeb
          - cubicweb
          - shell: "find . -name 'tmpdb*' -delete"
          - shell: "tox --hashseed noset"
          - email:

    It consists of two parts:

    • the scm section declares, well, SCM information, here the location of the review Mercurial repository, and,

    • a job section which consists of some metadata (project name), a reference of the SCM section declared above, some builders (here simple shell builders) and a publisher part to send results by email.

    Pretty simple. (Note that most test running configuration is here declared within the source repository, via tox (another story), so that the CI bot holds minimum knowledge and fetches information from the sources repository directly.)

    To automate the deployment of this kind of configurations, I made a jenkins_jobs-formula which takes care of:

    1. installing jenkins-job-builder,
    2. deploying YAML configurations,
    3. running jenkins-jobs update to push jobs into the Jenkins instance.

    In addition to installing the YAML file and triggering a jenkins-jobs update run upon changes of job files, the formula allows for job to list distribution packages that it would require for building.

    Wrapping things up, a pillar declaration of a Jenkins job looks like:

            file: <path to local cubicweb.yaml>
              - mercurial
              - python-dev
              - libgecode-dev

    where the file section indicates the source of the YAML file to install and pkgs lists build dependencies that are not managed by the job itself (typically non Python package in our case).

    So, as an end user, all is needed to provide is the YAML file and a pillar snippet similar to the above.


    This initial setup appears to be enough to greatly reduce the burden of managing a Jenkins server and to allow individual users to contribute jobs for their project based on simple contribution to a Salt configuration.

    Later on, there is a few things I'd like to extend on jenkins_jobs-formula side. Most notably the handling of distant sources for YAML configuration file (as well as maybe the packages list file). I'd also like to experiment on configuring slaves for the Jenkins server, possibly relying on Docker (taking advantage of another of my experiment...).

  • Running a local salt-master to orchestrate docker containers

    2015/05/20 by David Douard

    In a recent blog post, Denis explained how to build Docker containers using Salt.

    What's missing there is how to have a running salt-master dedicated to Docker containers.

    There is not need the salt-master run as root for this. A test config of mine looks like:

    david@perseus:~$ mkdir -p salt/etc/salt
    david@perseus:~$ cd salt
    david@perseus:~salt/$ cat << EOF >etc/salt/master
    user: david
    root_dir: /home/david/salt/
    pidfile: var/run/
    pki_dir: etc/salt/pki/master
    cachedir: var/cache/salt/master
    sock_dir: var/run/salt/master
        - /home/david/salt/states
        - /home/david/salt/formulas/cubicweb
        - /home/david/salt/pillar

    Here, is the ip of my docker0 bridge. Also note that path in file_roots and pillar_roots configs must be absolute (they are not relative to root_dir, see the salt-master configuration documentation).

    Now we can start a salt-master that will be accessible to Docker containers:

    david@perseus:~salt/$ /usr/bin/salt-master -c etc/salt


    with salt 2015.5.0, salt-master really wants to execute dmidecode, so add /usr/sbin to the $PATH variable before running the salt-master as non-root user.

    From there, you can talk to your test salt master by adding -c ~/salt/etc/salt option to all salt commands. Fortunately, you can also set the SALT_CONFIG_DIR environment variable:

    david@perseus:~salt/$ export SALT_CONFIG_DIR=~/salt/etc/salt
    david@perseus:~salt/$ salt-key
    Accepted Keys:
    Denied Keys:
    Unaccepted Keys:
    Rejected Keys:

    Now, you need to have a Docker images with salt-minion already installed, as explained in Denis' blog post. (I prefer using supervisord as PID 1 in my dockers, but that's not important here.)

    david@perseus:~salt/ docker run -d --add-host salt:  logilab/salted_debian:wheezy
    david@perseus:~salt/ docker run -d --name jessie1 --hostname jessie1 --add-host salt:  logilab/salted_debian:jessie
    david@perseus:~salt/ salt-key
    Accepted Keys:
    Denied Keys:
    Unaccepted Keys:
    Rejected Keys:
    david@perseus:~/salt$ salt-key -y -a 53bf7d8db530
    The following keys are going to be accepted:
    Unaccepted Keys:
    Key for minion 53bf7d8db530 accepted.
    david@perseus:~/salt$ salt-key -y -a jessie1
    The following keys are going to be accepted:
    Unaccepted Keys:
    Key for minion jessie1 accepted.
    david@perseus:~/salt$ salt '*'

    You can now build Docker images as explained by Denis, or test your sls config files in containers.

  • Mini-Debconf Lyon 2015

    2015/04/29 by Julien Cristau

    A couple of weeks ago I attended the mini-DebConf organized by Debian France in Lyon.

    It was a really nice week-end, and the first time a French mini-DebConf wasn't in Paris :)

    Among the highlights, Juliette Belin reported on her experience as a new contributor to Debian: she authored the awesome "Lines" theme which was selected as the default theme for Debian 8.


    As a non-developer and newcomer to the free software community, she had quite intesting insights and ideas about areas where development processes need to improve.

    And Raphael Geissert reported on the new service (previously, an http redirector to automagically pick the closest Debian archive mirror. So long, manual sources.list updates on laptops whenever travelling!


    Finally the mini-DebConf was a nice opportunity to celebrate the release of Debian 8, two weeks in advance.

    Now it's time to go and upgrade all our infrastructure to jessie.

  • Building Docker containers using Salt

    2015/04/07 by Denis Laxalde

    In this blog post, I'll talk about a way to use Salt to automate the build and configuration of Docker containers. I will not consider the deployment of Docker containers with Salt as this subject is already covered elsewhere (here for instance). The emphasis here is really on building (or configuring) a container for future deployment.


    Salt is a remote execution framework that can be used for configuration management. It's already widely used at Logilab to manage our infrastructure as well as on a semi-daily basis during our application development activities.

    Docker is a tool that helps automating the deployment of applications within Linux containers. It essentially provides a convenient abstraction and a set of utilities for system level virtualization on Linux. Amongst other things, Docker provides container build helpers around the concept of dockerfile.

    So, the first question is why would you use Salt to build Docker containers when you already have this dockerfile building tool. My first motivation is to encompass the limitations of the available declarations one could insert in a Dockerfile. First limitation: you can only execute instructions in a sequential manner using a Dockerfile, there's is no possibility of declaring dependencies between instructions or even of making an instruction conditional (apart from using the underlying shell conditional machinery of course). Then, you have only limited possibilities of specializing a Dockerfile. Finally, it's no so easy to apply a configuration step-by-step, for instance during the development of said configuration.

    That's enough for an introduction to lay down the underlying motivation of this post. Let's move on to more practical things!

    A Dockerfile for the base image

    Before jumping into the usage of Salt for the configuration of a Docker image, the first thing you need to do is to build a Docker container into a proper Salt minion.

    Assuming we're building on top of some a base image of Debian flavour subsequently referred to as <debian> (I won't tell you where it comes from, since you ought to build your own base image -- or find some friend you trust to provide you with one!), the following Dockerfile can be used to initialize a working image which will serve as the starting point for further configuration with Salt:

    FROM <debian>
    RUN apt-get update
    RUN apt-get install -y salt-minion

    Then, run docker build . docker_salt/debian_salt_minion and you're done.

    Plugin the minion container with the Salt master

    The next thing to do with our fresh Debian+salt-minion image is to turn it into a container running salt-minion, waiting for the Salt master to instruct it.

    docker run --add-host=salt: --hostname docker_minion \
        --name minion_container \
        docker_salt/debian/salt_minion salt-minion


    • --hostname is used to specify the network name of the container, for easier query by the Salt master,
    • is usually the IP address of the host, which in our example will serve as the Salt master,
    • --name is just used for easier book-keeping.


    salt-key -a docker_minion

    will register the new minion's key into the master's keyring.

    If all went well, the following command should succeed:

    salt 'docker_minion'

    Configuring the container with a Salt formula

    salt 'docker_minion' state.sls some_formula
    salt 'docker_minion' state.highstate

    Final steps: save the configured image and build a runnable image

    (Optional step, cleanup salt-minion installation.)

    Make a snapshot image of your configured container.

    docker stop minion_container
    docker commit -m 'Install something with Salt' \
        minion_container me/something

    Try out your new image:

    docker run -p 8080:80 me/something <entry point>

    where <entry point> will be the main program driving the service provided by the container (typically defined through the Salt formula).

    Make a fully configured image for you service:

    FROM me/something
    [...anything else you need, such as EXPOSE, etc...]
    CMD <entry point>

  • Monitoring our websites before we deploy them using Salt

    2015/03/11 by Arthur Lutz

    As you might have noticed we're quite big fans of Salt. One of the things that Salt enables us to do, it to apply what we're used to doing with code to our infrastructure. Let's look at TDD (Test Driven Development).

    Write the test first, make it fail, implement the code, test goes green, you're done.

    Apply the same thing to infrastructure and you get TDI (Test Driven Infrastructure).

    So before you deploy a service, you make sure that your supervision (shinken, nagios, incinga, salt based monitoring, etc.) is doing the correct test, you deploy and then your supervision goes green.

    Let's take a look at website supervision. At Logilab we weren't too satisfied with how our shinken/http_check were working so we started using uptime (nodejs + mongodb). Uptime has a simple REST API to get and add checks, so we wrote a salt execution module and a states module for it.

    For the sites that use the apache-formula we simply loop on the domains declared in the pillars to add checks :

    {% for domain in salt['pillar.get']('apache:sites').keys() %}
    uptime {{ domain }} (http):
        - name : http://{{ domain }}
    {% endfor %}

    For other URLs (specific URL such as sitemaps) we can list them in pillars and do :

    {% for url in salt['pillar.get']('uptime:urls') %}
    uptime {{ url }}:
        - name : {{ url }}
    {% endfor %}

    That's it. Monitoring comes before deployment.

    We've also contributed a formula for deploying uptime.

    Follow us if you are interested in Test Driven Infrastructure for we intend to write regular reports as we make progress exploring this new domain.

  • A report on the Salt Sprint 2015 in Paris

    2015/03/05 by Arthur Lutz

    On Wednesday the 4th of march 2015, Logilab hosted a sprint on salt on the same day as the sprint at SaltConf15. 7 people joined in and hacked on salt for a few hours. We collaboratively chose some subjects on a pad which is still available.


    We started off by familiarising those who had never used them to using tests in salt. Some of us tried to run the tests via tox which didn't work any more, a fix was found and will be submitted to the project.

    We organised in teams.

    Boris & Julien looked at the authorisation code and wrote a few issues (minion enumeration, acl documentation). On saltpad (client side) they modified the targeting to adapt to the permissions that the salt-api sends back.

    We discussed the salt permission model (external_auth) : where should the filter happen ? the master ? should the minion receive information about authorisation and not execute what is being asked for ? Boris will summarise some of the discussion about authorisations in a new issue.


    Sofian worked on some unification on execution modules (refresh_db which will be ignored for the modules that don't understand that). He will submit a pull request in the next few days.

    Georges & Paul added some tests to hg_pillar, the test creates a mercurial repository, adds a top.sls and a file and checks that they are visible. Here is the diff. They had some problems while debugging the tests.

    David & Arthur implemented the execution module for managing postgresql clusters (create, list, exists, remove) in debian. A pull request was submitted by the end of the day. A state module should follow shortly. On the way we removed some dead code in the postgres module.

    All in all, we had some interesting discussions about salt, it's architecture, shared tips about developing and using it and managed to get some code done. Thanks to all for participating and hopefully we'll sprint again soon...

  • Generate stats from your SaltStack infrastructure

    2014/12/15 by Arthur Lutz

    As presented at the November french meetup of saltstack users, we've published code to generate some statistics about a salstack infrastructure. We're using it, for the moment, to identify which parts of our infrastructure need attention. One of the tools we're using to monitor this distance is munin.

    You can grab the code at bitbucket salt-highstate-stats, fork it, post issues, discuss it on the mailing lists.

    If you're french speaking, you can also read the slides of the above presentation (mirrored on slideshare).

    Hope you find it useful.

  • Using Saltstack to limit impact of Poodle SSLv3 vulnerability

    2014/10/15 by Arthur Lutz

    Here at Logilab, we're big fans of SaltStack automation. As seen with Heartbleed, controlling your infrastructure and being able to fix your servers in a matter of a few commands as documented in this blog post. Same applies to Shellshock more recently with this blog post.

    Yesterday we got the news that a big vulnerability on SSL was going to be released. Code name : Poodle. This morning we got the details and started working on a fix through salt.

    So far, we've handled configuration changes and services restart for apache, nginx, postfix and user configuration for iceweasel (debian's firefox) and chromium (adapting to firefox and chrome should be a breeze). Some credit goes to mtpettyp for his answer on askubuntu.
    {% if salt['pkg.version']('apache2') %}
    poodle apache server restart:
            - name: apache2
      {% for foundfile in salt['']('rgrep -m 1 SSLProtocol /etc/apache*').split('\n') %}
        {% if 'No such file' not in foundfile and 'bak' not in foundfile and foundfile.strip() != ''%}
    poodle {{ foundfile.split(':')[0] }}:
            - name : {{ foundfile.split(':')[0] }}
            - pattern: "SSLProtocol all -SSLv2[ ]*$"
            - repl: "SSLProtocol all -SSLv2 -SSLv3"
            - backup: False
            - show_changes: True
            - watch_in:
                service: apache2
        {% endif %}
      {% endfor %}
    {% endif %}
    {% if salt['pkg.version']('nginx') %}
    poodle nginx server restart:
            - name: nginx
      {% for foundfile in salt['']('rgrep -m 1 ssl_protocols /etc/nginx/*').split('\n') %}
        {% if 'No such file' not in foundfile and 'bak' not in foundfile and foundfile.strip() != ''%}
    poodle {{ foundfile.split(':')[0] }}:
            - name : {{ foundfile.split(':')[0] }}
            - pattern: "ssl_protocols .*$"
            - repl: "ssl_protocols TLSv1 TLSv1.1 TLSv1.2;"
            - show_changes: True
            - watch_in:
                service: nginx
        {% endif %}
      {% endfor %}
    {% endif %}
    {% if salt['pkg.version']('postfix') %}
    poodle postfix server restart:
            - name: postfix
    poodle /etc/postfix/
    {% if '' in salt['']('grep smtpd_tls_mandatory_protocols /etc/postfix/') %}
            - pattern: "smtpd_tls_mandatory_protocols=.*"
            - repl: "smtpd_tls_mandatory_protocols=!SSLv2,!SSLv3"
    {% else %}
            - text: |
                # poodle fix
    {% endif %}
            - name: /etc/postfix/
            - watch_in:
                service: postfix
    {% endif %}
    {% if salt['pkg.version']('chromium') %}
            - pattern: Exec=/usr/bin/chromium %U
            - repl: Exec=/usr/bin/chromium --ssl-version-min=tls1 %U
    {% endif %}
    {% if salt['pkg.version']('iceweasel') %}
            - text : pref("security.tls.version.min", "1")
    {% endif %}

    The code is also published as a gist on github. Feel free to comment and fork the gist. There is room for improvement, and don't forget that by disabling SSLv3 you might prevent some users with "legacy" browsers from accessing your services.

  • Report from DebConf14

    2014/09/05 by Julien Cristau

    Last week I attended DebConf14 in Portland, Oregon. As usual the conference was a blur, with lots of talks, lots of new people, and lots of old friends. The organizers tried to do something different this year, with a longer conference (9 days instead of a week) and some dedicated hack time, instead of a pre-DebConf "DebCamp" week. That worked quite well for me, as it meant the schedule was not quite so full with talks, and even though I didn't really get any hacking done, it felt a bit more relaxed and allowed some more hallway track discussions.

    On the talks side, the keynotes from Zack and Biella provided some interesting thoughts. Some nice progress was made on making package builds reproducible.

    I gave two talks: an introduction to salt (odp),

    and a report on the Debian jessie release progress (pdf).

    And as usual all talks were streamed live and recorded, and many are already available thanks to the awesome DebConf video team. Also for a change, and because I'm a sucker for punishment, I came back with more stuff to do.

  • Logilab at Debconf 2014 - Debian annual conference

    2014/08/21 by Arthur Lutz

    Logilab is proud to contribute to the annual debian conference which will take place in Portland (USA) from the 23rd to the 31st of august.

    Julien Cristau (debian page) will be giving two talks at the conference :

    Logilab is also contributing to the conference as a sponsor for the event.

    Here is what we previously blogged about salt and the previous debconf . Stay tuned for a blog post about what we saw and heard at the conference.

  • Pylint 1.3 / Astroid 1.2 released

    2014/07/28 by Sylvain Thenault

    The EP14 Pylint sprint team (more on this here and there) is proud to announce they just released Pylint 1.3 together with its companion Astroid 1.2. As usual, this includes several new features as well and bug fixes. You'll find below some structured list of the changes.

    Packages are uploaded to pypi, debian/ubuntu packages should be soon provided by Logilab, until they get into the standard packaging system of your favorite distribution.

    Please notice Pylint 1.3 will be the last release branch support python 2.5 and 2.6. Starting from 1.4, we will only support python greater or equal to 2.7. This will be the occasion to do some great cleanup in the code base. Notice this is only about the Pylint's runtime, you should still be able to run Pylint on your Python 2.5 code, through using Python 2.7 at least.

    New checks

    • Add multiple checks for PEP 3101 advanced string formatting: 'bad-format-string', 'missing-format-argument-key', 'unused-format-string-argument', 'format-combined-specification', 'missing-format-attribute' and 'invalid-format-index'
    • New 'invalid-slice-index' and 'invalid-sequence-index' for invalid sequence and slice indices
    • New 'assigning-non-slot' warning, which detects assignments to attributes not defined in slots

    Improved checkers

    • Fixed 'fixme' false positive (#149)
    • Fixed 'unbalanced-iterable-unpacking' false positive when encountering starred nodes (#273)
    • Fixed 'bad-format-character' false positive when encountering the 'a' format on Python 3
    • Fixed 'unused-variable' false positive when the variable is assigned through an import (#196)
    • Fixed 'unused-variable' false positive when assigning to a nonlocal (#275)
    • Fixed 'pointless-string-statement' false positive for attribute docstrings (#193)
    • Emit 'undefined-variable' when using the Python 3 metaclass= argument. Also fix 'unused-import' false for that construction (#143)
    • Emit 'broad-except' and 'bare-except' even if the number of except handlers is different than 1. Fixes issue (#113)
    • Emit 'attribute-defined-outside-init' for all statements in the same module as the offended class, not just for the last assignment (#262, as well as a long standing output mangling problem in some edge cases)
    • Emit 'not-callable' when calling properties (#268)
    • Don't let ImportError propagate from the imports checker, leading to crash in some namespace package related cases (#203)
    • Don't emit 'no-name-in-module' for ignored modules (#223)
    • Don't emit 'unnecessary-lambda' if the body of the lambda call contains call chaining (#243)
    • Definition order is considered for classes, function arguments and annotations (#257)
    • Only emit 'attribute-defined-outside-init' for definition within the same module as the offended class, avoiding to mangle the output in some cases
    • Don't emit 'hidden-method' message when the attribute has been monkey-patched, you're on your own when you do that.

    Others changes

    • Checkers are now properly ordered to respect priority(#229)
    • Use the proper mode for pickle when opening and writing the stats file (#148)

    Astroid changes

    • Function nodes can detect decorator call chain and see if they are decorated with builtin descriptors (classmethod and staticmethod).
    • infer_call_result called on a subtype of the builtin type will now return a new Class rather than an Instance.
    • Class.metaclass() now handles module-level __metaclass__ declaration on python 2, and no longer looks at the __metaclass__ class attribute on python 3.
    • Add slots method to Class nodes, for retrieving the list of valid slots it defines.
    • Expose function annotation to astroid: Arguments node exposes 'varargannotation', 'kwargannotation' and 'annotations' attributes, while Function node has the 'returns' attribute.
    • Backported most of the logilab.common.modutils module there, as most things there are for pylint/astroid only and we want to be able to fix them without requiring a new logilab.common release
    • Fix names grabed using wildcard import in "absolute import mode" (i.e. with absolute_import activated from the __future__ or with python 3) (pylint issue #58)
    • Add support in brain for understanding enum classes.

  • EP14 Pylint sprint Day 2 and 3 reports

    2014/07/28 by Sylvain Thenault

    Here are the list of things we managed to achieve during those last two days at EuroPython.

    After several attempts, Michal managed to have pylint running analysis on several files in parallel. This is still in a pull request ( because of some limitations, so we decided it won't be part of the 1.3 release.

    Claudiu killed maybe 10 bugs or so and did some heavy issues cleanup in the trackers. He also demonstrated some experimental support of python 3 style annotation to drive a better inference. Pretty exciting! Torsten also killed several bugs, restored python 2.5 compat (though that will need a logilab-common release as well), introduced a new functional test framework that will replace the old one once all the existing tests will be backported. On wednesday, he did show us a near future feature they already have at Google: some kind of confidence level associated to messages so that you can filter out based on that. Sylvain fixed a couple of bugs (including which was annoying all the numpy community), started some refactoring of the PyLinter class so it does a little bit fewer things (still way too many though) and attempted to improve the pylint note on both pylint and astroid, which went down recently "thanks" to the new checks like 'bad-continuation'.

    Also, we merged the pylint-brain project into astroid to simplify things, so you should now submit your brain plugins directly to the astroid project. Hopefuly you'll be redirected there on attempt to use the old (removed) pylint-brain project on bitbucket.

    And, the good news is that now both Torsten and Claudiu have new powers: they should be able to do some releases of pylint and astroid. To celebrate that and the end of the sprint, we published Pylint 1.3 together with Astroid 1.2. More on this here.

  • EP14 Pylint sprint Day 1 report

    2014/07/24 by Sylvain Thenault

    We've had a fairly enjoyable and productive first day in our little hidden room at EuroPython in Berlin ! Below are some noticeable things we've worked on and discussed about.

    First, we discussed and agreed that while we should at some point cut the cord to the logilab.common package, it will take some time notably because of the usage logilab.common.configuration which would be somewhat costly to replace (and is working pretty well). There are some small steps we should do but basically we should mostly get back some pylint/astroid specific things from logilab.common to astroid or pylint. This should be partly done during the sprint, and remaining work will go to tickets in the tracker.

    We also discussed about release management. The point is that we should release more often, so every pylint maintainers should be able to do that easily. Sylvain will write some document about the release procedure and ensure access are granted to the pylint and astroid projects on pypi. We shall release pylint 1.3 / astroid 1.2 soon, and those releases branches will be the last one supporting python < 2.7.

    During this first day, we also had the opportunity to meet Carl Crowder, the guy behind, as well as David Halter which is building the Jedi completion library ( runs pylint on thousands of projects, and it would be nice if we could test beta release on some part of this panel. On the other hand, there are probably many code to share with the Jedi library like the parser and ast generation, as well as a static inference engine. That deserves a sprint on his own though, so we agreed that a nice first step would be to build a common library for import resolution without relying on the python interpreter for that, while handling most of the python dark import features like zip/egg import, .pth files and so one. Indeed that may be two nice future collaborations!

    Last but not least, we got some actual work done:

    • Michal Nowikowski from Intel in Poland joined us to work on the ability to run pylint in different processes so it may drastically improve performance on multiple cores box.
    • Torsten did continue some work on various improvements of the functionnal test framework.
    • Sylvain did merge logilab.common.modutils module into astroid as it's mostly driven by astroid and pylint needs. Also fixed the annoying namespace package crash.
    • Claudiu keep up the good work he does daily at improving and fixing pylint :)

  • Nazca notebooks

    2014/07/04 by Vincent Michel

    We have just published the following ipython notebooks explaining how to perform record linkage and entities matching with Nazca:

  • Open Legislative Data Conference 2014

    2014/06/10 by Nicolas Chauvat

    I was at the Open Legislative Data Conference on may 28 2014 in Paris, to present a simple demo I worked on since the same event that happened two years ago.

    The demo was called "Law is Code Rebooted with CubicWeb". It featured the use of the cubicweb-vcreview component to display the amendments of the hospital law ("loi hospitalière") gathered into a version control system (namely Mercurial).

    The basic idea is to compare writing code and writing law, for both are collaborative and distributed writing processes. Could we reuse for the second one the tools developed for the first?

    Here are the slides and a few screenshots.

    Statistics with queries embedded in report page.

    List of amendments.

    User comment on an amendment.

    While attending the conference, I enjoyed several interesting talks and chats with other participants, including:

    1. the study of co-sponsorship of proposals in the french parliament
    2. announcing their use of PostgreSQL and JSON.
    3. and last but not least, the great work done by RegardsCitoyens and SciencesPo MediaLab on visualizing the law making process.

    Thanks to the organisation team and the other speakers. Hope to see you again!

  • SaltStack Meetup with Thomas Hatch in Paris France

    2014/05/22 by Arthur Lutz

    This monday (19th of may 2014), Thomas Hatch was in Paris for dotScale 2014. After presenting SaltStack there (videos will be published at some point), he spent the evening with members of the French SaltStack community during a meetup set up by Logilab at IRILL.

    Here is a list of what we talked about :

    • Since Salt seems to have pushed ZMQ to its limits, SaltStack has been working on RAET (Reliable Asynchronous Event Transport Protocol ), a transport layer based on UDP and elliptic curve cryptography (Dan Berstein's CURVE-255-19) that works more like a stack than a socket and has reliability built in. RAET will be released as an optionnal beta feature in the next Salt release.
    • Folks from Dailymotion bumped into a bug that seems related to high latency networks and the auth_timeout. Updating to the very latest release should fix the issue.
    • Thomas told us about how a dedicated team at SaltStack handles pull requests and another team works on triaging github issues to input them into their internal SCRUM process. There are a lot of duplicate issues and old inactive issues that need attention and clutter the issue tracker. Help will be welcome.
    • Continuous integration is based on Jenkins and spins up VMs to test pull request. There is work in progress to test multiple clouds, various latencies and loads.
    • For the Docker integration, salt now keeps track of forwarded ports and relevant information about the containers.
    • salt-virt bumped into problems with chroots and timeouts due to ZMQ.
    • Multi-master: the problem lies with syncronisation of data which is sent to minions but also the data that is sent to the masters. Possible solutions to be explored are : the use of gitfs, there is no built-in solution for keys (salt-key has to be run on all masters), mine.send should send the data at both masters, for the jobs cache: one could use an external returner.
    • Thomas talked briefly about ioflo which should bring queuing, data hierarchy and data pub-sub to Salt.
    • About the rolling release question: versions in Salt are definitely not git snapshots, things get backported into previous versions. No clear definition yet of length of LTS versions.
    • salt-cloud and libcloud : in the next release, libcloud will not be a hard dependency. Some clouds didn't work in libcloud (for example AWS), so these providers got implemented directly in salt-cloud or by using third-party libraries (eg. python-boto).
    • Documentation: a sprint is planned next week. Reference documentation will not be completly revamped, but tutorial content will be added.

    Boris Feld showed a demo of vagrant images orchestrated by salt and a web UI to monitor a salt install.

    Thanks again to Thomas Hatch for coming and meeting up with (part of) the community here in France.

  • Salt April Meetup in Paris (France)

    2014/05/14 by Arthur Lutz

    On the 15th of april, in Paris (France), we took part in yet another Salt meetup. The community is now meeting up once every two months.

    We had two presentations:

    • Arthur Lutz made an introduction to returners and the scheduler using the SalMon monitoring system as an example. Salt is not only about configuration management Indeed!
    • The folks from Is Cool Entertainment did a presentation about how they are using salt-cloud to deploy and orchestrate clusters of EC2 machines (islands in their jargon) to reproduce parts of their production environment for testing and developement.

    More discussions about various salty subjects followed and were pursued in an Italian restaurant (photos here).

    In case it is not already in your diary : Thomas Hatch is coming to Paris next week, on Monday the 19th of May, and will be speaking at dotscale during the day and at a Salt meetup in the evening. The Salt Meetup will take place at IRILL (like the previous meetups, thanks again to them) and should start at 19h. The meetup is free and open to the public, but registering on this framadate would be appreciated.

show 208 results