Blog entries

  • Using branches in mercurial

    2008/10/14 by Arthur Lutz
    http://www.logilab.org/image/4873?vid=download&small=true

    The more we use mercurial to manage our code repositories, the more we enjoy its extended functionalities. Lately we've been playing and using branches which end up being very useful. We also use hgview instead of the built-in "hg view" command. And its latest release supports the branches functionality, you can filter out the branch you want to look at. Update your installation (apt-get upgrade ?) to enjoy this new functionality... or download it.

    http://www.selenic.com/hg-logo/logo-droplets-50.png

  • Fusionner dans le bon sens avec mercurial

    2008/11/27 by Nicolas Chauvat
    http://www.selenic.com/hg-logo/logo-droplets-50.png

    Contrairement à ce que l'on pourrait penser a priori, il y a un bon et un mauvais sens pour fusionner deux têtes dans en entrepôt mercurial.

    Prenons un exemple et admettons que l'on parte d'une révision 123. Alfred, Barnabé et Coruscant font des changements qui produisent l'arbre suivant:

    123 -> 124 -> 125 ---> 126 -> 129 --> 130
                       \-> 127 -> 128 /
    

    Si dans le même temps, Zoé part de la révision 123 et produit l'arbre:

    123 -> 131
    

    quand Zoé va vouloir faire son hg push sur l'entrepôt partagé, elle va avoir le message "ça crée des têtes, est-ce que vous êtes sûr de vouloir faire ce push". Zoé va se dire qu'il vaut mieux commencer par un 'pull' et un 'merge', mais c'est là qu'il faut se méfier.

    En effet, si Zoé est sur la révision 131 et qu'elle fusionne avec 130, le changeset sera énorme car il contiendra tous les changements qui permettent de passer de 123 à 130. En revanche, si Zoé passe sur la 130 et fusionne avec la 131, elle n'aura que les changements qu'elle vient d'apporter. Dans le premier cas, le 'merge' sera difficile à comprendre, alors qu'il sera bien plus simple dans le second.

    Au final, comment faire la fusion dans le "bon" sens ?

    1/ hg push -> ah tient, ça crée des têtes, donc je ne le fais pas

    2/ hg pull -u ou hg pull suivi de hg merge -> y'a eu un merge

    3/ hg status -> beaucoup de fichiers modifiés... oulàlà, c'est bizarre je vais regarder

    4/ hgview -> ah oui, sur l'autre tête y'a beaucoup plus de changements, donc je vais plutôt faire le 'merge' dans l'autre sens

    5/ hg up -C ab123_autre_tete -> je me retrouve sur l'autre tête

    6/ hg merge cd456_ma_tete -> un joli 'merge'

    7/ hg status -> seulement quelques fichiers modifés, voilà qui est mieux

    8/ hg ci -m merge -> youpi, je valide cette fusion

    9/ hg push -> tagada, je partage avec mes voisins

    Moralité: faites des hg status et des hgview aussi souvent que nécessaire pour préparer vos commits et parvenir à un historique facile à comprendre.

    Remarque: on peut aussi remplacer les étapes 5, 6 et 7 par hg diff -r ab123_autre_tete -r cd456_ma_tete pour afficher le diff de ma_tete par rapport à autre_tete, car l'opération de fusion doit être symétrique et donner le même résultat qu'on la lance depuis l'une ou l'autre tête, même si l'affichage par défaut de hg status et hg diff dépend de la tête qui constitue le premier parent (l'étape 5 ayant justement pour effet de changer ce premier parent).


  • hgview 0.10.0

    2008/12/30 by Graziella Toutoungis

    I have the pleasure of announcing that the version hgview 0.10.0 was posted on this site and is available for downloading. In this version we added some new functionalities like :

    • The possibility to order all revisions by date or author or description.....
    • Support for localtime.
    • Improve the message header when hg mv is used and fix the author base color
    • Integration of bboissin's fixes
    http://www.selenic.com/hg-logo/logo-droplets-50.png

    Finally : We have taken into account older versions. As pointed out by some users, mercurial version 1.1.x wasn't working very well with hgview, so we created patches which have to be applied according to the version of mercurial you are using.


  • hgview 1.0.0 released!

    2009/06/05 by David Douard

    I am pleased to introduce you to the latest kid of the Logilab team: hgview 1.0.0.

    hgview is a very helpful tool for daily work using the excellent DVCS Mercurial (which we heavily use at Logilab). It allows to easily and visually navigate your hg repository revision graphlog. It is written in Python and pyqt.

    This version is an almost complete rewrite of hgview 0.x which had two GUI backends, gtk and qt4. This 1.0 release drops the gtk backend (we may consider reintroducing it, we haven't decided yet... by the way, patches are always welcome). Some may not like this choice, but the immediate benefit of using qt4 is that hgview works like a charm on MacOS X systems.

    http://www.logilab.org/image/9269?vid=download

    Edit: there was a bug in hgview 1.0.0 on Ubuntu hardy. It's now fixed, and I've uploaded a 1.0.1 version deb package for hardy.

    Features

    • 4 different viewers:
      • repository navigator that displays the graphlog efficiently (works well with 10,000 changesets),
      • filelog navigator that displays the filelog of a file (follows files through renames),
      • filelog diff navigator that displays the filelog in diff mode to easily track changes between two revisions of a file,
      • manifest viewer that navigates in the files hierarchy as it was at a given revision.
    • Each viewer offers:
      • easy keyboard navigation:
        • up/down to change revision,
        • left/right to change file (for the repo navigator only),
        • return to display the diff viewer of the selected file,
      • search quickbar (Ctrl+F or /): search in graphlog (search as you type in the currently displayed file or diff, plus a cancellable background search in the revision tree),
      • goto quickbar (Ctrl+G): go to the given revision (accepts id or tag, with completion for tags),
      • navigation history: alt+left/alt+right to navigate backward/forward in the history,
    • can be used alone or as a hg extension,
    • can be configured using standard hg rc files (system, user or per repository),
    • possibility to declare users (with multiple mail addresses) and assign them a given color to make a given user look the same in all your repositories,

    Download and installation

    The source code is available as a tarball, or using our public hg repository of course.

    To use it from the sources, you just have to add a line in your .hgrc file, in the [extensions] section:

    hgext.hgview=/path/to/hgview/hgext/hgview.py

    Debian and Ubuntu users can also easily install hgview (and Logilab other free software tools) using our deb package repositories.


  • hgview 1.1.0 released

    2009/09/25 by David Douard

    I am pleased to announce the latest release of hgview 1.1.0.

    What is it?

    For the ones from the back of the classroom near the radiator, let me remind you that hgview is a very helpful tool for daily work using the excellent DVCS Mercurial (which we heavily use at Logilab). It allows to easily and visually navigate your hg repository revision graphlog. It is written in Python and pyqt.

    http://www.logilab.org/image/18210?vid=download

    What's new

    • user can now configure colors used in the diff area (and they now defaults to white on black)
    • indicate current working directory position by a square node
    • add many other configuration options (listed when typing hg help hgview)
    • removed 'hg hgview-options' command in favor of 'hg help hgview'
    • add ability to choose which parent to diff with for merge nodes
    • dramatically improved UI behaviour (shortcuts)
    • improved help and make it accessible from the GUI
    • make it possible not to display the diffstat column of the file list (which can dramatically improve performances on big repositories)
    • standalone application: improved command line options
    • indicate working directory position in the graph
    • add auto-reload feature (when the repo is modified due to a pull, a commit, etc., hgview detects it, reloads the repo and updates the graph)
    • fix many bugs, especially the file log navigator should now display the whole graph

    Download and installation

    The source code is available as a tarball, or using our public hg repository of course.

    To use it from the sources, you just have to add a line in your .hgrc file, in the [extensions] section:

    hgext.hgview=/path/to/hgview/hgext/hgview.py

    Debian and Ubuntu users can also easily install hgview (and Logilab other free software tools) using our deb package repositories.


  • hgview 1.2.0 released

    2010/01/21 by David Douard

    Here is at last the release of the version 1.2.0 of hgview.

    http://www.logilab.org/image/19894?vid=download

    In a nutshell, this release includes:

    • a basic support for mq extension,
    • a basic support for hg-bfiles extension,
    • working directory is now displayed as a node of the graph (if there are local modifications of course),
    • it's now possible to display only the subtree from a given revision (a bit like hg log -f)
    • it's also possible to activate an annotate view (make navigation slower however),
    • several improvements in the graph filling and rendering mecanisms,
    • I also added toolbar icons for the search and goto "quickbars" so they are not "hidden" any more to the one reluctant to user manuals,
    • it's now possible to go directly to the common ancestor of 2 revisions,
    • when on a merge node, it's now possible to choose the parent the diff is computed against,
    • make search also search in commit messages (it used to search only in diff contents),
    • and several bugfixes of course.
    Notes:
    there are packages for debian lenny, squeeze and sid, and for ubuntu hardy, interpid, jaunty and karmic. However, for lenny and hardy, provided packages won't work on pure distribs since hgview 1.2 depends on mercurial 1.1. Thus for these 2 distributions, packages will only work if you have installed backported mercurial packages.

  • We're happy to host the mercurial Sprint

    2010/02/02 by Arthur Lutz
    http://farm1.static.flickr.com/183/419945378_4ead41a76d_m.jpg

    We're very happy to be hosting the next mercurial sprint in our brand new offices in central Paris. It is quite an honor to be chosen when the other contender was Google.

    So a bunch of mercurial developers are heading out to our offices this coming Friday to sprint for three days on mercurial. We use mercurial a lot here over at Logilab and we also contribute a tool to visualize and manipulate a mercurial repository : hgview.

    To check out the things that we will be working on with the mercurial crew, check out the program of the sprint on their wiki.

    What is a sprint? "A sprint (sometimes called a Code Jam or hack-a-thon) is a short time period (three to five days) during which software developers work on a particular chunk of functionality. "The whole idea is to have a focused group of people make progress by the end of the week," explains Jeff Whatcott" [source]. For geographically distributed open source communities, it is also a way of physically meeting and working in the same room for a period of time.

    Sprinting is a practice that we encourage at Logilab, with CubicWeb we organize as often as possible open sprints, which is an opportunity for users and developers to come and code with us. We even use the sprint format for some internal stuff.

    photo by Sebastian Mary under creative commons licence.


  • Thoughts on the python3 conversion workflow

    2010/11/30 by Emile Anclin

    Python3

    The 2to3 script is a very useful tool. We can just use it to run over all code base, and end up with a python3 compatible code whilst keeping a python2 code base. To make our code python3 compatible, we do (or did) two things:

    • small python2 compatible modifications of our source code
    • run 2to3 over our code base to generate a python3 compatible version

    However, we not only want to have one python3 compatible version, but also keep developping our software. Hence, we want to be able to easily test it for both python2 and python3. Furthermore if we use patches to get nice commits, this is starting to be quite messy. Let's consider this in the case of Pylint. Indeed, the workflow described before proved to be unsatisfying.

    • I have two repositories, one for python2, one for python3. On the python3 side, I run 2to3 and store the modifications in a patch or a commit.

    • Whenever I implement a fix or a functionality on either side, I have to test if it still works on the other side; but as the 2to3 modifications are often quite heavy, directly creating patches on one side and applying them on the other side won't work most of the time.

    • Now say, I implement something in my python2 base and hold it in a patch or commit it. I can then pull it to my python3 repo:

      • running 2to3 on all Pylint is quite slow: around 30 sec for Pylint without the tests, and around 2 min with the tests. (I'd rather not imagine how long it would take for say CubicWeb).

      • even if I have all my 2to3 modifications on a patch, it takes 5-6 sec to "qpush" or "qpop" them all. Commiting the 2to3 changes instead and using:

        hg pull -u --rebase
        

        is not much faster. If I don't use --rebase, I will have merges on each pull up. Furthermore, we often have either a patch application failure, merge conflict or end up with something which is not python3 compatible (like a newly introduced "except Error, exc").

    • So quite often, I will have to fix it with:

      hg revert -r REV <broken_files>
      2to3 -nw <broken_files>
      hg qref # or hg resolve -m; hg rebase -c
      
    • Suppose that 2to3 transition worked fine, or that we fixed it. I run my tests with python3 and see it does not work; so I modify the patch: it all starts again; and the new patch or the patch modification will create a new head in my python3 repo...

    2to3 Fixers

    Considering all that, let's investigate 2to3: it comes with a lot of fixers that can be activated or desactived. Now, a lot of them fix just very seldom use cases or stuff deprecated since years. On the other hand, the 2to3 fixers work with regular expressions, so the more we remove, the faster 2to3 should be. Depending on the project, most cases will just not appear, and for the others, we should be able to find other means of disabling them. The lists proposed here after are just suggestions, it will depend on the source base and other overall considerations which and how fixers could actually be disabled.

    python2 compatible

    Following fixers are 2.x compatible and should be run once and for all (and can then be disabled on daily conversion usage):

    • apply
    • execfile (?)
    • exitfunc
    • getcwdu
    • has_key
    • idioms
    • ne
    • nonzero
    • paren
    • repr
    • standarderror
    • sys_exec
    • tuple_params
    • ws_comma

    compat

    This can be fixed using imports from a "compat" module like the logilab.common.compat module which holds convenient compatible objects.

    • callable
    • exec
    • filter (Wraps filter() usage in a list call)
    • input
    • intern
    • itertools_imports
    • itertools
    • map (Wraps map() in a list call)
    • raw_input
    • reduce
    • zip (Wraps zip() usage in a list call)

    strings and bytes

    Maybe they could also be handled by compat:

    • basestring
    • unicode
    • print

    For print for example, we could think of a once-and-for-all custom fixer, that would replace it by a convenient echo function (or whatever name you like) defined in compat.

    manually

    Following issues could probably be fixed manually:

    • dict (it fixes dict iterator methods; it should be possible to have code where we can disable this fixer)
    • import (Detects sibling imports; we could convert them to absolute import)
    • imports, imports2 (renamed modules)

    necessary

    These changes seem to be necessary:

    • except
    • long
    • funcattrs
    • future
    • isinstance (Fixes duplicate types in the second argument of isinstance(). For example, isinstance(x, (int, int)) is converted to isinstance(x, (int)))
    • metaclass
    • methodattrs
    • numliterals
    • next
    • raise

    Consider however that a lot of them might never be used in some projects, like long, funcattrs, methodattrs and numliterals or even metaclass. Also, isinstance is probably motivated by long to int and unicode to str conversions and hence might also be somehow avoided.

    don't know

    Can we fix these one also with compat ?

    • renames
    • throw
    • types
    • urllib
    • xrange
    • xreadlines

    2to3 and Pylint

    Pylint is a special case since its test suite has a lot of bad and deprecated code which should stay there. However, in order to have a reasonable work flow, it seems that something must be done to reduce the 1:30 minutes of 2to3 parsing of the tests. Probably nothing could be gained from the above considerations since most cases just should be in the tests, and actually are. Realise that We can expect to be supporting python2 and python3 for several years in parallel.

    After a quick look, we see that 90 % of the refactorings of test/input files are just concerning the print statements; more over most of them have nothing to do with the tested functionality. Hence a solution might be to avoid to run 2to3 on the test/input directory, since we already have a mechanism to select depending on python version whether a test file should be tested or not. To some extend, astng is a similar case, but the test suite and the whole project is much smaller.


  • Text mode makes it into hgview 1.4.0

    2011/10/06 by Alain Leufroy

    Here is at last the release of the version 1.4.0 of hgview.

    http://www.logilab.org/image/77974?vid=download

    Small description

    Besides the classic bugfixes this release introduces a new text based user interface thanks to the urwid library.

    Running hgview in a shell, in a terminal, over a ssh session is now possible! If you are trying not to use X (or use it less), have a geek mouse-killer window manager such as wmii/dwm/ion/awesome/... this is for you!

    This TUI (Text User Interface!) adopts the principal features of the Qt4 based GUI. Although only the main view has been implemented for now.

    In a nutshell, this interface includes the following features :

    • display the revision graph (with working directory as a node, and basic support for the mq extension),
    • display the files affected by a selected changeset (with basic support for the bfiles extension)
    • display diffs (with syntax highlighting thanks to pygments),
    • automatically refresh the displayed revision graph when the repository is being modified (requires pyinotify),
    • easy key-based navigation in revisions' history of a repo (same as the GUI),
    • a command system for special actions (see help)

    Installation

    There are packages for debian and ubuntu in the logilab's debian repository.

    Note:you have to install the hgview-curses package to get the text based interface.

    Or you can simply clone our Mercurial repository:

    hg clone http://hg.logilab.org/hgview

    (more on the hgview home page)

    Running the text based interface

    A new --interface option is now available to choose the interface:

    hgview --interface curses

    Or you can fix it in the [hgview] section of your ~/.hgrc:

    [hgview]
    interface = curses # or qt or raw
    

    Then run:

    hgview

    What's next

    We'll be working on including other features from the Qt4 interface and making it fully configurable.

    We'll also work on bugfixes and new features, so stay tuned! And feel free to file bugs and feature requests.


  • Introduction To Mercurial Phases (Part I)

    2012/02/02 by Pierre-Yves David
    credit: redshirtjosh, http://www.flickr.com/photos/43273828@N06/4111258568/

    On the behalf of Logilab I put a lot of efforts to include a new core feature named phases in Mercurial 2.1. Phases are a system for tracking which changesets have been or should be shared. This helps to prevent common mistakes when modifying history (for instance, with the mq or rebase extensions). It will transparently benefit to all users. This concept is the first step towards simple, safe and powerful rewritting mecanisms for history in mercurial.

    This serie of three blog entries will explain:

    1. how phases will help mercurial users,
    2. how one can control them,
    3. how older mercurial versions interact with newer versions that support phases.

    Preventing erroneous history rewriting

    credit: anita.priks, http://www.flickr.com/photos/46785534@N06/6358218623/

    History rewriting is a common practice in DVCS. However when done the wrong way the most common error results in duplicated history. The phase concept aims to make rewriting history safer. For this purpose Mercurial 2.1 introduces a distinction between the "past" part of your history (that is expected to stay there forever) and the "present" part of the history (that you are currently evolving). The old and immutable part is called public and the mutable part of your history is called draft.

    Let's see how this happens using a simple scenario.


    A new Mercurial user clones a repository:

    babar@Chessy ~ $ hg clone http://hg.celesteville.com/palace
    requesting all changes
    adding changesets
    adding manifests
    adding file changes
    added 2 changesets with 2 changes to 2 files
    updating to branch default
    2 files updated, 0 files merged, 0 files removed, 0 files unresolved
    babar@Chessy ~/palace $ cd palace
    babar@Chessy ~/palace $ hg log --graph
    @  changeset:   1:2afbcfd2af83
    |  tag:         tip
    |  user:        Celeste the Queen <Celeste@celesteville.com>
    |  date:        Wed Jan 25 16:41:56 2012 +0100
    |  summary:     We need a kitchen too.
    |
    o  changeset:   0:898889b143fb
       user:        Celeste the Queen <Celeste@celesteville.com>
       date:        Wed Jan 25 16:39:07 2012 +0100
       summary:     First description of the throne room
    

    The repository already contains some changesets. Our user makes some improvements and commits them:

    babar@Chessy ~/palace $ echo The wall shall be Blue >> throne-room
    babar@Chessy ~/palace $ hg ci -m 'Add wall color'
    babar@Chessy ~/palace $ echo In the middle stands a three meters round table >> kitchen
    babar@Chessy ~/palace $ hg ci -m 'Add a table in the kichen'
    

    But when he tries to push new changesets, he discovers that someone else already pushed one:

    babar@Chessy ~/palace $ hg push
    pushing to http://hg.celesteville.com/palace
    searching for changes
    abort: push creates new remote head bcd4d53319ec!
    (you should pull and merge or use push -f to force)
    babar@Chessy ~/palace $ hg pull
    pulling from http://hg.celesteville.com/palace
    searching for changes
    adding changesets
    adding manifests
    adding file changes
    added 1 changesets with 1 changes to 1 files (+1 heads)
    (run 'hg heads' to see heads, 'hg merge' to merge)
    babar@Chessy ~/palace $ hg log --graph
    o  changeset:   4:0a5b3d7e4e5f
    |  tag:         tip
    |  parent:      1:2afbcfd2af83
    |  user:        Celeste the Queen <Celeste@celesteville.com>
    |  date:        Wed Jan 25 16:58:23 2012 +0100
    |  summary:     Some bedroom description.
    |
    | @  changeset:   3:bcd4d53319ec
    | |  user:        Babar the King <babar@celesteville.com>
    | |  date:        Wed Jan 25 16:52:02 2012 +0100
    | |  summary:     Add a table in the kichen
    | |
    | o  changeset:   2:f9f14815935d
    |/   user:        Babar the King <babar@celesteville.com>
    |    date:        Wed Jan 25 16:51:51 2012 +0100
    |    summary:     Add wall color
    |
    o  changeset:   1:2afbcfd2af83
    |  user:        Celeste the Queen <Celeste@celesteville.com>
    |  date:        Wed Jan 25 16:41:56 2012 +0100
    |  summary:     We need a kitchen too.
    |
    o  changeset:   0:898889b143fb
       user:        Celeste the Queen <Celeste@celesteville.com>
       date:        Wed Jan 25 16:39:07 2012 +0100
       summary:     First description of the throne room
    

    Note

    From here on this scenario becomes very unlikely. Mercurial is simple enough for a new user not to be that confused by such a trivial situation. But we keep the example simple to focus on phases.

    Recently, our new user read some hype blog about "rebase" and the benefit of linear history. So, he decides to rewrite his history instead of merging.

    Despite reading the wonderful rebase help, our new user makes the wrong decision when it comes to using it. He decides to rebase the remote changeset 0a5b3d7e4e5f:"Some bedroom description." on top of his local changeset.

    With previous versions of mercurial, this mistake was allowed and would result in a duplication of the changeset 0a5b3d7e4e5f:"Some bedroom description."

    babar@Chessy ~/palace $ hg rebase -s 4 -d 3
    babar@Chessy ~/palace $ hg push
    pushing to http://hg.celesteville.com/palace
    searching for changes
    abort: push creates new remote head bcd4d53319ec!
    (you should pull and merge or use push -f to force)
    babar@Chessy ~/palace $ hg pull
    pulling from http://hg.celesteville.com/palace
    searching for changes
    adding changesets
    adding manifests
    adding file changes
    added 1 changesets with 1 changes to 1 files (+1 heads)
    (run 'hg heads' to see heads, 'hg merge' to merge)
    babar@Chessy ~/palace $ hg log --graph
    @  changeset:   5:55d9bae1e1cb
    |  tag:         tip
    |  parent:      3:bcd4d53319ec
    |  user:        Celeste the Queen <Celeste@celesteville.com>
    |  date:        Wed Jan 25 16:58:23 2012 +0100
    |  summary:     Some bedroom description.
    |
    | o  changeset:   4:0a5b3d7e4e5f
    | |  parent:      1:2afbcfd2af83
    | |  user:        Celeste the Queen <Celeste@celesteville.com>
    | |  date:        Wed Jan 25 16:58:23 2012 +0100
    | |  summary:     Some bedroom description.
    | |
    o |  changeset:   3:bcd4d53319ec
    | |  user:        Babar the King <babar@celesteville.com>
    | |  date:        Wed Jan 25 16:52:02 2012 +0100
    | |  summary:     Add a table in the kichen
    | |
    o |  changeset:   2:f9f14815935d
    |/   user:        Babar the King <babar@celesteville.com>
    |    date:        Wed Jan 25 16:51:51 2012 +0100
    |    summary:     Add wall color
    |
    o  changeset:   1:2afbcfd2af83
    |  user:        Celeste the Queen <Celeste@celesteville.com>
    |  date:        Wed Jan 25 16:41:56 2012 +0100
    |  summary:     We need a kitchen too.
    |
    o  changeset:   0:898889b143fb
       user:        Celeste the Queen <Celeste@celesteville.com>
       date:        Wed Jan 25 16:39:07 2012 +0100
       summary:     First description of the throne room
    

    In more complicated setups it's a fairly common mistake, Even in big and successful projects and with other DVCSs.

    In the new Mercurial version the user won't be able to make this mistake anymore. Trying to rebase the wrong way will result in:

    babar@Chessy ~/palace $ hg rebase -s 4 -d 3
    abort: can't rebase immutable changeset 0a5b3d7e4e5f
    (see hg help phases for details)
    

    The correct rebase still works as expected:

    babar@Chessy ~/palace $ hg rebase -s 2 -d 4
    babar@Chessy ~/palace $ hg log --graph
    @  changeset:   4:139ead8a540f
    |  tag:         tip
    |  user:        Babar the King <babar@celesteville.com>
    |  date:        Wed Jan 25 16:52:02 2012 +0100
    |  summary:     Add a table in the kichen
    |
    o  changeset:   3:0d1feb1bca54
    |  user:        Babar the King <babar@celesteville.com>
    |  date:        Wed Jan 25 16:51:51 2012 +0100
    |  summary:     Add wall color
    |
    o  changeset:   2:0a5b3d7e4e5f
    |  user:        Celeste the Queen <Celeste@celesteville.com>
    |  date:        Wed Jan 25 16:58:23 2012 +0100
    |  summary:     Some bedroom description.
    |
    o  changeset:   1:2afbcfd2af83
    |  user:        Celeste the Queen <Celeste@celesteville.com>
    |  date:        Wed Jan 25 16:41:56 2012 +0100
    |  summary:     We need a kitchen too.
    |
    o  changeset:   0:898889b143fb
       user:        Celeste the Queen <Celeste@celesteville.com>
       date:        Wed Jan 25 16:39:07 2012 +0100
       summary:     First description of the throne room
    

    What is happening here:

    • Changeset 0a5b3d7e4e5f from Celeste was set to the public phase because it was pulled from the outside. The public phase is immutable.
    • Changesets f9f14815935d and bcd4d53319ec (rebased as 0d1feb1bca54 and 139ead8a540f) have been commited locally and haven't been transmitted from this repository to another. As such, they are still in the draft phase. Unlike the public phase, the draft phase is mutable.

    Let's watch the whole action in slow motion, paying attention to phases:

    babar@Chessy ~ $ cat >> ~/.hgrc << EOF
    [ui]
    username=Babar the King <babar@celesteville.com>
    logtemplate='[{phase}] {desc} ({node|short})\\n'
    EOF
    

    First, changesets cloned from a public server are public:

    babar@Chessy ~ $ hg clone --quiet http://hg.celesteville.com/palace
    babar@Chessy ~/palace $ cd palace
    babar@Chessy ~/palace $ hg log --graph
    @  [public] We need a kitchen too. (2afbcfd2af83)
    |
    o  [public] First description of the throne room (898889b143fb)
    

    Second, new changesets committed locally are in the draft phase:

    babar@Chessy ~/palace $ echo The wall shall be Blue >> throne-room
    babar@Chessy ~/palace $ hg ci -m 'Add wall color'
    babar@Chessy ~/palace $ echo In the middle stand a three meters round table >> kitchen
    babar@Chessy ~/palace $ hg ci -m 'Add a table in the kichen'
    babar@Chessy ~/palace $ hg log --graph
    @  [draft] Add a table in the kichen (bcd4d53319ec)
    |
    o  [draft] Add wall color (f9f14815935d)
    |
    o  [public] We need a kitchen too. (2afbcfd2af83)
    |
    o  [public] First description of the throne room (898889b143fb)
    

    Third, changesets pulled from a public server are public:

    babar@Chessy ~/palace $ hg pull --quiet
    babar@Chessy ~/palace $ hg log --graph
    o  [public] Some bedroom description. (0a5b3d7e4e5f)
    |
    | @  [draft] Add a table in the kichen (bcd4d53319ec)
    | |
    | o  [draft] Add wall color (f9f14815935d)
    |/
    o  [public] We need a kitchen too. (2afbcfd2af83)
    |
    o  [public] First description of the throne room (898889b143fb)
    

    Note

    rebase preserves the phase of rebased changesets

    babar@Chessy ~/palace $ hg rebase -s 2 -d 4
    babar@Chessy ~/palace $ hg log --graph
    @  [draft] Add a table in the kichen (139ead8a540f)
    |
    o  [draft] Add wall color (0d1feb1bca54)
    |
    o  [public] Some bedroom description. (0a5b3d7e4e5f)
    |
    o  [public] We need a kitchen too. (2afbcfd2af83)
    |
    o  [public] First description of the throne room (898889b143fb)
    

    Finally, once pushed to the public server, changesets are set to the public (immutable) phase

    babar@Chessy ~/palace $ hg push
    pushing to http://hg.celesteville.com/palace
    searching for changes
    adding changesets
    adding manifests
    adding file changes
    added 2 changesets with 2 changes to 2 files
    babar@Chessy ~/palace $ hg log --graph
    
    @  [public] Add a table in the kichen (139ead8a540f)
    |
    o  [public] Add wall color (0d1feb1bca54)
    |
    o  [public] Some bedroom description. (0a5b3d7e4e5f)
    |
    o  [public] We need a kitchen too. (2afbcfd2af83)
    |
    o  [public] First description of the throne room (898889b143fb)
    

    To summarize:

    • Changesets exchanged with the outside are public and immutable.
    • Changesets committed locally are draft until exchanged with the outside.
    • As a user, you should not worry about phases. Phases move transparently.

    Preventing premature exchange of history

    credit: Richard Elzey, http://www.flickr.com/photos/elzey/3516256055/

    The public phases prevent user from accidentally rewriting public history. It's a good step forward but phases can go further. Phases can prevent you from accidentally making history public in the first place.

    For this purpose, a third phase is available, the secret phase. To explain it, I'll use the mq extension which is nicely integrated with this secret phase:

    Our fellow user enables the mq extension

    babar@Chessy ~/palace $ vim ~/.hgrc
    babar@Chessy ~/palace $ cat ~/.hgrc
    [ui]
    username=Babar the King <babar@celesteville.com>
    [extensions]
    # enable the mq extension included with Mercurial
    hgext.mq=
    [mq]
    # Enable secret phase integration.
    # This integration is off by default for backward compatibility.
    secret=true
    

    New patches (not general commits) are now created as secret

    babar@Chessy ~/palace $ echo A red carpet on the floor. >> throne-room
    babar@Chessy ~/palace $ hg qnew -m 'add a carpet' carpet.diff
    babar@Chessy ~/palace $ hg log --graph
    
    @  [secret] add a carpet (3c1b19d5d3f5)
    |
    @  [public] Add a table in the kichen (139ead8a540f)
    |
    o  [public] Add wall color (0d1feb1bca54)
    |
    
    

    this secret changeset is excluded from outgoing and push:

    babar@Chessy ~/palace $ hg outgoing
    comparing with http://hg.celesteville.com/palace
    searching for changes
    no changes found (ignored 1 secret changesets)
    babar@Chessy ~/palace $ hg push
    pushing to http://hg.celesteville.com/palace
    searching for changes
    no changes found (ignored 1 secret changesets)
    

    And other users do not see it:

    celeste@Chessy ~/palace $ hg incoming ~babar/palace/
    comparing with ~babar/palace
    searching for changes
    [public] Add wall color (0d1feb1bca54)
    [public] Add a table in the kichen (139ead8a540f)
    

    The mq integration take care of phase movement for the user. Changeset are made draft by qfinish

    babar@Chessy ~/palace $ hg qfinish .
    babar@Chessy ~/palace $ hg log --graph
    @  [draft] add a carpet (2afbcfd2af83)
    |
    o  [public] Add a table in the kichen (139ead8a540f)
    |
    o  [public] Add wall color (0d1feb1bca54)
    |
    
    

    And changesets are made secret again by qimport

    babar@Chessy ~/palace $ hg qimport -r 2afbcfd2af83
    babar@Chessy ~/palace $ hg log --graph
    @  [secret] add a carpet (2afbcfd2af83)
    |
    o  [public] Add a table in the kichen (139ead8a540f)
    |
    o  [public] Add wall color (0d1feb1bca54)
    |
    
    

    As expected, mq refuses to qimport public changesets

    babar@Chessy ~/palace $ hg qimport -r 139ead8a540f
    abort: revision 4 is not mutable
    

    In the next part I'll details how to control phases movement.


  • Introduction To Mercurial Phases (Part II)

    2012/02/02 by Pierre-Yves David

    This is the second part of a series of posts about the new phases feature we implemented for mercurial 2.1. The first part talks about how phases will help mercurial users, this second part explains how to control them.

    Controlling automatic phase movement

    Sometimes it may be desirable to push and pull changesets in the draft phase to share unfinished work. Below are some cases:

    • pushing to continuous integration,
    • pushing changesets for review,
    • user has multiple machines,
    • branch clone.

    You can disable publishing behavior in a repository configuration file [1]:

    [phases]
       publish=False
       

    When a repository is set to non-publishing, people push changesets without altering their phase. draft changesets are pushed as draft and public changesets are pushed as public:

    celeste@Chessy ~/palace $ hg showconfig phases
       phases.publish=False
       
    babar@Chessy ~/palace $ hg log --graph
       @  [draft] add a carpet (2afbcfd2af83)
       |
       o  [public] Add a table in the kichen (139ead8a540f)
       |
       o  [public] Add wall color (0d1feb1bca54)
       |
       
       babar@Chessy ~/palace $ hg outgoing ~celeste/palace/
       [public] Add wall color (0d1feb1bca54)
       [public] Add a table in the kichen (139ead8a540f)
       [draft] add a carpet (3c1b19d5d3f5)
       babar@Chessy ~/palace $ hg push ~celeste/palace/
       pushing to ~celeste/palace/
       searching for changes
       adding changesets
       adding manifests
       adding file changes
       added 3 changesets with 3 changes to 2 files
       babar@Chessy ~/palace $ hg log --graph
       @  [draft] add a carpet (2afbcfd2af83)
       |
       o  [public] Add a table in the kichen (139ead8a540f)
       |
       o  [public] Add wall color (0d1feb1bca54)
       |
       
       
    celeste@Chessy ~/palace $ hg log --graph
       o  [draft] add a carpet (2afbcfd2af83)
       |
       o  [public] Add a table in the kichen (139ead8a540f)
       |
       o  [public] Add wall color (0d1feb1bca54)
       |
       
       

    And pulling gives the phase as in the remote repository:

    celeste@Chessy ~/palace $ hg up 139ead8a540f
       celeste@Chessy ~/palace $ echo The wall will be decorated with portraits >> bedroom
       celeste@Chessy ~/palace $ hg ci -m 'Decorate the wall.'
       created new head
       celeste@Chessy ~/palace $ hg log --graph
       @  [draft] Decorate the wall. (3389164e92a1)
       |
       | o  [draft] add a carpet (3c1b19d5d3f5)
       |/
       o  [public] Add a table in the kichen (139ead8a540f)
       |
       o  [public] Add wall color (0d1feb1bca54)
       |
       
       ---
       babar@Chessy ~/palace $ hg pull ~celeste/palace/
       pulling from ~celeste/palace/
       searching for changes
       adding changesets
       adding manifests
       adding file changes
       added 1 changesets with 1 changes to 1 files (+1 heads)
       babar@Chessy ~/palace $ hg log --graph
       @  [draft] Decorate the wall. (3389164e92a1)
       |
       | o  [draft] add a carpet (3c1b19d5d3f5)
       |/
       o  [public] Add a table in the kichen (139ead8a540f)
       |
       o  [public] Add wall color (0d1feb1bca54)
       |
       
       

    Phase information is exchanged during pull and push operations. When a changeset exists on both sides but within different phases, its phase is unified to the lowest [2] phase. For instance, if a changeset is draft locally but public remotely, it is set public:

    celeste@Chessy ~/palace $ hg push -r 3389164e92a1
       pushing to http://hg.celesteville.com/palace
       searching for changes
       adding changesets
       adding manifests
       adding file changes
       added 1 changesets with 1 changes to 1 files
       celeste@Chessy ~/palace $ hg log --graph
       @  [public] Decorate the wall. (3389164e92a1)
       |
       | o  [draft] add a carpet (3c1b19d5d3f5)
       |/
       o  [public] Add a table in the kichen (139ead8a540f)
       |
       o  [public] Add wall color (0d1feb1bca54)
       |
       
       ---
       babar@Chessy ~/palace $ hg pull ~celeste/palace/
       pulling from ~celeste/palace/
       searching for changes
       no changes found
       babar@Chessy ~/palace $ hg log --graph
       @  [public] Decorate the wall. (3389164e92a1)
       |
       | o  [draft] add a carpet (3c1b19d5d3f5)
       |/
       o  [public] Add a table in the kichen (139ead8a540f)
       |
       o  [public] Add wall color (0d1feb1bca54)
       |
       
       

    Note

    pull is read-only operation and does not alter phases in remote repositories.

    You can also control the phase in which a new changeset is committed. If you don't want new changesets to be pushed without explicit consent, update your configuration with:

    [phases]
       new-commit=secret
       

    You will need to use manual phase movement before you can push them. See the next section for details:

    Note

    With what have been done so far for 2.1, the "most practical way to make a new commit secret" is to use:

       hg commit --config phases.new-commit=secret
       
    [1]You can use this setting in your user hgrc too.
    [2]Phases as ordered as follow: public < draft < secret

    Manual phase movement

    Most phase movements should be automatic and transparent. However it is still possible to move phase manually using the hg phase command:

    babar@Chessy ~/palace $ hg log --graph
       @    [draft] merge with Celeste works (f728ef4eba9f)
       |\
       o |  [draft] add a carpet (3c1b19d5d3f5)
       | |
       | o  [public] Decorate the wall. (3389164e92a1)
       |/
       o  [public] Add a table in the kichen (139ead8a540f)
       |
       
       babar@Chessy ~/palace $ hg phase --public 3c1b19d5d3f5
       babar@Chessy ~/palace $ hg log --graph
       @    [draft] merge with Celeste works (f728ef4eba9f)
       |\
       o |  [public] add a carpet (3c1b19d5d3f5)
       | |
       | o  [public] Decorate the wall. (3389164e92a1)
       |/
       o  [public] Add a table in the kichen (139ead8a540f)
       |
       
       

    Changesets only move to lower [#] phases during normal operation. By default, the phase command enforces this rule:

    babar@Chessy ~/palace $ hg phase --draft 3c1b19d5d3f5
       no phases changed
       babar@Chessy ~/palace $ hg log --graph
       @    [draft] merge with Celeste works (f728ef4eba9f)
       |\
       o |  [public] add a carpet (3c1b19d5d3f5)
       | |
       | o  [public] Decorate the wall. (3389164e92a1)
       |/
       o  [public] Add a table in the kichen (139ead8a540f)
       |
       
       

    If you are confident in what your are doing you can still use the --force switch to override this behavior:

    Warning

    Phases are designed to avoid forcing people to use hg phase --force. If you need to use --force on a regular basis, you are probably doing something wrong. Read the previous section again to see how to configure your environment for automatic phase movement suitable to your needs.

    babar@Chessy ~/palace $ hg phase --verbose --force --draft 3c1b19d5d3f5
       phase change for 1 changesets
       babar@Chessy ~/palace $ hg log --graph
       @    [draft] merge with Celeste works (f728ef4eba9f)
       |\
       o |  [draft] add a carpet (3c1b19d5d3f5)
       | |
       | o  [public] Decorate the wall. (3389164e92a1)
       |/
       o  [public] Add a table in the kichen (139ead8a540f)
       |
       
       

    Note that a phase defines a consistent set of revisions in your history graph. This means that to have a public (immutable) changeset all its ancestors need to be immutable too. Once you have a secret (not exchanged) changeset, all its descendant will be secret too.

    This means that changing the phase of a changeset may result in phase movement for other changesets:

    babar@Chessy ~/palace $ hg phase -v --public f728ef4eba9f # merge with Celeste works
       phase change for 2 changesets
       babar@Chessy ~/palace $ hg log --graph
       @    [public] merge with Celeste works (f728ef4eba9f)
       |\
       o |  [public] add a carpet (3c1b19d5d3f5)
       | |
       | o  [public] Decorate the wall. (3389164e92a1)
       |/
       o  [public] Add a table in the kichen (139ead8a540f)
       |
       
       babar@Chessy ~/palace $ hg phase -vf --draft 3c1b19d5d3f5 # add a carpet
       phase change for 2 changesets
       babar@Chessy ~/palace $ hg log --graph
       @    [draft] merge with Celeste works (f728ef4eba9f)
       |\
       o |  [draft] add a carpet (3c1b19d5d3f5)
       | |
       | o  [public] Decorate the wall. (3389164e92a1)
       |/
       o  [public] Add a table in the kichen (139ead8a540f)
       |
       
       

    The next and final post will explain how older mercurial versions interact with newer versions that support phases.

    [Images by Jimmy Smith (cc-by-nd) and Cory Doctorow (cc-by-sa)]


  • Introduction To Mercurial Phases (Part III)

    2012/02/03 by Pierre-Yves David

    This is the final part of a series of posts about the new phases feature we implemented for mercurial 2.1. The first part talks about how phases will help mercurial users, the second part explains how to control them. This one explains what people should take care of when upgrading.

    Important upgrade note and backward compatibility

    Phases do not require any conversion of your repos. Phase information is not stored in changesets. Everybody using a new client will take advantage of phases on any repository they touch.

    However there is some points you need to be aware of regarding interaction between the old world without phases and the new world with phases:

    Talking over the wire to a phaseless server using a phased client

    As ever, the Mercurial wire protocol (used to communicate through http and ssh) is fully backward compatible [1]. But as old Mercurial versions are not aware of phases, old servers will always be treated as publishing.

    Direct file system access to a phaseless repository using a phased client

    A new client has no way to determine which parts of the history should be immutable and which parts should not. In order to fail safely, a new repo will mark everything as public when no data is available. For example, in the scenario described in part I, if an old version of mercurial were used to clone and commit, a new version of mercurial will see them as public and refuse to rebase them.

    Note

    Some extensions (like mq) may provide smarter logic to set some changesets to the draft or even secret phases.

    The phased client will write phase data to the old repo on its first write operation.

    Direct file system access to a phased repository using a phaseless client

    Everything works fine except that the old client is unable to see or manipulate phases:

    • Changesets added to the repo inherit the phase of their parents, whatever the parents' phase. This could result in new commits being seen as public or pulled content seen as draft or even secret when a newer client uses the repo again!
    • Changesets pushed to a publishing server won't be set public.
    • Secret changesets are exchanged.
    • Old clients are willing to rewrite immutable changesets (as they don't know that they shouldn't).

    So, if you actively rewrite your history or use secret changesets, you should ensure that only new clients touch those repositories where the phase matters.

    Fixing phases error

    Several situations can result in bad phases in a repository:

    • When upgrading from phaseless to phased Mercurial, the default phases picked may be too restrictive.
    • When you let an old client touch your repository.
    • When you push to a publishing server that should not actually be publishing.

    The easiest way to restore a consistant state is to use the phase command. In most cases, changesets marked as public but absent from your real public server should be moved to draft:

    hg phase --force --draft 'public() and outgoing()'
    

    If you have multiple public servers, you can pull from the others to retrieve their phase data too.

    Conclusion

    Mercurial's phases are a simple concept that adds always on and transparent safety for most users while not preventing advanced ones from doing whatever they want.

    Behind this safety-enabling and useful feature, phases introduce in Mercurial code the concept of sharing mutable parts of history. The introduction of this feature paves the way for advanced history rewriting solutions while allowing safe and easy sharing of mutable parts of history. I'll post about those future features shortly.


    [1]You can expect the 0.9.0 version of Mercurial to interoperate cleanly with one released 5 years later.

    [Images by Crystian Cruz (cc-nd) and C.J. Peters (cc-by-sa)]


  • Mercurial 2.3 day 0

    2012/05/10 by Pierre-Yves David

    I'm now at Copenhagen to attend the mercurial "2.3" sprint.

    About twenty people are attending, including staff from Atlassian, Facebook, Google and Mozilla.

    I expect code and discussion about various topic among:

    • the development process of mercurial itself,
    • performance improvement on huge repository,
    • integration of Obsolete Markers into mercurial core,
    • improvement on various aspect (merge diff, moving some extension in core, ...)

    I'm of course very interested in the Obsolete Markers topic. I've been working on an experimental implementation for several months. An handful of people are using them at Logilab for two months and feedback are very promising.


  • Mercurial 2.3 sprint, Day 1-2-3

    2012/05/15 by Pierre-Yves David

    I'm now back from Copenhagen were I attended the mercurial 2.3 sprint with twenty other people. A huge amount of work was done in a very friendly atmosphere.

    Regarding mercurial's core:

    • Bookmark behaviour was improved to get closer to named branch's behaviour.
    • Several performance improvements regarding branches and heads caches. The heads cache refactoring improves rebase performance on huge repository (thanks to Facebook and Atlassian).
    • The concept I'm working on, Obsolete markers, was a highly discussed subject and is expected to get partly into the core in the near future. Thanks to my employer Logilab for paying me to work on this topic.
    • General code cleanup and lock validation.
    http://www.logilab.org/file/92956?vid=download

    Regarding the bundled extension :

    • Some fixes where made to progress which is now closer to getting into mercurial's core.
    • Histedit and keyring extensions are scheduled to be shipped with mercurial.
    • Some old and unmaintained extensions (children, hgtk) are now deprecated.
    • The LargeFile extension got some new features (thanks to the folks from Unity3D)
    • Rebase will use the --detach flag by default in the next release.
    http://www.logilab.org/file/92958?vid=download

    Regarding the project itself:

    http://www.logilab.org/file/92955?vid=download

    Regarding other extensions:

    http://www.logilab.org/file/92959?vid=download

    And I'm probably forgetting some stuff. Special thanks to Unity3D for hosting the sprint and providing power, network and food during these 3 days.


  • Logilab à PyConFR 2012 - compte rendu

    2012/10/09 by Alain Leufroy
    http://awesomeness.openphoto.me/custom/201209/4ed140-pycon3--1-of-37-_870x550.jpg

    Logilab était à la conférence PyConFR qui a pris place à Paris il y a deux semaines.

    Nous avons commencé par un sprint pylint, coordonné par Boris Feld, où pas mal de volontaires sont passés pour traquer des bogues ou ajouter des nouvelles fonctionnalités. Merci à tous!

    Pour ceux qui ne connaissent pas encore, pylint est un utilitaire pratique que nous avons dans notre forge. C'est un outil très puissant d'analyse statique de scripts python qui aide à améliorer/maintenir la qualité du code.

    Par la suite, après les "talks" des sponsors¸ où vous auriez pu voir Olivier, vous avons pu participer à quelques tutoriels et présentations vraiment excellentes. Il y avait des présentations pratiques avec, entre autres, les tests, scikit-learn ou les outils pour gérer des services (Cornice, Circus). Il y avait aussi des retours d'information sur le processus de développement de CPython, le développement communautaire ou un supercalculateur. Nous avons même pu faire de la musique avec python et un peu d'"embarqué" avec le Raspberry Pi et Arduino !

    Nous avons, avec Pierre-Yves, proposé deux tutoriels d'introduction au gestionnaire de versions décentralisé Mercurial. Le premier tutoriel abordait les bases avec des cas pratiques. Lors du second tutoriel, que l'on avait prévu initialement dans la continuité du premier, nous avons finalement abordé des utilisations plus avancées permettant de résoudre avec énormément d'efficacité des problématiques quotidiennes, comme les requêtes sur les dépôts, ou la recherche automatique de régression par bissection. Vous pouvez retrouver le support avec les exercices .

    Pierre-Yves a présenté une nouvelle propriété importante de Mercurial: l'obsolescence. Elle permet de mettre en place des outils d'édition d'historique en toute sécurité ! Parmi ces outils, Pierre-Yves a écrit une extension mutable-history qui vous offre une multitude de commandes très pratiques.

    La présentation est disponible en PDF et en consultation en ligne sur slideshare. Nous mettrons bientôt la vidéo en ligne.

    http://www.logilab.org/file/107770?vid=download

    Si le sujet vous intéresse et que vous avez raté cette présentation, Pierre-Yves reparlera de ce sujet à l'OSDC.

    Pour ceux qui en veulent plus, Tarek Ziadé à mis à disposition des photos de la conférence ici.


  • Retour Agile Tour Nantes 2012 - présentation et pistes à explorer

    2012/12/04 by Arthur Lutz

    Nous utilisons les méthodes agiles depuis la création de Logilab. Nous avons parfois pris des libertés avec le formalisme des méthodes connues, en adaptant nos pratiques à nos clients et nos particularités. Nous avons en chemin développé nos propres outils orientés vers notre activité de développement logiciel (gestion de version, processus sur les tickets, intégration continue, etc).

    https://www.logilab.org/file/113044?vid=download

    Il est parfois bon de se replonger dans la théorie et d'échanger les bonnes pratiques en terme d'agilité. C'est pour cette raison que nous avons participé à l'étape nantaise de l'Agile Tour.

    Logiciels libres et agilité

    Plutôt que d'être simples spectateurs, nous avons présenté nos pratiques agiles, fortement liées au logiciel libre, dont un avantage indéniable est la possibilité offerte à chacun de le modifier pour l'adapter à ses besoins.

    Premièrement, en utilisant la plate-forme web CubicWeb, nous avons pu construire une forge dont nous contrôlons le modèle de données. Les processus de gestion peuvent donc être spécifiques et les données des applications peuvent être étroitement intégrées. Par exemple, bien que la base logicielle soit la même, le circuit de validation des tickets sur l'extranet n'est pas identique à celui de nos forges publiques. Autre exemple, les versions livrées sur l'extranet apparaissent directement dans l'outil intranet de suivi des affaires et de décompte du temps (CRM/ERP).

    Deuxièmement, nous avons choisi mercurial (hg) en grande partie car il est écrit en python ce qui nous a permis de l'intégrer à nos autres outils, mais aussi d'y contribuer (cf evolve).

    Notre présentation est visible sur slideshare :

    http://www.logilab.org/file/113040?vid=download

    ou à télécharger en PDF.

    Behaviour Driven Development

    Le BDD (Behaviour Driven Development) se combine avec des tests fonctionnels haut niveau qui peuvent être décrits grâce à un formalisme syntaxique souvent associé au langage Gherkin. Ces scénarios de test peuvent ensuite être convertis en code et exécutés. Coté Python, nous avons trouvé behave et lettuce. De manière similaire à Selenium (scénarios de test de navigation Web), la difficulté de ce genre de tests est plutôt leur maintenance que l'écriture initiale.

    https://www.logilab.org/file/113042?vid=download

    Ce langage haut niveau peut néanmoins être un canal de communication avec un client écrivant des tests. À ce jour, nous avons eu plusieurs clients prenant le temps de faire des fiches de tests que nous "traduisons" ensuite en tests unitaires. Si le client n'est pas forcément prêt à apprendre le Python et leurs tests unitaires, il serait peut-être prêt à écrire des tests selon ce formalisme.


  • Retour OSDC 2012 - présentation mercurial DVCS

    2012/12/05 by Pierre-Yves David

    À la mi-octobre, j'ai participé à la conférence OSDC 2012 à Paris. Le but de cette conférence est de permettre à des développeurs de différentes communautés de se rencontrer dans une ambiance chaleureuse. De fait, j'ai découvert un certain nombre de projets et de pratiques intéressants.

    http://act.osdc.fr/osdc2012fr/css/logo.png

    Le samedi, j'ai découvert des outils javascript mettant l'accent sur les modèles de données comme AngularJS ou BackBone, Une présentation rapide du langage Go, le très prometteur portage des outils GCC sur Windows nommé MinGW ainsi que les nouveautés de GCC 4.8. La journée s'est conclut sur des présentations éclairs dont je retiendrai surtout la perversité des opérateurs secrets en Perl et le livre Javascript Éloquent intégralement en HTML qui en profite donc pour inclure exemples et exercices interactifs au fil du contenu.

    Le dimanche matin j'ai ouvert le bal en présentant mes travaux actuels dans le DVCS Mercurial: l'Évolution de Changeset (PDF de la présentation). Ce concept permet aux développeurs de découvrir la réécriture d'historique de manière simple et sûre. Les utilisateurs avancés ont accès de leur côté à des processus de travail et de revue encore inédits dans le monde des DVCS. Ma présentation fut suivie d'une introduction à la découverte automatique de bugs grâce à la bisection dans les DVCS.


    Changesets Evolution : Mercurial secoue le monde du dvcs par Pierre-Yves David

    La journée s'est poursuivie avec une présentation du langage Haskell, de la bibliothèque de visualisation sigmajs et la spécification SPORE apportant un peu d'espoir dans les spécifications de services Web REST.


  • Febuary 2013: Mercurial channel "tour"

    2013/01/22 by Pierre-Yves David

    The Release candidate version of Mercurial 2.5 was released last sunday.

    http://mercurial.selenic.com/images/mercurial-logo.png

    This new version makes a major change in the way "hidden" changesets are handled. In 2.4 only hg log (and a few others) would support effectively hiding "hidden" changesets. Now all hg commands are transparently compatible with the hidden revision concept. This is a considerable step towards changeset evolution, the next-generation collaboration technology that I'm developing for Mercurial.

    https://fosdem.org/2013/assets/flyer-thumb-0505d19dbf3cf6139bc7490525310f8e253e60448a29ed4313801b723d5b2ef1.png

    The 2.5 cycle is almost over, but there is no time to rest yet, Saturday the 2th of February, I will give a talk about changeset evolution concept at FOSDEM in the Mozilla Room. This talk in an updated version of the one I gave at OSDC.fr 2012 (video in french).

    The week after, I'm crossing the channel to attend the Mercurial 2.6 Sprint hosted by Facebook London. I expect a lot of discussion about the user interface and network access of changeset evolution.

    The HG 2.3 sprint

  • FOSDEM 2013

    2013/02/12 by Pierre-Yves David

    I was in Bruxelles for FOSDEM 2013. As with previous FOSDEM there were too many interesting talks and people to see. Here is a summary of what I saw:

    In the Mozilla's room:

    1. The html5 pdf viewer pdfjs is impressive. The PDF specification is really scary but this full featured "native" viewer is able to renders most of it with very good performance. Have a look at the pdfjs demo!
    1. Firefox debug tools overview with a specific focus of Firefox OS emulator in your browser.
    1. Introduction to webl10n: an internationalization format and library used in Firefox OS. A successful mix that results in a format that is idiot-proof enough for a duck to use, that relies on Unicode specifications to handle complex pluralization rules and that allows cascading translation definitions.
    typical webl10n user
    1. Status of html5 video and audio support in Firefox. The topic looks like a real headache but the team seems to be doing really well. Special mention for the reverse demo effect: The speaker expected some format to be still unsupported but someone else apparently implemented them over night.
    2. Last but not least I gave a talk about the changeset evolution concept that I'm putting in Mercurial. Thanks goes to Feth for asking me his not-scripted-at-all-questions during this talk. (slides)
    http://www.selenic.com/hg-logo/logo-droplets-150.png

    In the postgresql room:

    1. Insightful talk about more event trigger in postgresql engine and how this may becomes the perfect way to break your system.
    2. Full update of the capability of postgis 2.0. The postgis suite was already impressive for storing and querying 2D data, but it now have impressive capability regarding 3D data.
    http://upload.wikimedia.org/wikipedia/en/6/60/PostGIS_logo.png

    On python related topic:

    http://www.python.org/community/logos/python-logo-master-v3-TM-flattened.png
    • Victor Stinner has started an interesting project to improve CPython performance. The first one: astoptimizer breaks some of the language semantics to apply optimisation on compiling to byte code (lookup caching, constant folding,…). The other, registervm is a full redefinition of how the interpreter handles reference in byte code.

    After the FOSDEM, I crossed the channel to attend a Mercurial sprint in London. Expect more on this topic soon.


  • LMGC90 Sprint at Logilab in March 2013

    2013/03/28 by Vladimir Popescu

    LMGC90 Sprint at Logilab

    At the end of March 2013, Logilab hosted a sprint on the LMGC90 simulation code in Paris.

    LMGC90 is an open-source software developed at the LMGC ("Laboratoire de Mécanique et Génie Civil" -- "Mechanics and Civil Engineering Laboratory") of the CNRS, in Montpellier, France. LMGC90 is devoted to contact mechanics and is, thus, able to model large collections of deformable or undeformable physical objects of various shapes, with numerous interaction laws. LMGC90 also allows for multiphysics coupling.

    Sprint Participants

    https://www.logilab.org/file/143585/raw/logo_LMGC.jpg https://www.logilab.org/file/143749/raw/logo_SNCF.jpg https://www.logilab.org/file/143750/raw/logo_LaMSID.jpg https://www.logilab.org/file/143751/raw/logo_LOGILAB.jpg

    More than ten hackers joined in from:

    • the LMGC, which leads LMCG90 development and aims at constantly improving its architecture and usability;
    • the Innovation and Research Department of the SNCF (the French state-owned railway company), which uses LMGC90 to study railway mechanics, and more specifically, the ballast;
    • the LaMSID ("Laboratoire de Mécanique des Structures Industrielles Durables", "Laboratory for the Mechanics of Ageing Industrial Structures") laboratory of the EDF / CNRS / CEA , which has an strong expertise on Code_ASTER and LMGC90;
    • Logilab, as the developer, for the SNCF, of a CubicWeb-based platform dedicated to the simulation data and knowledge management.

    After a great introduction to LMGC90 by Frédéric Dubois and some preliminary discussions, teams were quickly constituted around the common areas of interest.

    Enhancing LMGC90's Python API to build core objects

    As of the sprint date, LMGC90 is mainly developed in Fortran, but also contains Python code for two purposes:

    • Exposing the Fortran functions and subroutines in the LMGC90 core to Python; this is achieved using Fortran 2003's ISO_C_BINDING module and Swig. These Python bindings are grouped in a module called ChiPy.
    • Making it easy to generate input data (so called "DATBOX" files) using Python. This is done through a module called Pre_LMGC.

    The main drawback of this approach is the double modelling of data that this architecture implies: once in the core and once in Pre_LMGC.

    It was decided to build a unique user-level Python layer on top of ChiPy, that would be able to build the computational problem description and write the DATBOX input files (currently achieved by using Pre_LMGC), as well as to drive the simulation and read the OUTBOX result files (currently by using direct ChiPy calls).

    This task has been met with success, since, in the short time span available (half a day, basically), the team managed to build some object types using ChiPy calls and save them into a DATBOX.

    Using the Python API to feed a computation data store

    This topic involved importing LMGC90 DATBOX data into the numerical platform developed by Logilab for the SNCF.

    This was achieved using ChiPy as a Python API to the Fortran core to get:

    • the bodies involved in the computation, along with their materials, behaviour laws (with their associated parameters), geometries (expressed in terms of zones);
    • the interactions between these bodies, along with their interaction laws (and associated parameters, e.g. friction coefficient) and body pair (each interaction is defined between two bodies);
    • the interaction groups, which contain interactions that have the same interaction law.

    There is still a lot of work to be done (notably regarding the charges applied to the bodies), but this is already a great achievement. This could only have occured in a sprint, were every needed expertise is available:

    • the SNCF experts were there to clarify the import needs and check the overall direction;

    • Logilab implemented a data model based on CubicWeb, and imported the data using the ChiPy bindings developed on-demand by the LMGC core developer team, using the usual-for-them ISO_C_BINDING/ Swig Fortran wrapping dance.

      https://www.logilab.org/file/143753/raw/logo_CubicWeb.jpg
    • Logilab undertook the data import; to this end, it asked the LMGC how the relevant information from LMGC90 can be exposed to Python via the ChiPy API.

    Using HDF5 as a data storage backend for LMGC90

    The main point of this topic was to replace the in-house DATBOX/OUTBOX textual format used by LMGC90 to store input and output data, with an open, standard and efficient format.

    Several formats have been considered, like HDF5, MED and NetCDF4.

    MED has been ruled out for the moment, because it lacks the support for storing body contact information. HDF5 was chosen at last because of the quality of its Python libraries, h5py and pytables, and the ease of use tools like h5fs provide.

    https://www.logilab.org/file/143754/raw/logo_HDF.jpg

    Alain Leufroy from Logilab quickly presented h5py and h5fs usage, and the team started its work, measuring the performance impact of the storage pattern of LMGC90 data. This was quickly achieved, as the LMGC experts made it easy to setup tests of various sizes, and as the Logilab developers managed to understand the concepts and implement the required code in a fast and agile way.

    Debian / Ubuntu Packaging of LMGC90

    This topic turned out to be more difficult than initially assessed, mainly because LMGC90 has dependencies to non-packaged external libraries, which thus had to be packaged first:

    • the Matlib linear algebra library, written in C,
    • the Lapack95 library, which is a Fortran95 interface to the Lapack library.

    Logilab kept working on this after the sprint and produced packages that are currently being tested by the LMGC team. Some changes are expected (for instance, Python modules should be prefixed with a proper namespace) before the packages can be submitted for inclusion into Debian. The expertise of Logilab regarding Debian packaging was of great help for this task. This will hopefully help to spread the use of LMGC90.

    https://www.logilab.org/file/143755/raw/logo_Debian.jpg

    Distributed Version Control System for LMGC90

    As you may know, Logilab is really fond of Mercurial as a DVCS. Our company invested a lot into the development of the great evolve extension, which makes Mercurial a very powerful tool to efficiently manage the team development of software in a clean fashion.

    This is why Logilab presented Mercurial's features and advantages over the current VCS used to manage LMGC90 sources, namely svn, to the other participants of the Sprint. This was appreciated and will hopefully benefit to LMGC90 ease of development and spread among the Open Source community.

    https://www.logilab.org/file/143756/raw/logo_HG.jpg

    Conclusions

    All in all, this two-day sprint on LMGC90, involving participants from several industrial and academic institutions has been a great success. A lot of code has been written but, more importantly, several stepping stones have been laid, such as:

    • the general LMGC90 data access architecture, with the Python layer on top of the LMGC90 core;
    • the data storage format, namely HDF5.

    Colaterally somehow, several other results have also been achieved:

    • partial LMGC90 data import into the SNCF CubicWeb-based numerical platform,
    • Debian / Ubuntu packaging of LMGC90 and dependencies.

    On a final note, one would say that we greatly appreciated the cooperation between the participants, which we found pleasant and efficient. We look forward to finding more occasions to work together.


  • Open Legislative Data Conference 2014

    2014/06/10 by Nicolas Chauvat

    I was at the Open Legislative Data Conference on may 28 2014 in Paris, to present a simple demo I worked on since the same event that happened two years ago.

    The demo was called "Law is Code Rebooted with CubicWeb". It featured the use of the cubicweb-vcreview component to display the amendments of the hospital law ("loi hospitalière") gathered into a version control system (namely Mercurial).

    The basic idea is to compare writing code and writing law, for both are collaborative and distributed writing processes. Could we reuse for the second one the tools developed for the first?

    Here are the slides and a few screenshots.

    http://www.logilab.org/file/253394/raw/lawiscode1.png

    Statistics with queries embedded in report page.

    http://www.logilab.org/file/253400/raw/lawiscode2.png

    List of amendments.

    http://www.logilab.org/file/253396/raw/lawiscode3.png

    User comment on an amendment.

    While attending the conference, I enjoyed several interesting talks and chats with other participants, including:

    1. the study of co-sponsorship of proposals in the french parliament
    2. data.senat.fr announcing their use of PostgreSQL and JSON.
    3. and last but not least, the great work done by RegardsCitoyens and SciencesPo MediaLab on visualizing the law making process.

    Thanks to the organisation team and the other speakers. Hope to see you again!


  • PyconFR 2014 : jour 1, frameworks web et gestion de source

    2014/11/04 by Arthur Lutz

    Suite de pyconfr 2014 jour 1 épisode 1.

    Performance des frameworks web : Python vs the world

    Ronan Amicel nous a présenté le travail de benchmark publié par TechEmpower. Ces tests et résultats sont forcement faux et biaisés, mais le code source des tests est publié en libre et il est donc possible d'apporter des corrections via le projet sur github

    Pour l'instant, Python3 serait plus lent que Python2, mais on peut espérer que Python3 rattrape son retard, puisqu'il est toujours développé. La comparaison avec pypy est surprenante, celui-ci est bien plus lent, l'hypothèse étant qu'il est ralenti lorsqu'il parle au driver mysql. En revanche, pour le test pypy + tornado, les performances peuvent être meilleures que nodejs car tornado est écrit en pur python il peut être optimisé par pypy.

    Dans le comparatif entre python et php, un acteur surprenant est phalcon qui a pris le parti de tout coder en C (plutôt qu'une partie seulement comme on peut le trouver dans nombre de projets python).

    Support de présentation : https://speakerdeck.com/ronnix/performance-des-frameworks-web-python-vs-the-world-v1-dot-1

    CubicWeb - Vos données ont du sens

    Nous attendions avec impatience cette présentation, et Christophe de Vienne a très bien présenté CubicWeb, le framework web dont Logilab est à l'origine.

    https://www.logilab.org/file/269991/raw/logo-cubicweb.png

    Après une courte introduction aux concepts du web sémantique (les URIS, les relations, le Linked Data), il a appuyé sur la nécéssité de donner du sens aux données que l'on stoque dans nos applications. Il a expliqué la finesse des réglages dans le moteur de permissions de CubicWeb.

    Il a expliqué certaines fonctionnalités intéressantes selon lui dans Cubicweb :

    • les hooks: équivalent des procédures stockées déclenchées par des triggers, ils sont écrits en Python et permettent de modifier des données en cascades, implémenter des règle de gestion ou générer des notifications.
    • les adaptateurs : permettent de maximiser la réutilisation de code en adaptant une entité à une nouvelle interface

    Selon Christophe, CubicWeb permet de développer une "base de donnée métier" strictement structurée, mais restant souple. Il a expliqué que l'interface par défaut n'est pas très sexy, mais qu'elle est néanmoins fonctionnelle comme backend d'édition.

    Une petite introduction aux cubes qui sont les "plugins" ou les "extensions" dans le monde CubicWeb, ils contiennent :

    • un schéma
    • du code métier
    • des vues
    • des contrôleurs

    Pour manipuler les données, CubicWeb utilise RQL, qui a été inventé avant SPARQL (langage de requête du web sémantique) et est plus pragmatique et lisible. Une fonctionnalité notable de RQL : plus besoin d'écrire des jointures SQL !

    Finalement Christophe a conclu en présentant le mariage de Pyramid et Cubicweb. Selon lui, en regardant dedans, ils ont des philosophies communes. Le code permettant de développer une application Pyramid sur une base CubicWeb est publié sur la forge de CubicWeb. Christophe a aussi expliqué qu'il pousse des modifications pour que CubicWeb soit plus accessible aux développeurs habitués aux modes de développement "à la python".

    Support de présentation : https://dl.dropboxusercontent.com/u/36590471/pyconfr-2014-pres-cubicweb/index.html

    La gestion de version, ce problème tellement simple…

    Pierre-Yves David (marmoute) nous a concocté un petit panorama des problèmes traités par les gestionnaires de source, avec des anecdotes de problèmes non-triviaux et quelques rappels historiques sur notre "science" informatique (merci les encodages!) Pierre-Yves s'est concentré sur les systèmes de gestion de version de "nouvelle génération", les outils décentralisés (hg, git, bzr). Forcément, étant donné qu'il travaille sur mercurial (et oui, celui écrit en python) il s'est concentré sur celui-là.

    http://mercurial.selenic.com/images/mercurial-logo.png

    Quand il travaillait chez Logilab, Pierre-Yves a notamment rajouté à Mercurial la notion de changeset obsolete et de phase pour faciliter la revue de code et le travail en équipe.

    Manipuler son code python avec RedBaron

    baron et RedBaron sont des projets assez prometteurs (et assez dingues) de manipulation de code en utilisant du code (plutôt que des éditeurs).

    Laurent Peuch est revenu sur les outils historiques du domaine : rope qui a pris la suite de bicycle repair man. Il y a aussi pyfmt par le même auteur, et autopep8 écrit par d'autres.

    Un exemple qui m'a parlé : ajouter @profile sur toutes les fonctions d'un script devient faisable en 3 lignes de python, et inversement pour les enlever. À suivre...

    Support de présentation : https://psycojoker.github.io/pyconfr-redbaron/presentation.html

    Prochain épisode

    Prochain épisode: jour 1, bus de communication, packaging et fin


  • Retour sur la journée conteneurs dans le cadre de Open Source Innovation Spring

    2015/04/07 by Arthur Lutz

    Logilab a co-organisé la demi-journée sur les conteneurs dans le cadre du Printemps de l'innovation open source (Open Source Innovation Spring). Voici une partie des choses qui y furent dites.

    Open Source Innovation Spring

    AlterWay a commencé par une introduction expliquant pourquoi docker est si hype en ce moment. Quelques bémols ont été placés sur les questions de sécurité et les systèmes de fichiers utilisés par défaut (AUFS n'est pas dans le kernel linux officiel, des alternatives sont à l'étude).

    Une partie de l'écosystème autour de Docker a été mentionné :

    Ensuite Normation a présenté la gestion de configuration et Docker, avec de grandes questions générales sur le déploiement de serveurs, leur durée de vie, leur transformation, etc.

    Logilab & Mozilla

    Logilab a présenté l'utilisation conjointe de Salt Mercurial et Docker pour appliquer les bonnes pratiques du développement logiciel à la gestion d'infrastructures. Les supports de présentation sont sur http://slides.logilab.fr/osis/osis (aussi sur slideshare).

    Normation a ensuite présenté les fondements techniques des conteneurs, à savoir les fonctionnalités du noyau linux qui ont permis leur essor. Petit historique sur les cgroups, avec les idées d'origine sur les processus dans Unix, mais aussi les bonnes idées apportées par Plan 9 (et qui ont ensuite été reprises par Linux). On a vu des choses sur les chroots, les namespaces, fakeroot, ip netns, les informations dans /proc/<pid>/ns, et les systèmes de fichier d'union utilisé par les conteneurs : aufs, unionfs, overlayfs, fuse.

    Intervenants de la journée

    Ensuite deux démonstrations ont été présentées :

    • Utilisation de docker et docker-swarm sur amazon ec2 pour déployer une application html5 : CircleCI lit le dépôt git de l'application, construit l'image Docker et l'ajoute au hub puis pilote docker-swarm pour qu'elle soit déployée.
    • Utilisation de plusieurs plate-formes de cloud (Azure, Numergy, CloudWatt) pour déployer un conteneur docker sur plusieurs clouds en parallèle.

    Deux retours d'expérience par Theodo et Deliverous ont conclu la journée.


  • Better code archaeology with Mercurial

    2017/09/21 by Denis Laxalde

    For about a year, Logilab has been involved in Mercurial development in the framework of a Mozilla Open Source Support (MOSS) program. Mercurial is a foundational technology for Mozilla, as the VCS used for the development of Firefox. As the main protagonist of the program, I'd first like to mention this has been a very nice experience for me, both from the social and technical perspectives. I've learned a lot and hope to continue working on this nice piece of software.

    The general objective of the program was to improve "code archaeology" workflows, especially when using hgweb (the web UI for Mercurial). Outcomes of program spread from versions 4.1 to 4.4 of Mercurial and I'm going to present the main (new) features in this post.

    Better navigation in "blame/annotate" view (hgweb)

    The first feature concerns the "annotate" view in hgweb; the idea was to improve navigation along "blamed" revisions, a process that is often tedious and involves a lot of clicks in the web UI (not easier from the CLI, for that matter). Basically, we added an hover box on the left side of file content displaying more context on blamed revision along with a couple of links to make navigation easier (links to parent changesets, diff and changeset views). See below for an example about mercurial.error module (to try it at https://www.mercurial-scm.org/repo/hg/annotate/4.3/mercurial/error.py)

    The hover box in annotate view in hgweb.

    Followlines

    While this wasn't exactly in the initial scope of the program, the most interesting result of my work is arguably the introduction of the "followlines" feature set. The idea is to build upon the log command instead of annotate to make it easier to follow changes across revisions and the new thing is to make this possible by filtering changes only affecting a block of lines in a particular file.

    The first component introduced a revset, named followlines, which accepts at least two arguments a file path and a line range written as fromline:toline (following Python slice syntax). For instance, say we are interested the history of LookupError class in mercurial/error.py module which, at tag 4.3, lives between line 43 and 59, we'll use the revset as follows:

    $ hg log -r 'followlines(mercurial/error.py, 43:59)'
    changeset:   7633:08cabecfa8a8
    user:        Matt Mackall <mpm@selenic.com>
    date:        Sun Jan 11 22:48:28 2009 -0600
    summary:     errors: move revlog errors
    
    changeset:   24038:10d02cd18604
    user:        Martin von Zweigbergk <martinvonz@google.com>
    date:        Wed Feb 04 13:57:35 2015 -0800
    summary:     error: store filename and message on LookupError for later
    
    changeset:   24137:dcfdfd63bde4
    user:        Siddharth Agarwal <sid0@fb.com>
    date:        Wed Feb 18 16:45:16 2015 -0800
    summary:     error.LookupError: rename 'message' property to something
    else
    
    changeset:   25945:147bd9e238a1
    user:        Gregory Szorc <gregory.szorc@gmail.com>
    date:        Sat Aug 08 19:09:09 2015 -0700
    summary:     error: use absolute_import
    
    changeset:   34016:6df193b5c437
    user:        Yuya Nishihara <yuya@tcha.org>
    date:        Thu Jun 01 22:43:24 2017 +0900
    summary:     py3: implement __bytes__() on most of our exception classes
    

    This only yielded changesets touching this class.

    This is not an exact science (because the algorithm works on diff hunks), but in many situations it gives interesting results way faster that with an iterative "annotate" process (in which one has to step from revision to revision and run annotate every times).

    The followlines() predicate accepts other arguments, in particular, descend which, in combination with the startrev, lets you walk the history of a block of lines in the descending direction of history. Below the detailed help (hg help revset.followlines) of this revset:

    $ hg help revsets.followlines
        "followlines(file, fromline:toline[, startrev=., descend=False])"
          Changesets modifying 'file' in line range ('fromline', 'toline').
    
          Line range corresponds to 'file' content at 'startrev' and should hence
          be consistent with file size. If startrev is not specified, working
          directory's parent is used.
    
          By default, ancestors of 'startrev' are returned. If 'descend' is True,
          descendants of 'startrev' are returned though renames are (currently)
          not followed in this direction.
    

    In hgweb

    The second milestone of the program was to port this followlines filtering feature into hgweb. This has been implemented as a line selection mechanism (using mouse) reachable from both the file and annotate views. There, you'll see a small green ± icon close to the line number. By clicking on this icon, you start the line selection process which can be completed by clicking on a similar icon on another line of the file.

    Starting a line selection for followlines in hgweb.

    After that, you'll see a box inviting you to follow the history of lines <selected range> , either in the ascending (older) or descending (newer) direction.

    Line selection completed for followlines in hgweb.

    Here clicking on the "newer" link, we get:

    Followlines result in hgweb.

    As you can notice, this gives a similar result as with the command line but it also displays the patch of each changeset. In these patch blocks, only the diff hunks affecting selected line range are displayed (sometimes with some extra context).

    What's next?

    At this point, the "followlines" feature is probably complete as far as hgweb is concerned. In the remainder of the MOSS program, I'd like to focus on a command-line interface producing a similar output as the one above in hgweb, i.e. filtering patches to only show followed lines. That would take the form of a --followlines/-L option to hg log command, e.g.:

    $ hg log -L mercurial/error.py,43:59 -p
    

    That's something I'd like to tackle at the upcoming Mercurial 4.4 sprint in Dublin!


  • Mercurial Sprint 4.4

    2017/10/10 by Denis Laxalde
    Mercurial

    In late September, I participated on behalf of Logilab to the Mercurial 4.4 sprint that held at Facebook Dublin. It was the opportunity to meet developers of the project, follow active topics and forthcoming plans for Mercurial. Here, I'm essentially summarizing the main points that held my attention.

    Amongst three days of sprint, the first two mostly dedicated to discussions and the last one to writing code. Overall, the organization was pretty smooth thanks to Facebook guys doing a fair amount of preparation. Amongst an attendance of 25 community members, some companies had a significant presence: noticeably Facebook had 10 employees and Google 5, the rest consisting of either unaffiliated people and single-person representing their affiliation. No woman, unfortunately, and a vast majority of people with English as a native or dominant usage language.

    The sprint started by with short talks presenting the state of the Mercurial project. Noticeably, in the state of the community talk, it was recalled that the project governance now holds on the steering committee while some things are still not completely formalized (in particular, the project does not have a code of conduct yet). Quite surprisingly, the committee made no explicit mention of the recurring tensions in the project which recently lead to a banishment of a major contributor.

    Facebook, Google, Mozilla and Unity then presented by turns the state of Mercurial usages in their organization. Both Facebook and Google have a significant set of tools around hg, most of them either aiming at making the user experience more smooth with respect to performance problems coming from their monorepos or towards GUI tools. Other than that, it's interesting to note most of these corporate users have now integrated evolve on the user side, either as is or with a convenience wrapper layer.

    After that, followed several "vision statements" presentations combined with breakout sessions. (Just presenting a partial selection.)

    The first statement was about streamlining the style of the code base: that one was fairly consensual as most people agreed upon the fact that something has to be done in this respect; it was finally decided (on the second day) to adopt a PEP-like process. Let's see how things evolve!

    Second, Boris Feld and I talked about the development of the evolve extension and the ongoing task about moving it into Mercurial core (slides). We talked about new usages and workflows around the changeset evolution concept and the topics concept. Despite the recent tensions on this technical topic in the community, we tried to feel positive and reaffirmed that the whole community has interests in moving evolve into core. On the same track, in another vision statement, Facebook presented their restack workflow (sort of evolve for "simple" cases) and suggested to push this into core: this is encouraging as it means that evolve-like workflows tend to get mainstream.

    Rust

    Another interesting vision statement was about using Rust in Mercurial. Most people agreed that Mercurial would benefit from porting its native C code in Rust, essentially for security reasons and hopefully to gain a bit of performance and maintainability. More radical ideas were also proposed such as making the hg executable a Rust program (thus embedding a Python interpreter along with its standard library) or reimplementing commands in Rust (which would pose problems with respect to the Python extension system). Later on, Facebook presented mononoke, a promising Mercurial server implemented in Rust that is expected to scale better with respect to high committing rates.

    Back to a community subject, we discussed about code review and related tooling. It was first recalled that the project would benefit from more reviewers, including people without a committer status. Then the discussion essentially focused on the Phabricator experiment that started in July. Introduction of a web-based review tool in Mercurial was arguably a surprise for the community at large since many reviewers and long-time contributors have always expressed a clear preference over email-based review. This experiment is apparently meant to lower the contribution barrier so it's nice to see the project moving forward on this topic and attempt to favor diversity by contribution. On the other hand, the choice of Phabricator was quite controversial. From the beginning (see replies to the announcement email linked above), several people expressed concerns (about notifications notably) and some reviewers also complained about the increase of review load and loss of efficiency induced by the new system. A survey recently addressed to the whole community apparently (no official report yet at the time of this writing) confirms that most employees from Facebook or Google seem pretty happy with the experiment while other community members generally dislike it. Despite that, it was decided to keep the "experiment" going while trying to improve the notification system (better threading support, more diff context, subscription-based notification, etc.). That may work, but there's a risk of community split as non-corporate members might feel left aside. Overall, adopting a consensus-based decision model on such important aspects would be beneficial.

    Yet another interesting discussion took place around branching models. Noticeably, Facebook and Google people presented their recommended (internal) workflow respectively based on remotenames and local bookmarks while Pulkit Goyal presented the current state to the topics extension and associated workflows. Bridging the gap between all these approaches would be nice and it seems that a first step would be to agree upon the definition of a stack of changesets to define a current line of work. In particular, both the topics extension and the show command have their own definition, which should be aligned.

    (Comments on reddit.)


  • Typing Mercurial with pytype

    2019/11/14 by Denis Laxalde

    Following the recent introduction of Python type annotations (aka "type hints") in Mercurial (see, e.g. this changeset by Augie Fackler), I've been playing a bit with this and pytype.

    pytype is a static type analyzer for Python code. It compares with the more popular mypy but I don't have enough perspective to make a meaningful comparison at the moment. In this post, I'll illustrate how I worked with pytype to gradually add type hints in a Mercurial module and while doing so, fix bugs!

    The module I focused on is mercurial.mail, which contains mail utilities and that I know quite well. Other modules are also being worked on, this one is a good starting point because it has a limited number of "internal" dependencies, which both makes it faster to iterate with pytype and reduces side effects of other modules not being correctly typed already.

    shell $ pytype mercurial/mail.py Computing dependencies Analyzing 1 sources with 36 local dependencies ninja: Entering directory `.pytype' [19/19] check mercurial.mail Success: no errors found

    The good news is that the module apparently already type-checks. Let's go deeper and merge the type annotations generated by pytype:

    $ merge-pyi -i mercurial/mail.py out/mercurial/mail.pyi
    

    (In practice, we'd use --as-comments option to write type hints as comments, so that the module is still usable on Python 2.)

    Now we have all declarations annotated with types. Typically, we'd get many things like:

    sourceCode def codec2iana(cs) -> Any: cs = pycompat.sysbytes(email.charset.Charset(cs).input_charset.lower()) # "latin1" normalizes to "iso8859-1", standard calls for "iso-8859-1" if cs.startswith(b"iso") and not cs.startswith(b"iso-"): return b"iso-" + cs[3:] return cs

    The function signature has been annotated with Any (omitted for parameters, explicit for return value). This somehow means that type inference failed to find the type of that function. As it's (quite) obvious, let's change that into:

    sourceCode def codec2iana(cs: bytes) -> bytes: ...

    And re-run pytype:

    $ pytype mercurial/mail.py
    Computing dependencies
    Analyzing 1 sources with 36 local dependencies
    ninja: Entering directory `.pytype'
    [1/1] check mercurial.mail
    FAILED: .pytype/pyi/mercurial/mail.pyi 
    pytype-single --imports_info .pytype/imports/mercurial.mail.imports --module-name mercurial.mail -V 3.7 -o .pytype/pyi/mercurial/mail.pyi --analyze-annotated --nofail --quick mercurial/mail.py
    File "mercurial/mail.py", line 253, in codec2iana: Function Charset.__init__ was called with the wrong arguments [wrong-arg-types]
      Expected: (self, input_charset: str = ...)
      Actually passed: (self, input_charset: bytes)
    
    For more details, see https://google.github.io/pytype/errors.html#wrong-arg-types.
    ninja: build stopped: subcommand failed.
    

    Interesting! email.charset.Charset is apparently instantiated with the wrong argument type. While it's not exactly a bug, because Python will handle bytes instead of str well in general, we can again change the signature (and code) to:

    sourceCode def codec2iana(cs: str) -> str: cs = email.charset.Charset(cs).input_charset.lower() # "latin1" normalizes to "iso8859-1", standard calls for "iso-8859-1" if cs.startswith("iso") and not cs.startswith("iso-"): return "iso-" + cs[3:] return cs

    Obviously, this involves a larger refactoring in client code of this simple function, see respective changeset for details.

    Another example is this function:

    sourceCode def _encode(ui, s, charsets) -> Any: '''Returns (converted) string, charset tuple. Finds out best charset by cycling through sendcharsets in descending order. Tries both encoding and fallbackencoding for input. Only as last resort send as is in fake ascii. Caveat: Do not use for mail parts containing patches!''' sendcharsets = charsets or _charsets(ui) if not isinstance(s, bytes): # We have unicode data, which we need to try and encode to # some reasonable-ish encoding. Try the encodings the user # wants, and fall back to garbage-in-ascii. for ocs in sendcharsets: try: return s.encode(pycompat.sysstr(ocs)), ocs except UnicodeEncodeError: pass except LookupError: ui.warn(_(b'ignoring invalid sendcharset: %s\n') % ocs) else: # Everything failed, ascii-armor what we've got and send it. return s.encode('ascii', 'backslashreplace') # We have a bytes of unknown encoding. We'll try and guess a valid # encoding, falling back to pretending we had ascii even though we # know that's wrong. try: s.decode('ascii') except UnicodeDecodeError: for ics in (encoding.encoding, encoding.fallbackencoding): ics = pycompat.sysstr(ics) try: u = s.decode(ics) except UnicodeDecodeError: continue for ocs in sendcharsets: try: return u.encode(pycompat.sysstr(ocs)), ocs except UnicodeEncodeError: pass except LookupError: ui.warn(_(b'ignoring invalid sendcharset: %s\n') % ocs) # if ascii, or all conversion attempts fail, send (broken) ascii return s, b'us-ascii'

    It quite clear from the return value (last line) that we can change its type signature to:

    sourceCode def _encode(ui, s:Union[bytes, str], charsets: List[bytes]) -> Tuple[bytes, bytes] ...

    And re-run pytype:

    $ pytype mercurial/mail.py
    Computing dependencies
    Analyzing 1 sources with 36 local dependencies
    ninja: Entering directory `.pytype'
    [1/1] check mercurial.mail
    FAILED: .pytype/pyi/mercurial/mail.pyi 
    pytype-single --imports_info .pytype/imports/mercurial.mail.imports --module-name mercurial.mail -V 3.7 -o .pytype/pyi/mercurial/mail.pyi --analyze-annotated --nofail --quick mercurial/mail.py
    File "mercurial/mail.py", line 342, in _encode: bad option in return type [bad-return-type]
      Expected: Tuple[bytes, bytes]
      Actually returned: bytes
    [...]
    
    For more details, see https://google.github.io/pytype/errors.html.
    ninja: build stopped: subcommand failed.
    

    That's a real bug. Line 371 contains return s.encode('ascii', 'backslashreplace') in the middle of the function. We indeed return a bytes value instead of a Tuple[bytes, bytes] as elsewhere in the function. The following changes fixes the bug:

    sourceCode diff --git a/mercurial/mail.py b/mercurial/mail.py --- a/mercurial/mail.py +++ b/mercurial/mail.py @@ -339,7 +339,7 @@ def _encode(ui, s, charsets) -> Any: ui.warn(_(b'ignoring invalid sendcharset: %s\n') % ocs) else: # Everything failed, ascii-armor what we've got and send it. - return s.encode('ascii', 'backslashreplace') + return s.encode('ascii', 'backslashreplace'), b'us-ascii' # We have a bytes of unknown encoding. We'll try and guess a valid # encoding, falling back to pretending we had ascii even though we # know that's wrong.

    Steps by steps, by replacing Any with real types, adjusting "wrong" types like in the first example and fixing bugs, we finally get the whole module (almost) completely annotated.