Blog entries by Pierre-Yves David [12]

FOSDEM 2013

2013/02/12 by Pierre-Yves David

I was in Bruxelles for FOSDEM 2013. As with previous FOSDEM there were too many interesting talks and people to see. Here is a summary of what I saw:

In the Mozilla's room:

  1. The html5 pdf viewer pdfjs is impressive. The PDF specification is really scary but this full featured "native" viewer is able to renders most of it with very good performance. Have a look at the pdfjs demo!
  1. Firefox debug tools overview with a specific focus of Firefox OS emulator in your browser.
  1. Introduction to webl10n: an internationalization format and library used in Firefox OS. A successful mix that results in a format that is idiot-proof enough for a duck to use, that relies on Unicode specifications to handle complex pluralization rules and that allows cascading translation definitions.
typical webl10n user
  1. Status of html5 video and audio support in Firefox. The topic looks like a real headache but the team seems to be doing really well. Special mention for the reverse demo effect: The speaker expected some format to be still unsupported but someone else apparently implemented them over night.
  2. Last but not least I gave a talk about the changeset evolution concept that I'm putting in Mercurial. Thanks goes to Feth for asking me his not-scripted-at-all-questions during this talk. (slides)
http://www.selenic.com/hg-logo/logo-droplets-150.png

In the postgresql room:

  1. Insightful talk about more event trigger in postgresql engine and how this may becomes the perfect way to break your system.
  2. Full update of the capability of postgis 2.0. The postgis suite was already impressive for storing and querying 2D data, but it now have impressive capability regarding 3D data.
http://upload.wikimedia.org/wikipedia/en/6/60/PostGIS_logo.png

On python related topic:

http://www.python.org/community/logos/python-logo-master-v3-TM-flattened.png
  • Victor Stinner has started an interesting project to improve CPython performance. The first one: astoptimizer breaks some of the language semantics to apply optimisation on compiling to byte code (lookup caching, constant folding,…). The other, registervm is a full redefinition of how the interpreter handles reference in byte code.

After the FOSDEM, I crossed the channel to attend a Mercurial sprint in London. Expect more on this topic soon.


Febuary 2013: Mercurial channel "tour"

2013/01/22 by Pierre-Yves David

The Release candidate version of Mercurial 2.5 was released last sunday.

http://mercurial.selenic.com/images/mercurial-logo.png

This new version makes a major change in the way "hidden" changesets are handled. In 2.4 only hg log (and a few others) would support effectively hiding "hidden" changesets. Now all hg commands are transparently compatible with the hidden revision concept. This is a considerable step towards changeset evolution, the next-generation collaboration technology that I'm developing for Mercurial.

https://fosdem.org/2013/assets/flyer-thumb-0505d19dbf3cf6139bc7490525310f8e253e60448a29ed4313801b723d5b2ef1.png

The 2.5 cycle is almost over, but there is no time to rest yet, Saturday the 2th of February, I will give a talk about changeset evolution concept at FOSDEM in the Mozilla Room. This talk in an updated version of the one I gave at OSDC.fr 2012 (video in french).

The week after, I'm crossing the channel to attend the Mercurial 2.6 Sprint hosted by Facebook London. I expect a lot of discussion about the user interface and network access of changeset evolution.

The HG 2.3 sprint

Mercurial 2.3 sprint, Day 1-2-3

2012/05/15 by Pierre-Yves David

I'm now back from Copenhagen were I attended the mercurial 2.3 sprint with twenty other people. A huge amount of work was done in a very friendly atmosphere.

Regarding mercurial's core:

  • Bookmark behaviour was improved to get closer to named branch's behaviour.
  • Several performance improvements regarding branches and heads caches. The heads cache refactoring improves rebase performance on huge repository (thanks to Facebook and Atlassian).
  • The concept I'm working on, Obsolete markers, was a highly discussed subject and is expected to get partly into the core in the near future. Thanks to my employer Logilab for paying me to work on this topic.
  • General code cleanup and lock validation.
http://www.logilab.org/file/92956?vid=download

Regarding the bundled extension :

  • Some fixes where made to progress which is now closer to getting into mercurial's core.
  • Histedit and keyring extensions are scheduled to be shipped with mercurial.
  • Some old and unmaintained extensions (children, hgtk) are now deprecated.
  • The LargeFile extension got some new features (thanks to the folks from Unity3D)
  • Rebase will use the --detach flag by default in the next release.
http://www.logilab.org/file/92958?vid=download

Regarding the project itself:

http://www.logilab.org/file/92955?vid=download

Regarding other extensions:

http://www.logilab.org/file/92959?vid=download

And I'm probably forgetting some stuff. Special thanks to Unity3D for hosting the sprint and providing power, network and food during these 3 days.


Mercurial 2.3 day 0

2012/05/10 by Pierre-Yves David

I'm now at Copenhagen to attend the mercurial "2.3" sprint.

About twenty people are attending, including staff from Atlassian, Facebook, Google and Mozilla.

I expect code and discussion about various topic among:

  • the development process of mercurial itself,
  • performance improvement on huge repository,
  • integration of Obsolete Markers into mercurial core,
  • improvement on various aspect (merge diff, moving some extension in core, ...)

I'm of course very interested in the Obsolete Markers topic. I've been working on an experimental implementation for several months. An handful of people are using them at Logilab for two months and feedback are very promising.


Introduction To Mercurial Phases (Part III)

2012/02/03 by Pierre-Yves David

This is the final part of a series of posts about the new phases feature we implemented for mercurial 2.1. The first part talks about how phases will help mercurial users, the second part explains how to control them. This one explains what people should take care of when upgrading.

Important upgrade note and backward compatibility

Phases do not require any conversion of your repos. Phase information is not stored in changesets. Everybody using a new client will take advantage of phases on any repository they touch.

However there is some points you need to be aware of regarding interaction between the old world without phases and the new world with phases:

Talking over the wire to a phaseless server using a phased client

As ever, the Mercurial wire protocol (used to communicate through http and ssh) is fully backward compatible [1]. But as old Mercurial versions are not aware of phases, old servers will always be treated as publishing.

Direct file system access to a phaseless repository using a phased client

A new client has no way to determine which parts of the history should be immutable and which parts should not. In order to fail safely, a new repo will mark everything as public when no data is available. For example, in the scenario described in part I, if an old version of mercurial were used to clone and commit, a new version of mercurial will see them as public and refuse to rebase them.

Note

Some extensions (like mq) may provide smarter logic to set some changesets to the draft or even secret phases.

The phased client will write phase data to the old repo on its first write operation.

Direct file system access to a phased repository using a phaseless client

Everything works fine except that the old client is unable to see or manipulate phases:

  • Changesets added to the repo inherit the phase of their parents, whatever the parents' phase. This could result in new commits being seen as public or pulled content seen as draft or even secret when a newer client uses the repo again!
  • Changesets pushed to a publishing server won't be set public.
  • Secret changesets are exchanged.
  • Old clients are willing to rewrite immutable changesets (as they don't know that they shouldn't).

So, if you actively rewrite your history or use secret changesets, you should ensure that only new clients touch those repositories where the phase matters.

Fixing phases error

Several situations can result in bad phases in a repository:

  • When upgrading from phaseless to phased Mercurial, the default phases picked may be too restrictive.
  • When you let an old client touch your repository.
  • When you push to a publishing server that should not actually be publishing.

The easiest way to restore a consistant state is to use the phase command. In most cases, changesets marked as public but absent from your real public server should be moved to draft:

hg phase --force --draft 'public() and outgoing()'

If you have multiple public servers, you can pull from the others to retrieve their phase data too.

Conclusion

Mercurial's phases are a simple concept that adds always on and transparent safety for most users while not preventing advanced ones from doing whatever they want.

Behind this safety-enabling and useful feature, phases introduce in Mercurial code the concept of sharing mutable parts of history. The introduction of this feature paves the way for advanced history rewriting solutions while allowing safe and easy sharing of mutable parts of history. I'll post about those future features shortly.


[1]You can expect the 0.9.0 version of Mercurial to interoperate cleanly with one released 5 years later.

[Images by Crystian Cruz (cc-nd) and C.J. Peters (cc-by-sa)]


Introduction To Mercurial Phases (Part II)

2012/02/02 by Pierre-Yves David

This is the second part of a series of posts about the new phases feature we implemented for mercurial 2.1. The first part talks about how phases will help mercurial users, this second part explains how to control them.

Controlling automatic phase movement

Sometimes it may be desirable to push and pull changesets in the draft phase to share unfinished work. Below are some cases:

  • pushing to continuous integration,
  • pushing changesets for review,
  • user has multiple machines,
  • branch clone.

You can disable publishing behavior in a repository configuration file [1]:

[phases]
   publish=False
   

When a repository is set to non-publishing, people push changesets without altering their phase. draft changesets are pushed as draft and public changesets are pushed as public:

celeste@Chessy ~/palace $ hg showconfig phases
   phases.publish=False
   
babar@Chessy ~/palace $ hg log --graph
   @  [draft] add a carpet (2afbcfd2af83)
   |
   o  [public] Add a table in the kichen (139ead8a540f)
   |
   o  [public] Add wall color (0d1feb1bca54)
   |
   
   babar@Chessy ~/palace $ hg outgoing ~celeste/palace/
   [public] Add wall color (0d1feb1bca54)
   [public] Add a table in the kichen (139ead8a540f)
   [draft] add a carpet (3c1b19d5d3f5)
   babar@Chessy ~/palace $ hg push ~celeste/palace/
   pushing to ~celeste/palace/
   searching for changes
   adding changesets
   adding manifests
   adding file changes
   added 3 changesets with 3 changes to 2 files
   babar@Chessy ~/palace $ hg log --graph
   @  [draft] add a carpet (2afbcfd2af83)
   |
   o  [public] Add a table in the kichen (139ead8a540f)
   |
   o  [public] Add wall color (0d1feb1bca54)
   |
   
   
celeste@Chessy ~/palace $ hg log --graph
   o  [draft] add a carpet (2afbcfd2af83)
   |
   o  [public] Add a table in the kichen (139ead8a540f)
   |
   o  [public] Add wall color (0d1feb1bca54)
   |
   
   

And pulling gives the phase as in the remote repository:

celeste@Chessy ~/palace $ hg up 139ead8a540f
   celeste@Chessy ~/palace $ echo The wall will be decorated with portraits >> bedroom
   celeste@Chessy ~/palace $ hg ci -m 'Decorate the wall.'
   created new head
   celeste@Chessy ~/palace $ hg log --graph
   @  [draft] Decorate the wall. (3389164e92a1)
   |
   | o  [draft] add a carpet (3c1b19d5d3f5)
   |/
   o  [public] Add a table in the kichen (139ead8a540f)
   |
   o  [public] Add wall color (0d1feb1bca54)
   |
   
   ---
   babar@Chessy ~/palace $ hg pull ~celeste/palace/
   pulling from ~celeste/palace/
   searching for changes
   adding changesets
   adding manifests
   adding file changes
   added 1 changesets with 1 changes to 1 files (+1 heads)
   babar@Chessy ~/palace $ hg log --graph
   @  [draft] Decorate the wall. (3389164e92a1)
   |
   | o  [draft] add a carpet (3c1b19d5d3f5)
   |/
   o  [public] Add a table in the kichen (139ead8a540f)
   |
   o  [public] Add wall color (0d1feb1bca54)
   |
   
   

Phase information is exchanged during pull and push operations. When a changeset exists on both sides but within different phases, its phase is unified to the lowest [2] phase. For instance, if a changeset is draft locally but public remotely, it is set public:

celeste@Chessy ~/palace $ hg push -r 3389164e92a1
   pushing to http://hg.celesteville.com/palace
   searching for changes
   adding changesets
   adding manifests
   adding file changes
   added 1 changesets with 1 changes to 1 files
   celeste@Chessy ~/palace $ hg log --graph
   @  [public] Decorate the wall. (3389164e92a1)
   |
   | o  [draft] add a carpet (3c1b19d5d3f5)
   |/
   o  [public] Add a table in the kichen (139ead8a540f)
   |
   o  [public] Add wall color (0d1feb1bca54)
   |
   
   ---
   babar@Chessy ~/palace $ hg pull ~celeste/palace/
   pulling from ~celeste/palace/
   searching for changes
   no changes found
   babar@Chessy ~/palace $ hg log --graph
   @  [public] Decorate the wall. (3389164e92a1)
   |
   | o  [draft] add a carpet (3c1b19d5d3f5)
   |/
   o  [public] Add a table in the kichen (139ead8a540f)
   |
   o  [public] Add wall color (0d1feb1bca54)
   |
   
   

Note

pull is read-only operation and does not alter phases in remote repositories.

You can also control the phase in which a new changeset is committed. If you don't want new changesets to be pushed without explicit consent, update your configuration with:

[phases]
   new-commit=secret
   

You will need to use manual phase movement before you can push them. See the next section for details:

Note

With what have been done so far for 2.1, the "most practical way to make a new commit secret" is to use:

   hg commit --config phases.new-commit=secret
   
[1]You can use this setting in your user hgrc too.
[2]Phases as ordered as follow: public < draft < secret

Manual phase movement

Most phase movements should be automatic and transparent. However it is still possible to move phase manually using the hg phase command:

babar@Chessy ~/palace $ hg log --graph
   @    [draft] merge with Celeste works (f728ef4eba9f)
   |\
   o |  [draft] add a carpet (3c1b19d5d3f5)
   | |
   | o  [public] Decorate the wall. (3389164e92a1)
   |/
   o  [public] Add a table in the kichen (139ead8a540f)
   |
   
   babar@Chessy ~/palace $ hg phase --public 3c1b19d5d3f5
   babar@Chessy ~/palace $ hg log --graph
   @    [draft] merge with Celeste works (f728ef4eba9f)
   |\
   o |  [public] add a carpet (3c1b19d5d3f5)
   | |
   | o  [public] Decorate the wall. (3389164e92a1)
   |/
   o  [public] Add a table in the kichen (139ead8a540f)
   |
   
   

Changesets only move to lower [#] phases during normal operation. By default, the phase command enforces this rule:

babar@Chessy ~/palace $ hg phase --draft 3c1b19d5d3f5
   no phases changed
   babar@Chessy ~/palace $ hg log --graph
   @    [draft] merge with Celeste works (f728ef4eba9f)
   |\
   o |  [public] add a carpet (3c1b19d5d3f5)
   | |
   | o  [public] Decorate the wall. (3389164e92a1)
   |/
   o  [public] Add a table in the kichen (139ead8a540f)
   |
   
   

If you are confident in what your are doing you can still use the --force switch to override this behavior:

Warning

Phases are designed to avoid forcing people to use hg phase --force. If you need to use --force on a regular basis, you are probably doing something wrong. Read the previous section again to see how to configure your environment for automatic phase movement suitable to your needs.

babar@Chessy ~/palace $ hg phase --verbose --force --draft 3c1b19d5d3f5
   phase change for 1 changesets
   babar@Chessy ~/palace $ hg log --graph
   @    [draft] merge with Celeste works (f728ef4eba9f)
   |\
   o |  [draft] add a carpet (3c1b19d5d3f5)
   | |
   | o  [public] Decorate the wall. (3389164e92a1)
   |/
   o  [public] Add a table in the kichen (139ead8a540f)
   |
   
   

Note that a phase defines a consistent set of revisions in your history graph. This means that to have a public (immutable) changeset all its ancestors need to be immutable too. Once you have a secret (not exchanged) changeset, all its descendant will be secret too.

This means that changing the phase of a changeset may result in phase movement for other changesets:

babar@Chessy ~/palace $ hg phase -v --public f728ef4eba9f # merge with Celeste works
   phase change for 2 changesets
   babar@Chessy ~/palace $ hg log --graph
   @    [public] merge with Celeste works (f728ef4eba9f)
   |\
   o |  [public] add a carpet (3c1b19d5d3f5)
   | |
   | o  [public] Decorate the wall. (3389164e92a1)
   |/
   o  [public] Add a table in the kichen (139ead8a540f)
   |
   
   babar@Chessy ~/palace $ hg phase -vf --draft 3c1b19d5d3f5 # add a carpet
   phase change for 2 changesets
   babar@Chessy ~/palace $ hg log --graph
   @    [draft] merge with Celeste works (f728ef4eba9f)
   |\
   o |  [draft] add a carpet (3c1b19d5d3f5)
   | |
   | o  [public] Decorate the wall. (3389164e92a1)
   |/
   o  [public] Add a table in the kichen (139ead8a540f)
   |
   
   

The next and final post will explain how older mercurial versions interact with newer versions that support phases.

[Images by Jimmy Smith (cc-by-nd) and Cory Doctorow (cc-by-sa)]


Introduction To Mercurial Phases (Part I)

2012/02/02 by Pierre-Yves David
credit: redshirtjosh, http://www.flickr.com/photos/43273828@N06/4111258568/

On the behalf of Logilab I put a lot of efforts to include a new core feature named phases in Mercurial 2.1. Phases are a system for tracking which changesets have been or should be shared. This helps to prevent common mistakes when modifying history (for instance, with the mq or rebase extensions). It will transparently benefit to all users. This concept is the first step towards simple, safe and powerful rewritting mecanisms for history in mercurial.

This serie of three blog entries will explain:

  1. how phases will help mercurial users,
  2. how one can control them,
  3. how older mercurial versions interact with newer versions that support phases.

Preventing erroneous history rewriting

credit: anita.priks, http://www.flickr.com/photos/46785534@N06/6358218623/

History rewriting is a common practice in DVCS. However when done the wrong way the most common error results in duplicated history. The phase concept aims to make rewriting history safer. For this purpose Mercurial 2.1 introduces a distinction between the "past" part of your history (that is expected to stay there forever) and the "present" part of the history (that you are currently evolving). The old and immutable part is called public and the mutable part of your history is called draft.

Let's see how this happens using a simple scenario.


A new Mercurial user clones a repository:

babar@Chessy ~ $ hg clone http://hg.celesteville.com/palace
requesting all changes
adding changesets
adding manifests
adding file changes
added 2 changesets with 2 changes to 2 files
updating to branch default
2 files updated, 0 files merged, 0 files removed, 0 files unresolved
babar@Chessy ~/palace $ cd palace
babar@Chessy ~/palace $ hg log --graph
@  changeset:   1:2afbcfd2af83
|  tag:         tip
|  user:        Celeste the Queen <Celeste@celesteville.com>
|  date:        Wed Jan 25 16:41:56 2012 +0100
|  summary:     We need a kitchen too.
|
o  changeset:   0:898889b143fb
   user:        Celeste the Queen <Celeste@celesteville.com>
   date:        Wed Jan 25 16:39:07 2012 +0100
   summary:     First description of the throne room

The repository already contains some changesets. Our user makes some improvements and commits them:

babar@Chessy ~/palace $ echo The wall shall be Blue >> throne-room
babar@Chessy ~/palace $ hg ci -m 'Add wall color'
babar@Chessy ~/palace $ echo In the middle stands a three meters round table >> kitchen
babar@Chessy ~/palace $ hg ci -m 'Add a table in the kichen'

But when he tries to push new changesets, he discovers that someone else already pushed one:

babar@Chessy ~/palace $ hg push
pushing to http://hg.celesteville.com/palace
searching for changes
abort: push creates new remote head bcd4d53319ec!
(you should pull and merge or use push -f to force)
babar@Chessy ~/palace $ hg pull
pulling from http://hg.celesteville.com/palace
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files (+1 heads)
(run 'hg heads' to see heads, 'hg merge' to merge)
babar@Chessy ~/palace $ hg log --graph
o  changeset:   4:0a5b3d7e4e5f
|  tag:         tip
|  parent:      1:2afbcfd2af83
|  user:        Celeste the Queen <Celeste@celesteville.com>
|  date:        Wed Jan 25 16:58:23 2012 +0100
|  summary:     Some bedroom description.
|
| @  changeset:   3:bcd4d53319ec
| |  user:        Babar the King <babar@celesteville.com>
| |  date:        Wed Jan 25 16:52:02 2012 +0100
| |  summary:     Add a table in the kichen
| |
| o  changeset:   2:f9f14815935d
|/   user:        Babar the King <babar@celesteville.com>
|    date:        Wed Jan 25 16:51:51 2012 +0100
|    summary:     Add wall color
|
o  changeset:   1:2afbcfd2af83
|  user:        Celeste the Queen <Celeste@celesteville.com>
|  date:        Wed Jan 25 16:41:56 2012 +0100
|  summary:     We need a kitchen too.
|
o  changeset:   0:898889b143fb
   user:        Celeste the Queen <Celeste@celesteville.com>
   date:        Wed Jan 25 16:39:07 2012 +0100
   summary:     First description of the throne room

Note

From here on this scenario becomes very unlikely. Mercurial is simple enough for a new user not to be that confused by such a trivial situation. But we keep the example simple to focus on phases.

Recently, our new user read some hype blog about "rebase" and the benefit of linear history. So, he decides to rewrite his history instead of merging.

Despite reading the wonderful rebase help, our new user makes the wrong decision when it comes to using it. He decides to rebase the remote changeset 0a5b3d7e4e5f:"Some bedroom description." on top of his local changeset.

With previous versions of mercurial, this mistake was allowed and would result in a duplication of the changeset 0a5b3d7e4e5f:"Some bedroom description."

babar@Chessy ~/palace $ hg rebase -s 4 -d 3
babar@Chessy ~/palace $ hg push
pushing to http://hg.celesteville.com/palace
searching for changes
abort: push creates new remote head bcd4d53319ec!
(you should pull and merge or use push -f to force)
babar@Chessy ~/palace $ hg pull
pulling from http://hg.celesteville.com/palace
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files (+1 heads)
(run 'hg heads' to see heads, 'hg merge' to merge)
babar@Chessy ~/palace $ hg log --graph
@  changeset:   5:55d9bae1e1cb
|  tag:         tip
|  parent:      3:bcd4d53319ec
|  user:        Celeste the Queen <Celeste@celesteville.com>
|  date:        Wed Jan 25 16:58:23 2012 +0100
|  summary:     Some bedroom description.
|
| o  changeset:   4:0a5b3d7e4e5f
| |  parent:      1:2afbcfd2af83
| |  user:        Celeste the Queen <Celeste@celesteville.com>
| |  date:        Wed Jan 25 16:58:23 2012 +0100
| |  summary:     Some bedroom description.
| |
o |  changeset:   3:bcd4d53319ec
| |  user:        Babar the King <babar@celesteville.com>
| |  date:        Wed Jan 25 16:52:02 2012 +0100
| |  summary:     Add a table in the kichen
| |
o |  changeset:   2:f9f14815935d
|/   user:        Babar the King <babar@celesteville.com>
|    date:        Wed Jan 25 16:51:51 2012 +0100
|    summary:     Add wall color
|
o  changeset:   1:2afbcfd2af83
|  user:        Celeste the Queen <Celeste@celesteville.com>
|  date:        Wed Jan 25 16:41:56 2012 +0100
|  summary:     We need a kitchen too.
|
o  changeset:   0:898889b143fb
   user:        Celeste the Queen <Celeste@celesteville.com>
   date:        Wed Jan 25 16:39:07 2012 +0100
   summary:     First description of the throne room

In more complicated setups it's a fairly common mistake, Even in big and successful projects and with other DVCSs.

In the new Mercurial version the user won't be able to make this mistake anymore. Trying to rebase the wrong way will result in:

babar@Chessy ~/palace $ hg rebase -s 4 -d 3
abort: can't rebase immutable changeset 0a5b3d7e4e5f
(see hg help phases for details)

The correct rebase still works as expected:

babar@Chessy ~/palace $ hg rebase -s 2 -d 4
babar@Chessy ~/palace $ hg log --graph
@  changeset:   4:139ead8a540f
|  tag:         tip
|  user:        Babar the King <babar@celesteville.com>
|  date:        Wed Jan 25 16:52:02 2012 +0100
|  summary:     Add a table in the kichen
|
o  changeset:   3:0d1feb1bca54
|  user:        Babar the King <babar@celesteville.com>
|  date:        Wed Jan 25 16:51:51 2012 +0100
|  summary:     Add wall color
|
o  changeset:   2:0a5b3d7e4e5f
|  user:        Celeste the Queen <Celeste@celesteville.com>
|  date:        Wed Jan 25 16:58:23 2012 +0100
|  summary:     Some bedroom description.
|
o  changeset:   1:2afbcfd2af83
|  user:        Celeste the Queen <Celeste@celesteville.com>
|  date:        Wed Jan 25 16:41:56 2012 +0100
|  summary:     We need a kitchen too.
|
o  changeset:   0:898889b143fb
   user:        Celeste the Queen <Celeste@celesteville.com>
   date:        Wed Jan 25 16:39:07 2012 +0100
   summary:     First description of the throne room

What is happening here:

  • Changeset 0a5b3d7e4e5f from Celeste was set to the public phase because it was pulled from the outside. The public phase is immutable.
  • Changesets f9f14815935d and bcd4d53319ec (rebased as 0d1feb1bca54 and 139ead8a540f) have been commited locally and haven't been transmitted from this repository to another. As such, they are still in the draft phase. Unlike the public phase, the draft phase is mutable.

Let's watch the whole action in slow motion, paying attention to phases:

babar@Chessy ~ $ cat >> ~/.hgrc << EOF
[ui]
username=Babar the King <babar@celesteville.com>
logtemplate='[{phase}] {desc} ({node|short})\\n'
EOF

First, changesets cloned from a public server are public:

babar@Chessy ~ $ hg clone --quiet http://hg.celesteville.com/palace
babar@Chessy ~/palace $ cd palace
babar@Chessy ~/palace $ hg log --graph
@  [public] We need a kitchen too. (2afbcfd2af83)
|
o  [public] First description of the throne room (898889b143fb)

Second, new changesets committed locally are in the draft phase:

babar@Chessy ~/palace $ echo The wall shall be Blue >> throne-room
babar@Chessy ~/palace $ hg ci -m 'Add wall color'
babar@Chessy ~/palace $ echo In the middle stand a three meters round table >> kitchen
babar@Chessy ~/palace $ hg ci -m 'Add a table in the kichen'
babar@Chessy ~/palace $ hg log --graph
@  [draft] Add a table in the kichen (bcd4d53319ec)
|
o  [draft] Add wall color (f9f14815935d)
|
o  [public] We need a kitchen too. (2afbcfd2af83)
|
o  [public] First description of the throne room (898889b143fb)

Third, changesets pulled from a public server are public:

babar@Chessy ~/palace $ hg pull --quiet
babar@Chessy ~/palace $ hg log --graph
o  [public] Some bedroom description. (0a5b3d7e4e5f)
|
| @  [draft] Add a table in the kichen (bcd4d53319ec)
| |
| o  [draft] Add wall color (f9f14815935d)
|/
o  [public] We need a kitchen too. (2afbcfd2af83)
|
o  [public] First description of the throne room (898889b143fb)

Note

rebase preserves the phase of rebased changesets

babar@Chessy ~/palace $ hg rebase -s 2 -d 4
babar@Chessy ~/palace $ hg log --graph
@  [draft] Add a table in the kichen (139ead8a540f)
|
o  [draft] Add wall color (0d1feb1bca54)
|
o  [public] Some bedroom description. (0a5b3d7e4e5f)
|
o  [public] We need a kitchen too. (2afbcfd2af83)
|
o  [public] First description of the throne room (898889b143fb)

Finally, once pushed to the public server, changesets are set to the public (immutable) phase

babar@Chessy ~/palace $ hg push
pushing to http://hg.celesteville.com/palace
searching for changes
adding changesets
adding manifests
adding file changes
added 2 changesets with 2 changes to 2 files
babar@Chessy ~/palace $ hg log --graph

@  [public] Add a table in the kichen (139ead8a540f)
|
o  [public] Add wall color (0d1feb1bca54)
|
o  [public] Some bedroom description. (0a5b3d7e4e5f)
|
o  [public] We need a kitchen too. (2afbcfd2af83)
|
o  [public] First description of the throne room (898889b143fb)

To summarize:

  • Changesets exchanged with the outside are public and immutable.
  • Changesets committed locally are draft until exchanged with the outside.
  • As a user, you should not worry about phases. Phases move transparently.

Preventing premature exchange of history

credit: Richard Elzey, http://www.flickr.com/photos/elzey/3516256055/

The public phases prevent user from accidentally rewriting public history. It's a good step forward but phases can go further. Phases can prevent you from accidentally making history public in the first place.

For this purpose, a third phase is available, the secret phase. To explain it, I'll use the mq extension which is nicely integrated with this secret phase:

Our fellow user enables the mq extension

babar@Chessy ~/palace $ vim ~/.hgrc
babar@Chessy ~/palace $ cat ~/.hgrc
[ui]
username=Babar the King <babar@celesteville.com>
[extensions]
# enable the mq extension included with Mercurial
hgext.mq=
[mq]
# Enable secret phase integration.
# This integration is off by default for backward compatibility.
secret=true

New patches (not general commits) are now created as secret

babar@Chessy ~/palace $ echo A red carpet on the floor. >> throne-room
babar@Chessy ~/palace $ hg qnew -m 'add a carpet' carpet.diff
babar@Chessy ~/palace $ hg log --graph

@  [secret] add a carpet (3c1b19d5d3f5)
|
@  [public] Add a table in the kichen (139ead8a540f)
|
o  [public] Add wall color (0d1feb1bca54)
|

this secret changeset is excluded from outgoing and push:

babar@Chessy ~/palace $ hg outgoing
comparing with http://hg.celesteville.com/palace
searching for changes
no changes found (ignored 1 secret changesets)
babar@Chessy ~/palace $ hg push
pushing to http://hg.celesteville.com/palace
searching for changes
no changes found (ignored 1 secret changesets)

And other users do not see it:

celeste@Chessy ~/palace $ hg incoming ~babar/palace/
comparing with ~babar/palace
searching for changes
[public] Add wall color (0d1feb1bca54)
[public] Add a table in the kichen (139ead8a540f)

The mq integration take care of phase movement for the user. Changeset are made draft by qfinish

babar@Chessy ~/palace $ hg qfinish .
babar@Chessy ~/palace $ hg log --graph
@  [draft] add a carpet (2afbcfd2af83)
|
o  [public] Add a table in the kichen (139ead8a540f)
|
o  [public] Add wall color (0d1feb1bca54)
|

And changesets are made secret again by qimport

babar@Chessy ~/palace $ hg qimport -r 2afbcfd2af83
babar@Chessy ~/palace $ hg log --graph
@  [secret] add a carpet (2afbcfd2af83)
|
o  [public] Add a table in the kichen (139ead8a540f)
|
o  [public] Add wall color (0d1feb1bca54)
|

As expected, mq refuses to qimport public changesets

babar@Chessy ~/palace $ hg qimport -r 139ead8a540f
abort: revision 4 is not mutable

In the next part I'll details how to control phases movement.


Interesting things seen at the Afpy Computer Camp

2011/11/28 by Pierre-Yves David

This summer I spent three days in Burgundy at the Afpy Computer Camps. This yearly meeting gathered French speaking python developers for talking and coding. The main points of this 2011 edition were:

http://www.afpy.org/_public/images/logo_afpy.png

The new IPython 0.11 was shown by Olivier Grisel. This new version contains lots of impressive feature like inline figures, asynchronous execution, exportable sessions, and a web-browser based client. IPython was also presented by its main author Fernando Perez during his keynote talk at EuroSciPy. Since then Logilab got involved with IPython. We contributed to the Debian packaging of iPython dependencies and we joined the discussion about Restructured Text formatting for note book.

http://ipython.org/ipython-doc/rel-0.11/_static/logo.png

Tarek Ziade bootstrapped his new Red Barrel project and small framework to build modern webservices with multiple back-end including the new socket.io protocol.

Alexis Métaireau and Feth Arezki discovered their common interest into account tracking application. The discussion's result is a first release of I hate money a few months later.

For my part, I spent most of my time working with Boris Feld on the Python Testing Infrastructure , a continuous integration tool to test python distributions available at PyPI.

http://master.pyti.org/data/pyti.ico.png

This yearly Afpy Computer Camps is an event intended for python developers but the Afpy also organize events for non python developer. The next one is tonight in Paris at La cantine : Vous reprendrez bien un peu de python ?. See you tonight ?


The Python Package Index is not a "Software Distribution"

2011/01/26 by Pierre-Yves David

Recent discussions on the #disutils irc channel and with my logilab co-workers led me to the following conclusions:

  • The Python Package Index is not a software distribution
  • There is more than one way to distribute python software
  • Distribution packagers are power users and need super cow-powers
  • Users want it to "just works"
  • The Python Package Index is used by many as a software distribution
  • Pypi has a lot of contributions because requirements are low.

The Python Package Index is not a software distribution

I would define a software distribution as :

  • Organised group of people
  • Who apply a Unified Quality process
  • To a finite set of software
  • Which includes all its dependencies
  • With a consistent set of versions that work together
  • For a finite set of platforms
  • Managed and installed by dedicated tools.

Pypi is a public index where:

  • Any python developer
  • Can upload any tarball containing something related
  • To any python package
  • Which might have external dependencies (outside Pypi)
  • The latest version of something is always available disregarding its compatibility with other packages.
  • Binary packages can be provided for any platform but are usually not.
  • There are several tools to install and manage python packages from pypi.

Pypi is not a software distribution, it is a software index.

Card File by Mr. Ducke / Matt

There is more than one way to distribute python software

There is a long way from the pure source used by the developer to the software installed on the system of the end user.

First, the source must be extracted from a (D)VCS to make a version tarball, while executing several release specific actions (eg: changelog generation from a tracker) Second, the version tarball is used to generate a platform independent build, while executing several build steps (eg, Cython compilation into C files or documentation generation). Third, the platform independent build is used to generate a platform dependant build, while executing several platforms dependant build (eg, compilation of C extension). Finally, the platform dependant build is installed and each file gets dispatched to its proper location during the installation process.

Pieces of software can be distributed as development snapshots taken from the (D)VCS, version tarballs, source packages, platform independent package or platform dependent package.

package! by Beck Gusler

Distribution packagers are power users and need super cow-powers

Distribution packagers usually have the necessary infrastructure and skills to build packages from version tarballs. Moreover they might have specific needs that require as much control as possible over the various build steps. For example:

  • Specific help system requiring a custom version of sphinx.
  • Specific security or platform constraint that require a specific version of Cython
Cheese Factory by James Yu

Users want it to "just work"

Standard users want it to "just work". They prefer simple and quick ways to install stuff. Build steps done on their machine increase the duration of the installation, add potential new dependencies and may trigger an error. Standard users are very disappointed when an installed failed because an error occurred while building the documentation. User give up when they have to download extra dependency and setup complicated compilation environment.

Users want as many build steps as possible to be done by someone else. That's why many users usually choose a distribution that do the job for them (eg, ubuntu, red-hat, python xy)

The Python Package Index is used by many as a software distribution

But there are several situations where the user can't rely on his distribution to install python software:

  • There is no distribution available for the platform (Windows, Mac OS X)
  • They want to install a python package outside of their distribution system (to test or because they do not have the credentials to install it system-wide)
  • The software or version they need is not included in the finite set of software included in their distribution.

When this happens, the user will use Pypi to fetch python packages. To help them, Pypi accepts binary packages of python modules and people have developed dedicated tools that ease installation of packages and their dependencies: pip, easy_install.

Pip + Pypi provides the tools of a distribution without its consistency. This is better than nothing.

Pypi has a lot of contributions because requirements are low

Pypi should contain version tarballs of all known python modules. It is the first purpose of an index. Version tarball should let distribution and power user perform as many build steps as possible. Pypi will continue to be used as a distribution by people without a better option. Packages provided to these users should require as little as possible to be installed, meaning they either have no build step to perform or have only platforms dependent build step (that could not be executed by the developer).

Thomas Fisher Rare Book Library by bookchen

If the incoming distutils2 provides a way to differentiate platform dependent build steps from platform independent ones, python developers will be able to upload three different kind of package on Pypi.

sdist:Pure source version released by upstream targeted at packagers and power users.
idist:Platform-independent package with platform independent build steps done (Cython, docs). If there is no such build step, the package is the same as sdist.
bdist:Platform-dependent package with all build steps performed. For package with no platform dependent build step this package is the same that idist.

(Image under creative commons Card File by-nc-nd by Mr. Ducke / Matt, Thomas Fisher Rare Book Library by bookchen, package! by Beck Gusler, Cheese Factory by James Yu)


Distutils2 January Sprint in Paris

2011/01/07 by Pierre-Yves David

At Logilab, we have the pleasure to host a distutils2 sprint in January. Sprinters are welcome in our Paris office from 9h on the 27th of January to 19h the 30th of January. This sprint will focus on polishing distutils2 for the next alpha release and on the install/remove scripts.

Distutils2 is an important project for Python. Every contribution will help to improve the current state of packaging in Python. See the wiki page on python.org for details about participation. If you can't attend or join us in Paris, you can participate on the #distutils channel of the freenode irc network

http://guide.python-distribute.org/_images/state_of_packaging.jpg

For additional details, see Tarek Ziadé's original announce, read the wiki page on python.org or contact us


Apycot for Mercurial

2010/02/11 by Pierre-Yves David
http://www.logilab.org/image/20439?vid=download

What is apycot

apycot is a highly extensible test automatization tool used for Continuous Integration. It can:

  • download the project from a version controlled repository (like SVN or Hg);
  • install it from scratch with all dependencies;
  • run various checkers;
  • store the results in a CubicWeb database;
  • post-process the results;
  • display the results in various format (html, xml, pdf, mail, RSS...);
  • repeat the whole procedure with various configurations;
  • get triggered by new changesets or run periodically.

For an example, take a look at the "test reports" tab of the logilab-common project.

Setting up an apycot for Mercurial

During the mercurial sprint, we set up a proof-of-concept environment running six different checkers:

  • Check syntax of all python files.
  • Check syntax of all documentation files.
  • Run pylint on the mercurial source code with the mercurial pylintrc.
  • Run the check-code.py script included in mercurial checking style and python errors
  • Run the Mercurial's test suite.
  • Run Mercurial's benchmark on a reference repository.

The first three checkers, shipped with apycot, were set up quickly. The last three are mercurial specific and required few additional tweaks to be integrated to apycot.

The bot was setup to run with all public mercurial repositories. Five checkers immediately proved useful as they pointed out some errors or warnings (on some rarely used contrib files it even found a syntax error).

Prospectives

A public instance is being set up. It will provide features that the community is looking forward to:

  • testing all python versions;
  • running pure python or the C variant;
  • code coverage of the test suite;
  • performance history.

Conclusion

apycot proved to be highly flexible and could quickly be adapted to Mercurial's test suite even for people new to apycot. The advantages of continuously running different long running tests is obvious. So apycot seems to be a very valuable tool for improving the software development process.


pylint bug day report

2009/12/04 by Pierre-Yves David
http://farm1.static.flickr.com/85/243306920_6a12bb48c7.jpg

The first pylint bug day took place on wednesday 25th. Four members of the Logilab crew and two other people spent the day working on pylint.

Several patches submitted before the bug day were processed and some tickets were closed.

Charles Hébert added James Lingard's patches for string formatting and is working on several improvements. Vincent Férotin submitted a patch for simple message listings. Sylvain Thenault fixed significant inference bugs in astng (an underlying module of pylint managing the syntax tree). Émile Anclin began a major astng refactoring to take advantage of new python2.6 functionality. For my part, I made several improvements to the test suite. I applied James Lingard patches for ++ operator and generalised it to -- too. I also added a new checker for function call arguments submitted by James Lingard once again. Finally I improved the message filtering of the --errors-only options.

We thank Maarten ter Huurne, Vincent Férotin for their participation and of course James Lingard for submitting numerous patches.

Another pylint bug day will be held in a few months.

image under creative commons by smccann


First contact with pupynere

2009/11/06 by Pierre-Yves David

I spent some time this week evaluating Pupynere, the PUre PYthon NEtcdf REader written by Roberto De Almeida. I see several advantages in pupynere.

First it's a pure Python module with no external dependency. It doesn't even depend on the NetCDF lib and it is therefore very easy to deploy.

http://www.unidata.ucar.edu/software/netcdf/netcdf1_sm.png

Second, it offers the same interface as Scientific Python's NetCDF bindings which makes transitioning from one module to another very easy.

Third pupynere is being integrated into Scipy as the scypi.io.netcdf module. Once integrated, this could ensure a wide adoption by the python community.

Finally it's easy to dig in this clear and small code base of about 600 lines. I have just sent several fixes and bug reports to the author.

http://docs.scipy.org/doc/_static/scipyshiny_small.png

However pupynere isn't mature yet. First it seems pupynere has been only used for simple cases so far. Many common cases are broken. Moreover there is no support for new NetCDF formats such as long-NetCDF and NetCDF4, and important features such as file update are still missing. In addition, The lack of a test suite is a serious issue. In my opinion, various bugs could already have been detected and fixed with simple unit tests. Contributions would be much more comfortable with the safety net offered by a test suite. I am not certain that the fixes and improvements I made this week did not introduce regressions.

To conclude, pupynere seems too young for production use. But I invite people to try it and provide feedback and fixes to the author. I'm looking forward to using this project in production in the future.