Blog entries

OpenData à Nantes: agrégateur des événements culturels
2011/12/12 by Arthur Lutz
Jeudi 8 décembre 2011 nous avons participé à la réunion de travail sur l'ouverture des données événementielles.

Problématique des licences

Un premier problème est que la licence proposée par LiberTIC est la CreativeCommons CC-BY, alors que les producteurs de données n'ont souvent pas les droits sur toutes les données qu'ils diffusent (par exemple la photo d'illustration d'un concert). Ils auront donc du mal à les publier en totalité sous licence CC-BY. Espérons que la licence Creative Commons rentre dans les habitudes et que cela ne va pas trop freiner le projet.

Aujourd'hui, l'utilisation ressemble à du Fair Use: on tolère la ré-utilisation de contenus protégés par le droit d'auteur car cela sert la diffusion de l'information.

Nous nous sommes demandé s'il est possible de mélanger deux licences dans un flux de données ou s'il faut faire deux flux séparés mais liés.

Problématique d'utilisation

Un deuxième problème est que les réutilisateurs ne seront pas intéréssés si les données sont trop pauvres et qu'elles n'incluent pas d'image ou de vidéo. Il faut donc trouver un socle commun qui satisfasse les producteurs et les réutilisateurs.

Import ou gros formulaires qui tâchent ?

Vu la complexité du modèle de données qui a émergé des discussions (beaucoup de cas particuliers), il a été proposé de fournir un formulaire de saisie d'un événement. A notre avis, la saisie "manuelle" doit rester un cas exceptionnel (un acteur culturel n'ayant pas de site pour publier par exemple), au risque de n'être pour les producteurs qu'un enième site à renseigner lors de la publication de son agenda.

Un exemple de bonnes pratiques est le très populaire GoodRelations qui offre un formulaire pour qu'un utilisateur qui n'a pas intégré le format à sa boutique en ligne puisse facilement générer son fichier et l'héberger chez lui, favorisant ainsi un modèle décentralisé calqué sur celui des moteurs de recherche.
Formats

Il nous semble donc important de se concentrer sur les formats standards qui pourraient être importés et exportés par la plateforme.

En voici une liste non exhaustive:
- ical
- xcal
- vcal
- schema.org / Event
- Event Ontology
Lectures supplémentaires

Cherchant à combiner des vocabulaires existants (afin de ne pas réinventer un format qui devra être traduit dans un autre vocabulaire pour être réutilisable) nous sommes tombés sur les articles suivants :
- le chapitre 15 du Mashup Guide explique comment du DublinCore peut être ajouté au sein de xcal. Il fournit aussi un exemple d'utilisation d'une API de requêtes sur un site populaire aux USA : http://upcoming.yahoo.com/
- La question de la licence "Machine readable" a été abordée http://wiki.creativecommons.org/Frequently_Asked_Questions#What_does_it_mean_that_Creative_Commons_licenses_are_.22machine-readable.22.3F et http://wiki.creativecommons.org/RDFa et http://wiki.creativecommons.org/XMP (property="cc:attributionName" rel="cc:attributionURL")
- LODE: An ontology for Linking Open Descriptions of Events http://linkedevents.org/ontology/
Conclusion

Il nous paraît important de ne pas se tromper dans les orientations choisies:
- utiliser des formats standards et combiner l'utilisation de namespaces existants plutôt que d'inventer un nouveau format
- proposer plusieurs formats d'export pour différentes utilisations (json, ical, etc) quitte à ne pas inclure tout le contenu disponible si le format ne s'y prête pas
- ne pas créer une API de plus et choisir de privilégier les standards du web sémantique en publiant du RDF et si possible en fournissant un accès SPARQL
- préférer la publication distribuée des données par leurs producteurs et leur agrégation par la plate-forme plutôt que d'attendre des producteurs qu'ils remplissent un formulaire de plus.
Nous attendons avec impatience la suite des travaux. Selon LiberTIC la plateforme sera developpée en logiciel libre avec des outils collaboratifs pour piloter le projet.

CubicWeb est une plateforme disponible en logiciel libre qui a déjà fait ses preuves et a été conçue pour développer des applications du type de l'aggrégateur décrit ci-dessus: import et export des données sous différents formats, utilisation des technologies standards du web sémantique. Nous espérons que ceux qui auront à réaliser l'agrégateur choisiront CubicWeb comme base technique pour ce projet.
0 Comment | tagged by creative commons, cubicweb, culture, innovation, opendata | blogged in Logilab.org - en VF
Interesting things seen at the Afpy Computer Camp
2011/11/28 by Pierre-Yves David
This summer I spent three days in Burgundy at the Afpy Computer Camps. This yearly meeting gathered French speaking python developers for talking and coding. The main points of this 2011 edition were:

The new IPython 0.11 was shown by Olivier Grisel. This new version contains lots of impressive feature like inline figures, asynchronous execution, exportable sessions, and a web-browser based client. IPython was also presented by its main author Fernando Perez during his keynote talk at EuroSciPy. Since then Logilab got involved with IPython. We contributed to the Debian packaging of iPython dependencies and we joined the discussion about Restructured Text formatting for note book.

Tarek Ziade bootstrapped his new Red Barrel project and small framework to build modern webservices with multiple back-end including the new socket.io protocol.

Alexis Métaireau and Feth Arezki discovered their common interest into account tracking application. The discussion's result is a first release of I hate money a few months later.

For my part, I spent most of my time working with Boris Feld on the Python Testing Infrastructure , a continuous integration tool to test python distributions available at PyPI.

This yearly Afpy Computer Camps is an event intended for python developers but the Afpy also organize events for non python developer. The next one is tonight in Paris at La cantine : Vous reprendrez bien un peu de python ?. See you tonight ?

0 Comment | tagged by afpy, afpy computer camps, ipython, pyti | blogged in Logilab.org - en
Rencontre Open Data à Nantes: Enjeux et opportunités pour le secteur culturel
2011/11/17 by Arthur Lutz
Nous étions présents à l'évenement organisé par Stereolux et Libertic consacré à l'OpenData dans le domaine de la culture à Nantes. Voici un court compte rendu des points que nous avons retenus de ces présentations.
Présentation générale de l'OpenData par Libertic

Il existe sur la toile assez d'articles sur l'Opendata pour qu'il ne nous semble pas nécessaire d'en donner une description, mais nous tenons à souligner que l'OpenData n'est pas simplement une mise à disposition des informations. Pour que des données puissent être qualifiées d'ouvertes, il faut qu'elles respectent une dizaine de principes parmi lesquels l'accessiblité, l'exploitabilité (données brutes), et la la réutilisablitié (licence).

Claire Gallon a cité plusieurs exemples d'OpenData dans le domaine culturel :
- la mise à disposition de données sur la fréquentation d'un musée permet de développer un service qui donnera la meilleure heure pour visiter ce musée. Voir When Should I visit Tate Modern
- Marseille-Provence 2013 (capitale culturelle européenne) ouvre ses données et attend que les acteurs écrivent des applications (mobiles notamment).
Un idée importante est que le service public doit s'adresser au plus grand nombre et ne peut pas consacrer ses ressources à la mise en place de services de niche. La mise à disposition des données permet à des tiers d'occuper ces niches.

En conclusion, Claire Gallon insiste sur la nécessité d'inclure la gestion de la communauté dans les démarches d'ouverture des données. La prochaine priorité des acteurs de l'OpenData sera la coproduction, à la fois pour l'écriture des applications et pour l'amélioration des données.
Présentation du projet data.bnf.fr par Romain Wenz

Romain Wenz de la Bibliothèque nationale de France a présenté http://data.bnf.fr sous l'angle de l'ouverture : l'ouverture à un public différent, l'ouverture à un mode de recherche différent (on cherche sur internet avant d'aller en bibliothèque) et l'ouverture sur les reseaux sociaux où le public partage des références à des contenus qu'il apprécie (twitter, facebook, etc.). Cette ouverture passe forcément par un web indexable, où l'on peut communiquer facilement une URL d'un contenu (exit les portails de recherche avec des sessions et variable http). Si un site n'est pas indexable, son contenu pourra être trouvé en s'y connectant directement, mais celui-ci restera dans le web "invisible" ou "profond".

Romain Wenz a insisté sur l'Importance des technologies utilisées : d'un coté les strandards ouverts et formalisés par le W3C, notamment en terme de web sémantique (RDF, RDFa, opengraph, schema.org, etc.) et de l'autre l'utilité de s'appuyer sur du logiciel libre. Dans le cas de http://data.bnf.fr il s'agit de CubicWeb.
Présentation des collaborations entre Wikimedia France et des institutions publiques à Toulouse

La transition entre la BnF et Wikimedia est facile : Wikisource (bibliothèque de livres libres de droits) a signé un partenariat avec Gallica qui lui a fourni des numérisations de livres tombés dans le domaine public.

Wikimedia France a présenté deux projets réussis en coproduction avec des institutions Toulousaines :
- le projet Phoebus a donné accès aux archives du Muséum de Toulouse à des bénévoles
- la communauté Wikimedia Commons a participé à l'enrichissement des metadonnées du fond consacré au photographe Eugène Trutat.
Présentation OpenData par la mairie de Nantes Métropole

Frédéric Vasse a briévement présenté la démarche de la Ville de Nantes en matière d'OpenData. Le lancement de la plateforme aura lieu lundi prochain à la Cantine Numérique de Nantes. Selon lui, l'objectif de Nantes est de réussir la coproduction avec les acteurs du territoire.

Conclusion et ouverture sur un projet concret d'OpenData pour les acteurs culturels

Libertic a conclu en proposant aux acteurs culturels un projet d'aggrégateur d'informations sur les événements culturels à Nantes. Nous espérons pouvoir vous donner prochainement plus d'informations sur ce projet.

Autre compte rendu (prises de notes) : http://www.scribd.com/doc/72810587/Opendata-Culture
1 Comment | tagged by cubicweb, data.bnf.fr, libertic, nantes, opendata | blogged in Logilab.org - en VF
Python in Finance (and Derivative Analytics)
2011/10/25 by Damien Garaud
The Logilab team attended (and co-organized) EuroScipy 2011, at the end of August in Paris.

We saw some interesting posters and a presentation dealing with Python in finance and derivative analytics [1].

In order to debunk the idea that "all computation libraries dedicated to financial applications must be written in C/C++ or some other compiled programming language", I would like to introduce a more Pythonic way.

You may know that financial applications such as risk management have in most cases high computational needs. For instance, it can be necessary to quickly perform a large number of Monte Carlo simulations to evaluate an American option in a few seconds.

The Python community provides several reliable and efficient libraries and packages dedicated to numerical computations:
- the well-known SciPy and NumPy libraries. They provide a complete set of tools to work with matrix, linear algebra operations, singular values decompositions, multi-variate regression models, ...
- scikits is a set of add-on toolkits for SciPy. For instance there are statistical models in statsmodels packages, a toolkit dedicated to timeseries manipulation and another one dedicated to numerical optimization;
- pandas is a recent Python package which provides "fast, flexible, and expressive data structures designed to make working with relational or labeled data both easy and intuitive.". pandas uses Cython to improve its performance. Moreover, pandas has been used extensively in production in financial applications;
- Cython is a way to write C extensions for the Python language. Since you write Cython code in the same way as you write Python code, it's easy to use it. Is it fast? Yes ! I compared a simple example from Cython's official documentation with a full Python code -- a piece of code which computes the first kth prime numbers. The Cython code is almost thirty times faster than the full-Python code (non-official). Furthermore, you can use NumPy in Cython code !
I believe that thanks to several useful tools and libraries, Python can be used in numerical computation, even in Finance (both research and production). It is easy-to-maintain without sacrificing performances.

Note that you can find some other references on Visixion webpages:
- slides and paper about automating numerical research with Python at PyPhy 2011 ;
- Rapid Financial Engineering and Automating Numerical Research at PyCon DE 2011.
1 Comment | tagged by cython, euroscipy, numpy, pandas, python, scikit, scipy | blogged in Logilab.org - en
Rss feeds aggregator based on Scikits.learn and CubicWeb
2011/10/17 by Vincent Michel
During Euroscipy, the Logilab Team presented an original approach for querying news using semantic information: "Rss feeds aggregator based on Scikits.learn and CubicWeb" by Vincent Michel This work is based on two major pieces of software:
- CubicWeb, the pythonic semantic web framework, is used to store and query Dbpedia information. CubicWeb is able to reconstruct links from rdf/nt files, and can easily execute complex queries in a database with more than 8 millions entities and 75 millions links when using a PostgreSQL backend.
- Scikit.learn is a cutting-edge python toolbox for machine learning. It provides algorithms that are simple and easy to use.
Based on these tools, we built a pure Python application to query the news:
- Named Entities are extracted from RSS articles of a few mainstream English newspapers (New York Times, Reuteurs, BBC News, etc.), for each group of words in an article, we check if a Dbpedia entry has the same label. If so, we create a semantic link between the article and the Dbpedia entry.
- An occurrence matrix of "RSS Articles" times "Named Entities" is constructed and may be used against several machine learning algorithms (MeanShift algorithm, Hierachical Clustering) in order to provide original and informative views of recent events.
Moreover, queries may be used jointly with semantic information from Dbpedia:
- All musical artists in the news:
```
DISTINCT Any E, R WHERE E appears_in_rss R, E has_type T, T label "musical artist"
```
- All living office holder persons in the news:
```
DISTINCT Any E WHERE E appears_in_rss R, E has_type T, T label "office holder", E has_subject C, C label "Living people"
```
- All news that talk about Barack Obama and any scientist:
```
DISTINCT Any R WHERE E1 label "Barack Obama", E1 appears_in_rss R, E2 appears_in_rss R, E2 has_type T, T label "scientist"
```
- All news that talk about a drug:
```
Any X, R WHERE X appears_in_rss R, X has_type T, T label "drug"
```
Such a tool may be used for informetrics and news analysis. Feel free to download the complete slides of the presentation.
0 Comment | tagged by dbpedia, euroscipy, machine learning, news, postgresql, rss, scikit, semantic | blogged in Logilab.org - en
Helping pylint to understand things it doesn't
2011/10/10 by Sylvain Thenault
The latest release of logilab-astng (0.23), the underlying source code representation library used by PyLint, provides a new API that may change pylint users' life in the near future...

It aims to allow registration of functions that will be called after a module has been parsed. While this sounds dumb, it gives a chance to fix/enhance the understanding PyLint has about your code.

I see this as a major step towards greatly enhanced code analysis, improving the situation where PyLint users know that when running it against code using their favorite framework (who said CubicWeb? :p ), they should expect a bunch of false positives because of black magic in the ORM or in decorators or whatever else. There are also places in the Python standard library where dynamic code can cause false positives in PyLint.
The problem

Let's take a simple example, and see how we can improve things using the new API. The following code:
```
import hashlib

def hexmd5(value):
    """"return md5 checksum hexadecimal digest of the given value"""
    return hashlib.md5(value).hexdigest()

def hexsha1(value):
    """"return sha1 checksum hexadecimal digest of the given value"""
    return hashlib.sha1(value).hexdigest()
```
gives the following output when analyzed through pylint:
```
[syt@somewhere ~]$ pylint -E example.py
No config file found, using default configuration
************* Module smarter_astng
E:  5,11:hexmd5: Module 'hashlib' has no 'md5' member
E:  9,11:hexsha1: Module 'hashlib' has no 'sha1' member
```
However:
```
[syt@somewhere ~]$ python
Python 2.7.1+ (r271:86832, Apr 11 2011, 18:13:53)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import smarter_astng
>>> smarter_astng.hexmd5('hop')
'5f67b2845b51a17a7751f0d7fd460e70'
>>> smarter_astng.hexsha1('hop')
'cffb6b20e0eef296772f6c1457cdde0049bdfb56'
```
The code runs fine... Why does pylint bother me then? If we take a look at the hashlib module, we see that there are no sha1 or md5 defined in there. They are defined dynamically according to Openssl library availability in order to use the fastest available implementation, using code like:
```
for __func_name in __always_supported:
    # try them all, some may not work due to the OpenSSL
    # version not supporting that algorithm.
    try:
        globals()[__func_name] = __get_hash(__func_name)
    except ValueError:
        import logging
        logging.exception('code for hash %s was not found.', __func_name)
```
Honestly I don't blame PyLint for not understanding this kind of magic. The situation on this particular case could be improved, but that's some tedious work, and there will always be "similar but different" case that won't be understood.
The solution

The good news is that thanks to the new astng callback, I can help it be smarter! See the code below:
```
from logilab.astng import MANAGER, scoped_nodes

def hashlib_transform(module):
    if module.name == 'hashlib':
        for hashfunc in ('sha1', 'md5'):
            module.locals[hashfunc] = [scoped_nodes.Class(hashfunc, None)]

def register(linter):
    """called when loaded by pylint --load-plugins, register our tranformation
    function here
    """
    MANAGER.register_transformer(hashlib_transform)
```
What's in there?
- A function that will be called with each astng module built during a pylint execution, i.e. not only the one that you analyses, but also those accessed for type inference.
- This transformation function is fairly simple: if the module is the 'hashlib' module, it will insert into its locals dictionary a fake class node for each desired name.
- It is registered using the register_transformer method of astng's MANAGER (the central access point to built syntax tree). This is done in the pylint plugin API register callback function (called when module is imported using 'pylint --load-plugins'.
Now let's try it! Suppose I stored the above code in a 'astng_hashlib.py' module in my PYTHONPATH, I can now run pylint with the plugin activated:
```
[syt@somewhere ~]$ pylint -E --load-plugins astng_hashlib example.py
No config file found, using default configuration
************* Module smarter_astng
E:  5,11:hexmd5: Instance of 'md5' has no 'hexdigest' member
E:  9,11:hexsha1: Instance of 'sha1' has no 'hexdigest' member
```
Huum. We have now a different error :( Pylint grasp there are some md5 and sha1 classes but it complains they don't have a hexdigest method. Indeed, we didn't give a clue about that.

We could continue on and on to give it a full representation of hashlib public API using the astng nodes API. But that would be painful, trust me. Or we could do something clever using some higher level astng API:
```
from logilab.astng import MANAGER
from logilab.astng.builder import ASTNGBuilder

def hashlib_transform(module):
    if module.name == 'hashlib':
    fake = ASTNGBuilder(MANAGER).string_build('''

class md5(object):
  def __init__(self, value): pass
  def hexdigest(self):
    return u''

class sha1(object):
  def __init__(self, value): pass
  def hexdigest(self):
    return u''

''')
    for hashfunc in ('sha1', 'md5'):
        module.locals[hashfunc] = fake.locals[hashfunc]

def register(linter):
    """called when loaded by pylint --load-plugins, register our tranformation
    function here
    """
    MANAGER.register_transformer(hashlib_transform)
```
The idea is to write a fake python implementation only documenting the prototype of the desired class, and to get an astng from it, using the string_build method of the astng builder. This method will return a Module node containing the astng for the given string. It's then easy to replace or insert additional information into the original module, as you can see in the above example.

Now if I run pylint using the updated plugin:
```
[syt@somewhere ~]$ pylint -E --load-plugins astng_hashlib example.py
No config file found, using default configuration
```
No error anymore, great!
What's next?

This fairly simple change could quickly provide great enhancements. We should probably improve the astng manipulation API now that it's exposed like that. But we can also easily imagine a code base of such pylint plugins maintained by each community around a python library or framework. One could then use a plugins stack matching stuff used by its software, and have a greatly enhanced experience of using pylint.

For a start, it would be great if pylint could be shipped with a plugin that explains all the magic found in the standard library, wouldn't it? Left as an exercice to the reader!
5 Comments | tagged by astng, pylint, quality | blogged in Logilab.org - en
Text mode makes it into hgview 1.4.0
2011/10/06 by Alain Leufroy
Here is at last the release of the version 1.4.0 of hgview.
Small description

Besides the classic bugfixes this release introduces a new text based user interface thanks to the urwid library.

Running hgview in a shell, in a terminal, over a ssh session is now possible! If you are trying not to use X (or use it less), have a geek mouse-killer window manager such as wmii/dwm/ion/awesome/... this is for you!

This TUI (Text User Interface!) adopts the principal features of the Qt4 based GUI. Although only the main view has been implemented for now.

In a nutshell, this interface includes the following features :
- display the revision graph (with working directory as a node, and basic support for the mq extension),
- display the files affected by a selected changeset (with basic support for the bfiles extension)
- display diffs (with syntax highlighting thanks to pygments),
- automatically refresh the displayed revision graph when the repository is being modified (requires pyinotify),
- easy key-based navigation in revisions' history of a repo (same as the GUI),
- a command system for special actions (see help)
Installation

There are packages for debian and ubuntu in the logilab's debian repository.

Note: you have to install the hgview-curses package to get the text based interface.

Or you can simply clone our Mercurial repository:

hg clone http://hg.logilab.org/hgview

(more on the hgview home page)
Running the text based interface

A new --interface option is now available to choose the interface:

hgview --interface curses

Or you can fix it in the [hgview] section of your ~/.hgrc:
```
[hgview]
interface = curses # or qt or raw
```
Then run:

hgview
What's next

We'll be working on including other features from the Qt4 interface and making it fully configurable.

We'll also work on bugfixes and new features, so stay tuned! And feel free to file bugs and feature requests.
1 Comment | tagged by console, hgview, mercurial, python, release, terminal | blogged in Logilab.org - en
Drawing UML diagrams with Python
2011/09/26 by Nicolas Chauvat

It started with a desire to draw diagrams of hierarchical systems with Python. Since this is similar to what we do in CubicWeb with schemas of the data model, I read the code and realized we had that graph submodule in the logilab.common library. This module uses dot from graphviz as a backend to draw the diagrams.

Reading about UML diagrams drawn with GraphViz, I learned about UMLGraph, that uses GNU Pic to draw sequence diagrams. Pic is a language based on groff and the pic2plot tool is part of plotutils (apt-get install plotutils). Here is a tutorial. I have found some Python code wrapping pic2plot available as plugin to wikipad. It is worth noticing that TeX seems to have a nice package for UML sequence diagrams called pgf-umlsd.

Since nowadays everything is moving into the web browser, I looked for a javascript library that does what graphviz does and I found canviz which looks nice.

If (only) I had time, I would extend pyreverse to draw sequence diagrams and not only class diagrams...

1 Comment | blogged in Logilab.org - en
EuroSciPy'11 - Annual European Conference for Scientists using Python.
2011/08/24 by Alain Leufroy

The EuroScipy2011 conference will be held in Paris at the Ecole Normale Supérieure from August 25th to 28th and is co-organized and sponsored by INRIA, Logilab and other companies.

The conference is dedicated to cross-disciplinary gathering focused on the use and development of the Python language in scientific research.

August 25th and 26th are dedicated to tutorial tracks -- basic and advanced tutorials. August 27th and 28th are dedicated to talks, posters and demos sessions.

Damien Garaud, Vincent Michel and Alain Leufroy (and others) from Logilab will be there. We will talk about a RSS feeds aggregator based on Scikits.learn and CubicWeb and we have a poster about LibAster (a python library for thermomechanical simulation based on Code_Aster).

0 Comment | tagged by euroscipy, libaster, science | blogged in Logilab.org - en
Pylint 0.24 / logilab-astng 0.22
2011/07/21 by Sylvain Thenault
Hi there!

I'm pleased to announce new releases of pylint and its underlying library logilab-astng. See http://www.logilab.org/project/pylint/0.24.0 and http://www.logilab.org/project/logilab-astng/0.22.0 for more info.

Those releases include mostly fixes and a few enhancements. Python 2.6 relative / absolute imports should now work fine and Python 3 support has been enhanced. There are still two remaining failures in astng test suite when using python 3, but we're unfortunatly missing resources to fix them yet.

Many thanks to everyone who contributed to this release by submitting patches or by participating to the latest bugs day.

0 Comment | tagged by astng, pylint, release | blogged in Logilab.org - en

Note:	you have to install the hgview-curses package to get the text based interface.

show 317 results