Blog entries

  • Reinteract: un outil intéressant pour faire du numpy/scipy

    2008/05/27 by Arthur Lutz

    Il existe un outil, Reinteract, qui permet d'avoir une sorte de d'éditeur/shell Python, où l'on peut aisément modifier et réinterpreter une ligne de code.

    Sachant qu'il sait aussi afficher des plots, etc, il est possible de s'en servir avantageusement pour faire des sessions Matlab-like.

    Je pense donc que c'est un outil à présenter à nos chers apprenants qui sont intéressés par le couple python/numpy comme substitut à Matlab ©®.

    Ex:

    http://fishsoup.net/software/reinteract/reinteract-demo.png

    écrit par David Douard


  • EuroSciPy'09 (part 1/2): The Need For Speed

    2009/07/29 by Nicolas Chauvat
    http://www.logilab.org/image/9852?vid=download

    The EuroSciPy2009 conference was held in Leipzig at the end of July and was sponsored by Logilab and other companies. It started with three talks about speed.

    Starving CPUs

    In his keynote, Fransesc Alted talked about starving CPUs. Thirty years back, memory and CPU frequencies where about the same. Memory speed kept up for about ten years with the evolution of CPU speed before falling behind. Nowadays, memory is about a hundred times slower than the cache which is itself about twenty times slower than the CPU. The direct consequence is that CPUs are starving and spend many clock cycles waiting for data to process.

    In order to improve the performance of programs, it is now required to know about the multiple layers of computer memory, from disk storage to CPU. The common architecture will soon count six levels: mechanical disk, solid state disk, ram, cache level 3, cache level 2, cache level 1.

    Using optimized array operations, taking striding into account, processing data blocks of the right size and using compression to diminish the amount of data that is transfered from one layer to the next are four techniques that go a long way on the road to high performance. Compression algorithms like Blosc increase throughput for they strike the right balance between being fast and providing good compression ratios. Blosc compression will soon be available in PyTables.

    Fransesc also mentions the numexpr extension to numpy, and its combination with PyTables named tables.Expr, that nicely and easily accelerates the computation of some expressions involving numpy arrays. In his list of references, Fransesc cites Ulrich Drepper article What every programmer should know about memory.

    Using PyPy's JIT for science

    Maciej Fijalkowski started his talk with a general presentation of the PyPy framework. One uses PyPy to describe an interpreter in RPython, then generate the actual interpreter code and its JIT.

    Since PyPy is has become more of a framework to write interpreters than a reimplementation of Python in Python, I suggested to change its misleading name to something like gcgc the Generic Compiler for Generating Compilers. Maciej answered that there are discussions on the mailing list to split the project in two and make the implementation of the Python interpreter distinct from the GcGc framework.

    Maciej then focused his talk on his recent effort to rewrite in RPython the part of numpy that exposes the underlying C library to Python. He says the benefits of using PyPy's JIT to speedup that wrapping layer are already visible. He has details on the PyPy blog. Gaël Varoquaux added that David Cournapeau has started working on making the C/Python split in numpy cleaner, which would further ease the job of rewriting it in RPython.

    CrossTwine Linker

    Damien Diederen talked about his work on CrossTwine Linker and compared it with the many projects that are actively attacking the problem of speed that dynamic and interpreted languages have been dragging along for years. Parrot tries to be the über virtual machine. Psyco offers very nice acceleration, but currently only on 32bits system. PyPy might be what he calls the Right Approach, but still needs a lot of work. Jython and IronPython modify the language a bit but benefit from the qualities of the JVM or the CLR. Unladen Swallow is probably the one that's most similar to CrossTwine.

    CrossTwine considers CPython as a library and uses a set of C++ classes to generate efficient interpreters that make calls to CPython's internals. CrossTwine is a tool that helps improving performance by hand-replacing some code paths with very efficient code that does the same operations but bypasses the interpreter and its overhead. An interpreter built with CrossTwine can be viewed as a JIT'ed branch of the official Python interpreter that should be feature-compatible (and bug-compatible) with CPython. Damien calls he approach "punching holes in C substrate to get more speed" and says it could probably be combined with Psyco for even better results.

    CrossTwine works on 64bit systems, but it is not (yet?) free software. It focuses on some use cases to greatly improve speed and is not to be considered a general purpose interpreter able to make any Python code faster.

    More readings

    Cython is a language that makes writing C extensions for the Python language as easy as Python itself. It replaces the older Pyrex.

    The SciPy2008 conference had at least two papers talking about speeding Python: Converting Python Functions to Dynamically Compiled C and unPython: Converting Python Numerical Programs into C.

    David Beazley gave a very interesting talk in 2009 at a Chicago Python Users group meeting about the effects of the GIL on multicore machines.

    I will continue my report on the conference with the second part titled "Applications And Open Questions".


  • Python in Finance (and Derivative Analytics)

    2011/10/25 by Damien Garaud

    The Logilab team attended (and co-organized) EuroScipy 2011, at the end of August in Paris.

    We saw some interesting posters and a presentation dealing with Python in finance and derivative analytics [1].

    In order to debunk the idea that "all computation libraries dedicated to financial applications must be written in C/C++ or some other compiled programming language", I would like to introduce a more Pythonic way.

    You may know that financial applications such as risk management have in most cases high computational needs. For instance, it can be necessary to quickly perform a large number of Monte Carlo simulations to evaluate an American option in a few seconds.

    The Python community provides several reliable and efficient libraries and packages dedicated to numerical computations:

    http://numpy.scipy.org/_static/numpy_logo.png https://scikits.appspot.com/static/images/scipyshiny_small.png
    • the well-known SciPy and NumPy libraries. They provide a complete set of tools to work with matrix, linear algebra operations, singular values decompositions, multi-variate regression models, ...
    • scikits is a set of add-on toolkits for SciPy. For instance there are statistical models in statsmodels packages, a toolkit dedicated to timeseries manipulation and another one dedicated to numerical optimization;
    • pandas is a recent Python package which provides "fast, flexible, and expressive data structures designed to make working with relational or labeled data both easy and intuitive.". pandas uses Cython to improve its performance. Moreover, pandas has been used extensively in production in financial applications;
    http://docs.cython.org/_static/cython-logo-light.png
    • Cython is a way to write C extensions for the Python language. Since you write Cython code in the same way as you write Python code, it's easy to use it. Is it fast? Yes ! I compared a simple example from Cython's official documentation with a full Python code -- a piece of code which computes the first kth prime numbers. The Cython code is almost thirty times faster than the full-Python code (non-official). Furthermore, you can use NumPy in Cython code !

    I believe that thanks to several useful tools and libraries, Python can be used in numerical computation, even in Finance (both research and production). It is easy-to-maintain without sacrificing performances.

    Note that you can find some other references on Visixion webpages:


  • What's New in Pandas 0.13?

    2014/01/19 by Damien Garaud
    http://www.logilab.org/file/203841/raw/pandas_logo.png

    Do you know pandas, a Python library for data analysis? Version 0.13 came out on January the 16th and this post describes a few new features and improvements that I think are important.

    Each release has its list of bug fixes and API changes. You may read the full release note if you want all the details, but I will just focus on a few things.

    You may be interested in one of my previous blog post that showed a few useful Pandas features with datasets from the Quandl website and came with an IPython Notebook for reproducing the results.

    Let's talk about some new and improved Pandas features. I suppose that you have some knowledge of Pandas features and main objects such as Series and DataFrame. If not, I suggest you watch the tutorial video by Wes McKinney on the main page of the project or to read 10 Minutes to Pandas in the documentation.

    Refactoring

    I welcome the refactoring effort: the Series type, subclassed from ndarray, has now the same base class as DataFrame and Panel, i.e. NDFrame. This work unifies methods and behaviors for these classes. Be aware that you can hit two potential incompatibilities with versions less that 0.13. See internal refactoring for more details.

    Timeseries

    to_timedelta()

    Function pd.to_timedelta to convert a string, scalar or array of strings to a Numpy timedelta type (np.timedelta64 in nanoseconds). It requires a Numpy version >= 1.7. You can handle an array of timedeltas, divide it by an other timedelta to carry out a frequency conversion.

    from datetime import timedelta
    import numpy as np
    import pandas as pd
    
    # Create a Series of timedelta from two DatetimeIndex.
    dr1 = pd.date_range('2013/06/23', periods=5)
    dr2 = pd.date_range('2013/07/17', periods=5)
    td = pd.Series(dr2) - pd.Series(dr1)
    
    # Set some Na{N,T} values.
    td[2] -= np.timedelta64(timedelta(minutes=10, seconds=7))
    td[3] = np.nan
    td[4] += np.timedelta64(timedelta(hours=14, minutes=33))
    td
    
    0   24 days, 00:00:00
    1   24 days, 00:00:00
    2   23 days, 23:49:53
    3                 NaT
    4   24 days, 14:33:00
    dtype: timedelta64[ns]
    

    Note the NaT type (instead of the well-known NaN). For day conversion:

    td / np.timedelta64(1, 'D')
    
    0    24.000000
    1    24.000000
    2    23.992975
    3          NaN
    4    24.606250
    dtype: float64
    

    You can also use the DateOffSet as:

    td + pd.offsets.Minute(10) - pd.offsets.Second(7) + pd.offsets.Milli(102)
    

    Nanosecond Time

    Support for nanosecond time as an offset. See pd.offsets.Nano. You can use N of this offset in the pd.date_range function as the value of the argument freq.

    Daylight Savings

    The tz_localize method can now infer a fall daylight savings transition based on the structure of the unlocalized data. This method, as the tz_convert method is available for any DatetimeIndex, Series and DataFrame with a DatetimeIndex. You can use it to localize your datasets thanks to the pytz module or convert your timeseries to a different time zone. See the related documentation about time zone handling. To use the daylight savings inference in the method tz_localize, set the infer_dst argument to True.

    DataFrame Features

    New Method isin()

    New DataFrame method isin which is used for boolean indexing. The argument to this method can be an other DataFrame, a Series, or a dictionary of a list of values. Comparing two DataFrame with isin is equivalent to do df1 == df2. But you can also check if values from a list occur in any column or check if some values for a few specific columns occur in the DataFrame (i.e. using a dict instead of a list as argument):

    df = pd.DataFrame({'A': [3, 4, 2, 5],
                       'Q': ['f', 'e', 'd', 'c'],
                       'X': [1.2, 3.4, -5.4, 3.0]})
    
       A  Q    X
    0  3  f  1.2
    1  4  e  3.4
    2  2  d -5.4
    3  5  c  3.0
    

    and then:

    df.isin(['f', 1.2, 3.0, 5, 2, 'd'])
    
           A      Q      X
    0   True   True   True
    1  False  False  False
    2   True   True  False
    3   True  False   True
    

    Of course, you can use the previous result as a mask for the current DataFrame.

    mask = _
    df[mask.any(1)]
    
          A  Q    X
       0  3  f  1.2
       2  2  d -5.4
       3  5  c  3.0
    
    When you pass a dictionary to the ``isin`` method, you can specify the column
    labels for each values.
    
    mask = df.isin({'A': [2, 3, 5], 'Q': ['d', 'c', 'e'], 'X': [1.2, -5.4]})
    df[mask]
    
        A    Q    X
    0   3  NaN  1.2
    1 NaN    e  NaN
    2   2    d -5.4
    3   5    c  NaN
    

    See the related documentation for more details or different examples.

    New Method str.extract

    The new vectorized extract method from the StringMethods object, available with the suffix str on Series or DataFrame. Thus, it is possible to extract some data thanks to regular expressions as followed:

    s = pd.Series(['doe@umail.com', 'nobody@post.org', 'wrong.mail', 'pandas@pydata.org', ''])
    # Extract usernames.
    s.str.extract(r'(\w+)@\w+\.\w+')
    

    returns:

    0       doe
    1    nobody
    2       NaN
    3    pandas
    4       NaN
    dtype: object
    

    Note that the result is a Series with the re match objects. You can also add more groups as:

    # Extract usernames and domain.
    s.str.extract(r'(\w+)@(\w+\.\w+)')
    
            0           1
    0     doe   umail.com
    1  nobody    post.org
    2     NaN         NaN
    3  pandas  pydata.org
    4     NaN         NaN
    

    Elements that do no math return NaN. You can use named groups. More useful if you want a more explicit column names (without NaN values in the following example):

    # Extract usernames and domain with named groups.
    s.str.extract(r'(?P<user>\w+)@(?P<at>\w+\.\w+)').dropna()
    
         user          at
    0     doe   umail.com
    1  nobody    post.org
    3  pandas  pydata.org
    

    Thanks to this part of the documentation, I also found out other useful strings methods such as split, strip, replace, etc. when you handle a Series of str for instance. Note that the most of them have already been available in 0.8.1. Take a look at the string handling API doc (recently added) and some basics about vectorized strings methods.

    Interpolation Methods

    DataFrame has a new interpolate method, similar to Series. It was possible to interpolate missing data in a DataFrame before, but it did not take into account the dates if you had index timeseries. Now, it is possible to pass a specific interpolation method to the method function argument. You can use scipy interpolation functions such as slinear, quadratic, polynomial, and others. The time method is used to take your index timeseries into account.

    from datetime import date
    # Arbitrary timeseries
    ts = pd.DatetimeIndex([date(2006,5,2), date(2006,12,23), date(2007,4,13),
                           date(2007,6,14), date(2008,8,31)])
    df = pd.DataFrame(np.random.randn(5, 2), index=ts, columns=['X', 'Z'])
    # Fill the DataFrame with missing values.
    df['X'].iloc[[1, -1]] = np.nan
    df['Z'].iloc[3] = np.nan
    df
    
                       X         Z
    2006-05-02  0.104836 -0.078031
    2006-12-23       NaN -0.589680
    2007-04-13 -1.751863  0.543744
    2007-06-14  1.210980       NaN
    2008-08-31       NaN  0.566205
    

    Without any optional argument, you have:

    df.interpolate()
    
                       X         Z
    2006-05-02  0.104836 -0.078031
    2006-12-23 -0.823514 -0.589680
    2007-04-13 -1.751863  0.543744
    2007-06-14  1.210980  0.554975
    2008-08-31  1.210980  0.566205
    

    With the time method, you obtain:

    df.interpolate(method='time')
    
                       X         Z
    2006-05-02  0.104836 -0.078031
    2006-12-23 -1.156217 -0.589680
    2007-04-13 -1.751863  0.543744
    2007-06-14  1.210980  0.546496
    2008-08-31  1.210980  0.566205
    

    I suggest you to read more examples in the missing data doc part and the scipy documentation about the module interpolate.

    Misc

    Convert a Series to a single-column DataFrame with its method to_frame.

    Misc & Experimental Features

    Retrieve R Datasets

    Not a killing feature but very pleasant: the possibility to load into a DataFrame all R datasets listed at http://stat.ethz.ch/R-manual/R-devel/library/datasets/html/00Index.html

    import pandas.rpy.common as com
    titanic = com.load_data('Titanic')
    titanic.head()
    
      Survived    Age     Sex Class value
    0       No  Child    Male   1st   0.0
    1       No  Child    Male   2nd   0.0
    2       No  Child    Male   3rd  35.0
    3       No  Child    Male  Crew   0.0
    4       No  Child  Female   1st   0.0
    

    for the datasets about survival of passengers on the Titanic. You can find several and different datasets about New York air quality measurements, body temperature series of two beavers, plant growth results or the violent crime rates by US state for instance. Very useful if you would like to show pandas to a friend, a colleague or your Grandma and you do not have a dataset with you.

    And then three great experimental features.

    Eval and Query Experimental Features

    The eval and query methods which use numexpr which can fastly evaluate array expressions as x - 0.5 * y. For numexpr, x and y are Numpy arrays. You can use this powerfull feature in pandas to evaluate different DataFrame columns. By the way, we have already talked about numexpr a few years ago in EuroScipy 09: Need for Speed.

    df = pd.DataFrame(np.random.randn(10, 3), columns=['x', 'y', 'z'])
    df.head()
    
              x         y         z
    0 -0.617131  0.460250 -0.202790
    1 -1.943937  0.682401 -0.335515
    2  1.139353  0.461892  1.055904
    3 -1.441968  0.477755  0.076249
    4 -0.375609 -1.338211 -0.852466
    
    df.eval('x + 0.5 * y - z').head()
    
    0   -0.184217
    1   -1.267222
    2    0.314395
    3   -1.279340
    4   -0.192248
    dtype: float64
    

    About the query method, you can select elements using a very simple query syntax.

    df.query('x >= y > z')
    
              x         y         z
    9  2.560888 -0.827737 -1.326839
    

    msgpack Serialization

    New reading and writing functions to serialize your data with the great and well-known msgpack library. Note this experimental feature does not have a stable storage format. You can imagine to use zmq to transfer msgpack serialized pandas objects over TCP, IPC or SSH for instance.

    Google BigQuery

    A recent module pandas.io.gbq which provides a way to load into and extract datasets from the Google BigQuery Web service. I've not installed the requirements for this feature now. The example of the release note shows how you can select the average monthly temperature in the year 2000 across the USA. You can also read the related pandas documentation. Nevertheless, you will need a BigQuery account as the other Google's products.

    Take Your Keyboard

    Give it a try, play with some data, mangle and plot them, compute some stats, retrieve some patterns or whatever. I'm convinced that pandas will be more and more used and not only for data scientists or quantitative analysts. Open an IPython Notebook, pick up some data and let yourself be tempted by pandas.

    I think I will use more the vectorized strings methods that I found out about when writing this post. I'm glad to learn more about timeseries because I know that I'll use these features. I'm looking forward to the two experimental features such as eval/query and msgpack serialization.

    You can follow me on Twitter (@jazzydag). See also Logilab (@logilab_org).


  • Lecture de données tabulées avec Numpy -- Retour d'expérience sur dtype

    2013/12/17 by Damien Garaud

    Ce billet est un petit retour d'expérience sur l'utilisation de Numpy pour lire et extraire des données tabulées depuis des fichiers texte.

    Chaque section, hormis les objectifs ou la conclusion, correspond soit à une difficulté rencontrée, une remarque technique, des explications et références vers la documentation officielle sur un point précis qui m'a fait patauger quelques temps. Il y a de forte chance, pour certains d'entre vous, que les points décrits ici vous paraissent évidents, que vous vous disiez "mais qui ne sait pas ça ?!". J'étais moi-même le premier étonné, depuis que je connais Numpy, de ne pas savoir ce genre de choses. Je l'étais moins quand autour de moi, mes camarades ne semblaient pas non plus connaître les petites histoires numpysiennes que je vais vous conter.

    http://www.logilab.org/file/203839/raw/numpylogo.png

    Objectifs

    Le Pourquoi et le Où on va au fait ?

    J'avais sous la main des fichiers aux données tabulées, type CSV, où les types de données par colonne étaient clairement identifiés. Je ne souhaitais pas passer du temps avec le module csv de la bibliothèque standard à convertir chaque élément en type de base, str, flottants ou dates. Numpy étant déjà une dépendance du projet, et connaissant la fonction np.genfromtxt, j'ai évidemment souhaité l'utiliser.

    Il était nécessaire de ne lire que certaines colonnes. Je souhaitais associer un nom à chaque colonne. L'objectif était ensuite d'itérer sur ces données ligne par ligne et les traiter dans des générateurs Python. Je n'utilise pas ici Numpy pour faire des opérations mathématiques sur ces tableaux à deux dimensions avec des types hétérogènes. Et je ne pense d'ailleurs pas qu'il soit pertinent d'utiliser ce type de tableau pour faire ces opérations.

    dtypes différents, str et extraction de chaînes vides

    On utilise ici l'argument dtype des fonctions telles que np.genfromtxt pour lire des fichiers tabulés dont les colonnes sont de types différents.

    Attention au dtype à passer à np.genfromtxt ou np.recfromtxt quand on parse des données tabulée (file ou stream). Pour parser une colonne de chaînes de caratères, lui passer [('colname', str)] renvoie des chaînes vides si les autres dtypes sont de types différents.

    Il faut préciser la taille :

    dtype=[('colname', str, 10)]
    # or
    dtype=[('colname', 'S10')]
    

    Ou alors prendre un "vrai" objet str Python :

    dtype=[('colname', object)]
    

    aussi équivalent à:

    dtype=[('colname', 'object')]
    

    Et oui, je suis littéralement tombé sur l'évidence, les "types Numpy", c'est du type C. Et les chaînes, c'est du char * et il y a donc besoin de la taille. La solitude s'est fait moindre quand j'ai su que je n'étais pas le seul à être tombé sur des données tronquées voire vides.

    dtype et tableau à zéro dimension

    Attention au tableau Numpy 0D quand le contenu tabulé à parser n'a qu'une seule ligne (cas d'un np.[rec]array avec plusieurs dtypes). Impossible d'itérer sur les éléments puisque dimension nulle.

    Supposons que vous ayez un fichier tabulé d'une seule ligne :

    Name,Play,Age
    Coltrane,Saxo,28
    

    J'utilise np.genfromtxt en précisant le type des colonnes que je souhaite récupérer (je ne prends pas en compte ici la première ligne).

    data = np.genfromtxt(filename, delimiter=',',
                         dtype=[('name', 'S12'), ('play', object), ('age', int)],
                         skip_header=1)
    

    Voici la représentation de mon array :

    array(('Coltrane', 'Saxo', 28),
        dtype=[('name', 'S12'), ('play', 'O'), ('age', '<i8')])
    

    Si dans votre code, vous avez eu la bonne idée de parcourir vos données avec :

    for name, instrument, age in data:
        # ...
    

    vous pourrez obenir un malheureux TypeError: 'numpy.int64' object is not iterable par exemple. Vous n'avez pas eu de chance, votre tableau Numpy est à zéro dimension et une shape nulle (i.e. un tuple vide). Effectivement, itérer sur un objet de dimension nulle n'est pas chose aisée. Ce que je veux, c'est un tableau à une dimension avec un seul élément (ici un tuple avec mes trois champs) sur lequel il est possible d'itérer.

    Pour cela, faire:

    >>> print data
    array(('Coltrane', 'Saxo', 28),
          dtype=[('name', 'S12'), ('play', 'O'), ('age', '<i8')])array(('babar', 42.), dytpe=[('f0', 'S5'), ('f1', '<f8')])
    >>> print data.shape, data.ndim
    (), 0
    >>> data = data[np.newaxis]
    >>> print data.shape, data.ndim
    (1,), 1
    

    dtype et str : chararray ou ndarray de strings ?

    Pour les chararray, lire help(np.chararray) ou http://docs.scipy.org/doc/numpy/reference/generated/numpy.chararray.html. En particulier:

    The chararray class exists for backwards compatibility with Numarray, it is not recommended for new development. Starting from numpy 1.4, if one needs arrays of strings, it is recommended to use arrays of dtype object_, string_ or unicode_, and use the free functions in the numpy.char module for fast vectorized string operations.

    On fera donc la distinction entre:

    # ndarray of str
    na = np.array(['babar', 'celeste'], dtype=np.str_)
    # chararray
    ca = np.chararray(2)
    ca[0], ca[1] = 'babar', 'celeste'
    

    Le type de tableau est ici différent : np.ndarray pour le premier et np.chararray pour le second. Malheureusement pour np.recfromtxt et en particulier pour np.recarray, si on transpose le label de la colonne en tant qu'attribut, np.recarray il est transformé en chararray avec le bon type Numpy --- |S7 dans notre cas --- au lieu de conserver un np.ndarray de type |S7.

    Exemple :

    from StringIO import StringIO
    rawtxt = 'babar,36\nceleste,12'
    a = np.recfromtxt(StringIO(rawtxt), delimiter=',', dtype=[('name', 'S7'), ('age', int)])
    print(type(a.name))
    

    Le print rend bien un objet de type chararray. Alors que :

    a = np.genfromtxt(StringIO(rawtxt), delimiter=',', dtype=[('name', 'S7'), ('age', int)])
    print(type(a['name']))
    

    affiche ndarray. J'aimerais que tout puisse être du même type, peu importe la fonction utilisée. Au vue de la documentation et de l'aspect déprécié du type charray, on souhaiterait avoir que du ndarray de type np.str. J'ai par ailleurs ouvert le ticket Github 3993 qui n'a malheureusement que peu de succès :-(

    Tableau de chaînes : quel dtype ?

    Si certains se demandent quoi mettre pour représenter le type "une chaîne de caractères" dans un tableau numpy, ils ont le choix :

    np.array(['coltrane', 'hancock'], dtype=np.str)
    np.array(['coltrane', 'hancock'], dtype=np.str_)
    np.array(['coltrane', 'hancock'], dtype=np.string_)
    np.array(['coltrane', 'hancock'], dtype='S')
    np.array(['coltrane', 'hancock'], dtype='S10')
    np.array(['coltrane', 'hancock'], dtype='|S10')
    

    Les questions peuvent être multiples : est-ce la même chose ? pourquoi tant de choses différentes ? Pourquoi tant de haine quand on lit la doc Numpy et que l'info ne saute pas aux yeux ? À savoir que le tableau construit sera identique dans chacun des cas. Il existe peut-être même d'autres moyens de construire un tableau de type identique en lui passant encore un n-ième argument dtype différent.

    • np.str représente le type str de Python. Il est converti automatiquement en type chaines de caractère Numpy dont la longueur correspond à la longueur maximale rencontrée.
    • np.str_ et np.string_ sont équivalents et représentent le type "chaîne de caractères" pour Numpy (longueur = longueur max.).
    • Les trois autres utilisent la représentation sous forme de chaîne de caractères du type np.string_.
      • S ne précise pas la taille de la chaîne (Numpy prend donc la chaîne la plus longue)
      • S10 précise la taille de la chaîne (données tronquées si la taille est plus petite que la chaîne la plus longue)
      • |S10 est strictement identique au précédent. Il faut savoir qu'il existe cette notation <f8 ou >f8 pour représenter un flottant. Les chevrons signifient little endian ou big endian respectivement. Le | sert à dire "pas pertinent". Source: la section typestr sur la page http://docs.scipy.org/doc/numpy/reference/arrays.interface.html

    À noter que j'ai particulièrement apprécié l'utilisation d'un symbole pour spécifier une information non pertinente --- depuis le temps que je me demandais ce que voulait bien pouvoir dire ce pipe devant le 'S'.

    Conclusion (et pourquoi pas pandas ?)

    http://www.logilab.org/file/203841/raw/pandas_logo.png

    Pandas, bibliothèque Python d'analyse de données basée sur Numpy, propose, via sa fonction read_csv, le même genre de fonctionnalités. Il a l'avantage de convertir les types par colonne sans lui donner d'information de type, qu'on lise toutes les colonnes ou seulement quelques unes. Pour les colonnes de type "chaîne de caractères", il prend un dtype=object et n'essaie pas de deviner la longueur maximale pour le type np.str_. Vous ne rencontrerez donc pas "le coup des chaînes vides/tronquées" comme avec dtype='S'.

    Je ne m'étalerai pas sur tout le bien que je pense de cette bibliothèque. Je vous invite par ailleurs à lire/ parcourir un billet de novembre qui expose un certain nombre de fonctionnalités croustillantes et accompagné d'un IPython Notebook.

    Et pourquoi pas Pandas ? Il ne me semble pas pertinent de dépendre d'une nouvelle bibliothèque, aussi bien soit-elle, pour une fonction, aussi utile soit-elle. Pandas est un projet intéressant, mais jeune, qui ne se distribue pas aussi bien que Numpy pour l'instant. De plus, le projet sur lequel je travaillais utilisait déjà Numpy. Je n'avais besoin de rien d'autre pour réaliser mon travail, et dépendre de Pandas ne me semblait pas très pertinent. Je me suis donc contenté des fonctions np.{gen,rec}fromtxt qui font très bien le boulot, avec un dtype comme il faut, tout en retenant les boulettes que j'ai faites.