Blog entries

  • Instrumentation of google appengine's datastore.

    2008/06/23 by Sylvain Thenault

    Here is a piece of code I've written which I thought may be useful to some other people...

    You'll find here a simple python module to use with the Google AppEngine SDK to monkey patch the datastore API in order to get an idea of the calls performed by your application.

    To instrument of the datastore, put at the top level of your handler file

    import instrdatastore
    

    Note that it's important to have this before any other import in your application or in the google package to avoid that some modules will use the unpatched version of datastore functions (and hence calls to those functions wouldn't be considered).

    Then add at the end of your handler function

    instrdatastore.print_info()
    

    The handler file should look like this:

    """my handler file with datastore instrumenting activated"""
    import instrdatastore
    
    # ... other initialization code
    
    # main function so this handler module is cached
    def main():
      from wsgiref.handlers import CGIHandler
      from ginco.wsgi.handler import ErudiWSGIApplication
      application = ErudiWSGIApplication(config, vreg=vreg)
      CGIHandler().run(application)
      instrdatastore.print_info()
    
    if __name__ == "__main__":
      main()
    

    Now you should see in your logs the number of Get/Put/Delete/Query which has been done during request processing

    2008-06-23 06:59:12 - (root) WARNING: datastore access information
    2008-06-23 06:59:12 - (root) WARNING: nb Get: 2
    2008-06-23 06:59:12 - (root) WARNING: arguments (args, kwargs):
    ((datastore_types.Key.from_path('EGroup', u'key_users', _app=u'winecellar'),), {})
    ((datastore_types.Key.from_path('EUser', u'key_test@example.com', _app=u'winecellar'),), {})
    2008-06-23 06:59:12 - (root) WARNING: nb Query: 1
    2008-06-23 06:59:12 - (root) WARNING: arguments (args, kwargs):
    (({'for_user =': None}, 'EProperty'), {})
    2008-06-23 06:59:58 - (root) WARNING: nb Put: 1
    2008-06-23 06:59:58 - (root) WARNING: arguments (args, kwargs):
    (({u'login': None, u'last_usage_time': 1214204398.2022741, u'data': ""},), {})
    

    I'll probably extend this as the time goes. Also notice you may encounter some problems with the automatic reloading feature of the dev app server when instrumentation is activated, in which case you should simply restart the web server.


  • Javascript date support

    2008/11/27 by Adrien Di Mascio

    Coming from the python and mx.DateTime world, the javascript Date object is not really appealing. For me, the most disturbing things are :

    • The year parameter in the Date constructor is always considered as a XXe century year if year < 100. (this goes along with the getYear / getFullYear distinction).
    • The inconsistency between months and days indexes : months indexes starts at 0 whereas days indexes starts at 1.
    • The lack of decent strptime / strftime functions (even basic ones not taking locales into account).

    Recently, I've worked with the great Timeline project which makes an heavy use of dates and I had the need for a very basic strptime implementation. This can by no mean be considered as a comprehensive implementation, but it might help so here it is:

    var _DATE_FORMAT_REGXES = {
        'Y': new RegExp('^-?[0-9]+'),
        'd': new RegExp('^[0-9]{1,2}'),
        'm': new RegExp('^[0-9]{1,2}'),
        'H': new RegExp('^[0-9]{1,2}'),
        'M': new RegExp('^[0-9]{1,2}')
    }
    
    /*
     * _parseData does the actual parsing job needed by `strptime`
     */
    function _parseDate(datestring, format) {
        var parsed = {};
        for (var i1=0,i2=0;i1<format.length;i1++,i2++) {
        var c1 = format[i1];
        var c2 = datestring[i2];
        if (c1 == '%') {
            c1 = format[++i1];
            var data = _DATE_FORMAT_REGXES[c1].exec(datestring.substring(i2));
            if (!data.length) {
                return null;
            }
            data = data[0];
            i2 += data.length-1;
            var value = parseInt(data, 10);
            if (isNaN(value)) {
                return null;
            }
            parsed[c1] = value;
            continue;
        }
        if (c1 != c2) {
            return null;
        }
        }
        return parsed;
    }
    
    /*
     * basic implementation of strptime. The only recognized formats
     * defined in _DATE_FORMAT_REGEXES (i.e. %Y, %d, %m, %H, %M)
     */
    function strptime(datestring, format) {
        var parsed = _parseDate(datestring, format);
        if (!parsed) {
        return null;
        }
        // create initial date (!!! year=0 means 1900 !!!)
        var date = new Date(0, 0, 1, 0, 0);
        date.setFullYear(0); // reset to year 0
        if (parsed.Y) {
        date.setFullYear(parsed.Y);
        }
        if (parsed.m) {
        if (parsed.m < 1 || parsed.m > 12) {
            return null;
        }
        // !!! month indexes start at 0 in javascript !!!
        date.setMonth(parsed.m - 1);
        }
        if (parsed.d) {
        if (parsed.m < 1 || parsed.m > 31) {
            return null;
        }
        date.setDate(parsed.d);
        }
        if (parsed.H) {
        if (parsed.H < 0 || parsed.H > 23) {
            return null;
        }
        date.setHours(parsed.H);
        }
        if (parsed.M) {
        if (parsed.M < 0 || parsed.M > 59) {
            return null;
        }
        date.setMinutes(parsed.M);
        }
        return date;
    }
    
    // and now monkey patch the Timeline's parser ...
    /* provide our own custom date parser since the default
     * one only understands iso8601 and gregorian dates
     */
    Timeline.NativeDateUnit.getParser = function(format) {
        if (typeof format == "string") {
        if (format.indexOf('%') != -1) {
            return function(datestring) {
                if (datestring) {
                    return strptime(datestring, format);
                }
                return null;
            };
        }
            format = format.toLowerCase();
        }
        if (format == "iso8601" || format == "iso 8601") {
        return Timeline.DateTime.parseIso8601DateTime;
        }
        return Timeline.DateTime.parseGregorianDateTime;
    };