subscribe to this blog

Logilab.org - en

News from Logilab and our Free Software projects, as well as on topics dear to our hearts (Python, Debian, Linux, the semantic web, scientific computing...)

show 184 results
  • Running a local salt-master to orchestrate docker containers

    2015/05/20 by David Douard

    In a recent blog post, Denis explained how to build Docker containers using Salt.

    What's missing there is how to have a running salt-master dedicated to Docker containers.

    There is not need the salt-master run as root for this. A test config of mine looks like:

    david@perseus:~$ mkdir -p salt/etc/salt
    david@perseus:~$ cd salt
    david@perseus:~salt/$ cat << EOF >etc/salt/master
    interface: 192.168.127.1
    user: david
    
    root_dir: /home/david/salt/
    pidfile: var/run/salt-master.pid
    pki_dir: etc/salt/pki/master
    cachedir: var/cache/salt/master
    sock_dir: var/run/salt/master
    
    file_roots:
      base:
        - /home/david/salt/states
        - /home/david/salt/formulas/cubicweb
    
    pillar_roots:
      base:
        - /home/david/salt/pillar
    EOF
    

    Here, 192.168.127.1 is the ip of my docker0 bridge. Also note that path in file_roots and pillar_roots configs must be absolute (they are not relative to root_dir, see the salt-master configuration documentation).

    Now we can start a salt-master that will be accessible to Docker containers:

    david@perseus:~salt/$ /usr/bin/salt-master -c etc/salt
    

    Warning

    with salt 2015.5.0, salt-master really wants to execute dmidecode, so add /usr/sbin to the $PATH variable before running the salt-master as non-root user.

    From there, you can talk to your test salt master by adding -c ~/salt/etc/salt option to all salt commands. Fortunately, you can also set the SALT_CONFIG_DIR environment variable:

    david@perseus:~salt/$ export SALT_CONFIG_DIR=~/salt/etc/salt
    david@perseus:~salt/$ salt-key
    Accepted Keys:
    Denied Keys:
    Unaccepted Keys:
    Rejected Keys:
    

    Now, you need to have a Docker images with salt-minion already installed, as explained in Denis' blog post. (I prefer using supervisord as PID 1 in my dockers, but that's not important here.)

    david@perseus:~salt/ docker run -d --add-host salt:192.168.127.1  logilab/salted_debian:wheezy
    53bf7d8db53001557e9ae25f5141cd9f2caf7ad6bcb7c2e3442fcdbb1caf5144
    david@perseus:~salt/ docker run -d --name jessie1 --hostname jessie1 --add-host salt:192.168.127.1  logilab/salted_debian:jessie
    3da874e58028ff6dcaf3999b29e2563e1bc4d6b1b7f2f0b166f9a8faffc8aa47
    david@perseus:~salt/ salt-key
    Accepted Keys:
    Denied Keys:
    Unaccepted Keys:
    53bf7d8db530
    jessie1
    Rejected Keys:
    david@perseus:~/salt$ salt-key -y -a 53bf7d8db530
    The following keys are going to be accepted:
    Unaccepted Keys:
    53bf7d8db530
    Key for minion 53bf7d8db530 accepted.
    david@perseus:~/salt$ salt-key -y -a jessie1
    The following keys are going to be accepted:
    Unaccepted Keys:
    jessie1
    Key for minion jessie1 accepted.
    david@perseus:~/salt$ salt '*' test.ping
    jessie1:
        True
    53bf7d8db530:
        True
    

    You can now build Docker images as explained by Denis, or test your sls config files in containers.


  • Mini-Debconf Lyon 2015

    2015/04/29 by Julien Cristau
    //www.logilab.org/file/291628/raw/debian-france.png

    A couple of weeks ago I attended the mini-DebConf organized by Debian France in Lyon.

    It was a really nice week-end, and the first time a French mini-DebConf wasn't in Paris :)

    Among the highlights, Juliette Belin reported on her experience as a new contributor to Debian: she authored the awesome "Lines" theme which was selected as the default theme for Debian 8.

    //www.logilab.org/file/291626/raw/juliette.jpg

    As a non-developer and newcomer to the free software community, she had quite intesting insights and ideas about areas where development processes need to improve.

    And Raphael Geissert reported on the new httpredir.debian.org service (previously http.debian.net), an http redirector to automagically pick the closest Debian archive mirror. So long, manual sources.list updates on laptops whenever travelling!

    //www.logilab.org/file/291627/raw/raphael.jpg

    Finally the mini-DebConf was a nice opportunity to celebrate the release of Debian 8, two weeks in advance.

    Now it's time to go and upgrade all our infrastructure to jessie.


  • Building Docker containers using Salt

    2015/04/07 by Denis Laxalde

    In this blog post, I'll talk about a way to use Salt to automate the build and configuration of Docker containers. I will not consider the deployment of Docker containers with Salt as this subject is already covered elsewhere (here for instance). The emphasis here is really on building (or configuring) a container for future deployment.

    Motivation

    Salt is a remote execution framework that can be used for configuration management. It's already widely used at Logilab to manage our infrastructure as well as on a semi-daily basis during our application development activities.

    Docker is a tool that helps automating the deployment of applications within Linux containers. It essentially provides a convenient abstraction and a set of utilities for system level virtualization on Linux. Amongst other things, Docker provides container build helpers around the concept of dockerfile.

    So, the first question is why would you use Salt to build Docker containers when you already have this dockerfile building tool. My first motivation is to encompass the limitations of the available declarations one could insert in a Dockerfile. First limitation: you can only execute instructions in a sequential manner using a Dockerfile, there's is no possibility of declaring dependencies between instructions or even of making an instruction conditional (apart from using the underlying shell conditional machinery of course). Then, you have only limited possibilities of specializing a Dockerfile. Finally, it's no so easy to apply a configuration step-by-step, for instance during the development of said configuration.

    That's enough for an introduction to lay down the underlying motivation of this post. Let's move on to more practical things!

    A Dockerfile for the base image

    Before jumping into the usage of Salt for the configuration of a Docker image, the first thing you need to do is to build a Docker container into a proper Salt minion.

    Assuming we're building on top of some a base image of Debian flavour subsequently referred to as <debian> (I won't tell you where it comes from, since you ought to build your own base image -- or find some friend you trust to provide you with one!), the following Dockerfile can be used to initialize a working image which will serve as the starting point for further configuration with Salt:

    FROM <debian>
    RUN apt-get update
    RUN apt-get install -y salt-minion
    

    Then, run docker build . docker_salt/debian_salt_minion and you're done.

    Plugin the minion container with the Salt master

    The next thing to do with our fresh Debian+salt-minion image is to turn it into a container running salt-minion, waiting for the Salt master to instruct it.

    docker run --add-host=salt:10.1.1.1 --hostname docker_minion \
        --name minion_container \
        docker_salt/debian/salt_minion salt-minion
    

    Here:

    • --hostname is used to specify the network name of the container, for easier query by the Salt master,
    • 10.1.1.1 is usually the IP address of the host, which in our example will serve as the Salt master,
    • --name is just used for easier book-keeping.

    Finally,

    salt-key -a docker_minion
    

    will register the new minion's key into the master's keyring.

    If all went well, the following command should succeed:

    salt 'docker_minion' test.ping
    

    Configuring the container with a Salt formula

    salt 'docker_minion' state.sls some_formula
    salt 'docker_minion' state.highstate
    

    Final steps: save the configured image and build a runnable image

    (Optional step, cleanup salt-minion installation.)

    Make a snapshot image of your configured container.

    docker stop minion_container
    docker commit -m 'Install something with Salt' \
        minion_container me/something
    

    Try out your new image:

    docker run -p 8080:80 me/something <entry point>
    

    where <entry point> will be the main program driving the service provided by the container (typically defined through the Salt formula).

    Make a fully configured image for you service:

    FROM me/something
    [...anything else you need, such as EXPOSE, etc...]
    CMD <entry point>
    

  • Monitoring our websites before we deploy them using Salt

    2015/03/11 by Arthur Lutz

    As you might have noticed we're quite big fans of Salt. One of the things that Salt enables us to do, it to apply what we're used to doing with code to our infrastructure. Let's look at TDD (Test Driven Development).

    Write the test first, make it fail, implement the code, test goes green, you're done.

    Apply the same thing to infrastructure and you get TDI (Test Driven Infrastructure).

    So before you deploy a service, you make sure that your supervision (shinken, nagios, incinga, salt based monitoring, etc.) is doing the correct test, you deploy and then your supervision goes green.

    Let's take a look at website supervision. At Logilab we weren't too satisfied with how our shinken/http_check were working so we started using uptime (nodejs + mongodb). Uptime has a simple REST API to get and add checks, so we wrote a salt execution module and a states module for it.

    https://www.logilab.org/file/288174/raw/68747470733a2f2f7261772e6769746875622e636f6d2f667a616e696e6f74746f2f757074696d652f646f776e6c6f6164732f636865636b5f64657461696c732e706e67.png

    For the sites that use the apache-formula we simply loop on the domains declared in the pillars to add checks :

    {% for domain in salt['pillar.get']('apache:sites').keys() %}
    uptime {{ domain }} (http):
      uptime.monitored:
        - name : http://{{ domain }}
    {% endfor %}
    

    For other URLs (specific URL such as sitemaps) we can list them in pillars and do :

    {% for url in salt['pillar.get']('uptime:urls') %}
    uptime {{ url }}:
      uptime.monitored:
        - name : {{ url }}
    {% endfor %}
    

    That's it. Monitoring comes before deployment.

    We've also contributed a formula for deploying uptime.

    Follow us if you are interested in Test Driven Infrastructure for we intend to write regular reports as we make progress exploring this new domain.


  • A report on the Salt Sprint 2015 in Paris

    2015/03/05 by Arthur Lutz

    On Wednesday the 4th of march 2015, Logilab hosted a sprint on salt on the same day as the sprint at SaltConf15. 7 people joined in and hacked on salt for a few hours. We collaboratively chose some subjects on a pad which is still available.

    //www.logilab.org/file/248336/raw/Salt-Logo.png

    We started off by familiarising those who had never used them to using tests in salt. Some of us tried to run the tests via tox which didn't work any more, a fix was found and will be submitted to the project.

    We organised in teams.

    Boris & Julien looked at the authorisation code and wrote a few issues (minion enumeration, acl documentation). On saltpad (client side) they modified the targeting to adapt to the permissions that the salt-api sends back.

    We discussed the salt permission model (external_auth) : where should the filter happen ? the master ? should the minion receive information about authorisation and not execute what is being asked for ? Boris will summarise some of the discussion about authorisations in a new issue.

    //www.logilab.org/file/288010/raw/IMG_3034.JPG

    Sofian worked on some unification on execution modules (refresh_db which will be ignored for the modules that don't understand that). He will submit a pull request in the next few days.

    Georges & Paul added some tests to hg_pillar, the test creates a mercurial repository, adds a top.sls and a file and checks that they are visible. Here is the diff. They had some problems while debugging the tests.

    David & Arthur implemented the execution module for managing postgresql clusters (create, list, exists, remove) in debian. A pull request was submitted by the end of the day. A state module should follow shortly. On the way we removed some dead code in the postgres module.

    All in all, we had some interesting discussions about salt, it's architecture, shared tips about developing and using it and managed to get some code done. Thanks to all for participating and hopefully we'll sprint again soon...


  • Generate stats from your SaltStack infrastructure

    2014/12/15 by Arthur Lutz

    As presented at the November french meetup of saltstack users, we've published code to generate some statistics about a salstack infrastructure. We're using it, for the moment, to identify which parts of our infrastructure need attention. One of the tools we're using to monitor this distance is munin.

    You can grab the code at bitbucket salt-highstate-stats, fork it, post issues, discuss it on the mailing lists.

    If you're french speaking, you can also read the slides of the above presentation (mirrored on slideshare).

    Hope you find it useful.


  • Using Saltstack to limit impact of Poodle SSLv3 vulnerability

    2014/10/15 by Arthur Lutz

    Here at Logilab, we're big fans of SaltStack automation. As seen with Heartbleed, controlling your infrastructure and being able to fix your servers in a matter of a few commands as documented in this blog post. Same applies to Shellshock more recently with this blog post.

    Yesterday we got the news that a big vulnerability on SSL was going to be released. Code name : Poodle. This morning we got the details and started working on a fix through salt.

    So far, we've handled configuration changes and services restart for apache, nginx, postfix and user configuration for iceweasel (debian's firefox) and chromium (adapting to firefox and chrome should be a breeze). Some credit goes to mtpettyp for his answer on askubuntu.

    http://www.logilab.org/file/267853/raw/saltstack_poodlebleed.jpg
    {% if salt['pkg.version']('apache2') %}
    poodle apache server restart:
        service.running:
            - name: apache2
      {% for foundfile in salt['cmd.run']('rgrep -m 1 SSLProtocol /etc/apache*').split('\n') %}
        {% if 'No such file' not in foundfile and 'bak' not in foundfile and foundfile.strip() != ''%}
    poodle {{ foundfile.split(':')[0] }}:
        file.replace:
            - name : {{ foundfile.split(':')[0] }}
            - pattern: "SSLProtocol all -SSLv2[ ]*$"
            - repl: "SSLProtocol all -SSLv2 -SSLv3"
            - backup: False
            - show_changes: True
            - watch_in:
                service: apache2
        {% endif %}
      {% endfor %}
    {% endif %}
    
    {% if salt['pkg.version']('nginx') %}
    poodle nginx server restart:
        service.running:
            - name: nginx
      {% for foundfile in salt['cmd.run']('rgrep -m 1 ssl_protocols /etc/nginx/*').split('\n') %}
        {% if 'No such file' not in foundfile and 'bak' not in foundfile and foundfile.strip() != ''%}
    poodle {{ foundfile.split(':')[0] }}:
        file.replace:
            - name : {{ foundfile.split(':')[0] }}
            - pattern: "ssl_protocols .*$"
            - repl: "ssl_protocols TLSv1 TLSv1.1 TLSv1.2;"
            - show_changes: True
            - watch_in:
                service: nginx
        {% endif %}
      {% endfor %}
    {% endif %}
    
    {% if salt['pkg.version']('postfix') %}
    poodle postfix server restart:
        service.running:
            - name: postfix
    poodle /etc/postfix/main.cf:
    {% if 'main.cf' in salt['cmd.run']('grep smtpd_tls_mandatory_protocols /etc/postfix/main.cf') %}
        file.replace:
            - pattern: "smtpd_tls_mandatory_protocols=.*"
            - repl: "smtpd_tls_mandatory_protocols=!SSLv2,!SSLv3"
    {% else %}
        file.append:
            - text: |
                # poodle fix
                smtpd_tls_mandatory_protocols=!SSLv2,!SSLv3
    {% endif %}
            - name: /etc/postfix/main.cf
            - watch_in:
                service: postfix
    {% endif %}
    
    {% if salt['pkg.version']('chromium') %}
    /usr/share/applications/chromium.desktop:
        file.replace:
            - pattern: Exec=/usr/bin/chromium %U
            - repl: Exec=/usr/bin/chromium --ssl-version-min=tls1 %U
    {% endif %}
    
    {% if salt['pkg.version']('iceweasel') %}
    /etc/iceweasel/pref/poodle.js:
        file.managed:
            - text : pref("security.tls.version.min", "1")
    {% endif %}
    

    The code is also published as a gist on github. Feel free to comment and fork the gist. There is room for improvement, and don't forget that by disabling SSLv3 you might prevent some users with "legacy" browsers from accessing your services.


  • Report from DebConf14

    2014/09/05 by Julien Cristau

    Last week I attended DebConf14 in Portland, Oregon. As usual the conference was a blur, with lots of talks, lots of new people, and lots of old friends. The organizers tried to do something different this year, with a longer conference (9 days instead of a week) and some dedicated hack time, instead of a pre-DebConf "DebCamp" week. That worked quite well for me, as it meant the schedule was not quite so full with talks, and even though I didn't really get any hacking done, it felt a bit more relaxed and allowed some more hallway track discussions.

    http://www.logilab.org/file/264666/raw/Screenshot%20from%202014-09-05%2015%3A09%3A38.png

    On the talks side, the keynotes from Zack and Biella provided some interesting thoughts. Some nice progress was made on making package builds reproducible.

    I gave two talks: an introduction to salt (odp),

    http://www.logilab.org/file/264663/raw/slide2.jpg

    and a report on the Debian jessie release progress (pdf).

    http://www.logilab.org/file/264665/raw/slide3.jpg

    And as usual all talks were streamed live and recorded, and many are already available thanks to the awesome DebConf video team. Also for a change, and because I'm a sucker for punishment, I came back with more stuff to do.


  • Logilab at Debconf 2014 - Debian annual conference

    2014/08/21 by Arthur Lutz

    Logilab is proud to contribute to the annual debian conference which will take place in Portland (USA) from the 23rd to the 31st of august.

    Julien Cristau (debian page) will be giving two talks at the conference :

    http://www.logilab.org/file/263602/raw/debconf2014.png

    Logilab is also contributing to the conference as a sponsor for the event.

    Here is what we previously blogged about salt and the previous debconf . Stay tuned for a blog post about what we saw and heard at the conference.

    https://www.debian.org/logos/openlogo-100.png

  • Pylint 1.3 / Astroid 1.2 released

    2014/07/28 by Sylvain Thenault

    The EP14 Pylint sprint team (more on this here and there) is proud to announce they just released Pylint 1.3 together with its companion Astroid 1.2. As usual, this includes several new features as well and bug fixes. You'll find below some structured list of the changes.

    Packages are uploaded to pypi, debian/ubuntu packages should be soon provided by Logilab, until they get into the standard packaging system of your favorite distribution.

    Please notice Pylint 1.3 will be the last release branch support python 2.5 and 2.6. Starting from 1.4, we will only support python greater or equal to 2.7. This will be the occasion to do some great cleanup in the code base. Notice this is only about the Pylint's runtime, you should still be able to run Pylint on your Python 2.5 code, through using Python 2.7 at least.

    New checks

    • Add multiple checks for PEP 3101 advanced string formatting: 'bad-format-string', 'missing-format-argument-key', 'unused-format-string-argument', 'format-combined-specification', 'missing-format-attribute' and 'invalid-format-index'
    • New 'invalid-slice-index' and 'invalid-sequence-index' for invalid sequence and slice indices
    • New 'assigning-non-slot' warning, which detects assignments to attributes not defined in slots

    Improved checkers

    • Fixed 'fixme' false positive (#149)
    • Fixed 'unbalanced-iterable-unpacking' false positive when encountering starred nodes (#273)
    • Fixed 'bad-format-character' false positive when encountering the 'a' format on Python 3
    • Fixed 'unused-variable' false positive when the variable is assigned through an import (#196)
    • Fixed 'unused-variable' false positive when assigning to a nonlocal (#275)
    • Fixed 'pointless-string-statement' false positive for attribute docstrings (#193)
    • Emit 'undefined-variable' when using the Python 3 metaclass= argument. Also fix 'unused-import' false for that construction (#143)
    • Emit 'broad-except' and 'bare-except' even if the number of except handlers is different than 1. Fixes issue (#113)
    • Emit 'attribute-defined-outside-init' for all statements in the same module as the offended class, not just for the last assignment (#262, as well as a long standing output mangling problem in some edge cases)
    • Emit 'not-callable' when calling properties (#268)
    • Don't let ImportError propagate from the imports checker, leading to crash in some namespace package related cases (#203)
    • Don't emit 'no-name-in-module' for ignored modules (#223)
    • Don't emit 'unnecessary-lambda' if the body of the lambda call contains call chaining (#243)
    • Definition order is considered for classes, function arguments and annotations (#257)
    • Only emit 'attribute-defined-outside-init' for definition within the same module as the offended class, avoiding to mangle the output in some cases
    • Don't emit 'hidden-method' message when the attribute has been monkey-patched, you're on your own when you do that.

    Others changes

    • Checkers are now properly ordered to respect priority(#229)
    • Use the proper mode for pickle when opening and writing the stats file (#148)

    Astroid changes

    • Function nodes can detect decorator call chain and see if they are decorated with builtin descriptors (classmethod and staticmethod).
    • infer_call_result called on a subtype of the builtin type will now return a new Class rather than an Instance.
    • Class.metaclass() now handles module-level __metaclass__ declaration on python 2, and no longer looks at the __metaclass__ class attribute on python 3.
    • Add slots method to Class nodes, for retrieving the list of valid slots it defines.
    • Expose function annotation to astroid: Arguments node exposes 'varargannotation', 'kwargannotation' and 'annotations' attributes, while Function node has the 'returns' attribute.
    • Backported most of the logilab.common.modutils module there, as most things there are for pylint/astroid only and we want to be able to fix them without requiring a new logilab.common release
    • Fix names grabed using wildcard import in "absolute import mode" (i.e. with absolute_import activated from the __future__ or with python 3) (pylint issue #58)
    • Add support in brain for understanding enum classes.

show 184 results