Blog entries

  • Experiments on building a Jenkins CI service with Salt

    2015/06/17 by Denis Laxalde

    In this blog post, I'll talk about my recent experiments on building a continuous integration service with Jenkins that is, as much as possible, managed through Salt. We've been relying on a Jenkins platform for quite some time at Logilab (Tolosa team). The service was mostly managed by me with sporadic help from other team-mates but I've never been entirely satisfied about the way it was managed because it involved a lot of boilerplate configuration through Jenkins user interface and this does not scale very well nor does it make long term maintenance easy.

    So recently, I've taken a stance and decided to go through a Salt-based configuration and management of our Jenkins CI platform. There are actually two aspects here. The first concerns the setup of Jenkins itself (this includes installation, security configuration, plugins management amongst other things). The second concerns the management of client projects (or jobs in Jenkins jargon). For this second aspect, one of the design goals was to enable easy configuration of jobs by users not necessarily familiar with Jenkins setup and to make collaborative maintenance easy. To tackle these two aspects I've essentially been using (or developing) two distinct Salt formulas which I'll detail hereafter.

    Jenkins jobs salt

    Core setup: the jenkins formula

    The core setup of Jenkins is based on an existing Salt formula, the jenkins-formula which I extended a bit to support map.jinja and which was further improved to support installation of plugins by Yann and Laura (see 3b524d4).

    With that, deploying a Jenkins server is as simple as adding the following to your states and pillars top.sls files:

    base:
      "jenkins":
        - jenkins
        - jenkins.plugins
    

    Base pillar configuration is used to declare anything that differs from the default Jenkins settings in a jenkins section, e.g.:

    jenkins:
      lookup:
        - home: /opt/jenkins
    

    Plugins configuration is declared in plugins subsection as follows:

    jenkins:
      lookup:
        plugins:
          scm-api:
            url: 'http://updates.jenkins-ci.org/download/plugins/scm-api/0.2/scm-api.hpi'
            hash: 'md5=9574c07bf6bfd02a57b451145c870f0e'
          mercurial:
            url: 'http://updates.jenkins-ci.org/download/plugins/mercurial/1.54/mercurial.hpi'
            hash: 'md5=1b46e2732be31b078001bcc548149fe5'
    

    (Note that plugins dependency is not handled by Jenkins when installing from the command line, neither by this formula. So in the preceding example, just having an entry for the Mercurial plugin would have not been enough because this plugin depends on scm-api.)

    Other aspects (such as security setup) are not handled yet (neither by the original formula, nor by our extension), but I tend to believe that this is acceptable to manage this "by hand" for now.

    Jobs management : the jenkins_jobs formula

    For this task, I leveraged the excellent jenkins-job-builder tool which makes it possible to configure jobs using a declarative YAML syntax. The tool takes care of installing the job and also handles any housekeeping tasks such as checking configuration validity or deleting old configurations. With this tool, my goal was to let end-users of the Jenkins service add their own project by providing a minima a YAML job description file. So for instance, a simple Job description for a CubicWeb job could be:

    - scm:
        name: cubicweb
        scm:
          - hg:
             url: http://hg.logilab.org/review/cubicweb
             clean: true
    
    - job:
        name: cubicweb
        display-name: CubicWeb
        scm:
          - cubicweb
        builders:
          - shell: "find . -name 'tmpdb*' -delete"
          - shell: "tox --hashseed noset"
        publishers:
          - email:
              recipients: cubicweb@lists.cubicweb.org
    

    It consists of two parts:

    • the scm section declares, well, SCM information, here the location of the review Mercurial repository, and,

    • a job section which consists of some metadata (project name), a reference of the SCM section declared above, some builders (here simple shell builders) and a publisher part to send results by email.

    Pretty simple. (Note that most test running configuration is here declared within the source repository, via tox (another story), so that the CI bot holds minimum knowledge and fetches information from the sources repository directly.)

    To automate the deployment of this kind of configurations, I made a jenkins_jobs-formula which takes care of:

    1. installing jenkins-job-builder,
    2. deploying YAML configurations,
    3. running jenkins-jobs update to push jobs into the Jenkins instance.

    In addition to installing the YAML file and triggering a jenkins-jobs update run upon changes of job files, the formula allows for job to list distribution packages that it would require for building.

    Wrapping things up, a pillar declaration of a Jenkins job looks like:

    jenkins_jobs:
      lookup:
        jobs:
          cubicweb:
            file: <path to local cubicweb.yaml>
            pkgs:
              - mercurial
              - python-dev
              - libgecode-dev
    

    where the file section indicates the source of the YAML file to install and pkgs lists build dependencies that are not managed by the job itself (typically non Python package in our case).

    So, as an end user, all is needed to provide is the YAML file and a pillar snippet similar to the above.

    Outlook

    This initial setup appears to be enough to greatly reduce the burden of managing a Jenkins server and to allow individual users to contribute jobs for their project based on simple contribution to a Salt configuration.

    Later on, there is a few things I'd like to extend on jenkins_jobs-formula side. Most notably the handling of distant sources for YAML configuration file (as well as maybe the packages list file). I'd also like to experiment on configuring slaves for the Jenkins server, possibly relying on Docker (taking advantage of another of my experiment...).


  • Running a local salt-master to orchestrate docker containers

    2015/05/20 by David Douard

    In a recent blog post, Denis explained how to build Docker containers using Salt.

    What's missing there is how to have a running salt-master dedicated to Docker containers.

    There is not need the salt-master run as root for this. A test config of mine looks like:

    david@perseus:~$ mkdir -p salt/etc/salt
    david@perseus:~$ cd salt
    david@perseus:~salt/$ cat << EOF >etc/salt/master
    interface: 192.168.127.1
    user: david
    
    root_dir: /home/david/salt/
    pidfile: var/run/salt-master.pid
    pki_dir: etc/salt/pki/master
    cachedir: var/cache/salt/master
    sock_dir: var/run/salt/master
    
    file_roots:
      base:
        - /home/david/salt/states
        - /home/david/salt/formulas/cubicweb
    
    pillar_roots:
      base:
        - /home/david/salt/pillar
    EOF
    

    Here, 192.168.127.1 is the ip of my docker0 bridge. Also note that path in file_roots and pillar_roots configs must be absolute (they are not relative to root_dir, see the salt-master configuration documentation).

    Now we can start a salt-master that will be accessible to Docker containers:

    david@perseus:~salt/$ /usr/bin/salt-master -c etc/salt
    

    Warning

    with salt 2015.5.0, salt-master really wants to execute dmidecode, so add /usr/sbin to the $PATH variable before running the salt-master as non-root user.

    From there, you can talk to your test salt master by adding -c ~/salt/etc/salt option to all salt commands. Fortunately, you can also set the SALT_CONFIG_DIR environment variable:

    david@perseus:~salt/$ export SALT_CONFIG_DIR=~/salt/etc/salt
    david@perseus:~salt/$ salt-key
    Accepted Keys:
    Denied Keys:
    Unaccepted Keys:
    Rejected Keys:
    

    Now, you need to have a Docker images with salt-minion already installed, as explained in Denis' blog post. (I prefer using supervisord as PID 1 in my dockers, but that's not important here.)

    david@perseus:~salt/ docker run -d --add-host salt:192.168.127.1  logilab/salted_debian:wheezy
    53bf7d8db53001557e9ae25f5141cd9f2caf7ad6bcb7c2e3442fcdbb1caf5144
    david@perseus:~salt/ docker run -d --name jessie1 --hostname jessie1 --add-host salt:192.168.127.1  logilab/salted_debian:jessie
    3da874e58028ff6dcaf3999b29e2563e1bc4d6b1b7f2f0b166f9a8faffc8aa47
    david@perseus:~salt/ salt-key
    Accepted Keys:
    Denied Keys:
    Unaccepted Keys:
    53bf7d8db530
    jessie1
    Rejected Keys:
    david@perseus:~/salt$ salt-key -y -a 53bf7d8db530
    The following keys are going to be accepted:
    Unaccepted Keys:
    53bf7d8db530
    Key for minion 53bf7d8db530 accepted.
    david@perseus:~/salt$ salt-key -y -a jessie1
    The following keys are going to be accepted:
    Unaccepted Keys:
    jessie1
    Key for minion jessie1 accepted.
    david@perseus:~/salt$ salt '*' test.ping
    jessie1:
        True
    53bf7d8db530:
        True
    

    You can now build Docker images as explained by Denis, or test your sls config files in containers.


  • Building Docker containers using Salt

    2015/04/07 by Denis Laxalde

    In this blog post, I'll talk about a way to use Salt to automate the build and configuration of Docker containers. I will not consider the deployment of Docker containers with Salt as this subject is already covered elsewhere (here for instance). The emphasis here is really on building (or configuring) a container for future deployment.

    Motivation

    Salt is a remote execution framework that can be used for configuration management. It's already widely used at Logilab to manage our infrastructure as well as on a semi-daily basis during our application development activities.

    Docker is a tool that helps automating the deployment of applications within Linux containers. It essentially provides a convenient abstraction and a set of utilities for system level virtualization on Linux. Amongst other things, Docker provides container build helpers around the concept of dockerfile.

    So, the first question is why would you use Salt to build Docker containers when you already have this dockerfile building tool. My first motivation is to encompass the limitations of the available declarations one could insert in a Dockerfile. First limitation: you can only execute instructions in a sequential manner using a Dockerfile, there's is no possibility of declaring dependencies between instructions or even of making an instruction conditional (apart from using the underlying shell conditional machinery of course). Then, you have only limited possibilities of specializing a Dockerfile. Finally, it's no so easy to apply a configuration step-by-step, for instance during the development of said configuration.

    That's enough for an introduction to lay down the underlying motivation of this post. Let's move on to more practical things!

    A Dockerfile for the base image

    Before jumping into the usage of Salt for the configuration of a Docker image, the first thing you need to do is to build a Docker container into a proper Salt minion.

    Assuming we're building on top of some a base image of Debian flavour subsequently referred to as <debian> (I won't tell you where it comes from, since you ought to build your own base image -- or find some friend you trust to provide you with one!), the following Dockerfile can be used to initialize a working image which will serve as the starting point for further configuration with Salt:

    FROM <debian>
    RUN apt-get update
    RUN apt-get install -y salt-minion
    

    Then, run docker build . docker_salt/debian_salt_minion and you're done.

    Plugin the minion container with the Salt master

    The next thing to do with our fresh Debian+salt-minion image is to turn it into a container running salt-minion, waiting for the Salt master to instruct it.

    docker run --add-host=salt:10.1.1.1 --hostname docker_minion \
        --name minion_container \
        docker_salt/debian/salt_minion salt-minion
    

    Here:

    • --hostname is used to specify the network name of the container, for easier query by the Salt master,
    • 10.1.1.1 is usually the IP address of the host, which in our example will serve as the Salt master,
    • --name is just used for easier book-keeping.

    Finally,

    salt-key -a docker_minion
    

    will register the new minion's key into the master's keyring.

    If all went well, the following command should succeed:

    salt 'docker_minion' test.ping
    

    Configuring the container with a Salt formula

    salt 'docker_minion' state.sls some_formula
    salt 'docker_minion' state.highstate
    

    Final steps: save the configured image and build a runnable image

    (Optional step, cleanup salt-minion installation.)

    Make a snapshot image of your configured container.

    docker stop minion_container
    docker commit -m 'Install something with Salt' \
        minion_container me/something
    

    Try out your new image:

    docker run -p 8080:80 me/something <entry point>
    

    where <entry point> will be the main program driving the service provided by the container (typically defined through the Salt formula).

    Make a fully configured image for you service:

    FROM me/something
    [...anything else you need, such as EXPOSE, etc...]
    CMD <entry point>
    

  • Retour sur la journée conteneurs dans le cadre de Open Source Innovation Spring

    2015/04/07 by Arthur Lutz

    Logilab a co-organisé la demi-journée sur les conteneurs dans le cadre du Printemps de l'innovation open source (Open Source Innovation Spring). Voici une partie des choses qui y furent dites.

    Open Source Innovation Spring

    AlterWay a commencé par une introduction expliquant pourquoi docker est si hype en ce moment. Quelques bémols ont été placés sur les questions de sécurité et les systèmes de fichiers utilisés par défaut (AUFS n'est pas dans le kernel linux officiel, des alternatives sont à l'étude).

    Une partie de l'écosystème autour de Docker a été mentionné :

    Ensuite Normation a présenté la gestion de configuration et Docker, avec de grandes questions générales sur le déploiement de serveurs, leur durée de vie, leur transformation, etc.

    Logilab & Mozilla

    Logilab a présenté l'utilisation conjointe de Salt Mercurial et Docker pour appliquer les bonnes pratiques du développement logiciel à la gestion d'infrastructures. Les supports de présentation sont sur http://slides.logilab.fr/osis/osis (aussi sur slideshare).

    Normation a ensuite présenté les fondements techniques des conteneurs, à savoir les fonctionnalités du noyau linux qui ont permis leur essor. Petit historique sur les cgroups, avec les idées d'origine sur les processus dans Unix, mais aussi les bonnes idées apportées par Plan 9 (et qui ont ensuite été reprises par Linux). On a vu des choses sur les chroots, les namespaces, fakeroot, ip netns, les informations dans /proc/<pid>/ns, et les systèmes de fichier d'union utilisé par les conteneurs : aufs, unionfs, overlayfs, fuse.

    Intervenants de la journée

    Ensuite deux démonstrations ont été présentées :

    • Utilisation de docker et docker-swarm sur amazon ec2 pour déployer une application html5 : CircleCI lit le dépôt git de l'application, construit l'image Docker et l'ajoute au hub puis pilote docker-swarm pour qu'elle soit déployée.
    • Utilisation de plusieurs plate-formes de cloud (Azure, Numergy, CloudWatt) pour déployer un conteneur docker sur plusieurs clouds en parallèle.

    Deux retours d'expérience par Theodo et Deliverous ont conclu la journée.


  • Monitoring our websites before we deploy them using Salt

    2015/03/11 by Arthur Lutz

    As you might have noticed we're quite big fans of Salt. One of the things that Salt enables us to do, it to apply what we're used to doing with code to our infrastructure. Let's look at TDD (Test Driven Development).

    Write the test first, make it fail, implement the code, test goes green, you're done.

    Apply the same thing to infrastructure and you get TDI (Test Driven Infrastructure).

    So before you deploy a service, you make sure that your supervision (shinken, nagios, incinga, salt based monitoring, etc.) is doing the correct test, you deploy and then your supervision goes green.

    Let's take a look at website supervision. At Logilab we weren't too satisfied with how our shinken/http_check were working so we started using uptime (nodejs + mongodb). Uptime has a simple REST API to get and add checks, so we wrote a salt execution module and a states module for it.

    https://www.logilab.org/file/288174/raw/68747470733a2f2f7261772e6769746875622e636f6d2f667a616e696e6f74746f2f757074696d652f646f776e6c6f6164732f636865636b5f64657461696c732e706e67.png

    For the sites that use the apache-formula we simply loop on the domains declared in the pillars to add checks :

    {% for domain in salt['pillar.get']('apache:sites').keys() %}
    uptime {{ domain }} (http):
      uptime.monitored:
        - name : http://{{ domain }}
    {% endfor %}
    

    For other URLs (specific URL such as sitemaps) we can list them in pillars and do :

    {% for url in salt['pillar.get']('uptime:urls') %}
    uptime {{ url }}:
      uptime.monitored:
        - name : {{ url }}
    {% endfor %}
    

    That's it. Monitoring comes before deployment.

    We've also contributed a formula for deploying uptime.

    Follow us if you are interested in Test Driven Infrastructure for we intend to write regular reports as we make progress exploring this new domain.


  • A report on the Salt Sprint 2015 in Paris

    2015/03/05 by Arthur Lutz

    On Wednesday the 4th of march 2015, Logilab hosted a sprint on salt on the same day as the sprint at SaltConf15. 7 people joined in and hacked on salt for a few hours. We collaboratively chose some subjects on a pad which is still available.

    //www.logilab.org/file/248336/raw/Salt-Logo.png

    We started off by familiarising those who had never used them to using tests in salt. Some of us tried to run the tests via tox which didn't work any more, a fix was found and will be submitted to the project.

    We organised in teams.

    Boris & Julien looked at the authorisation code and wrote a few issues (minion enumeration, acl documentation). On saltpad (client side) they modified the targeting to adapt to the permissions that the salt-api sends back.

    We discussed the salt permission model (external_auth) : where should the filter happen ? the master ? should the minion receive information about authorisation and not execute what is being asked for ? Boris will summarise some of the discussion about authorisations in a new issue.

    //www.logilab.org/file/288010/raw/IMG_3034.JPG

    Sofian worked on some unification on execution modules (refresh_db which will be ignored for the modules that don't understand that). He will submit a pull request in the next few days.

    Georges & Paul added some tests to hg_pillar, the test creates a mercurial repository, adds a top.sls and a file and checks that they are visible. Here is the diff. They had some problems while debugging the tests.

    David & Arthur implemented the execution module for managing postgresql clusters (create, list, exists, remove) in debian. A pull request was submitted by the end of the day. A state module should follow shortly. On the way we removed some dead code in the postgres module.

    All in all, we had some interesting discussions about salt, it's architecture, shared tips about developing and using it and managed to get some code done. Thanks to all for participating and hopefully we'll sprint again soon...


  • SaltStack Meetup with Thomas Hatch in Paris France

    2014/05/22 by Arthur Lutz

    This monday (19th of may 2014), Thomas Hatch was in Paris for dotScale 2014. After presenting SaltStack there (videos will be published at some point), he spent the evening with members of the French SaltStack community during a meetup set up by Logilab at IRILL.

    http://www.logilab.org/file/248338/raw/thomas-hatch.png

    Here is a list of what we talked about :

    • Since Salt seems to have pushed ZMQ to its limits, SaltStack has been working on RAET (Reliable Asynchronous Event Transport Protocol ), a transport layer based on UDP and elliptic curve cryptography (Dan Berstein's CURVE-255-19) that works more like a stack than a socket and has reliability built in. RAET will be released as an optionnal beta feature in the next Salt release.
    • Folks from Dailymotion bumped into a bug that seems related to high latency networks and the auth_timeout. Updating to the very latest release should fix the issue.
    • Thomas told us about how a dedicated team at SaltStack handles pull requests and another team works on triaging github issues to input them into their internal SCRUM process. There are a lot of duplicate issues and old inactive issues that need attention and clutter the issue tracker. Help will be welcome.
    http://www.logilab.org/file/248336/raw/Salt-Logo.png
    • Continuous integration is based on Jenkins and spins up VMs to test pull request. There is work in progress to test multiple clouds, various latencies and loads.
    • For the Docker integration, salt now keeps track of forwarded ports and relevant information about the containers.
    • salt-virt bumped into problems with chroots and timeouts due to ZMQ.
    • Multi-master: the problem lies with syncronisation of data which is sent to minions but also the data that is sent to the masters. Possible solutions to be explored are : the use of gitfs, there is no built-in solution for keys (salt-key has to be run on all masters), mine.send should send the data at both masters, for the jobs cache: one could use an external returner.
    • Thomas talked briefly about ioflo which should bring queuing, data hierarchy and data pub-sub to Salt.
    http://www.logilab.org/file/248335/raw/ioflo.png
    • About the rolling release question: versions in Salt are definitely not git snapshots, things get backported into previous versions. No clear definition yet of length of LTS versions.
    • salt-cloud and libcloud : in the next release, libcloud will not be a hard dependency. Some clouds didn't work in libcloud (for example AWS), so these providers got implemented directly in salt-cloud or by using third-party libraries (eg. python-boto).
    • Documentation: a sprint is planned next week. Reference documentation will not be completly revamped, but tutorial content will be added.

    Boris Feld showed a demo of vagrant images orchestrated by salt and a web UI to monitor a salt install.

    http://www.vagrantup.com/images/logo_vagrant-81478652.png

    Thanks again to Thomas Hatch for coming and meeting up with (part of) the community here in France.


  • Compte rendu présentation Salt à Solution Linux

    2014/05/21 by Arthur Lutz

    Logilab était à l'édition 2014 de Solutions Linux qui se déroulait au CNIT à Paris. David Douard participait à la table ronde sur les outils libres pour la supervision lors de la session Administration Système, Devops, au cours de laquelle un certain nombre de projets libres ont été mentionnés : nagios, shinken, graphite, ElasticSearch, logstash, munin, saltstack, kibana, centreon, rsyslog.

    http://www.logilab.org/file/248048/raw/solutionlinux.png

    Suite à des présentations sur OpenLDAP, LXC, btrfs et ElasticSearch David Douard a présenté notre approche agile de l'administration système articulée autour de Salt et en particulier le principe de l'administration système pilotée par les tests (diapos) (Test-Driven Infrastructure).

    https://www.logilab.org/file/248098/raw/Screenshot%20from%202014-05-21%2017%3A55%3A35.png

    Merci aux organisateurs de Solutions Linux pour cette édition 2014.


  • Salt April Meetup in Paris (France)

    2014/05/14 by Arthur Lutz

    On the 15th of april, in Paris (France), we took part in yet another Salt meetup. The community is now meeting up once every two months.

    We had two presentations:

    • Arthur Lutz made an introduction to returners and the scheduler using the SalMon monitoring system as an example. Salt is not only about configuration management Indeed!
    • The folks from Is Cool Entertainment did a presentation about how they are using salt-cloud to deploy and orchestrate clusters of EC2 machines (islands in their jargon) to reproduce parts of their production environment for testing and developement.

    More discussions about various salty subjects followed and were pursued in an Italian restaurant (photos here).

    In case it is not already in your diary : Thomas Hatch is coming to Paris next week, on Monday the 19th of May, and will be speaking at dotscale during the day and at a Salt meetup in the evening. The Salt Meetup will take place at IRILL (like the previous meetups, thanks again to them) and should start at 19h. The meetup is free and open to the public, but registering on this framadate would be appreciated.


  • Mini compte rendu Meetup Debian à Nantes

    2014/03/13 by Arthur Lutz

    Hier soir, je suis allé au premier meetup Debian à Nantes. C'était bien sympatique, une vingtaine de personnes ont répondu présent à l'appel de Damien Raude-Morvan et Thomas Vincent. Merci à eux d'avoir lancé l'initiative (le pad d'organisation).

    //www.logilab.org/file/228927/raw/debian-france.jpg

    Après un tour de table des participants, et de quelques discussions sur debian en général (et une explication par Damien de l'état de Java dans Debian), Damien a présenté l'association Debian France ainsi que le concours du nouveau contributeur Debian. La liste d'idées est longue et sympatique n'hésitez pas à aller jeter un oeil et faire une contribution.

    Ensuite Thomas nous a présenté l'équipe de traduction francaise de debian et ses principles de fonctionnement (qualité avant quantité, listes de discussion, IRC, processus de traduction, etc.).

    //www.logilab.org/file/228931/raw/saltstack_logo.jpg

    Finalement, j'ai rapidement présenté Salt et sa place dans Debian. Pour l'archive publique : les diapos de la présentation.

    À la prochaine !

    Pour faire un commentaire, il faut s'authentifier ou s'enregistrer.


  • Retour sur MiniDebConf Paris 2014

    2014/03/05 by Arthur Lutz
    http://www.logilab.org/file/226609/raw/200px-Mini-debconf-paris.png

    Nous sommes heureux d'avoir participé à la MiniDebConf Paris.

    Nous avons sponsorisé l'évenement mais aussi effectué deux présentations :

    Avec une cinquantaine de participants sur les deux jours, c'est toujours agréable de rencontrer la communauté francaise autour de Debian. Merci donc à l'association Debian France d'avoir organisé cette conférence.


  • Second Salt Meetup builds the french community

    2014/03/04 by Arthur Lutz

    On the 6th of February, the Salt community in France met in Paris to discuss Salt and choose the tools to federate itself. The meetup was kindly hosted by IRILL.

    There were two formal presentations :

    • Logilab did a short introduction of Salt,
    • Majerti presented a feedback of their experience with Salt in various professional contexts.

    The presentation space was then opened to other participants and Boris Feld did a short presentation of how Salt was used at NovaPost.

    http://www.logilab.org/file/226420/raw/saltstack_meetup.jpeg

    We then had a short break to share some pizza (sponsored by Logilab).

    After the break, we had some open discussion about various subjects, including "best practices" in Salt and some specific use cases. Regis Leroy talked about the states that Makina Corpus has been publishing on github. The idea of reconciling the documentation and the monitoring of an infrastructure was brought up by Logilab, that calls it "Test Driven Infrastructure".

    The tools we collectively chose to form the community were the following :

    • a mailing-list kindly hosted by the AFPY (a pythonic french organization)
    • a dedicated #salt-fr IRC channel on freenode

    We decided that the meetup would take place every two months, hence the third one will be in April. There is already some discussion about organizing events to tell as many people as possible about Salt. It will probably start with an event at NUMA in March.

    After the meetup was officially over, a few people went on to have some drinks nearby. Thank you all for coming and your participation.

    login or register to comment on this blog


  • A Salt Configuration for C++ Development

    2014/01/24 by Damien Garaud
    http://www.logilab.org/file/204916/raw/SaltStack-Logo.png

    At Logilab, we've been using Salt for one year to manage our own infrastructure. I wanted to use it to manage a specific configuration: C++ development. When I instantiate a Virtual Machine with a Debian image, I don't want to spend time to install and configure a system which fits my needs as a C++ developer:

    This article is a very simple recipe to get a C++ development environment, ready to use, ready to hack.

    Give Me an Editor and a DVCS

    Quite simple: I use the YAML file format used by Salt to describe what I want. To install these two editors, I just need to write:

    vim-nox:
      pkg.installed
    
    emacs23-nox:
      pkg.installed
    

    For Mercurial, you'll guess:

    mercurial:
     pkg.installed
    

    You can write these lines in the same init.sls file, but you can also decide to split your configuration into different subdirectories: one place for each thing. I decided to create a dev and editor directories at the root of my salt config with two init.sls inside.

    That's all for the editors. Next step: specific C++ development packages.

    Install Several "C++" Packages

    In a cpp folder, I write a file init.sls with this content:

    gcc:
        pkg.installed
    
    g++:
        pkg.installed
    
    gdb:
        pkg.installed
    
    cmake:
        pkg.installed
    
    automake:
        pkg.installed
    
    libtool:
        pkg.installed
    
    pkg-config:
        pkg.installed
    
    colorgcc:
        pkg.installed
    

    The choice of these packages is arbitrary. You add or remove some as you need. There is not a unique right solution. But I want more. I want some LLVM packages. In a cpp/llvm.sls, I write:

    llvm:
     pkg.installed
    
    clang:
        pkg.installed
    
    libclang-dev:
        pkg.installed
    
    {% if not grains['oscodename'] == 'wheezy' %}
    lldb-3.3:
        pkg.installed
    {% endif %}
    

    The last line specifies that you install the lldb package if your Debian release is not the stable one, i.e. jessie/testing or sid in my case. Now, just include this file in the init.sls one:

    # ...
    # at the end of 'cpp/init.sls'
    include:
      - .llvm
    

    Organize your sls files according to your needs. That's all for packages installation. You Salt configuration now looks like this:

    .
    |-- cpp
    |   |-- init.sls
    |   `-- llvm.sls
    |-- dev
    |   `-- init.sls
    |-- edit
    |   `-- init.sls
    `-- top.sls
    

    Launching Salt

    Start your VM and install a masterless Salt on it (e.g. apt-get install salt-minion). For launching Salt locally on your naked VM, you need to copy your configuration (through scp or a DVCS) into /srv/salt/ directory and to write the file top.sls:

    base:
      '*':
        - dev
        - edit
        - cpp
    

    Then just launch:

    > salt-call --local state.highstate
    

    as root.

    And What About Configuration Files?

    You're right. At the beginning of the post, I talked about a "ready to use" Mercurial with some HG extensions. So I use and copy the default /etc/mercurial/hgrc.d/hgext.rc file into the dev directory of my Salt configuration. Then, I edit it to set some extensions such as color, rebase, pager. As I also need Evolve, I have to clone the source code from https://bitbucket.org/marmoute/mutable-history. With Salt, I can tell "clone this repo and copy this file" to specific places.

    So, I add some lines to dev/init.sls.

    https://bitbucket.org/marmoute/mutable-history:
        hg.latest:
          - rev: tip
          - target: /opt/local/mutable-history
          - require:
             - pkg: mercurial
    
    /etc/mercurial/hgrc.d/hgext.rc:
        file.managed:
          - source: salt://dev/hgext.rc
          - user: root
          - group: root
          - mode: 644
    

    The require keyword means "install (if necessary) this target before cloning". The other lines are quite self-explanatory.

    In the end, you have just six files with a few lines. Your configuration now looks like:

    .
    |-- cpp
    |   |-- init.sls
    |   `-- llvm.sls
    |-- dev
    |   |-- hgext.rc
    |   `-- init.sls
    |-- edit
    |   `-- init.sls
    `-- top.sls
    

    You can customize it and share it with your teammates. A step further would be to add some configuration files for your favorite editor. You can also imagine to install extra packages that your library depends on. Quite simply add a subdirectory amazing_lib and write your own init.sls. I know I often need Boost libraries for example. When your Salt configuration has changed, just type: salt-call --local state.highstate.

    As you can see, setting up your environment on a fresh system will take you only a couple commands at the shell before you are ready to compile your C++ library, debug it, fix it and commit your modifications to your repository.


  • SaltStack Paris Meetup on Feb 6th, 2014 - (S01E02)

    2013/12/20 by Nicolas Chauvat

    Logilab has set up the second meetup for salt users in Paris on Feb 6th, 2014 at IRILL, near Place d'Italie, starting at 18:00. The address is 23 avenue d'Italie, 75013 Paris.

    Here is the announce in french http://www.logilab.fr/blogentry/1981

    Please forward it to whom may be interested, underlining that pizzas will be offered to refuel the chatters ;)

    Conveniently placed a week after the Salt Conference, topics will include anything related to salt and its uses, demos, new ideas, exchange of salt formulas, commenting the talks/videos of the saltconf, etc.

    If you are interested in Salt, Python and Devops and will be in Paris at that time, we hope to see you there !


  • SaltStack Paris Meetup - some of what was said

    2013/10/09 by Arthur Lutz

    Last week, on the first day of OpenWorldForum 2013, we met up with Thomas Hatch of SaltStack to have a talk about salt. He was in Paris to give two talks the following day (1 & 2), and it was a great opportunity to meet him and physically meet part of the French Salt community. Since Logilab hosted the Great Salt Sprint in Paris, we offered to co-organise the meetup at OpenWorldForum.

    http://saltstack.com/images/SaltStack-Logo.pnghttp://openworldforum.org/static/pictures/Calque1.png

    Introduction

    About 15 people gathered in Montrouge (near Paris) and we all took turns to present ourselves and how or why we used salt. Some people wanted to migrate from BCFG2 to salt. Some people told the story of working a month with CFEngine and meeting the same functionnality in two days with salt and so decided to go for that instead. Some like salt because they can hack its python code. Some use salt to provision pre-defined AMI images for the clouds (salt-ami-cloud-builder). Some chose salt over Ansible. Some want to use salt to pilot temporary computation clusters in the cloud (sort of like what StarCluster does with boto and ssh).

    When Paul from Logilab introduced salt-ami-cloud-builder, Thomas Hatch said that some work is being done to go all the way : build an image from scratch from a state definition. On the question of Debian packaging, some efforts could be done to have salt into wheezy-backports. Julien Cristau from Logilab who is a debian developer might help with that.

    Some untold stories where shared : some companies that replaced puppet by salt, some companies use salt to control an HPC cluster, some companies use salt to pilot their existing puppet system.

    We had some discussions around salt-cloud, which will probably be merged into salt at some point. One idea for salt-cloud was raised : have a way of defining a "minimum" type of configuration which translates into the profiles according to which provider is used (an issue should be added shortly). The expression "pushing states" was often used, it is probably a good way of looking a the combination of using salt-cloud and the masterless mode available with salt-ssh. salt-cloud controls an existing cloud, but Thomas Hatch points to the fact that with salt-virt, salt is becoming a cloud controller itself, more on that soon.

    Mixing pillar definition between 'public' and 'private' definitions can be tricky. Some solutions exist with multiple gitfs (or mercurial) external pillar definitions, but more use cases will drive more flexible functionalities in the future.

    Presentation and live demo

    For those in the audience that were not (yet) users of salt, Thomas went back to explaining a few basics about it. Salt should be seen as a "toolkit to solve problems in a infrastructure" says Thomas Hatch. Why is it fast ? It is completely asynchronous and event driven.

    He gave a quick presentation about the new salt-ssh which was introduced in 0.17, which allows the application of salt recipes to machines that don't have a minion connected to the master.

    The peer communication can be used so as to add a condition for a state on the presence of service on a different minion.

    While doing demos or even hacking on salt, one can use salt/test/minionswarm.py which makes fake minions, not everyone has hundreds of servers in at their fingertips.

    Smart modules are loaded dynamically, for example, the git module that gets loaded if a state installs git and then in the same highstate uses the git modules.

    Thomas explained the difference between grains and pillars : grains is data about a minion that lives on the minion, pillar is data about the minion that lives on the master. When handling grains, the grains.setval can be useful (it writes in /etc/salt/grains as yaml, so you can edit it separately). If a minion is not reachable one can obtain its grains information by replacing test=True by cache=True.

    Thomas shortly presented saltstack-formulas : people want to "program" their states, and formulas answer this need, some of the jinja2 is overly complicated to make them flexible and programmable.

    While talking about the unified package commands (a salt command often has various backends according to what system runs the minion), for example salt-call --local pkg.install vim, Thomas told this funny story : ironically, salt was nominated for "best package manager" at some linux magazine competition. (so you don't have to learn how to use FreeBSD packaging tools).

    While hacking salt, one can take a look at the Event Bus (see test/eventlisten.py), many applications are possible when using the data on this bus. Thomas talks about a future IOflow python module where a complex logic can be implemented in the reactor with rules and a state machine. One example use of this would be if the load is high on X number of servers and the number of connexions Y on these servers then launch extra machines.

    To finish on a buzzword, someone asked "what is the overlap of salt and docker" ? The answer is not simple, but Thomas thinks that in the long run there will be a lot of overlap, one can check out the existing lxc modules and states.

    Wrap up

    To wrap up, Thomas announced a salt conference planned for January 2014 in Salt Lake City.

    Logilab proposes to bootstrap the French community around salt. As the group suggest this could take the form of a mailing list, an irc channel, a meetup group , some sprints, or a combination of all the above. On that note, next international sprint will probably take place in January 2014 around the salt conference.