Blog entries july 2016 [2]
  • Testing salt formulas with testinfra

    2016/07/21 by Philippe Pepiot

    In a previous post we talked about an environment to develop salt formulas. To add some spicy requirements, the formula must now handle multiple target OS (Debian and Centos), have tests and a continuous integration (CI) server setup.

    http://testinfra.readthedocs.io/en/latest/_static/logo.png

    I started a year ago to write a framework to this purpose, it's called testinfra and is used to execute commands on remote systems and make assertions on the state and the behavior of the system. The modules API provides a pythonic way to inspect the system. It has a smooth integration with pytest that adds some useful features out of the box like parametrization to run tests against multiple systems.

    Writing useful tests is not an easy task, my advice is to test code that triggers implicit actions, code that has caused issues in the past or simply test the application is working correctly like you would do in the shell.

    For instance this is one of the tests I wrote for the saemref formula

    def test_saemref_running(Process, Service, Socket, Command):
        assert Service("supervisord").is_enabled
    
        supervisord = Process.get(comm="supervisord")
        # Supervisor run as root
        assert supervisord.user == "root"
        assert supervisord.group == "root"
    
        cubicweb = Process.get(ppid=supervisord.pid)
        # Cubicweb should run as saemref user
        assert cubicweb.user == "saemref"
        assert cubicweb.group == "saemref"
        assert cubicweb.comm == "uwsgi"
        # Should have 2 worker process with 8 thread each and 1 http proccess with one thread
        child_threads = sorted([c.nlwp for c in Process.filter(ppid=cubicweb.pid)])
        assert child_threads == [1, 8, 8]
    
        # uwsgi should bind on all ipv4 adresses
        assert Socket("tcp://0.0.0.0:8080").is_listening
    
        html = Command.check_output("curl http://localhost:8080")
        assert "<title>accueil (Référentiel SAEM)</title>" in html
    

    Now we can run tests against a running container by giving its name or docker id to testinfra:

    % testinfra --hosts=docker://1a8ddedf8164 test_saemref.py
    [...]
    test/test_saemref.py::test_saemref_running[docker:/1a8ddedf8164] PASSED
    

    The immediate advantage of writing such test is that you can reuse it for monitoring purpose, testinfra can behave like a nagios plugin:

    % testinfra -qq --nagios --hosts=ssh://prod test_saemref.py
    TESTINFRA OK - 1 passed, 0 failed, 0 skipped in 2.31 seconds
    .
    

    We can now integrate the test suite in our run-tests.py by adding some code to build and run a provisioned docker image and add a test command that runs testinfra tests against it.

    provision_option = click.option('--provision', is_flag=True, help="Provision the container")
    
    @cli.command(help="Build an image")
    @image_choice
    @provision_option
    def build(image, provision=False):
        dockerfile = "test/{0}.Dockerfile".format(image)
        tag = "{0}-formula:{1}".format(formula, image)
        if provision:
            dockerfile_content = open(dockerfile).read()
            dockerfile_content += "\n" + "\n".join([
                "ADD test/minion.conf /etc/salt/minion.d/minion.conf",
                "ADD {0} /srv/formula/{0}".format(formula),
                "RUN salt-call --retcode-passthrough state.sls {0}".format(formula),
            ]) + "\n"
            dockerfile = "test/{0}_provisioned.Dockerfile".format(image)
            with open(dockerfile, "w") as f:
                f.write(dockerfile_content)
            tag += "-provisioned"
        subprocess.check_call(["docker", "build", "-t", tag, "-f", dockerfile, "."])
        return tag
    
    
    @cli.command(help="Spawn an interactive shell in a new container")
    @image_choice
    @provision_option
    @click.pass_context
    def dev(ctx, image, provision=False):
        tag = ctx.invoke(build, image=image, provision=provision)
        subprocess.call([
            "docker", "run", "-i", "-t", "--rm", "--hostname", image,
            "-v", "{0}/test/minion.conf:/etc/salt/minion.d/minion.conf".format(BASEDIR),
            "-v", "{0}/{1}:/srv/formula/{1}".format(BASEDIR, formula),
            tag, "/bin/bash",
        ])
    
    
    @cli.command(help="Run tests against a provisioned container",
                 context_settings={"allow_extra_args": True})
    @click.pass_context
    @image_choice
    def test(ctx, image):
        import pytest
        tag = ctx.invoke(build, image=image, provision=True)
        docker_id = subprocess.check_output([
            "docker", "run", "-d", "--hostname", image,
            "-v", "{0}/test/minion.conf:/etc/salt/minion.d/minion.conf".format(BASEDIR),
            "-v", "{0}/{1}:/srv/formula/{1}".format(BASEDIR, formula),
            tag, "tail", "-f", "/dev/null",
        ]).strip()
        try:
            ctx.exit(pytest.main(["--hosts=docker://" + docker_id] + ctx.args))
        finally:
            subprocess.check_call(["docker", "rm", "-f", docker_id])
    

    Tests can be run on a local CI server or on travis, they "just" require a docker server, here is an example of .travis.yml

    sudo: required
    services:
      - docker
    language: python
    python:
      - "2.7"
    env:
      matrix:
        - IMAGE=centos7
        - IMAGE=jessie
    install:
      - pip install testinfra
    script:
      - python run-tests.py test $IMAGE -- -v
    

    I wrote a dummy formula with the above code, feel free to use it as a template for your own formula or open pull requests and break some tests.

    There is a highly enhanced version of this code in the saemref formula repository, including:

    • Building a provisioned docker image with custom pillars, we use it to run an online demo
    • Destructive tests where each test is run in a dedicated "fresh" container
    • Run Systemd in the containers to get a system close to the production one (this enables the use of Salt service module)
    • Run a postgresql container linked to the tested container for specific tests like upgrading a Cubicweb instance.

    Destructive tests rely on advanced pytest features that may produce weird bugs when mixed together, too much magic involved here. Also, handling Systemd in docker is really painful and adds a lot of complexity, for instance some systemctl commands require a running systemd as PID 1 and this is not the case during the docker build phase. So the trade-off between complexity and these features may not be worth.

    There is also a lot of quite new tools to develop and test infrastructure code that you could include in your stack like test-kitchen, serverspec, and goss. Choose your weapon and go test your infrastructure code.


  • Developing salt formulas with docker

    2016/07/21 by Philippe Pepiot
    https://www.logilab.org/file/248336/raw/Salt-Logo.png

    While developing salt formulas I was looking for a simple and reproducible environment to allow faster development, less bugs and more fun. The formula must handle multiple target OS (Debian and Centos).

    The first barrier is the master/minion installation of Salt, but fortunately Salt has a masterless mode. The idea is quite simple, bring up a virtual machine, install a Salt minion on it, expose the code inside the VM and call Salt states.

    https://www.logilab.org/file/7159870/raw/docker.png

    At Logilab we like to work with docker, a lightweight OS-level virtualization solution. One of the key features is docker volumes to share local files inside the container. So I started to write a simple Python script to build a container with a Salt minion installed and run it with formula states and a few config files shared inside the VM.

    The formula I was working on is used to deploy the saemref project, which is a Cubicweb based application:

    % cat test/centos7.Dockerfile
    FROM centos:7
    RUN yum -y install epel-release && \
        yum -y install https://repo.saltstack.com/yum/redhat/salt-repo-latest-1.el7.noarch.rpm && \
        yum clean expire-cache && \
        yum -y install salt-minion
    
    % cat test/jessie.Dockerfile
    FROM debian:jessie
    RUN apt-get update && apt-get -y install wget
    RUN wget -O - https://repo.saltstack.com/apt/debian/8/amd64/latest/SALTSTACK-GPG-KEY.pub | apt-key add -
    RUN echo "deb http://repo.saltstack.com/apt/debian/8/amd64/latest jessie main" > /etc/apt/sources.list.d/saltstack.list
    RUN apt-get update && apt-get -y install salt-minion
    
    % cat test/minion.conf
    file_client: local
    file_roots:
      base:
        - /srv/salt
        - /srv/formula
    

    And finally the run-tests.py file, using the beautiful click module

    #!/usr/bin/env python
    import os
    import subprocess
    
    import click
    
    @click.group()
    def cli():
        pass
    
    formula = "saemref"
    BASEDIR = os.path.abspath(os.path.dirname(__file__))
    
    image_choice = click.argument("image", type=click.Choice(["centos7", "jessie"]))
    
    
    @cli.command(help="Build an image")
    @image_choice
    def build(image):
        dockerfile = "test/{0}.Dockerfile".format(image)
        tag = "{0}-formula:{1}".format(formula, image)
        subprocess.check_call(["docker", "build", "-t", tag, "-f", dockerfile, "."])
        return tag
    
    
    @cli.command(help="Spawn an interactive shell in a new container")
    @image_choice
    @click.pass_context
    def dev(ctx, image):
        tag = ctx.invoke(build, image=image)
        subprocess.call([
            "docker", "run", "-i", "-t", "--rm", "--hostname", image,
            "-v", "{0}/test/minion.conf:/etc/salt/minion.d/minion.conf".format(BASEDIR),
            "-v", "{0}/{1}:/srv/formula/{1}".format(BASEDIR, formula),
            tag, "/bin/bash",
        ])
    
    
    if __name__ == "__main__":
        cli()
    

    Now I can run quickly multiple containers and test my Salt states inside the containers while editing the code locally:

    % ./run-tests.py dev centos7
    [root@centos7 /]# salt-call state.sls saemref
    
    [ ... ]
    
    [root@centos7 /]# ^D
    % # The container is destroyed when it exits
    

    Notice that we could add some custom pillars and state files simply by adding specific docker shared volumes.

    With a few lines we created a lightweight vagrant like, but faster, with docker instead of virtualbox and it remain fully customizable for future needs.