In a previous post we talked about an environment to develop salt formulas. To add some spicy requirements, the formula must now handle multiple target OS (Debian and Centos), have tests and a continuous integration (CI) server setup.

I started a year ago to write a framework to this purpose, it's called testinfra and is used to execute commands on remote systems and make assertions on the state and the behavior of the system. The modules API provides a pythonic way to inspect the system. It has a smooth integration with pytest that adds some useful features out of the box like parametrization to run tests against multiple systems.

Writing useful tests is not an easy task, my advice is to test code that triggers implicit actions, code that has caused issues in the past or simply test the application is working correctly like you would do in the shell.

For instance this is one of the tests I wrote for the saemref formula

def test_saemref_running(Process, Service, Socket, Command):
    assert Service("supervisord").is_enabled

    supervisord = Process.get(comm="supervisord")
    # Supervisor run as root
    assert supervisord.user == "root"
    assert == "root"

    cubicweb = Process.get(
    # Cubicweb should run as saemref user
    assert cubicweb.user == "saemref"
    assert == "saemref"
    assert cubicweb.comm == "uwsgi"
    # Should have 2 worker process with 8 thread each and 1 http proccess with one thread
    child_threads = sorted([c.nlwp for c in Process.filter(])
    assert child_threads == [1, 8, 8]

    # uwsgi should bind on all ipv4 adresses
    assert Socket("tcp://").is_listening

    html = Command.check_output("curl http://localhost:8080")
    assert "<title>accueil (Référentiel SAEM)</title>" in html

Now we can run tests against a running container by giving its name or docker id to testinfra:

% testinfra --hosts=docker://1a8ddedf8164
test/[docker:/1a8ddedf8164] PASSED

The immediate advantage of writing such test is that you can reuse it for monitoring purpose, testinfra can behave like a nagios plugin:

% testinfra -qq --nagios --hosts=ssh://prod
TESTINFRA OK - 1 passed, 0 failed, 0 skipped in 2.31 seconds

We can now integrate the test suite in our by adding some code to build and run a provisioned docker image and add a test command that runs testinfra tests against it.

provision_option = click.option('--provision', is_flag=True, help="Provision the container")

@cli.command(help="Build an image")
def build(image, provision=False):
    dockerfile = "test/{0}.Dockerfile".format(image)
    tag = "{0}-formula:{1}".format(formula, image)
    if provision:
        dockerfile_content = open(dockerfile).read()
        dockerfile_content += "\n" + "\n".join([
            "ADD test/minion.conf /etc/salt/minion.d/minion.conf",
            "ADD {0} /srv/formula/{0}".format(formula),
            "RUN salt-call --retcode-passthrough state.sls {0}".format(formula),
        ]) + "\n"
        dockerfile = "test/{0}_provisioned.Dockerfile".format(image)
        with open(dockerfile, "w") as f:
        tag += "-provisioned"
    subprocess.check_call(["docker", "build", "-t", tag, "-f", dockerfile, "."])
    return tag

@cli.command(help="Spawn an interactive shell in a new container")
def dev(ctx, image, provision=False):
    tag = ctx.invoke(build, image=image, provision=provision)[
        "docker", "run", "-i", "-t", "--rm", "--hostname", image,
        "-v", "{0}/test/minion.conf:/etc/salt/minion.d/minion.conf".format(BASEDIR),
        "-v", "{0}/{1}:/srv/formula/{1}".format(BASEDIR, formula),
        tag, "/bin/bash",

@cli.command(help="Run tests against a provisioned container",
             context_settings={"allow_extra_args": True})
def test(ctx, image):
    import pytest
    tag = ctx.invoke(build, image=image, provision=True)
    docker_id = subprocess.check_output([
        "docker", "run", "-d", "--hostname", image,
        "-v", "{0}/test/minion.conf:/etc/salt/minion.d/minion.conf".format(BASEDIR),
        "-v", "{0}/{1}:/srv/formula/{1}".format(BASEDIR, formula),
        tag, "tail", "-f", "/dev/null",
        ctx.exit(pytest.main(["--hosts=docker://" + docker_id] + ctx.args))
        subprocess.check_call(["docker", "rm", "-f", docker_id])

Tests can be run on a local CI server or on travis, they "just" require a docker server, here is an example of .travis.yml

sudo: required
  - docker
language: python
  - "2.7"
    - IMAGE=centos7
    - IMAGE=jessie
  - pip install testinfra
  - python test $IMAGE -- -v

I wrote a dummy formula with the above code, feel free to use it as a template for your own formula or open pull requests and break some tests.

There is a highly enhanced version of this code in the saemref formula repository, including:

  • Building a provisioned docker image with custom pillars, we use it to run an online demo
  • Destructive tests where each test is run in a dedicated "fresh" container
  • Run Systemd in the containers to get a system close to the production one (this enables the use of Salt service module)
  • Run a postgresql container linked to the tested container for specific tests like upgrading a Cubicweb instance.

Destructive tests rely on advanced pytest features that may produce weird bugs when mixed together, too much magic involved here. Also, handling Systemd in docker is really painful and adds a lot of complexity, for instance some systemctl commands require a running systemd as PID 1 and this is not the case during the docker build phase. So the trade-off between complexity and these features may not be worth.

There is also a lot of quite new tools to develop and test infrastructure code that you could include in your stack like test-kitchen, serverspec, and goss. Choose your weapon and go test your infrastructure code.

blog entry of