In a previous post we talked about an environment to develop salt
formulas. To add some spicy requirements, the formula must now handle multiple
target OS (Debian and Centos), have tests and a continuous integration (CI)
server setup.
I started a year ago to write a framework to this purpose, it's called
testinfra and is used to execute commands on remote systems and make
assertions on the state and the behavior of the system. The modules API
provides a pythonic way to inspect the system. It has a smooth integration with
pytest that adds some useful features out of the box like parametrization to
run tests against multiple systems.
Writing useful tests is not an easy task, my advice is to test code that
triggers implicit actions, code that has caused issues in the past or simply
test the application is working correctly like you would do in the shell.
For instance this is one of the tests I wrote for the saemref formula
def test_saemref_running(Process, Service, Socket, Command):
assert Service("supervisord").is_enabled
supervisord = Process.get(comm="supervisord")
assert supervisord.user == "root"
assert supervisord.group == "root"
cubicweb = Process.get(ppid=supervisord.pid)
assert cubicweb.user == "saemref"
assert cubicweb.group == "saemref"
assert cubicweb.comm == "uwsgi"
child_threads = sorted([c.nlwp for c in Process.filter(ppid=cubicweb.pid)])
assert child_threads == [1, 8, 8]
assert Socket("tcp://0.0.0.0:8080").is_listening
html = Command.check_output("curl http://localhost:8080")
assert "<title>accueil (Référentiel SAEM)</title>" in html
Now we can run tests against a running container by giving its name or docker id to testinfra:
% testinfra --hosts=docker://1a8ddedf8164 test_saemref.py
[...]
test/test_saemref.py::test_saemref_running[docker:/1a8ddedf8164] PASSED
The immediate advantage of writing such test is that you can reuse it for
monitoring purpose, testinfra can behave like a nagios plugin:
% testinfra -qq --nagios --hosts=ssh://prod test_saemref.py
TESTINFRA OK - 1 passed, 0 failed, 0 skipped in 2.31 seconds
.
We can now integrate the test suite in our run-tests.py by adding some code
to build and run a provisioned docker image and add a test command that runs
testinfra tests against it.
provision_option = click.option('--provision', is_flag=True, help="Provision the container")
@cli.command(help="Build an image")
@image_choice
@provision_option
def build(image, provision=False):
dockerfile = "test/{0}.Dockerfile".format(image)
tag = "{0}-formula:{1}".format(formula, image)
if provision:
dockerfile_content = open(dockerfile).read()
dockerfile_content += "\n" + "\n".join([
"ADD test/minion.conf /etc/salt/minion.d/minion.conf",
"ADD {0} /srv/formula/{0}".format(formula),
"RUN salt-call --retcode-passthrough state.sls {0}".format(formula),
]) + "\n"
dockerfile = "test/{0}_provisioned.Dockerfile".format(image)
with open(dockerfile, "w") as f:
f.write(dockerfile_content)
tag += "-provisioned"
subprocess.check_call(["docker", "build", "-t", tag, "-f", dockerfile, "."])
return tag
@cli.command(help="Spawn an interactive shell in a new container")
@image_choice
@provision_option
@click.pass_context
def dev(ctx, image, provision=False):
tag = ctx.invoke(build, image=image, provision=provision)
subprocess.call([
"docker", "run", "-i", "-t", "--rm", "--hostname", image,
"-v", "{0}/test/minion.conf:/etc/salt/minion.d/minion.conf".format(BASEDIR),
"-v", "{0}/{1}:/srv/formula/{1}".format(BASEDIR, formula),
tag, "/bin/bash",
])
@cli.command(help="Run tests against a provisioned container",
context_settings={"allow_extra_args": True})
@click.pass_context
@image_choice
def test(ctx, image):
import pytest
tag = ctx.invoke(build, image=image, provision=True)
docker_id = subprocess.check_output([
"docker", "run", "-d", "--hostname", image,
"-v", "{0}/test/minion.conf:/etc/salt/minion.d/minion.conf".format(BASEDIR),
"-v", "{0}/{1}:/srv/formula/{1}".format(BASEDIR, formula),
tag, "tail", "-f", "/dev/null",
]).strip()
try:
ctx.exit(pytest.main(["--hosts=docker://" + docker_id] + ctx.args))
finally:
subprocess.check_call(["docker", "rm", "-f", docker_id])
Tests can be run on a local CI server or on travis, they "just" require a
docker server, here is an example of .travis.yml
sudo: required
services:
- docker
language: python
python:
- "2.7"
env:
matrix:
- IMAGE=centos7
- IMAGE=jessie
install:
- pip install testinfra
script:
- python run-tests.py test $IMAGE -- -v
I wrote a dummy formula with the above code, feel free to use it as a
template for your own formula or open pull requests and break some tests.
There is a highly enhanced version of this code in the saemref formula repository, including:
- Building a provisioned docker image with custom pillars, we use it to run an online demo
- Destructive tests where each test is run in a dedicated "fresh" container
- Run Systemd in the containers to get a system close to the production one (this enables the use of Salt service module)
- Run a postgresql container linked to the tested container for specific tests like upgrading a Cubicweb instance.
Destructive tests rely on advanced pytest features that may produce weird
bugs when mixed together, too much magic involved here. Also, handling Systemd
in docker is really painful and adds a lot of complexity, for instance some
systemctl commands require a running systemd as PID 1 and this is not the
case during the docker build phase. So the trade-off between complexity and
these features may not be worth.
There is also a lot of quite new tools to develop and test infrastructure code
that you could include in your stack like test-kitchen, serverspec, and
goss. Choose your weapon and go test your infrastructure code.
