In this blog post, I'll talk about my recent experiments on building a
continuous integration service with Jenkins that is, as much as possible,
managed through Salt. We've been relying on a Jenkins platform for quite some
time at Logilab (Tolosa team). The service was mostly managed by me with
sporadic help from other team-mates but I've never been entirely satisfied
about the way it was managed because it involved a lot of boilerplate
configuration through Jenkins user interface and this does not scale very well
nor does it make long term maintenance easy.
So recently, I've taken a stance and decided to go through a Salt-based
configuration and management of our Jenkins CI platform. There are actually
two aspects here. The first concerns the setup of Jenkins itself (this
includes installation, security configuration, plugins management amongst
other things). The second concerns the management of client projects (or jobs
in Jenkins jargon). For this second aspect, one of the design goals was to
enable easy configuration of jobs by users not necessarily familiar with
Jenkins setup and to make collaborative maintenance easy. To tackle these two
aspects I've essentially been using (or developing) two distinct Salt
formulas which I'll detail hereafter.
Core setup: the
The core setup of Jenkins is based on an existing Salt formula, the
jenkins-formula which I extended a bit to support
map.jinja and which
was further improved to support installation of plugins by Yann and Laura (see
With that, deploying a Jenkins server is as simple as adding the following to
your states and pillars
Base pillar configuration is used to declare anything that differs from the
default Jenkins settings in a
jenkins section, e.g.:
- home: /opt/jenkins
Plugins configuration is declared in
plugins subsection as follows:
(Note that plugins dependency is not handled by Jenkins when installing from
the command line, neither by this formula. So in the preceding example, just
having an entry for the Mercurial plugin would have not been enough because
this plugin depends on scm-api.)
Other aspects (such as security setup) are not handled yet (neither by the
original formula, nor by our extension), but I tend to believe that this is
acceptable to manage this "by hand" for now.
Jobs management : the
For this task, I leveraged the excellent jenkins-job-builder tool which
makes it possible to configure jobs using a declarative YAML syntax. The tool
takes care of installing the job and also handles any housekeeping tasks such
as checking configuration validity or deleting old configurations. With this
tool, my goal was to let end-users of the Jenkins service add their own
project by providing a minima a YAML job description file. So for instance,
a simple Job description for a CubicWeb job could be:
- shell: "find . -name 'tmpdb*' -delete"
- shell: "tox --hashseed noset"
It consists of two parts:
scm section declares, well, SCM information, here the location of the
review Mercurial repository, and,
job section which consists of some metadata (project name), a reference
of the SCM section declared above, some
builders (here simple shell
builders) and a
publisher part to send results by email.
Pretty simple. (Note that most test running configuration is here declared
within the source repository, via tox (another story), so that the CI bot
holds minimum knowledge and fetches information from the sources repository
To automate the deployment of this kind of configurations, I made a
jenkins_jobs-formula which takes care of:
- installing jenkins-job-builder,
- deploying YAML configurations,
jenkins-jobs update to push jobs into the Jenkins instance.
In addition to installing the YAML file and triggering a
run upon changes of job files, the formula allows for job to list distribution
packages that it would require for building.
Wrapping things up, a pillar declaration of a Jenkins job looks like:
file: <path to local cubicweb.yaml>
file section indicates the source of the YAML file to install and
pkgs lists build dependencies that are not managed by the job itself
(typically non Python package in our case).
So, as an end user, all is needed to provide is the YAML file and a pillar
snippet similar to the above.
This initial setup appears to be enough to greatly reduce the burden of
managing a Jenkins server and to allow individual users to contribute jobs for
their project based on simple contribution to a Salt configuration.
Later on, there is a few things I'd like to extend on
jenkins_jobs-formula side. Most notably the handling of distant sources
for YAML configuration file (as well as maybe the packages list file).
I'd also like to experiment on configuring slaves for the Jenkins server,
possibly relying on Docker (taking advantage of another of my