In this blog post, I'll talk about my recent experiments on building a continuous integration service with Jenkins that is, as much as possible, managed through Salt. We've been relying on a Jenkins platform for quite some time at Logilab (Tolosa team). The service was mostly managed by me with sporadic help from other team-mates but I've never been entirely satisfied about the way it was managed because it involved a lot of boilerplate configuration through Jenkins user interface and this does not scale very well nor does it make long term maintenance easy.
So recently, I've taken a stance and decided to go through a Salt-based configuration and management of our Jenkins CI platform. There are actually two aspects here. The first concerns the setup of Jenkins itself (this includes installation, security configuration, plugins management amongst other things). The second concerns the management of client projects (or jobs in Jenkins jargon). For this second aspect, one of the design goals was to enable easy configuration of jobs by users not necessarily familiar with Jenkins setup and to make collaborative maintenance easy. To tackle these two aspects I've essentially been using (or developing) two distinct Salt formulas which I'll detail hereafter.
Core setup: the
The core setup of Jenkins is based on an existing Salt formula, the
jenkins-formula which I extended a bit to support
map.jinja and which
was further improved to support installation of plugins by Yann and Laura (see
With that, deploying a Jenkins server is as simple as adding the following to
your states and pillars
base: "jenkins": - jenkins - jenkins.plugins
Base pillar configuration is used to declare anything that differs from the
default Jenkins settings in a
jenkins section, e.g.:
jenkins: lookup: - home: /opt/jenkins
Plugins configuration is declared in
plugins subsection as follows:
jenkins: lookup: plugins: scm-api: url: 'http://updates.jenkins-ci.org/download/plugins/scm-api/0.2/scm-api.hpi' hash: 'md5=9574c07bf6bfd02a57b451145c870f0e' mercurial: url: 'http://updates.jenkins-ci.org/download/plugins/mercurial/1.54/mercurial.hpi' hash: 'md5=1b46e2732be31b078001bcc548149fe5'
(Note that plugins dependency is not handled by Jenkins when installing from the command line, neither by this formula. So in the preceding example, just having an entry for the Mercurial plugin would have not been enough because this plugin depends on scm-api.)
Other aspects (such as security setup) are not handled yet (neither by the original formula, nor by our extension), but I tend to believe that this is acceptable to manage this "by hand" for now.
Jobs management : the
For this task, I leveraged the excellent jenkins-job-builder tool which makes it possible to configure jobs using a declarative YAML syntax. The tool takes care of installing the job and also handles any housekeeping tasks such as checking configuration validity or deleting old configurations. With this tool, my goal was to let end-users of the Jenkins service add their own project by providing a minima a YAML job description file. So for instance, a simple Job description for a CubicWeb job could be:
- scm: name: cubicweb scm: - hg: url: http://hg.logilab.org/review/cubicweb clean: true - job: name: cubicweb display-name: CubicWeb scm: - cubicweb builders: - shell: "find . -name 'tmpdb*' -delete" - shell: "tox --hashseed noset" publishers: - email: recipients: firstname.lastname@example.org
It consists of two parts:
scmsection declares, well, SCM information, here the location of the review Mercurial repository, and,
jobsection which consists of some metadata (project name), a reference of the SCM section declared above, some
builders(here simple shell builders) and a
publisherpart to send results by email.
Pretty simple. (Note that most test running configuration is here declared within the source repository, via tox (another story), so that the CI bot holds minimum knowledge and fetches information from the sources repository directly.)
To automate the deployment of this kind of configurations, I made a jenkins_jobs-formula which takes care of:
- installing jenkins-job-builder,
- deploying YAML configurations,
jenkins-jobs updateto push jobs into the Jenkins instance.
In addition to installing the YAML file and triggering a
run upon changes of job files, the formula allows for job to list distribution
packages that it would require for building.
Wrapping things up, a pillar declaration of a Jenkins job looks like:
jenkins_jobs: lookup: jobs: cubicweb: file: <path to local cubicweb.yaml> pkgs: - mercurial - python-dev - libgecode-dev
file section indicates the source of the YAML file to install and
pkgs lists build dependencies that are not managed by the job itself
(typically non Python package in our case).
So, as an end user, all is needed to provide is the YAML file and a pillar snippet similar to the above.
This initial setup appears to be enough to greatly reduce the burden of managing a Jenkins server and to allow individual users to contribute jobs for their project based on simple contribution to a Salt configuration.
Later on, there is a few things I'd like to extend on jenkins_jobs-formula side. Most notably the handling of distant sources for YAML configuration file (as well as maybe the packages list file). I'd also like to experiment on configuring slaves for the Jenkins server, possibly relying on Docker (taking advantage of another of my experiment...).