Blog entries by David Douard [6]
  • Running a local salt-master to orchestrate docker containers

    2015/05/20 by David Douard

    In a recent blog post, Denis explained how to build Docker containers using Salt.

    What's missing there is how to have a running salt-master dedicated to Docker containers.

    There is not need the salt-master run as root for this. A test config of mine looks like:

    david@perseus:~$ mkdir -p salt/etc/salt
    david@perseus:~$ cd salt
    david@perseus:~salt/$ cat << EOF >etc/salt/master
    interface: 192.168.127.1
    user: david
    
    root_dir: /home/david/salt/
    pidfile: var/run/salt-master.pid
    pki_dir: etc/salt/pki/master
    cachedir: var/cache/salt/master
    sock_dir: var/run/salt/master
    
    file_roots:
      base:
        - /home/david/salt/states
        - /home/david/salt/formulas/cubicweb
    
    pillar_roots:
      base:
        - /home/david/salt/pillar
    EOF
    

    Here, 192.168.127.1 is the ip of my docker0 bridge. Also note that path in file_roots and pillar_roots configs must be absolute (they are not relative to root_dir, see the salt-master configuration documentation).

    Now we can start a salt-master that will be accessible to Docker containers:

    david@perseus:~salt/$ /usr/bin/salt-master -c etc/salt
    

    Warning

    with salt 2015.5.0, salt-master really wants to execute dmidecode, so add /usr/sbin to the $PATH variable before running the salt-master as non-root user.

    From there, you can talk to your test salt master by adding -c ~/salt/etc/salt option to all salt commands. Fortunately, you can also set the SALT_CONFIG_DIR environment variable:

    david@perseus:~salt/$ export SALT_CONFIG_DIR=~/salt/etc/salt
    david@perseus:~salt/$ salt-key
    Accepted Keys:
    Denied Keys:
    Unaccepted Keys:
    Rejected Keys:
    

    Now, you need to have a Docker images with salt-minion already installed, as explained in Denis' blog post. (I prefer using supervisord as PID 1 in my dockers, but that's not important here.)

    david@perseus:~salt/ docker run -d --add-host salt:192.168.127.1  logilab/salted_debian:wheezy
    53bf7d8db53001557e9ae25f5141cd9f2caf7ad6bcb7c2e3442fcdbb1caf5144
    david@perseus:~salt/ docker run -d --name jessie1 --hostname jessie1 --add-host salt:192.168.127.1  logilab/salted_debian:jessie
    3da874e58028ff6dcaf3999b29e2563e1bc4d6b1b7f2f0b166f9a8faffc8aa47
    david@perseus:~salt/ salt-key
    Accepted Keys:
    Denied Keys:
    Unaccepted Keys:
    53bf7d8db530
    jessie1
    Rejected Keys:
    david@perseus:~/salt$ salt-key -y -a 53bf7d8db530
    The following keys are going to be accepted:
    Unaccepted Keys:
    53bf7d8db530
    Key for minion 53bf7d8db530 accepted.
    david@perseus:~/salt$ salt-key -y -a jessie1
    The following keys are going to be accepted:
    Unaccepted Keys:
    jessie1
    Key for minion jessie1 accepted.
    david@perseus:~/salt$ salt '*' test.ping
    jessie1:
        True
    53bf7d8db530:
        True
    

    You can now build Docker images as explained by Denis, or test your sls config files in containers.


  • Building Debian images for an OpenStack (private) cloud

    2012/12/23 by David Douard

    Now I have a working OpenStack cloud at Logilab, I want to provide my fellow collegues a bunch of ready-made images to create instances.

    Strangely, there are no really usable ready-made UEC Debian images available out there. There have been recent efforts made to provide Debian images on Amazon Market Place, and the toolsuite used to build these is available as a collection of bash shell scripts from a github repository. There are also some images for Eucalyptus, but I have not been able to make them boot properly on my kvm-based OpenStack install.

    So I have tried to build my own set of Debian images to upload in my glance shop.

    Vocabulary

    A bit of vocabulary may be useful for the one not very accustomed with OpenStack nor AWS jargons.

    When you want to create an instance of an image, ie. boot a virtual machine in a cloud, you generally choose from a set of ready made system images, then you choose a virtual machine flavor (ie. a combination of a number of virtual CPUs, an amount of RAM, and a harddrive size used as root device). Generally, you have to choose between tiny (1 CPU, 512MB, no disk), small (1 CPU, 2G of RAM, 20G of disk), etc.

    In the cloud world, an instance is not meant to be sustainable. What is sustainable is a volume that can be attached to a running instance.

    If you want your instance to be sustainable, there are 2 choices:

    • you can snapshot a running instance and upload it as a new image ; so it is not really a sustainable instance, instead, it's the ability to configure an instance that is then the base for booting other instances,
    • or you can boot an instance from a volume (which is the sustainable part of a virtual machine in a cloud).

    In the Amazon world, a "standard" image (the one that is instanciated when creating a new instance) is called an instance store-backed AMI images, also called an UEC image, and a volume image is called an EBS-backed AMI image (EBS stands for Elastic Block Storage). So an AMI images stored in a volume cannot be instanciated, it can be booted once and only once at a time. But it is sustainable. Different usage.

    An UEC or AMI image consist in a triplet: a kernel, an init ramdisk and a root file system image. An EBS-backed image is just the raw image disk to be booted on a virtulization host (a kvm raw or qcow2 image, etc.)

    Images in OpenStack

    In OpenStack, when you create an instance from a given image, what happens depends on the kind of image.

    In fact, in OpenStack, one can upload traditional UEC AMI images (need to upload the 3 files, the kernel, the initial ramdisk and the root filesystem as a raw image). But one can also upload bare images. These kind of images are booted directly by the virtualization host. So it is some kind of hybrid between a boot from volume (an EBS-backed boot in the Amazon world) and the traditional instanciation from an UEC image.

    Instanciating an AMI image

    When one creates an instance from an AMI image in an OpenStack cloud:

    • the kernel is copied to the virtualization host,
    • the initial ramdisk is copied to the virtualization host,
    • the root FS image is copied to the virtualization host,
    • then, the root FS image is :
      • duplicated (instanciated),
      • resized (the file is increased if needed) to the size of the asked instance flavor,
      • the file system is resized to the new size of the file,
      • the contained filesystem is mounted (using qemu-nbd) and the configured SSH acces key is added to /root/.ssh/authorized_keys
      • the nbd volume is then unmounted
    • a libvirt domain is created, configured to boot from the given kernel and init ramdisk, using the resized and modified image disk as root filesystem,
    • the libvirt domain is then booted.

    Instantiating a BARE image

    When one creates an instance from a BARE image in an OpenStack cloud:

    • the VM image file is copied on the virtualization host,
    • the VM image file is duplicated (instantiated),
    • a libvirt domain is created, configured to boot from this copied image disk as root filesystem,
    • the libvirt domain is then booted.

    Differences between the 2 instantiation methods

    Instantiating a BARE image:
    • Involves a much simpler process.
    • Allows to boot a non-linux system (depends on the virtualization system, especially true when using kvm vitualization).
    • Is slower to boot and consumes more resources, since the virtual machine image must be the size of the required/wanted virtual machine (but can remain minimal if using a qcow2 image format). If you use a 10G raw image, then 10G of data will be copied from the image provider to the virtualization host, and this big file will be duplicated each time you instantiate this image.
    • The root filesystem size corresponding to the flavor of the instance is not honored; the filesystem size is the one of the BARE images.
    Instantiating an AMI image:
    • Honours the flavor.
    • Generally allows quicker instance creation process.
    • Less resource consumption.
    • Can only boot Linux guests.

    If one wants to boot a Windows guest in OpenStack, the only solution (as far as I know) is to use a BARE image of an installed Windows system. It works (I have succeeded in doing so), but a minimal Windows 7 install is several GB, so instantiating such a BARE image is very slow, because the image needs to be uploaded on the virtualization host.

    Building a Debian AMI image

    So I wanted to provide a minimal Debian image in my cloud, and to provide it as an AMI image so the flavor is honoured, and so the standard cloud injection mechanisms (like setting up the ssh key to access the VM) work without having to tweak the rc.local script or use cloud-init in my guest.

    Here is what I did.

    1. Install a Debian system in a standard libvirt/kvm guest.

    david@host:~$ virt-install  --connect qemu+tcp://virthost/system   \
                     -n openstack-squeeze-amd64 -r 512 \
                     -l http://ftp2.fr.debian.org/pub/debian/dists/stable/main/installer-amd64/ \
                     --disk pool=default,bus=virtio,type=qcow2,size=5 \
                     --network bridge=vm7,model=virtio  --nographics  \
                     --extra-args='console=tty0 console=ttyS0,115200'
    

    This creates a new virtual machine, launch the Debian installer directly downloaded from a Debian mirror, and start the usual Debian installer in a virtual serial console (I don't like VNC very much).

    I then followed the installation procedure. When asked for the partitioning and so, I chose to create only one primary partition (ie. with no swap partition; it wont be necessary here). I also chose only "Default system" and "SSH server" to be installed.

    2. Configure the system

    After the installation process, the VM is rebooted, I log into it (by SSH or via the console), so I can configure a bit the system.

    david@host:~$ ssh root@openstack-squeeze-amd64.vm.logilab.fr
    Linux openstack-squeeze-amd64 2.6.32-5-amd64 #1 SMP Sun Sep 23 10:07:46 UTC 2012 x86_64
    
    The programs included with the Debian GNU/Linux system are free software;
    the exact distribution terms for each program are described in the
    individual files in /usr/share/doc/*/copyright.
    
    Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
    permitted by applicable law.
    Last login: Sun Dec 23 20:14:24 2012 from 192.168.1.34
    root@openstack-squeeze-amd64:~# apt-get update
    root@openstack-squeeze-amd64:~# apt-get install vim curl parted # install some must have packages
    [...]
    root@openstack-squeeze-amd64:~# dpkg-reconfigure locales # I like to have fr_FR and en_US in my locales
    [...]
    root@openstack-squeeze-amd64:~# echo virtio_baloon >> /etc/modules
    root@openstack-squeeze-amd64:~# echo acpiphp >> /etc/modules
    root@openstack-squeeze-amd64:~# update-initramfs -u
    root@openstack-squeeze-amd64:~# apt-get clean
    root@openstack-squeeze-amd64:~# rm /etc/udev/rules.d/70-persistent-net.rules
    root@openstack-squeeze-amd64:~# rm .bash_history
    root@openstack-squeeze-amd64:~# poweroff
    

    What we do here is to install some packages, do some configurations. The important part is adding the acpiphp module so the volume attachment will work in our instances. We also clean some stuffs up before shutting the VM down.

    3. Convert the image into an AMI image

    Since I created the VM image as a qcow2 image, I needed to convert it back to a raw image:

    david@host:~$ scp root@virthost:/var/lib/libvirt/images/openstack-squeeze-amd64.img .
    david@host:~$ qemu-img convert -O raw openstack-squeeze-amd64.img openstack-squeeze-amd64.raw
    

    Then, as I want a minimal-sized disk image, the filesystem must be resized to minimal. I did this like described below, but I think there are simpler methods to do so.

    david@host:~$ fdisk -l openstack-squeeze-amd64.raw  # display the partition location in the disk
    
    Disk openstack-squeeze-amd64.raw: 5368 MB, 5368709120 bytes
    149 heads, 8 sectors/track, 8796 cylinders, total 10485760 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disk identifier: 0x0001fab7
    
                       Device Boot      Start         End      Blocks   Id  System
    debian-squeeze-amd64.raw1            2048    10483711     5240832   83  Linux
    david@host:~$ # extract the filesystem from the image
    david@host:~$ dd if=openstack-squeeze-amd64.raw of=openstack-squeeze-amd64.ami bs=1024 skip=1024 count=5240832
    david@host:~$ losetup /dev/loop1 openstack-squeeze-amd64.ami
    david@host:~$ mkdir /tmp/img
    david@host:~$ mount /dev/loop1 /tmp/img
    david@host:~$ cp /tmp/img/boot/vmlinuz-2.6.32-5-amd64 .
    david@host:~$ cp /tmp/img/boot/initrd.img-2.6.32-5-amd64 .
    david@host:~$ umount /tmp/img
    david@host:~$ e2fsck -f /dev/loop1 # required before a resize
    
    e2fsck 1.42.5 (29-Jul-2012)
    Pass 1: Checking inodes, blocks, and sizes
    Pass 2: Checking directory structure
    Pass 3: Checking directory connectivity
    Pass 4: Checking reference counts
    Pass 5: Checking group summary information
    /dev/loop1: 26218/327680 files (0.2% non-contiguous), 201812/1310208 blocks
    david@host:~$ resize2fs -M /dev/loop1 # minimize the filesystem
    
    resize2fs 1.42.5 (29-Jul-2012)
    Resizing the filesystem on /dev/loop1 to 191461 (4k) blocks.
    The filesystem on /dev/loop1 is now 191461 blocks long.
    david@host:~$ # note the new size ^^^^ and the block size above (4k)
    david@host:~$ losetup -d /dev/loop1 # detach the lo device
    david@host:~$ dd if=debian-squeeze-amd64.ami of=debian-squeeze-amd64-reduced.ami bs=4096 count=191461
    

    4. Upload in OpenStack

    After all this, you have a kernel image, a init ramdisk file and a minimized root filesystem image file. So you just have to upload them to your OpenStack image provider (glance):

    david@host:~$ glance add disk_format=aki container_format=aki name="debian-squeeze-uec-x86_64-kernel" \
                     < vmlinuz-2.6.32-5-amd64
    Uploading image 'debian-squeeze-uec-x86_64-kernel'
    ==================================================================================[100%] 24.1M/s, ETA  0h  0m  0s
    Added new image with ID: 644e59b8-1503-403f-a4fe-746d4dac2ff8
    david@host:~$ glance add disk_format=ari container_format=ari name="debian-squeeze-uec-x86_64-initrd" \
                     < initrd.img-2.6.32-5-amd64
    Uploading image 'debian-squeeze-uec-x86_64-initrd'
    ==================================================================================[100%] 26.7M/s, ETA  0h  0m  0s
    Added new image with ID: 6f75f1c9-1e27-4cb0-bbe0-d30defa8285c
    david@host:~$ glance add disk_format=ami container_format=ami name="debian-squeeze-uec-x86_64" \
                     kernel_id=644e59b8-1503-403f-a4fe-746d4dac2ff8 ramdisk_id=6f75f1c9-1e27-4cb0-bbe0-d30defa8285c \
                     < debian-squeeze-amd64-reduced.ami
    Uploading image 'debian-squeeze-uec-x86_64'
    ==================================================================================[100%] 42.1M/s, ETA  0h  0m  0s
    Added new image with ID: 4abc09ae-ea34-44c5-8d54-504948e8d1f7
    
    http://www.logilab.org/file/115220?vid=download

    And that's it (!). I now have a working Debian squeeze image in my cloud that works fine:

    http://www.logilab.org/file/115221?vid=download

  • Openstack, Wheezy and ZFS on Linux

    2012/12/19 by David Douard

    Openstack, Wheezy and ZFS on Linux

    A while ago, I started the install of an OpenStack cluster at Logilab, so our developers can play easily with any kind of environment. We are planning to improve our Apycot automatic testing platform so it can use "elastic power". And so on.

    http://www.openstack.org/themes/openstack/images/open-stack-cloud-computing-logo-2.png

    I first tried a Ubuntu Precise based setup, since at that time, Debian packages were not really usable. The setup never reached a point where it could be relased as production ready, due to the fact I tried a too complex and bleeding edge configuration (involving Quantum, openvswitch, sheepdog)...

    Meanwhile, we went really short of storage capacity. For now, it mainly consists in hard drives distributed in our 19" Dell racks (generally with hardware RAID controllers). So I recently purchased a low-cost storage bay (SuperMicro SC937 with a 6Gb/s JBOD-only HBA) with 18 spinning hard drives and 4 SSDs. This storage bay being driven by ZFS on Linux (tip: the SSD-stored ZIL is a requirement to get decent performances). This storage setup is still under test for now.

    http://zfsonlinux.org/images/zfs-linux.png

    I also went to the last Mini-DebConf in Paris, where Loic Dachary presented the status of the OpenStack packaging effort in Debian. This gave me the will to give a new try to OpenStack using Wheezy and a bit simpler setup. But I could not consider not to use my new ZFS-based storage as a nova volume provider. It is not available for now in OpenStack (there is a backend for Solaris, but not for ZFS on Linux). However, this is Python and in fact, the current ISCSIDriver backend needs very little to make it work with zfs instead of lvm as "elastics" block-volume provider and manager.

    So, I wrote a custom nova volume driver to handle this. As I don't want the nova-volume daemon to run on my ZFS SAN, I wrote this backend mixing the SanISCSIDriver (which manages the storage system via SSH) and the standard ISCSIDriver (which uses standard Linux isci target tools). I'm not very fond of the API of the VolumeDriver (especially the fact that the ISCSIDriver is responsible for 2 roles: managing block-level volumes and exporting block-level volumes). This small design flaw (IMHO) is the reason I had to duplicate some code (not much but...) to implement my ZFSonLinuxISCSIDriver...

    So here is the setup I made:

    Infrastructure

    My OpenStack Essex "cluster" consists for now in:

    • one control node, running in a "normal" libvirt-controlled virtual machine; it is a Wheezy that runs:
      • nova-api
      • nova-cert
      • nova-network
      • nova-scheduler
      • nova-volume
      • glance
      • postgresql
      • OpenStack dashboard
    • one computing node (Dell R310, Xeon X3480, 32G, Wheezy), which runs:
      • nova-api
      • nova-network
      • nova-compute
    • ZFS-on-Linux SAN (3x raidz1 poools made of 6 1T drives, 2x (mirrored) 32G SLC SDDs, 2x 120G MLC SSDs for cache); for now, the storage is exported to the SAN via one 1G ethernet link.

    OpensStack Essex setup

    I mainly followed the Debian HOWTO to setup my private cloud. I mainly tuned the network settings to match my environement (and the fact my control node lives in a VM, with VLAN stuff handled by the host).

    I easily got a working setup (I must admit that I think my previous experiment with OpenStack helped a lot when dealing with custom configurations... and vocabulary; I'm not sure I would have succeded "easily" following the HOWTO, but hey, it is a functionnal HOWTO, meaning if you do not follow the instructions because you want special tunings, don't blame the HOWTO).

    Compared to the HOWTO, my nova.conf looks like (as of today):

    [DEFAULT]
    logdir=/var/log/nova
    state_path=/var/lib/nova
    lock_path=/var/lock/nova
    root_helper=sudo nova-rootwrap
    auth_strategy=keystone
    dhcpbridge_flagfile=/etc/nova/nova.conf
    dhcpbridge=/usr/bin/nova-dhcpbridge
    sql_connection=postgresql://novacommon:XXX@control.openstack.logilab.fr/nova
    
    ##  Network config
    # A nova-network on each compute node
    multi_host=true
    # VLan manger
    network_manager=nova.network.manager.VlanManager
    vlan_interface=eth1
    # My ip
    my-ip=172.17.10.2
    public_interface=eth0
    # Dmz & metadata things
    dmz_cidr=169.254.169.254/32
    ec2_dmz_host=169.254.169.254
    metadata_host=169.254.169.254
    
    ## More general things
    # The RabbitMQ host
    rabbit_host=control.openstack.logilab.fr
    
    ## Glance
    image_service=nova.image.glance.GlanceImageService
    glance_api_servers=control.openstack.logilab.fr:9292
    use-syslog=true
    ec2_host=control.openstack.logilab.fr
    
    novncproxy_base_url=http://control.openstack.logilab.fr:6080/vnc_auto.html
    vncserver_listen=0.0.0.0
    vncserver_proxyclient_address=127.0.0.1
    

    Volume

    I had a bit more work to do to make nova-volume work. First, I got hit by this nasty bug #695791 which is trivial to fix... when you know how to fix it (I noticed the bug report after I fixed it by myself).

    Then, as I wanted the volumes to be stored and exported by my shiny new ZFS-on-Linux setup, I had to write my own volume driver, which was quite easy, since it is Python, and the logic to implement was already provided by the ISCSIDriver class on the one hand, and by the SanISCSIDrvier on the other hand. So I ended with this firt implementation. This file should be copied to nova volumes package directory (nova/volume/zol.py):

    # vim: tabstop=4 shiftwidth=4 softtabstop=4
    
    # Copyright 2010 United States Government as represented by the
    # Administrator of the National Aeronautics and Space Administration.
    # Copyright 2011 Justin Santa Barbara
    # Copyright 2012 David DOUARD, LOGILAB S.A.
    # All Rights Reserved.
    #
    #    Licensed under the Apache License, Version 2.0 (the "License"); you may
    #    not use this file except in compliance with the License. You may obtain
    #    a copy of the License at
    #
    #         http://www.apache.org/licenses/LICENSE-2.0
    #
    #    Unless required by applicable law or agreed to in writing, software
    #    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
    #    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
    #    License for the specific language governing permissions and limitations
    #    under the License.
    """
    Driver for ZFS-on-Linux-stored volumes.
    
    This is mainly a custom version of the ISCSIDriver that uses ZFS as
    volume provider, generally accessed over SSH.
    """
    
    import os
    
    from nova import exception
    from nova import flags
    from nova import utils
    from nova import log as logging
    from nova.openstack.common import cfg
    from nova.volume.driver import _iscsi_location
    from nova.volume import iscsi
    from nova.volume.san import SanISCSIDriver
    
    
    LOG = logging.getLogger(__name__)
    
    san_opts = [
        cfg.StrOpt('san_zfs_command',
                   default='/sbin/zfs',
                   help='The ZFS command.'),
        ]
    
    FLAGS = flags.FLAGS
    FLAGS.register_opts(san_opts)
    
    
    class ZFSonLinuxISCSIDriver(SanISCSIDriver):
        """Executes commands relating to ZFS-on-Linux-hosted ISCSI volumes.
    
        Basic setup for a ZoL iSCSI server:
    
        XXX
    
        Note that current implementation of ZFS on Linux does not handle:
    
          zfs allow/unallow
    
        For now, needs to have root access to the ZFS host. The best is to
        use a ssh key with ssh authorized_keys restriction mechanisms to
        limit root access.
    
        Make sure you can login using san_login & san_password/san_private_key
        """
        ZFSCMD = FLAGS.san_zfs_command
    
        _local_execute = utils.execute
    
        def _getrl(self):
            return self._runlocal
        def _setrl(self, v):
            if isinstance(v, basestring):
                v = v.lower() in ('true', 't', '1', 'y', 'yes')
            self._runlocal = v
        run_local = property(_getrl, _setrl)
    
        def __init__(self):
            super(ZFSonLinuxISCSIDriver, self).__init__()
            self.tgtadm.set_execute(self._execute)
            LOG.info("run local = %s (%s)" % (self.run_local, FLAGS.san_is_local))
    
        def set_execute(self, execute):
            LOG.debug("override local execute cmd with %s (%s)" %
                      (repr(execute), execute.__module__))
            self._local_execute = execute
    
        def _execute(self, *cmd, **kwargs):
            if self.run_local:
                LOG.debug("LOCAL execute cmd %s (%s)" % (cmd, kwargs))
                return self._local_execute(*cmd, **kwargs)
            else:
                LOG.debug("SSH execute cmd %s (%s)" % (cmd, kwargs))
                check_exit_code = kwargs.pop('check_exit_code', None)
                command = ' '.join(cmd)
                return self._run_ssh(command, check_exit_code)
    
        def _create_volume(self, volume_name, sizestr):
            zfs_poolname = self._build_zfs_poolname(volume_name)
    
            # Create a zfs volume
            cmd = [self.ZFSCMD, 'create']
            if FLAGS.san_thin_provision:
                cmd.append('-s')
            cmd.extend(['-V', sizestr])
            cmd.append(zfs_poolname)
            self._execute(*cmd)
    
        def _volume_not_present(self, volume_name):
            zfs_poolname = self._build_zfs_poolname(volume_name)
            try:
                out, err = self._execute(self.ZFSCMD, 'list', '-H', zfs_poolname)
                if out.startswith(zfs_poolname):
                    return False
            except Exception as e:
                # If the volume isn't present
                return True
            return False
    
        def create_volume_from_snapshot(self, volume, snapshot):
            """Creates a volume from a snapshot."""
            zfs_snap = self._build_zfs_poolname(snapshot['name'])
            zfs_vol = self._build_zfs_poolname(snapshot['name'])
            self._execute(self.ZFSCMD, 'clone', zfs_snap, zfs_vol)
            self._execute(self.ZFSCMD, 'promote', zfs_vol)
    
        def delete_volume(self, volume):
            """Deletes a volume."""
            if self._volume_not_present(volume['name']):
                # If the volume isn't present, then don't attempt to delete
                return True
            zfs_poolname = self._build_zfs_poolname(volume['name'])
            self._execute(self.ZFSCMD, 'destroy', zfs_poolname)
    
        def create_export(self, context, volume):
            """Creates an export for a logical volume."""
            self._ensure_iscsi_targets(context, volume['host'])
            iscsi_target = self.db.volume_allocate_iscsi_target(context,
                                                                volume['id'],
                                                          volume['host'])
            iscsi_name = "%s%s" % (FLAGS.iscsi_target_prefix, volume['name'])
            volume_path = self.local_path(volume)
    
            # XXX (ddouard) this code is not robust: does not check for
            # existing iscsi targets on the host (ie. not created by
            # nova), but fixing it require a deep refactoring of the iscsi
            # handling code (which is what have been done in cinder)
            self.tgtadm.new_target(iscsi_name, iscsi_target)
            self.tgtadm.new_logicalunit(iscsi_target, 0, volume_path)
    
            if FLAGS.iscsi_helper == 'tgtadm':
                lun = 1
            else:
                lun = 0
            if self.run_local:
                iscsi_ip_address = FLAGS.iscsi_ip_address
            else:
                iscsi_ip_address = FLAGS.san_ip
            return {'provider_location': _iscsi_location(
                    iscsi_ip_address, iscsi_target, iscsi_name, lun)}
    
        def remove_export(self, context, volume):
            """Removes an export for a logical volume."""
            try:
                iscsi_target = self.db.volume_get_iscsi_target_num(context,
                                                               volume['id'])
            except exception.NotFound:
                LOG.info(_("Skipping remove_export. No iscsi_target " +
                           "provisioned for volume: %d"), volume['id'])
                return
    
            try:
                # ietadm show will exit with an error
                # this export has already been removed
                self.tgtadm.show_target(iscsi_target)
            except Exception as e:
                LOG.info(_("Skipping remove_export. No iscsi_target " +
                           "is presently exported for volume: %d"), volume['id'])
                return
    
            self.tgtadm.delete_logicalunit(iscsi_target, 0)
            self.tgtadm.delete_target(iscsi_target)
    
        def check_for_export(self, context, volume_id):
            """Make sure volume is exported."""
            tid = self.db.volume_get_iscsi_target_num(context, volume_id)
            try:
                self.tgtadm.show_target(tid)
            except exception.ProcessExecutionError, e:
                # Instances remount read-only in this case.
                # /etc/init.d/iscsitarget restart and rebooting nova-volume
                # is better since ensure_export() works at boot time.
                LOG.error(_("Cannot confirm exported volume "
                            "id:%(volume_id)s.") % locals())
                raise
    
        def local_path(self, volume):
            zfs_poolname = self._build_zfs_poolname(volume['name'])
            zvoldev = '/dev/zvol/%s' % zfs_poolname
            return zvoldev
    
        def _build_zfs_poolname(self, volume_name):
            zfs_poolname = '%s%s' % (FLAGS.san_zfs_volume_base, volume_name)
            return zfs_poolname
    

    To configure my nova-volume instance (which runs on the control node, since it's only a manager), I added these to my nova.conf file:

    # nove-volume config
    volume_driver=nova.volume.zol.ZFSonLinuxISCSIDriver
    iscsi_ip_address=172.17.1.7
    iscsi_helper=tgtadm
    san_thin_provision=false
    san_ip=172.17.1.7
    san_private_key=/etc/nova/sankey
    san_login=root
    san_zfs_volume_base=data/openstack/volume/
    san_is_local=false
    verbose=true
    

    Note that the private key (/etc/nova/sankey here) is stored in clear and that it must be readable by the nova user.

    This key being stored in clear and giving root acces to my ZFS host, I have limited a bit this root access by using a custom command wrapper in the .ssh/authorized_keys file.

    Something like (naive implementation):

    [root@zfshost ~]$ cat /root/zfswrapper
    #!/bin/sh
    CMD=`echo $SSH_ORIGINAL_COMMAND | awk '{print $1}'`
    if [ "$CMD" != "/sbin/zfs" && "$CMD" != "tgtadm" ]; then
      echo "Can do only zfs/tgtadm stuff here"
      exit 1
    fi
    
    echo "[`date`] $SSH_ORIGINAL_COMMAND" >> .zfsopenstack.log
    exec $SSH_ORIGINAL_COMMAND
    

    Using this in root's .ssh/authorized_keys file:

    [root@zfshost ~]$ cat /root/.ssh/authorized_keys | grep control
    from="control.openstack.logilab.fr",no-pty,no-port-forwarding,no-X11-forwarding, \
          no-agent-forwarding,command="/root/zfswrapper" ssh-rsa AAAA[...] root@control
    

    I had to set the iscsi_ip_address (the ip address of the ZFS host), but I think this is a result of something mistakenly implemented in my ZFSonLinux driver.

    Using this config, I can boot an image, create a volume on my ZFS storage, and attach it to the running image.

    I have to test things like snapshot, (live?) migration and so. This is a very first draft implementation which needs to be refined, improved and tested.

    What's next

    Besides the fact that it needs more tests, I plan to use salt for my OpenStack deployment (first to add more compute nodes in my cluster), and on the other side, I'd like to try the salt-cloud so I have a bunch of Debian images that "just work" (without the need of porting the cloud-init Ubuntu package).

    On the side of my zol driver, I need to port it to Cinder, but I do not have a Folsom install to test it...


  • hgview 1.2.0 released

    2010/01/21 by David Douard

    Here is at last the release of the version 1.2.0 of hgview.

    http://www.logilab.org/image/19894?vid=download

    In a nutshell, this release includes:

    • a basic support for mq extension,
    • a basic support for hg-bfiles extension,
    • working directory is now displayed as a node of the graph (if there are local modifications of course),
    • it's now possible to display only the subtree from a given revision (a bit like hg log -f)
    • it's also possible to activate an annotate view (make navigation slower however),
    • several improvements in the graph filling and rendering mecanisms,
    • I also added toolbar icons for the search and goto "quickbars" so they are not "hidden" any more to the one reluctant to user manuals,
    • it's now possible to go directly to the common ancestor of 2 revisions,
    • when on a merge node, it's now possible to choose the parent the diff is computed against,
    • make search also search in commit messages (it used to search only in diff contents),
    • and several bugfixes of course.
    Notes:
    there are packages for debian lenny, squeeze and sid, and for ubuntu hardy, interpid, jaunty and karmic. However, for lenny and hardy, provided packages won't work on pure distribs since hgview 1.2 depends on mercurial 1.1. Thus for these 2 distributions, packages will only work if you have installed backported mercurial packages.

  • Solution to a common Mercurial task

    2009/12/10 by David Douard

    An interesting question has just been sent by Greg Ward on the Mercurial devel mailing-list (as a funny coincidence, it happened that I had to solve this problem a few days ago).

    Let me quote his message:

    here's my problem: imagine a customer is running software built from
    changeset A, and we want to upgrade them to a new version, built from
    changeset B.  So I need to know what bugs are fixed in B that were not
    fixed in A.  I have already implemented a changeset/bug mapping, so I
    can trivially lookup the bugs fixed by any changeset.  (It even handles
    "ongoing" and "reverted" bugs in addition to "fixed".)
    

    And he gives an example of situation where a tricky case may be found:

                    +--- 75 -- 78 -- 79 ------------+
                   /                                 \
                  /     +-- 77 -- 80 ---------- 84 -- 85
                 /     /                        /
    0 -- ... -- 74 -- 76                       /
                       \                      /
                        +-- 81 -- 82 -- 83 --+
    

    So what is the problem?

    Imagine the lastest distributed stable release is built on rev 81. Now, I need to publish a new bugfix release based on this latest stable version, including every changeset that is a bugfix, but that have not yet been applied at revision 81.

    So the first problem we need to solve is answering: what are the revisions ancestors of revision 85 that are not ancestor of revision 81?

    Command line solution

    Using hg commands, the solution is proposed by Steve Losh:

    hg log --template '{rev}\n' --rev 85:0 --follow --prune 81
    

    or better, as suggested by Matt:

    hg log -q --template '{rev}\n' --rev 85:0 --follow --prune 81
    

    The second is better since it does only read the index, and thus is much faster. But on big repositories, this command remains quite slow (with Greg's situation, a repo of more than 100000 revisions, the command takes more than 2 minutes).

    Python solution

    Using Python, one may think about using revlog.nodesbetween(), but it won't work as wanted here, not listing revisions 75, 78 and 79.

    On the mailing list, Matt gave the most simple and efficient solution:

    cl = repo.changelog
    a = set(cl.ancestors(81))
    b = set(cl.ancestors(85))
    revs = b - a
    

    Idea for a new extension

    Using this simple python code, it should be easy to write a nice Mercurial extension (which could be named missingrevisions) to do this job.

    Then, it should be interesting to also implement some filtering feature. For example, if there are simple conventions used in commit messages, eg. using something like "[fix #1245]" or "[close #1245]" in the commit message when the changeset is a fix for a bug listed in the bugtracker, then we may type commands like:

    hg missingrevs REV -f bugfix
    

    or:

    hg missingrevs REV -h HEADREV -f bugfix
    

    to find bugfix revisions ancestors of HEADREV that are not ancestors of REV.

    With filters (bugfix here) may be configurables in hgrc using regexps.


  • hgview 1.1.0 released

    2009/09/25 by David Douard

    I am pleased to announce the latest release of hgview 1.1.0.

    What is it?

    For the ones from the back of the classroom near the radiator, let me remind you that hgview is a very helpful tool for daily work using the excellent DVCS Mercurial (which we heavily use at Logilab). It allows to easily and visually navigate your hg repository revision graphlog. It is written in Python and pyqt.

    http://www.logilab.org/image/18210?vid=download

    What's new

    • user can now configure colors used in the diff area (and they now defaults to white on black)
    • indicate current working directory position by a square node
    • add many other configuration options (listed when typing hg help hgview)
    • removed 'hg hgview-options' command in favor of 'hg help hgview'
    • add ability to choose which parent to diff with for merge nodes
    • dramatically improved UI behaviour (shortcuts)
    • improved help and make it accessible from the GUI
    • make it possible not to display the diffstat column of the file list (which can dramatically improve performances on big repositories)
    • standalone application: improved command line options
    • indicate working directory position in the graph
    • add auto-reload feature (when the repo is modified due to a pull, a commit, etc., hgview detects it, reloads the repo and updates the graph)
    • fix many bugs, especially the file log navigator should now display the whole graph

    Download and installation

    The source code is available as a tarball, or using our public hg repository of course.

    To use it from the sources, you just have to add a line in your .hgrc file, in the [extensions] section:

    hgext.hgview=/path/to/hgview/hgext/hgview.py

    Debian and Ubuntu users can also easily install hgview (and Logilab other free software tools) using our deb package repositories.


  • hgview 1.0.0 released!

    2009/06/05 by David Douard

    I am pleased to introduce you to the latest kid of the Logilab team: hgview 1.0.0.

    hgview is a very helpful tool for daily work using the excellent DVCS Mercurial (which we heavily use at Logilab). It allows to easily and visually navigate your hg repository revision graphlog. It is written in Python and pyqt.

    This version is an almost complete rewrite of hgview 0.x which had two GUI backends, gtk and qt4. This 1.0 release drops the gtk backend (we may consider reintroducing it, we haven't decided yet... by the way, patches are always welcome). Some may not like this choice, but the immediate benefit of using qt4 is that hgview works like a charm on MacOS X systems.

    http://www.logilab.org/image/9269?vid=download

    Edit: there was a bug in hgview 1.0.0 on Ubuntu hardy. It's now fixed, and I've uploaded a 1.0.1 version deb package for hardy.

    Features

    • 4 different viewers:
      • repository navigator that displays the graphlog efficiently (works well with 10,000 changesets),
      • filelog navigator that displays the filelog of a file (follows files through renames),
      • filelog diff navigator that displays the filelog in diff mode to easily track changes between two revisions of a file,
      • manifest viewer that navigates in the files hierarchy as it was at a given revision.
    • Each viewer offers:
      • easy keyboard navigation:
        • up/down to change revision,
        • left/right to change file (for the repo navigator only),
        • return to display the diff viewer of the selected file,
      • search quickbar (Ctrl+F or /): search in graphlog (search as you type in the currently displayed file or diff, plus a cancellable background search in the revision tree),
      • goto quickbar (Ctrl+G): go to the given revision (accepts id or tag, with completion for tags),
      • navigation history: alt+left/alt+right to navigate backward/forward in the history,
    • can be used alone or as a hg extension,
    • can be configured using standard hg rc files (system, user or per repository),
    • possibility to declare users (with multiple mail addresses) and assign them a given color to make a given user look the same in all your repositories,

    Download and installation

    The source code is available as a tarball, or using our public hg repository of course.

    To use it from the sources, you just have to add a line in your .hgrc file, in the [extensions] section:

    hgext.hgview=/path/to/hgview/hgext/hgview.py

    Debian and Ubuntu users can also easily install hgview (and Logilab other free software tools) using our deb package repositories.


  • qgpibplotter is (hopefully) working

    2008/09/05 by David Douard

    My latest personal project, pygpibtoolkit, holds a simple HPGL plotter trying to emulate the HP7470A GPIB plotter, using the very nice and cheap Prologix USB-GPIB dongle. This tool is (for now) called qgpibplotter (since it is using the Qt4 toolkit).

    Tonight, I took (at last) the time to make it work nicely. Well, nicely with the only device I own which is capable of plotting on the GPIB bus, my HP3562A DSA.

    Now, you just have to press the "Plot" button of your test equipment, and bingo! you can see the plot on your computer.

    http://www.logilab.org/image/5837?vid=download

  • ion, dock and screen configuration

    2008/07/04 by David Douard

    I have a laptop I use at work (with a docking station), in the train and at home (with an external display), on which my environment is ion3.

    As I use suspend-to-RAM all the time, I have added some keybindings to automatically reconfigure my screen when I plug/unplug an external display (on the dock as well as direct VGA connection).

    The lua code to paste in your .ion3/cfg_ion.lua for the bindings looks like:

    function autoscreen_on()
            local f = io.popen('/home/david/bin/autoscreen -c', 'r')
          if not f then
              return
          end
          local s = f:read('*a')
          f:close()
        ioncore.restart()
    end
    
    function autoscreen_off()
            local f = io.popen('/home/david/bin/autoscreen -d', 'r')
          if not f then
              return
          end
          local s = f:read('*a')
          f:close()
        ioncore.restart()
    end
    
    defbindings("WMPlex.toplevel", {
        bdoc("Turn on any external display and tell ion to reconfigure itself"),
        kpress(META.."F10",
               "autoscreen_on()"),
    })
    
    defbindings("WMPlex.toplevel", {
        bdoc("Turn off any external display and tell ion to reconfigure itself"),
        kpress(META.."F11",
               "autoscreen_off()"),
    })
    

    It makes use of the following python script (named /home/david/bin/autoscreen in the lua code above):

    #!/usr/bin/env python
    
    import sys
    import os
    import re
    from subprocess import Popen, PIPE
    import optparse
    parser = optparse.OptionParser("A simple automatic screen configurator (using xrandr)")
    parser.add_option('-c', '--connect', action="store_true",
                      dest='connect',
                      default=False,
                      help="configure every connected screens")
    parser.add_option('-d', '--disconnect', action="store_true",
                      dest='disconnect',
                      default=False,
                      help="unconfigure every connected screens other than LVDS (laptop screen)")
    parser.add_option('', '--main-display',
                      dest='maindisplay',
                      default="LVDS",
                      help="main display identifier (typically, the laptop LCD screen; defaults to LVDS)")
    
    options, args = parser.parse_args()
    
    if int(options.connect) + int(options.disconnect) > 1:
        print "ERROR: only one option -c or -d at a time"
        parser.print_help()
        sys.exit(1)
    
    
    xrandr = Popen("xrandr", shell=True, bufsize=0, stdout=PIPE).stdout.read()
    
    connected = re.findall(r'([a-zA-Z0-9-]*) connected', xrandr)
    connected = [c for c in connected if c != options.maindisplay]
    
    cmd = "xrandr --output %s %s"
    
    if options.connect or options.disconnect:
        for c in connected:
            if options.connect:
                action = "--auto"
            elif options.disconnect:
                action = "--off"
    
            p = Popen(cmd % (c, action), shell=True)
            sts = os.waitpid(p.pid, 0)