Blog entries

  • Building Debian images for an OpenStack (private) cloud

    2012/12/23 by David Douard

    Now I have a working OpenStack cloud at Logilab, I want to provide my fellow collegues a bunch of ready-made images to create instances.

    Strangely, there are no really usable ready-made UEC Debian images available out there. There have been recent efforts made to provide Debian images on Amazon Market Place, and the toolsuite used to build these is available as a collection of bash shell scripts from a github repository. There are also some images for Eucalyptus, but I have not been able to make them boot properly on my kvm-based OpenStack install.

    So I have tried to build my own set of Debian images to upload in my glance shop.

    Vocabulary

    A bit of vocabulary may be useful for the one not very accustomed with OpenStack nor AWS jargons.

    When you want to create an instance of an image, ie. boot a virtual machine in a cloud, you generally choose from a set of ready made system images, then you choose a virtual machine flavor (ie. a combination of a number of virtual CPUs, an amount of RAM, and a harddrive size used as root device). Generally, you have to choose between tiny (1 CPU, 512MB, no disk), small (1 CPU, 2G of RAM, 20G of disk), etc.

    In the cloud world, an instance is not meant to be sustainable. What is sustainable is a volume that can be attached to a running instance.

    If you want your instance to be sustainable, there are 2 choices:

    • you can snapshot a running instance and upload it as a new image ; so it is not really a sustainable instance, instead, it's the ability to configure an instance that is then the base for booting other instances,
    • or you can boot an instance from a volume (which is the sustainable part of a virtual machine in a cloud).

    In the Amazon world, a "standard" image (the one that is instanciated when creating a new instance) is called an instance store-backed AMI images, also called an UEC image, and a volume image is called an EBS-backed AMI image (EBS stands for Elastic Block Storage). So an AMI images stored in a volume cannot be instanciated, it can be booted once and only once at a time. But it is sustainable. Different usage.

    An UEC or AMI image consist in a triplet: a kernel, an init ramdisk and a root file system image. An EBS-backed image is just the raw image disk to be booted on a virtulization host (a kvm raw or qcow2 image, etc.)

    Images in OpenStack

    In OpenStack, when you create an instance from a given image, what happens depends on the kind of image.

    In fact, in OpenStack, one can upload traditional UEC AMI images (need to upload the 3 files, the kernel, the initial ramdisk and the root filesystem as a raw image). But one can also upload bare images. These kind of images are booted directly by the virtualization host. So it is some kind of hybrid between a boot from volume (an EBS-backed boot in the Amazon world) and the traditional instanciation from an UEC image.

    Instanciating an AMI image

    When one creates an instance from an AMI image in an OpenStack cloud:

    • the kernel is copied to the virtualization host,
    • the initial ramdisk is copied to the virtualization host,
    • the root FS image is copied to the virtualization host,
    • then, the root FS image is :
      • duplicated (instanciated),
      • resized (the file is increased if needed) to the size of the asked instance flavor,
      • the file system is resized to the new size of the file,
      • the contained filesystem is mounted (using qemu-nbd) and the configured SSH acces key is added to /root/.ssh/authorized_keys
      • the nbd volume is then unmounted
    • a libvirt domain is created, configured to boot from the given kernel and init ramdisk, using the resized and modified image disk as root filesystem,
    • the libvirt domain is then booted.

    Instantiating a BARE image

    When one creates an instance from a BARE image in an OpenStack cloud:

    • the VM image file is copied on the virtualization host,
    • the VM image file is duplicated (instantiated),
    • a libvirt domain is created, configured to boot from this copied image disk as root filesystem,
    • the libvirt domain is then booted.

    Differences between the 2 instantiation methods

    Instantiating a BARE image:
    • Involves a much simpler process.
    • Allows to boot a non-linux system (depends on the virtualization system, especially true when using kvm vitualization).
    • Is slower to boot and consumes more resources, since the virtual machine image must be the size of the required/wanted virtual machine (but can remain minimal if using a qcow2 image format). If you use a 10G raw image, then 10G of data will be copied from the image provider to the virtualization host, and this big file will be duplicated each time you instantiate this image.
    • The root filesystem size corresponding to the flavor of the instance is not honored; the filesystem size is the one of the BARE images.
    Instantiating an AMI image:
    • Honours the flavor.
    • Generally allows quicker instance creation process.
    • Less resource consumption.
    • Can only boot Linux guests.

    If one wants to boot a Windows guest in OpenStack, the only solution (as far as I know) is to use a BARE image of an installed Windows system. It works (I have succeeded in doing so), but a minimal Windows 7 install is several GB, so instantiating such a BARE image is very slow, because the image needs to be uploaded on the virtualization host.

    Building a Debian AMI image

    So I wanted to provide a minimal Debian image in my cloud, and to provide it as an AMI image so the flavor is honoured, and so the standard cloud injection mechanisms (like setting up the ssh key to access the VM) work without having to tweak the rc.local script or use cloud-init in my guest.

    Here is what I did.

    1. Install a Debian system in a standard libvirt/kvm guest.

    david@host:~$ virt-install  --connect qemu+tcp://virthost/system   \
                     -n openstack-squeeze-amd64 -r 512 \
                     -l http://ftp2.fr.debian.org/pub/debian/dists/stable/main/installer-amd64/ \
                     --disk pool=default,bus=virtio,type=qcow2,size=5 \
                     --network bridge=vm7,model=virtio  --nographics  \
                     --extra-args='console=tty0 console=ttyS0,115200'
    

    This creates a new virtual machine, launch the Debian installer directly downloaded from a Debian mirror, and start the usual Debian installer in a virtual serial console (I don't like VNC very much).

    I then followed the installation procedure. When asked for the partitioning and so, I chose to create only one primary partition (ie. with no swap partition; it wont be necessary here). I also chose only "Default system" and "SSH server" to be installed.

    2. Configure the system

    After the installation process, the VM is rebooted, I log into it (by SSH or via the console), so I can configure a bit the system.

    david@host:~$ ssh root@openstack-squeeze-amd64.vm.logilab.fr
    Linux openstack-squeeze-amd64 2.6.32-5-amd64 #1 SMP Sun Sep 23 10:07:46 UTC 2012 x86_64
    
    The programs included with the Debian GNU/Linux system are free software;
    the exact distribution terms for each program are described in the
    individual files in /usr/share/doc/*/copyright.
    
    Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
    permitted by applicable law.
    Last login: Sun Dec 23 20:14:24 2012 from 192.168.1.34
    root@openstack-squeeze-amd64:~# apt-get update
    root@openstack-squeeze-amd64:~# apt-get install vim curl parted # install some must have packages
    [...]
    root@openstack-squeeze-amd64:~# dpkg-reconfigure locales # I like to have fr_FR and en_US in my locales
    [...]
    root@openstack-squeeze-amd64:~# echo virtio_baloon >> /etc/modules
    root@openstack-squeeze-amd64:~# echo acpiphp >> /etc/modules
    root@openstack-squeeze-amd64:~# update-initramfs -u
    root@openstack-squeeze-amd64:~# apt-get clean
    root@openstack-squeeze-amd64:~# rm /etc/udev/rules.d/70-persistent-net.rules
    root@openstack-squeeze-amd64:~# rm .bash_history
    root@openstack-squeeze-amd64:~# poweroff
    

    What we do here is to install some packages, do some configurations. The important part is adding the acpiphp module so the volume attachment will work in our instances. We also clean some stuffs up before shutting the VM down.

    3. Convert the image into an AMI image

    Since I created the VM image as a qcow2 image, I needed to convert it back to a raw image:

    david@host:~$ scp root@virthost:/var/lib/libvirt/images/openstack-squeeze-amd64.img .
    david@host:~$ qemu-img convert -O raw openstack-squeeze-amd64.img openstack-squeeze-amd64.raw
    

    Then, as I want a minimal-sized disk image, the filesystem must be resized to minimal. I did this like described below, but I think there are simpler methods to do so.

    david@host:~$ fdisk -l openstack-squeeze-amd64.raw  # display the partition location in the disk
    
    Disk openstack-squeeze-amd64.raw: 5368 MB, 5368709120 bytes
    149 heads, 8 sectors/track, 8796 cylinders, total 10485760 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disk identifier: 0x0001fab7
    
                       Device Boot      Start         End      Blocks   Id  System
    debian-squeeze-amd64.raw1            2048    10483711     5240832   83  Linux
    david@host:~$ # extract the filesystem from the image
    david@host:~$ dd if=openstack-squeeze-amd64.raw of=openstack-squeeze-amd64.ami bs=1024 skip=1024 count=5240832
    david@host:~$ losetup /dev/loop1 openstack-squeeze-amd64.ami
    david@host:~$ mkdir /tmp/img
    david@host:~$ mount /dev/loop1 /tmp/img
    david@host:~$ cp /tmp/img/boot/vmlinuz-2.6.32-5-amd64 .
    david@host:~$ cp /tmp/img/boot/initrd.img-2.6.32-5-amd64 .
    david@host:~$ umount /tmp/img
    david@host:~$ e2fsck -f /dev/loop1 # required before a resize
    
    e2fsck 1.42.5 (29-Jul-2012)
    Pass 1: Checking inodes, blocks, and sizes
    Pass 2: Checking directory structure
    Pass 3: Checking directory connectivity
    Pass 4: Checking reference counts
    Pass 5: Checking group summary information
    /dev/loop1: 26218/327680 files (0.2% non-contiguous), 201812/1310208 blocks
    david@host:~$ resize2fs -M /dev/loop1 # minimize the filesystem
    
    resize2fs 1.42.5 (29-Jul-2012)
    Resizing the filesystem on /dev/loop1 to 191461 (4k) blocks.
    The filesystem on /dev/loop1 is now 191461 blocks long.
    david@host:~$ # note the new size ^^^^ and the block size above (4k)
    david@host:~$ losetup -d /dev/loop1 # detach the lo device
    david@host:~$ dd if=debian-squeeze-amd64.ami of=debian-squeeze-amd64-reduced.ami bs=4096 count=191461
    

    4. Upload in OpenStack

    After all this, you have a kernel image, a init ramdisk file and a minimized root filesystem image file. So you just have to upload them to your OpenStack image provider (glance):

    david@host:~$ glance add disk_format=aki container_format=aki name="debian-squeeze-uec-x86_64-kernel" \
                     < vmlinuz-2.6.32-5-amd64
    Uploading image 'debian-squeeze-uec-x86_64-kernel'
    ==================================================================================[100%] 24.1M/s, ETA  0h  0m  0s
    Added new image with ID: 644e59b8-1503-403f-a4fe-746d4dac2ff8
    david@host:~$ glance add disk_format=ari container_format=ari name="debian-squeeze-uec-x86_64-initrd" \
                     < initrd.img-2.6.32-5-amd64
    Uploading image 'debian-squeeze-uec-x86_64-initrd'
    ==================================================================================[100%] 26.7M/s, ETA  0h  0m  0s
    Added new image with ID: 6f75f1c9-1e27-4cb0-bbe0-d30defa8285c
    david@host:~$ glance add disk_format=ami container_format=ami name="debian-squeeze-uec-x86_64" \
                     kernel_id=644e59b8-1503-403f-a4fe-746d4dac2ff8 ramdisk_id=6f75f1c9-1e27-4cb0-bbe0-d30defa8285c \
                     < debian-squeeze-amd64-reduced.ami
    Uploading image 'debian-squeeze-uec-x86_64'
    ==================================================================================[100%] 42.1M/s, ETA  0h  0m  0s
    Added new image with ID: 4abc09ae-ea34-44c5-8d54-504948e8d1f7
    
    http://www.logilab.org/file/115220?vid=download

    And that's it (!). I now have a working Debian squeeze image in my cloud that works fine:

    http://www.logilab.org/file/115221?vid=download

  • About salt-ami-cloud-builder

    2013/06/07 by Paul Tonelli

    What

    At Logilab we are big fans of SaltStack, we use it quite extensivelly to centralize, configure and automate deployments.

    http://www.logilab.org/file/145398/raw/SaltStack-Logo.png

    We've talked on this blog about how to build a Debian AMI "by hand" and we wanted to automate this fully. Hence the salt way seemed to be the obvious way to go.

    So we wrote salt-ami-cloud-builder. It is mainly glue between existing pieces of software that we use and like. If you already have some definition of a type of host that you provision using salt-stack, salt-ami-cloud-builder should be able to generate the corresponding AMI.

    http://www.logilab.org/file/145397/raw/open-stack-cloud-computing-logo-2.png

    Why

    Building a Debian based OpenStack private cloud using salt made us realize that we needed a way to generate various flavours of AMIs for the following reasons:

    • Some of our openstack users need "preconfigured" AMIs (for example a Debian system with Postgres 9.1 and the appropriate Python bindings) without doing the modifications by hand or waiting for an automated script to do the job at AMI boot time.
    • Some cloud use cases require that you boot many (hundreds for instance) machines with the same configuration. While tools like salt automate the job, waiting while the same download and install takes place hundreds of times is a waste of resources. If the modifications have already been integrated into a specialized ami, you save a lot of computing time. And especially in the Amazon (or other pay-per-use cloud infrastructures), these resources are not free.
    • Sometimes one needs to repeat a computation on an instance with the very same packages and input files, possibly years after the first run. Freezing packages and files in one preconfigured AMI helps this a lot. When relying only on a salt configuration the installed packages may not be (exactly) the same from one run to the other.

    Relation to other projects

    While multiple tools like build-debian-cloud exist, their objective is to build a vanilla AMI from scratch. The salt-ami-cloud-builder starts from such vanilla AMIs to create variations. Other tools like salt-cloud focus instead on the boot phase of the deployment of (multiple) machines.

    Chef & Puppet do the same job as Salt, however Salt being already extensively deployed at Logilab, we continue to build on it.

    Get it now !

    Grab the code here: http://hg.logilab.org/master/salt-ami-cloud-builder

    The project page is http://www.logilab.org/project/salt-ami-cloud-builder

    The docs can be read here: http://docs.logilab.org/salt-ami-cloud-builder

    We hope you find it useful. Bug reports and contributions are welcome.

    The logilab-salt-ami-cloud-builder team :)