Now I have a working OpenStack cloud at Logilab, I want to provide my fellow collegues a bunch of ready-made images to create instances.

Strangely, there are no really usable ready-made UEC Debian images available out there. There have been recent efforts made to provide Debian images on Amazon Market Place, and the toolsuite used to build these is available as a collection of bash shell scripts from a github repository. There are also some images for Eucalyptus, but I have not been able to make them boot properly on my kvm-based OpenStack install.

So I have tried to build my own set of Debian images to upload in my glance shop.


A bit of vocabulary may be useful for the one not very accustomed with OpenStack nor AWS jargons.

When you want to create an instance of an image, ie. boot a virtual machine in a cloud, you generally choose from a set of ready made system images, then you choose a virtual machine flavor (ie. a combination of a number of virtual CPUs, an amount of RAM, and a harddrive size used as root device). Generally, you have to choose between tiny (1 CPU, 512MB, no disk), small (1 CPU, 2G of RAM, 20G of disk), etc.

In the cloud world, an instance is not meant to be sustainable. What is sustainable is a volume that can be attached to a running instance.

If you want your instance to be sustainable, there are 2 choices:

  • you can snapshot a running instance and upload it as a new image ; so it is not really a sustainable instance, instead, it's the ability to configure an instance that is then the base for booting other instances,
  • or you can boot an instance from a volume (which is the sustainable part of a virtual machine in a cloud).

In the Amazon world, a "standard" image (the one that is instanciated when creating a new instance) is called an instance store-backed AMI images, also called an UEC image, and a volume image is called an EBS-backed AMI image (EBS stands for Elastic Block Storage). So an AMI images stored in a volume cannot be instanciated, it can be booted once and only once at a time. But it is sustainable. Different usage.

An UEC or AMI image consist in a triplet: a kernel, an init ramdisk and a root file system image. An EBS-backed image is just the raw image disk to be booted on a virtulization host (a kvm raw or qcow2 image, etc.)

Images in OpenStack

In OpenStack, when you create an instance from a given image, what happens depends on the kind of image.

In fact, in OpenStack, one can upload traditional UEC AMI images (need to upload the 3 files, the kernel, the initial ramdisk and the root filesystem as a raw image). But one can also upload bare images. These kind of images are booted directly by the virtualization host. So it is some kind of hybrid between a boot from volume (an EBS-backed boot in the Amazon world) and the traditional instanciation from an UEC image.

Instanciating an AMI image

When one creates an instance from an AMI image in an OpenStack cloud:

  • the kernel is copied to the virtualization host,
  • the initial ramdisk is copied to the virtualization host,
  • the root FS image is copied to the virtualization host,
  • then, the root FS image is :
    • duplicated (instanciated),
    • resized (the file is increased if needed) to the size of the asked instance flavor,
    • the file system is resized to the new size of the file,
    • the contained filesystem is mounted (using qemu-nbd) and the configured SSH acces key is added to /root/.ssh/authorized_keys
    • the nbd volume is then unmounted
  • a libvirt domain is created, configured to boot from the given kernel and init ramdisk, using the resized and modified image disk as root filesystem,
  • the libvirt domain is then booted.

Instantiating a BARE image

When one creates an instance from a BARE image in an OpenStack cloud:

  • the VM image file is copied on the virtualization host,
  • the VM image file is duplicated (instantiated),
  • a libvirt domain is created, configured to boot from this copied image disk as root filesystem,
  • the libvirt domain is then booted.

Differences between the 2 instantiation methods

Instantiating a BARE image:
  • Involves a much simpler process.
  • Allows to boot a non-linux system (depends on the virtualization system, especially true when using kvm vitualization).
  • Is slower to boot and consumes more resources, since the virtual machine image must be the size of the required/wanted virtual machine (but can remain minimal if using a qcow2 image format). If you use a 10G raw image, then 10G of data will be copied from the image provider to the virtualization host, and this big file will be duplicated each time you instantiate this image.
  • The root filesystem size corresponding to the flavor of the instance is not honored; the filesystem size is the one of the BARE images.
Instantiating an AMI image:
  • Honours the flavor.
  • Generally allows quicker instance creation process.
  • Less resource consumption.
  • Can only boot Linux guests.

If one wants to boot a Windows guest in OpenStack, the only solution (as far as I know) is to use a BARE image of an installed Windows system. It works (I have succeeded in doing so), but a minimal Windows 7 install is several GB, so instantiating such a BARE image is very slow, because the image needs to be uploaded on the virtualization host.

Building a Debian AMI image

So I wanted to provide a minimal Debian image in my cloud, and to provide it as an AMI image so the flavor is honoured, and so the standard cloud injection mechanisms (like setting up the ssh key to access the VM) work without having to tweak the rc.local script or use cloud-init in my guest.

Here is what I did.

1. Install a Debian system in a standard libvirt/kvm guest.

david@host:~$ virt-install  --connect qemu+tcp://virthost/system   \
                 -n openstack-squeeze-amd64 -r 512 \
                 -l \
                 --disk pool=default,bus=virtio,type=qcow2,size=5 \
                 --network bridge=vm7,model=virtio  --nographics  \
                 --extra-args='console=tty0 console=ttyS0,115200'

This creates a new virtual machine, launch the Debian installer directly downloaded from a Debian mirror, and start the usual Debian installer in a virtual serial console (I don't like VNC very much).

I then followed the installation procedure. When asked for the partitioning and so, I chose to create only one primary partition (ie. with no swap partition; it wont be necessary here). I also chose only "Default system" and "SSH server" to be installed.

2. Configure the system

After the installation process, the VM is rebooted, I log into it (by SSH or via the console), so I can configure a bit the system.

david@host:~$ ssh
Linux openstack-squeeze-amd64 2.6.32-5-amd64 #1 SMP Sun Sep 23 10:07:46 UTC 2012 x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sun Dec 23 20:14:24 2012 from
root@openstack-squeeze-amd64:~# apt-get update
root@openstack-squeeze-amd64:~# apt-get install vim curl parted # install some must have packages
root@openstack-squeeze-amd64:~# dpkg-reconfigure locales # I like to have fr_FR and en_US in my locales
root@openstack-squeeze-amd64:~# echo virtio_baloon >> /etc/modules
root@openstack-squeeze-amd64:~# echo acpiphp >> /etc/modules
root@openstack-squeeze-amd64:~# update-initramfs -u
root@openstack-squeeze-amd64:~# apt-get clean
root@openstack-squeeze-amd64:~# rm /etc/udev/rules.d/70-persistent-net.rules
root@openstack-squeeze-amd64:~# rm .bash_history
root@openstack-squeeze-amd64:~# poweroff

What we do here is to install some packages, do some configurations. The important part is adding the acpiphp module so the volume attachment will work in our instances. We also clean some stuffs up before shutting the VM down.

3. Convert the image into an AMI image

Since I created the VM image as a qcow2 image, I needed to convert it back to a raw image:

david@host:~$ scp root@virthost:/var/lib/libvirt/images/openstack-squeeze-amd64.img .
david@host:~$ qemu-img convert -O raw openstack-squeeze-amd64.img openstack-squeeze-amd64.raw

Then, as I want a minimal-sized disk image, the filesystem must be resized to minimal. I did this like described below, but I think there are simpler methods to do so.

david@host:~$ fdisk -l openstack-squeeze-amd64.raw  # display the partition location in the disk

Disk openstack-squeeze-amd64.raw: 5368 MB, 5368709120 bytes
149 heads, 8 sectors/track, 8796 cylinders, total 10485760 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0001fab7

                   Device Boot      Start         End      Blocks   Id  System
debian-squeeze-amd64.raw1            2048    10483711     5240832   83  Linux
david@host:~$ # extract the filesystem from the image
david@host:~$ dd if=openstack-squeeze-amd64.raw of=openstack-squeeze-amd64.ami bs=1024 skip=1024 count=5240832
david@host:~$ losetup /dev/loop1 openstack-squeeze-amd64.ami
david@host:~$ mkdir /tmp/img
david@host:~$ mount /dev/loop1 /tmp/img
david@host:~$ cp /tmp/img/boot/vmlinuz-2.6.32-5-amd64 .
david@host:~$ cp /tmp/img/boot/initrd.img-2.6.32-5-amd64 .
david@host:~$ umount /tmp/img
david@host:~$ e2fsck -f /dev/loop1 # required before a resize

e2fsck 1.42.5 (29-Jul-2012)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/loop1: 26218/327680 files (0.2% non-contiguous), 201812/1310208 blocks
david@host:~$ resize2fs -M /dev/loop1 # minimize the filesystem

resize2fs 1.42.5 (29-Jul-2012)
Resizing the filesystem on /dev/loop1 to 191461 (4k) blocks.
The filesystem on /dev/loop1 is now 191461 blocks long.
david@host:~$ # note the new size ^^^^ and the block size above (4k)
david@host:~$ losetup -d /dev/loop1 # detach the lo device
david@host:~$ dd if=debian-squeeze-amd64.ami of=debian-squeeze-amd64-reduced.ami bs=4096 count=191461

4. Upload in OpenStack

After all this, you have a kernel image, a init ramdisk file and a minimized root filesystem image file. So you just have to upload them to your OpenStack image provider (glance):

david@host:~$ glance add disk_format=aki container_format=aki name="debian-squeeze-uec-x86_64-kernel" \
                 < vmlinuz-2.6.32-5-amd64
Uploading image 'debian-squeeze-uec-x86_64-kernel'
==================================================================================[100%] 24.1M/s, ETA  0h  0m  0s
Added new image with ID: 644e59b8-1503-403f-a4fe-746d4dac2ff8
david@host:~$ glance add disk_format=ari container_format=ari name="debian-squeeze-uec-x86_64-initrd" \
                 < initrd.img-2.6.32-5-amd64
Uploading image 'debian-squeeze-uec-x86_64-initrd'
==================================================================================[100%] 26.7M/s, ETA  0h  0m  0s
Added new image with ID: 6f75f1c9-1e27-4cb0-bbe0-d30defa8285c
david@host:~$ glance add disk_format=ami container_format=ami name="debian-squeeze-uec-x86_64" \
                 kernel_id=644e59b8-1503-403f-a4fe-746d4dac2ff8 ramdisk_id=6f75f1c9-1e27-4cb0-bbe0-d30defa8285c \
                 < debian-squeeze-amd64-reduced.ami
Uploading image 'debian-squeeze-uec-x86_64'
==================================================================================[100%] 42.1M/s, ETA  0h  0m  0s
Added new image with ID: 4abc09ae-ea34-44c5-8d54-504948e8d1f7

And that's it (!). I now have a working Debian squeeze image in my cloud that works fine:
blog entry of