Now I have a working OpenStack cloud at Logilab, I want to provide my fellow collegues a bunch of ready-made images to create instances.
Strangely, there are no really usable ready-made UEC Debian images available out there. There have been recent efforts made to provide Debian images on Amazon Market Place, and the toolsuite used to build these is available as a collection of bash shell scripts from a github repository. There are also some images for Eucalyptus, but I have not been able to make them boot properly on my kvm-based OpenStack install.
So I have tried to build my own set of Debian images to upload in my glance shop.
Vocabulary
A bit of vocabulary may be useful for the one not very accustomed with OpenStack nor AWS jargons.
When you want to create an instance of an image, ie. boot a virtual machine in a cloud, you generally choose from a set of ready made system images, then you choose a virtual machine flavor (ie. a combination of a number of virtual CPUs, an amount of RAM, and a harddrive size used as root device). Generally, you have to choose between tiny (1 CPU, 512MB, no disk), small (1 CPU, 2G of RAM, 20G of disk), etc.
In the cloud world, an instance is not meant to be sustainable. What is sustainable is a volume that can be attached to a running instance.
If you want your instance to be sustainable, there are 2 choices:
- you can snapshot a running instance and upload it as a new image ; so it is not really a sustainable instance, instead, it's the ability to configure an instance that is then the base for booting other instances,
- or you can boot an instance from a volume (which is the sustainable part of a virtual machine in a cloud).
In the Amazon world, a "standard" image (the one that is instanciated when creating a new instance) is called an instance store-backed AMI images, also called an UEC image, and a volume image is called an EBS-backed AMI image (EBS stands for Elastic Block Storage). So an AMI images stored in a volume cannot be instanciated, it can be booted once and only once at a time. But it is sustainable. Different usage.
An UEC or AMI image consist in a triplet: a kernel, an init ramdisk and a root file system image. An EBS-backed image is just the raw image disk to be booted on a virtulization host (a kvm raw or qcow2 image, etc.)
Images in OpenStack
In OpenStack, when you create an instance from a given image, what happens depends on the kind of image.
In fact, in OpenStack, one can upload traditional UEC AMI images (need to upload the 3 files, the kernel, the initial ramdisk and the root filesystem as a raw image). But one can also upload bare images. These kind of images are booted directly by the virtualization host. So it is some kind of hybrid between a boot from volume (an EBS-backed boot in the Amazon world) and the traditional instanciation from an UEC image.
Instanciating an AMI image
When one creates an instance from an AMI image in an OpenStack cloud:
- the kernel is copied to the virtualization host,
- the initial ramdisk is copied to the virtualization host,
- the root FS image is copied to the virtualization host,
- then, the root FS image is :
- duplicated (instanciated),
- resized (the file is increased if needed) to the size of the asked instance flavor,
- the file system is resized to the new size of the file,
- the contained filesystem is mounted (using qemu-nbd) and the configured SSH acces key is added to /root/.ssh/authorized_keys
- the nbd volume is then unmounted
- a libvirt domain is created, configured to boot from the given kernel and init ramdisk, using the resized and modified image disk as root filesystem,
- the libvirt domain is then booted.
Instantiating a BARE image
When one creates an instance from a BARE image in an OpenStack cloud:
- the VM image file is copied on the virtualization host,
- the VM image file is duplicated (instantiated),
- a libvirt domain is created, configured to boot from this copied image disk as root filesystem,
- the libvirt domain is then booted.
Differences between the 2 instantiation methods
- Instantiating a BARE image:
- Involves a much simpler process.
- Allows to boot a non-linux system (depends on the virtualization system, especially true when using kvm vitualization).
- Is slower to boot and consumes more resources, since the virtual machine image must be the size of the required/wanted virtual machine (but can remain minimal if using a qcow2 image format). If you use a 10G raw image, then 10G of data will be copied from the image provider to the virtualization host, and this big file will be duplicated each time you instantiate this image.
- The root filesystem size corresponding to the flavor of the instance is not honored; the filesystem size is the one of the BARE images.
- Instantiating an AMI image:
- Honours the flavor.
- Generally allows quicker instance creation process.
- Less resource consumption.
- Can only boot Linux guests.
If one wants to boot a Windows guest in OpenStack, the only solution (as far as I know) is to use a BARE image of an installed Windows system. It works (I have succeeded in doing so), but a minimal Windows 7 install is several GB, so instantiating such a BARE image is very slow, because the image needs to be uploaded on the virtualization host.
Building a Debian AMI image
So I wanted to provide a minimal Debian image in my cloud, and to provide it as an AMI image so the flavor is honoured, and so the standard cloud injection mechanisms (like setting up the ssh key to access the VM) work without having to tweak the rc.local script or use cloud-init in my guest.
Here is what I did.
1. Install a Debian system in a standard libvirt/kvm guest.
david@host:~$ virt-install --connect qemu+tcp://virthost/system \
-n openstack-squeeze-amd64 -r 512 \
-l http://ftp2.fr.debian.org/pub/debian/dists/stable/main/installer-amd64/ \
--disk pool=default,bus=virtio,type=qcow2,size=5 \
--network bridge=vm7,model=virtio --nographics \
--extra-args='console=tty0 console=ttyS0,115200'
This creates a new virtual machine, launch the Debian installer directly downloaded from a Debian mirror, and start the usual Debian installer in a virtual serial console (I don't like VNC very much).
I then followed the installation procedure. When asked for the partitioning and so, I chose to create only one primary partition (ie. with no swap partition; it wont be necessary here). I also chose only "Default system" and "SSH server" to be installed.
2. Configure the system
After the installation process, the VM is rebooted, I log into it (by SSH or via the console), so I can configure a bit the system.
david@host:~$ ssh root@openstack-squeeze-amd64.vm.logilab.fr Linux openstack-squeeze-amd64 2.6.32-5-amd64 #1 SMP Sun Sep 23 10:07:46 UTC 2012 x86_64 The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Last login: Sun Dec 23 20:14:24 2012 from 192.168.1.34 root@openstack-squeeze-amd64:~# apt-get update root@openstack-squeeze-amd64:~# apt-get install vim curl parted # install some must have packages [...] root@openstack-squeeze-amd64:~# dpkg-reconfigure locales # I like to have fr_FR and en_US in my locales [...] root@openstack-squeeze-amd64:~# echo virtio_baloon >> /etc/modules root@openstack-squeeze-amd64:~# echo acpiphp >> /etc/modules root@openstack-squeeze-amd64:~# update-initramfs -u root@openstack-squeeze-amd64:~# apt-get clean root@openstack-squeeze-amd64:~# rm /etc/udev/rules.d/70-persistent-net.rules root@openstack-squeeze-amd64:~# rm .bash_history root@openstack-squeeze-amd64:~# poweroff
What we do here is to install some packages, do some configurations. The important part is adding the acpiphp module so the volume attachment will work in our instances. We also clean some stuffs up before shutting the VM down.
3. Convert the image into an AMI image
Since I created the VM image as a qcow2 image, I needed to convert it back to a raw image:
david@host:~$ scp root@virthost:/var/lib/libvirt/images/openstack-squeeze-amd64.img . david@host:~$ qemu-img convert -O raw openstack-squeeze-amd64.img openstack-squeeze-amd64.raw
Then, as I want a minimal-sized disk image, the filesystem must be resized to minimal. I did this like described below, but I think there are simpler methods to do so.
david@host:~$ fdisk -l openstack-squeeze-amd64.raw # display the partition location in the disk
Disk openstack-squeeze-amd64.raw: 5368 MB, 5368709120 bytes
149 heads, 8 sectors/track, 8796 cylinders, total 10485760 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0001fab7
Device Boot Start End Blocks Id System
debian-squeeze-amd64.raw1 2048 10483711 5240832 83 Linux
david@host:~$ # extract the filesystem from the image
david@host:~$ dd if=openstack-squeeze-amd64.raw of=openstack-squeeze-amd64.ami bs=1024 skip=1024 count=5240832
david@host:~$ losetup /dev/loop1 openstack-squeeze-amd64.ami
david@host:~$ mkdir /tmp/img
david@host:~$ mount /dev/loop1 /tmp/img
david@host:~$ cp /tmp/img/boot/vmlinuz-2.6.32-5-amd64 .
david@host:~$ cp /tmp/img/boot/initrd.img-2.6.32-5-amd64 .
david@host:~$ umount /tmp/img
david@host:~$ e2fsck -f /dev/loop1 # required before a resize
e2fsck 1.42.5 (29-Jul-2012)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/loop1: 26218/327680 files (0.2% non-contiguous), 201812/1310208 blocks
david@host:~$ resize2fs -M /dev/loop1 # minimize the filesystem
resize2fs 1.42.5 (29-Jul-2012)
Resizing the filesystem on /dev/loop1 to 191461 (4k) blocks.
The filesystem on /dev/loop1 is now 191461 blocks long.
david@host:~$ # note the new size ^^^^ and the block size above (4k)
david@host:~$ losetup -d /dev/loop1 # detach the lo device
david@host:~$ dd if=debian-squeeze-amd64.ami of=debian-squeeze-amd64-reduced.ami bs=4096 count=191461
4. Upload in OpenStack
After all this, you have a kernel image, a init ramdisk file and a minimized root filesystem image file. So you just have to upload them to your OpenStack image provider (glance):
david@host:~$ glance add disk_format=aki container_format=aki name="debian-squeeze-uec-x86_64-kernel" \
< vmlinuz-2.6.32-5-amd64
Uploading image 'debian-squeeze-uec-x86_64-kernel'
==================================================================================[100%] 24.1M/s, ETA 0h 0m 0s
Added new image with ID: 644e59b8-1503-403f-a4fe-746d4dac2ff8
david@host:~$ glance add disk_format=ari container_format=ari name="debian-squeeze-uec-x86_64-initrd" \
< initrd.img-2.6.32-5-amd64
Uploading image 'debian-squeeze-uec-x86_64-initrd'
==================================================================================[100%] 26.7M/s, ETA 0h 0m 0s
Added new image with ID: 6f75f1c9-1e27-4cb0-bbe0-d30defa8285c
david@host:~$ glance add disk_format=ami container_format=ami name="debian-squeeze-uec-x86_64" \
kernel_id=644e59b8-1503-403f-a4fe-746d4dac2ff8 ramdisk_id=6f75f1c9-1e27-4cb0-bbe0-d30defa8285c \
< debian-squeeze-amd64-reduced.ami
Uploading image 'debian-squeeze-uec-x86_64'
==================================================================================[100%] 42.1M/s, ETA 0h 0m 0s
Added new image with ID: 4abc09ae-ea34-44c5-8d54-504948e8d1f7
And that's it (!). I now have a working Debian squeeze image in my cloud that works fine:
Comments
Hello, i followed those instructions but at the end i had to add meta-datas to the image (for pointing to kernel_id and ramdisk_id) for successfully launching an intances (then 2, then 3 .... :D).
But thanks a lot, it was a very good guide.
Not sure to understand what you mean by "add meta-datas"... I specify the kernel_id and the ramdisk_id when I create the AMI images in glance (the 3rd one).
Thanks for the article. You probably mean `modprobe acpiphp'.
fixed, thanks.