Questions and Answers

Table of contents

Who should I contact in case information or additional help is needed?

Please send any question to, also referred to as service administrator in this document. Alternatively, you can use the LRZ Service Desk and open a ticket.

How do I access the cloud platform?

The easiest way is to use the web GUI Sunstone, available at More details are available in this section.

My virtual machine does not boot

Please check the section dedicated to the status of virtual machines to understand if too many resources have been booked. As a general rule, the Log tab in the Virtual Machines view should be checked for details.

What are the resource limits?

Please refer to this section to learn about the resources a single VM or a project group can book.

All my resources are used up, but I powered off all my VMs. How can this be?

From the physical machines under your desk you are used to the fact that you use the command shutdown now or poweroff to really shut down your machines and afterwards they are really powered off (but they are still physically there, you still own them). When using virtual machines, things are a bit different. Here is why:

When you call poweroff in a virtual machine, it will shut down and be unreachable, but ONE does not know that the machine has been shut down. For ONE it still exists, it still uses an IP, it still uses your CPUs, RAM, etc., and you are still being charged for all of this just the same way as if your VM was still running, because no other user can access these resources: they are still allocated to you, you still own this VM! In short: this is a very unfavourable state and you should really avoid it, for your own good and for the good of the Cloud. Do never use poweroff or shutdown inside your VM! Instead use the stop or undeploy buttons in the ONE frontend; this will free your resources, you no longer own the VM, other users can use your resources (but your hard disk will be preserved), and you will not be charged any more. The only safe command is the reboot command; you can use this without problems from inside your VM.

Once you shut down a physical computer, you have to manually push the power-on button on the device to bring it back to life. There is no such button on a VM! In other words: you can not power on again a VM that has been shutdown! It is lost! And continues to use your resources. All you can now do is to delete your VM for good.

"But there is the 'play' button in ONE, doesn't this restart my powered off machine?" you will ask. The answer is no, because this button will only resume/restart VMs that have been suspended/powered off/stopped/undeployed from the ONE frontend.

An overview over the various states of a VM is given in the manual here

The VNC window does not open

The rationale behind VNC and the correct setup is explained here. Anyhow, a very common problem is the firewall on the client side. Though we chose a port in the privileged range (below 1024, namely we use port 600), in certain case it is filtered as well. In case of problem, please point your browser to the URL The following page should be seen (it could be necessary, though unlikely, to accept the certificate):

OpenNebula noVNC error

If a blank page, or any other error, shows up, then the port is blocked. Please check your firewall or ask your network admnistrator. If connected to the MWN through a laptop (or any other device equipped with a WLAN card), a very quick solution is to switch to Eduroam, after unplugging the cable. On Eduroam, port 600 is free, so the given error message should appear and it should be possible to start the VNC session from within the web GUI.

The VM can not detect the disk and/or the network card I just attached

Disk(s) and virtual network card(s) can be added on the fly to existing VMs. This operation is performed by ONE through the hypervisor. Nonetheless, the guest OS running in the VM should be aware of the change. In order to force the periodic scan of the PCI bus, please log into the VM (via VNC and/or SSH) and issue the following command:

echo 1 > /sys/bus/pci/rescan

as user root. In order to find the the block device used for the new virtual hardware, please consult the logs (i.e. use the dmesg command).

My disk image is in an error state, what can I do?

If a virtual machine is not properly shut down, the file system on the disk images mounted there could be corrupted, hence the error state. This situation can be solved by opening the Images view, selecting the image in an error state, and choose the Enable option from the More drop down menu.

OpenNebula enable image option

If the image is mounted autmatically when the virtual machine boots, then the guest operating system will check the filesystem for errors. In case the disk image is not mounted at boot time by the virtual machine, it is a good idea to check it right away after the startup phase.

How do I connect my virtual machine to a network?

The section dedicated to Virtual Networks contains all the details about the available subnets.

How to configure a network interface of a VM

As a general rule, the interface eth0 is associated to the first network card defined in the template, eth1 to the second one and so on, unless the udev facility of the VM is changing the criteria.

If the VM's template contains only one interface, then the DHCP assignment works out of the box. With two or more interfaces, there's a problem. Every interface tries to set the default route, but there can be only one. So, the first interface to be set up wins, and it's not given that it is the first one specified in the template. The solution consists in specifying the priority, or metric, of the interfaces. Each one sets the default gateway given by the DHCP but with a different priority.

As already described, there are two virtual networks defined, private (MWN_access) and public. (Internet_access). If you plan to have both, then give maximum priority (or lowest metric) to the public one, otherwise the VM won't be reachable. If you don't specify anything, then the priority is maximum and the metric is the minimum. If the VM has only network card, then there is no need to specify a metric.

Now, some details on how to configure the network card of a VM. We consider here some well-know Linux distributions.

SUSE: for each interface there is a script named /etc/sysconfig/network/ifcfg-<if name>, such as ifcfg-eth0, ifcfg-eth1 and so on. The content should be

   DEVICE=eth0 #this is an example, change to eth1 in ifcfg-eth1 ...

The value of the DEVICE directive should be adapted accordingly (i.e., eth1) while  DHCLIENT_PRIMARY_DEVICE should be changed to no, then save the new file in a consistent way, i.e., ifcfg-eth1.

Remember: the device attached to the public network needs DHCLIENT_PRIMARY_DEVICE=yes, for all the others it should be set to no.

Fedora and derivatives: the approach is very similar to SUSE. For each interface there is a script located in /etc/sysconfig/network-scripts, named ifcfg-<inteface name>, such as /etc/sysconfig/network-scripts/ifcfg-eth0 and so on.

The content should be


For every other interface, the DEVICE and the file name should be adapted, while the parameter METRIC=1 should be added. METRIC increases for every new interface. A higher metric means a decreasing priority of the route associated to that interface. The public interface has to be the one without the METRIC parameter, that is the one whose default route is the preferred.

Debian and derivatives: Debian and Ubuntu have only one file, /etc/network/interfaces. A typical example is

   auto lo
   iface lo inet loopback
   auto eth0
   iface eht0 inet dhcp

It is obvious that the loopback interface and eth0 are brought up automatically at boot (auto) and then configured.

For each additional interface, the auto statement, the iface definition, and the entry

   metric 1

should be added.

Increase the metric for each interface. The public interface, once again, should have no metric definition (it is 0 by default, identifying the default gateway, too). Note: there is a bug in Debian 7 which prevents the hostname setup via DHCP. In order to fix it, it is necessary to create in the guest the file /etc/dhcp/dhclient-exit-hooks.d/hostname with the following content:

# Update hostname from DHCP
if [ "$reason" != BOUND ] && [ "$reason" != RENEW ] && [ "$reason" != REBIND ] && [ "$reason" != REBOOT ]

hostname $new_host_name
echo $new_host_name > /etc/hostname

Is it possible to resize a Virtual Machine?

Yes, it is. First of all, undeploy your VM. The resources (namely, the worker node) will not be freed but the VM state will be lost, so the VM will boot from scratch again, at resume time.

Once the VM reaches the UNDEPLOYED status, then click on it to open the window with all its details, as in the following picture.

OpenNebula resize VM dialogue box

In the Capacity tab the button Resize is now available.

Click on it and a dialogue box very similar to the General section of the Template view will open, where it is possible to modify the number of CPUs, the number of Virtual CPUs and/or the RAM memory assigned to the VM.

OpenNebula resize VM dialogue box

Once done, click on the green Resize button and restart the VM by means of the Play button.

In case the worker node does not have enough resources (i.e., you asked for too many CPUs or too much memory), a warning message will be shown. In this case, the only alternative is to shut down the machine and reschedule it with the new requirements.

This topic is also explained by the official ONE documentation.

How can I reduce the boot time of the VMs?

See next question for answer.

What is the correct way to manage the disk images?

See next question for answer.

When should I use a volatile disk and when an image in the datastore?

Most of the time needed to have a running VM is spent copying images from the image datastore to the system datastore, where the worker nodes put the disk images of the running VM(s). As already mentioned, the two datastores (the image repository and the system space) are NFS mounts: the copy operation involves a network transfer, though on a dedicated link, but still subject to (eventual) traffic congestion and queues. From this it is clear that:

  • using a compressed format, such as qcow2, for images is a must. Smaller files mean reduced transfer times;
  • every image that is picked from the datastore brings on a file transfer, so volatile disks should be used every time that it is possible.

From the storage point of view, in a VM we can recognize four components: the OS (including the kernel, the basic system, all the executables and applications, ...), the swap partition, a scratch space (for temporary results together with input and output sets) and a long term archive (longer than the lifetime of the VM). In more details, few guidelines on how to fulfill in a wise way the different needs just mentioned:

OS: this is the core of the VM, including, among the others, the bootable files, the kernel, the basic commands, the executables needed for the jobs. This is a typical case of an image kept in the datastore and all the considerations previously exposed apply. The image type is OS and the qcow2 compressed format is highly recommended. The suggested (uncompressed) size for the image is between 5 GB and 10 GB, taking into account that a typical command line only Linux installation only need 2 GB, just the double for a basic graphic system. For sure the image should not contain any swap partition and also the /home folder should be kept as small as possible, just for the configuration files of the applications, eventually. Please remember also that the remarks mentioned when discussing the advanced options of images still apply. Namely, the default bus is virtio (instead of IDE), it is good practice to specify vda as TARGET (since this is the boot drive), DRIVER has to be left blank, the CACHE is none if nothing specified (as suggested) and the IO model is native. This image can be created offline, i. e. on the user's pc using a virtualization platform such as VMWare rather than Virtual Box or the Linux's desktop application Virt Manager. The resulting file can be then converted and compressed in qcow2 format to be uploaded to ONE (minding to remove the unnecessary partitions). An alternative could be:

  • add to the datastore an installation CD of the preferred Linux distribution. When creating an image in the datastore, it is as easy as selecting CDROM as image type and choosing the Upload button for the image location, typing or pasting the address where to download the bootable CD in the path field.
  • create an empty disk (or data block image) in the datastore (5 GB to 10 GB), mainly as persistent and specifying vda as TARGET.
  • add both to a template, specifying the CD as boot device (OS Booting section of the template, Boot dropdown menu). Please note that ONE expects to find the boot CD on the first channel of the IDE bus, so, in order to avoid confusion, in the Storage section of the templace referring to the CDROM image, it is safer to enter hda as Target.
  • follow the installation procedure, partitioning the disk without a swap space, that will be added later.
  • after testing, remove the installation CD image from the template and switch off the persistency flag from the image where the system has just been installed.

There is a tutorial available. It is mainly target to Windows, but the procedure to install a Linux OS is very similar: the turning points are highlighted by the label Hint for Linux users.

swap: this is the typical case of a volatile disk. It makes no sense to include this partition in the OS image since it can be created on the fly on the worker node. It is only necessary to include it in the template definition

OpenNebula template add volatile swap

For clarity, being the second disk attached to the VM, it is advisable to specify vdb as TARGET in the advanced options. In order to mount the swap automatically the /etc/fstab file of the VM (in the OS image) should be updated accordingly, for example by means of the following line

/dev/vdb none            swap    sw              0       0

This can be done when the OS image is created or lately, for example during the configuration of the VM, with the persistent flag activated.

scratch space: this is the work space, where to place files at runtime. The suggestion is to use again a volatile disk

OpenNebula template add volatile disk

entering the size and selecting the file system as needed. As seen for the swap partition, also a volatile disk with a file system can be auto mounted by the VM, editing the /etc/fstab file accordingly. For example, supposing that the volatile disk is vdc (always a good idea to specify it in the TARGET field of the advanced options), a consistent entry would be:

/dev/vdc /media           ext3    defaults        0       2

Please remember that disk snapshot operations do not apply to volatile disks.

One could be also tempted to use a datastore image (DATABLOCK type) as scratch space.

It is possible, but beware the fact that it has to be deployed on the worker node, and, if set as persistent, it is also copied back at the end of the VM's lifetime.

Long term storage: the LRZ cloud is a computational facility and it is not really meant to store files beyond the life time of the VMs. This does not mean that the users are forbidden to do that, but the impact on the overall experience would be high, experiencing long delays in the deployment of the VMs (as explained in the previous sub-paragraph). At the moment, LRZ does not provide any form of dedicated long term storage, such as a NAS mount, even though it is planned as a later service. What we can suggest so far is to move your results into our TSM (tape) facility or to the LRZ Linux cluster (usually the accounts which are Cloud enabled also have access to the Cluster platform). Users can choose between the classic scp/sftp tools or use the higher performing GridFTP protocol. The latter case requires the installation of additional software in the VM, such as the globus-ftp-client-progs for command line interaction with a GridFTP server or Globus Connect personal for Globus Online. Please note that LRZ can provide its customers with a temporary certificate valid for the usage of its grid resources.

The upload of VM images is very slow

The upload of a VM image form the user's computer can be a time consuming (i.e., up to one hour for a 5 GB file) and unclear process. Here we'll try to give some insights and suggestions.

As already explained, the dialogue box to upload an image is accessible via the Images menu (on the left pane), clicking on the green button (+) at the top left corner. In this particular case, we suppose that the user selected Upload as Image location (in the middle of the window) and a file on user's computer was already picked by means of the Choose button, as shown in the following picture.

OpenNebula upload image file

After clicking on the Create button, the upload will start. The user is brought back to the list of images and a progress bar will appear at the bottom.

OpenNebula upload image progress bar

The transfer consists of 3 different parts:

  • upload to the frontend machine: the file is moved to the web server where Sunstone is running, and stored in a buffer. The progress bar is moving to the right end;
  • file dumping: the content of the file is moved into a file. This is transparent to the user and the progress bar is stuck at the right edge;
  • registration into the datastore: the file is moved to the datastore, via NFS. The progress bar is still present.

During the 3 phases, the user should not close the browser, otherwise the operation will fail. It is obvious that a direct upload of a local file is convenient for small images, i.e., no larger than a few gigabytes. As an alternative for larger images we suggest to upload the file directly to the frontend machine ( and then register the image from there.

Each user has a folder in /media/scratch named after his/her account, for example /media/scratch/regularuser. Files can be uploaded there, via GridFTP. The address to use is The GridFTP server (listening on the default 2811 port) can be accessed via command line, installing the package globus-ftp-client-progs, or by means of Globus Connect personal for Globus Online (the endpoint is lrz#ONE). The LRZ temporary certificates are also accepted. Please note that:

  • users should not remove read permissions for all from the uploaded files because ONE is running under an unprivileged account. The permissions of /media/scratch prevent other users than the owner and the ONE account to access the files in the subfolders. In other words, a user can't access the content of someone else's folder and the files uploaded are kept private. This also means that /media/scratch can not be used as a share area to exchange files;
  • files older than 24 hours are deleted automatically, so that users do not have to cleanup their temporary area.

Once the image is in place, it can be registered via Sunstone. It is only necessary to choose Provide a path as Image location in the Create Image dialogue box, as shown in the image.

OpenNebula upload image path

The path should point to the file in the scratch area, for example /media/scratch/regularuser/fc18.qcow2. After the green Create button is pressed, Sunstone returns to the list of images where the new image will appear in the status LOCKED, while it's being moved to the datastore. The user can logout anytime, there is no need to wait for the image to switch to the READY state.

How can I change the root password of my VM?

If the root (or any other user) password of a VM has been forgotten, there are a couple of methods to recover it. This also applies in case the user did not have the possibility to setup the root and user password during the installation of the guest OS, for example because employing an image prepared by someone else. What is explained here is an emergency procedure in case SSH key injection for root does not work (for example, the SSH daemon is not running or root access via SSH has been disabled and the user has no other possibility to take control of the VM).

Offline method

A warning first: in this section we are asking you to work on your own machine, giving commands as root, which could be potentially destructive for your system. We cannot be held accountable in case of damage or loss of files. If, after reading this paragraph, you do not feel comfortable in doing what you are asked but you still wish to work offline (or maybe it is the only possibility), then it is always possible to boot the VM image locally by means of KVM or Virtual Box and boot it in single user mode as described in the Online section. If unsure, please contact us and provide more information.

The basic assumption here is that the user has the VM image in his/her own local Linux PC (this prerequisite is fundamental since many Linux tools are going to be used). The basic idea is to mount the VM image locally as a loop device and then work on it in order to change the password.

The very first step is to mount the VM image. Most likely the following commands should be issued as root, or at least by means of sudo.

If the format is (uncompressed) raw, then the kpartx and losetup tools are going to be used. Please refer to the software manager of your Linux distribution to find out the name of the packages to be installed.

Assuming that ttylinux.img is the name of the local file containing the VM image, please start with the following two commands

kpartx -a ttylinux.img 
kpartx -l ttylinux.img

The output (only the latter has one) is something like

loop0p1 : 0 81585 /dev/loop0 63

stating the loop device kpartx is using. The message shows that the first available loop device (automatically picked) is /dev/loop0, in particular the first partition found in the VM image (the only one present) will be assigned to /dev/mapper/loop0p1 (please check the documentation of your Linux distribution to be sure about the full path, this is implementation dependent). Should the VM image contain multiple partitions, the second one will be accessible via /dev/mapper/loop0p2 and so on.

It is now possible to access the filesystem on the VM image. Use

fdisk -l /dev/loop0

to inspect the partitions, choose those whose Id is 83 and mount them till the / (or root) partition is found. The mount operation is performed as usual, for example, to mount the first partition of the VM disk in the local /mnt folder, just type

mount /dev/mapper/loop0p1 /mnt

There is only one exception to this rule, if the VM image uses logical volumes (or LVM). The partition type is 8e and it is necessary to activate the partition before mounting it (vgchange -ay) and to use the logical name of the volume (shown by lvs) instead of those of the block device.

If everything is correct, ls -lh /mnt will show the content of the first parition of the VM image. The remaining instructions will be given after dealing with the qcow2 image format.

If the image is in qcow2 format, it is not necessary to convert it to raw, it is possible to work on that on the fly by means of the qemu-nbd tool.

First of all, it is necessary to load the network block device (nbd) module typing modprobe nbd max_part=15. The max_part (number of partitions per block device) option is fundamental since by default it is 0 and it would result in a failure when trying to mount the image. Then, the VM image should be connected to a network block device:

qemu-nbd -c /dev/nbd0 ttylinux.qcow2 

where ttylinux.qcow2 is the file containing the VM filesystem and /dev/nbd0 is one of the network block devices created when the kernel module has been loaded. The devices are numbered through 0 to 15 (given the value assigned to max_part) and the target has to be chosen manually. In case the block device is in use, qemu-nbd returns an error and it will be necessary to choose the next one.

The first partition of the VM image is available under /dev/nbd0p1, the second one uses /dev/nbd0p2 and so on. Once again,

fdisk -l /dev/nbd0

shows the available partitions. The goal is still to identify the / partition of the VM and mount it, for example, if the choice is the first one

mount /dev/nbd0p1 /mnt

Now that the VM disk image is accessible on the local platform, it is possible to update it. Instead of manually modfying the /mnt/etc/shadow file (password are not in clear text, they're rather hashed or encrypted), it is advisable to use chroot. Entering

chroot /mnt

a new shell session will open but the root of the filesystem is now /mnt, that is the root of the VM. At this point all commands issued will affect only the filesystem of the VM (the image mounted) and in fact the shell uses the operating system of the VM. It is clear that a passwd root will affect the root user of the VM. At the same time, it is possible to add new users to the VM simply typing

adduser user1

This approach minimises the errors in tampering with the shadow file, given that the user is in the root jail (i.e., the chroot command has been issued), so please be sure that you are not affecting your own system.

Once all the changes have been carried out, type


to end the chroot session and return to the local machine, then unmount the VM disk

umount /mnt

to have a clean file system. In case the VM has a LVM disk, then the logical volume(s) should be deactivated (vgchange -an <volume name>). Please note that fumbling with LVM without a clear idea of what one is doing can be potentially destructive, so, please, read a guide first if unsure or ask for more detailed help. Last but not least, unbind the image from the loop device:

kpartx -d ttylinux.img

if the image is in raw format, or

qemu-nbd -d /dev/nbd0

for the qcow2 case (please change the target network block device according to what you used during the binding operation).

At this point the image can be transferred to OpenNebula and instantiated.

Online method - Single user mode

The first solution we propose is the so called single user mode. The VM has to be booted, or rebooted, in order to gain full privileged access without being asked for the root password. This is a standard feature of GRUB 2, nowadays one of the most adopted bootloaders by many Linux distributions. We will limit our discussion to this case, nonetheless, many guides are available for other softwares, such as LILO.

Please note that if the VM image is not persistent or a disk snapshot is not taken, then the change (i.e., the new root password) is not permanent, so make the disk persistent!

First of all, boot or reboot the VM whose password has to be recovered. Shortly after, the user is presented with a menu from which he can choose the kernel to start, as in the picture.

GRUB 2 boot menu

Usually there is a time of few seconds (5 or 10) after which a default choice is made. In this interval, the user can interact with the bootloader. In particular, the kernel entry can be edited. Select the default option (usually the first one) or use the arrow keys to highlight the desired kernel. Then, press e to enter the edit mode. The new screen details the options to boot the operating system, such as

echo    'Loading Linux 2.6-32-686-pae ...'
linux   /boot/vmlinuz-2.6-32-686-pae root=UUID=2ee67dfe-875f-4cf1-b972-b3f27fc8cfda ro  video=i915 modeset=0 pci=msi
echo    'Loading initial ramdisk ...'
initrd  /boot/initrd.img-2.6-32-686-pae

The important line to look for is the one starting with the keywork linux. Move the cursor at its end and append the following text: init=/bin/bash, so that it looks like

linux   /boot/vmlinuz-2.6-32-686-pae root=UUID=2ee67dfe-875f-4cf1-b972-b3f27fc8cfda ro  video=i915 modeset=0 pci=msi init=/bin/bash

Hit F10 or the combination Ctrl and x. This can slightly vary, just follow the instructions at the bottom of the bootloader screen. The changes will take effect only once, in other words they are not saved in the bootloader menu.

The VM will reboot and after a short while the root prompt will be given. If needed, remount the filesystem in read/write mode, mount / -o remount,rw, then change the root password ( passwd root) or add users as desired. Reboot again, typing reboot or shutdown -r now at the prompt. The VM will start over again as usual (i.e., no single user mode is triggered anymore). Remove the persistent flag from the image, if it is the case, and/or take a disk snapshot as needed.

Online method - VM mount and chroot

An alternative to what was explained so far involves the usage of a VM, under the full control of the user, where to mount the image of the VM without root access.

Attaching a disk to a running VM is easy: from the VMs summary select a VM and hit the tab Storage in the middle bar. Clicking on the button Attach new disk, a pop up will open, letting the user choose an image among those available and accessible in the datastore. Select the image of the VM whose root password needs to be changed (before this step, assign the persistent flag to that disk, so that changes are not lost). At the end of the operation OpenNebula will show the device (/dev/vd*) that has to be used to mount the new disk.

Supposing that the running VM can access the new disk by means of the /dev/vdb device, the first partition is bound to /dev/vdb1, the second one to /dev/vdb2 and so on. Use fdisk -l /dev/vdb to check the partition table of the new disk and to know about the partitions' type. As explained before, look for partitions of type 83 and/or 8e (the latter is LVM, so activate the logical volumes by means of vgchange -ay and learn about the logical volumes through lvs) and mount them for inspection, for example:

mount /dev/vdb1 /mnt

Once the root of the file system (/) has been found, it is possible to edit /mnt/etc/shadow or better, change root to /mnt, i.e. chroot /mnt. In the root jail, change the root password (passwd root) and/or add/remove users. As usual, be sure to affect the mounted filesystem rather the one of the running VM. Finally, exit from the root jail, unmount the disk, detach it via OpenNebula (or take a disk snapshot to be used instead of the old image without root access) and finally remove the persistent flag.

How can I stage in files directly into the VM?

In this section we will propose an easy method to upload files and folders from any location on the Internet to a running VM without additional components on the VM itself.

Generally speaking, this task would require a public IP for the VM and a file server such as FTP, GridFTP, or SSH itself to manage the transfer. This means an additional daemon to take care of and to secure. LRZ implemented the possibility to automatically share a folder between each VM and the outside world, accessible through the host ( In particular:

  • the setup is completely optional, later on precise instructions will be given on how to do it;
  • each VM has its own folder, sharing across multiple VMs is not supported. This is a limitation of the solution employed, 9P VirtFS;
  • it is not possible to attach the folder on the fly while the VM is running, again due to the limitation of the protocol. It is allowed only during the deployment phase, when the VM is instantiated;
  • the folder is exposed to the VM in read only mode while it is accessible in read/write via GridFTP, sftp or gsisftp on, only by the owner of the VM (that is the account used to start it). The group ownership of the folder is fixed and it is needed for the reading operations. This sharing solution is provided by the hypervisor (KVM), which runs as unprivileged user. As such, this unprivileged user should be able to read the folder itself and all its content, in order to feed the VM with it;
  • the folder is erased once the VM is deleted or shut down, since it is not meant for permanent storage.

This feature is not setup by default, but if the user wants to include it, the first step is specifying the related key/value pair in the Context section of the VM template. Please

  • open the template of the VM,
  • select the Context section from the top row (green square in the following picture),
  • on the left pane pick Custom vars,
  • in the proposed form, enter HOSTSHARE as KEY and yes as VALUE,
  • do not forget to click on the Add button (yellow square), otherwise the attribute will not be included in the template,
  • finally confirm the modified template by means of the button Update (red square).

Setup of the VM - frontend shared folder

If the template of a VM contains the HOSTSHARE attribute, then, when the machine is started, a folder named after the numerical ID of the VM is created on, in /media/stagein, to be precise. The numerical ID of a VM is reported as the column ID in the Virtual Machines view, under the Virtual Resources menu on the left pane. The green square in the following picture provides an example.

VM numerical ID

Following this example, the shared folder on will be /media/stagein/6. The file permissions are:

$ ls -lhd /media/stagein/6
drwxr-x--- 2 regularuser oneadmin 4.0K Aug 15 16:47 /media/stagein/6 

The folder is accessible only by the owner of the VM and by the oneadmin group. Please do not modify it, otherwise the sharing mechanism will stop working. Moreover, the security level is adequate, no other users or groups will be able to enter it.

The VM can access the location by means of a mount operation but the availability of modules 9p and 9p_net (kernel greater or equal than 2.6.32) are a prerequisite. The exact command to enter on the VM is:

mount -t 9p -o trans=virtio,version=9p2000.L hostonly <mount point>

or, in /etc/fstab 

hostonly   <mount point>	9p	trans=virtio,version=9p2000.L,ro,noauto    0    0

The noauto option has been given to prevent the OS to mount the partition at boot time, since in some cases it fails. Nonetheless, the <mount point> has been associated to the shared folder and it can be mounted simply by means of mount <mount point> without any further option.

The tag hostonly identifies the folder and it is always the same, for each VM. In fact, as already said, the folder is specific to the VM and for convenience we decided not to change the identifier, so that it is easy to manage the mount operation at boot. Of course<mount point> is a folder on the VM, for example /mnt.

A few last details on the limits and on the usage of the folder from inside the VM. The access type is read only, so it is not possible to remove files or folders, nor use it to stage out data. This decision makes file management much easier since the mechanism is governed by a process which, for security reasons, does not run as root, but as an unprivileged user. The root user of the VM can of course read the whole content, but looking closely at the mount point (/mnt, in the example proposed), always from the VM's perspective, we see the following:

# ls -hld /mnt/
drwxr-x--- 3 129037 110 4.0K Aug 16  2014 /mnt/

The folder is owned and accessible by the account whose ID is 129037, which corresponds to the VM owner (regularuser in our case) on the ONE frontend. Now, if the administrator of the VM wishes to access the shared folder as unprivileged user, it is necessary to create an account with the shown numerical ID. It is easy to realise that the permission scheme of /media/stagein/6 on and the /mnt mount point inside the VM correspond (since the two folders are the same). It is also obvious that setting a broader permission (i.e., 755 or, in other words read and execute for all users) on /media/stagein/6 is not a good idea, given that it is then world reachable by all LRZ Cloud users.

Finally, a comment on how to use the shared folder from the outside, through The mentioned machine hosts

  • a GridFTP endpoint (port 2811)
  • a sftp (file transfer protocol via SSH, on port 22) server
  • a gsisftp (the "Grid certificate" aware counterpart of sftp, on port 22 as well) server

Command line users (mainly on Linux/Unix/Mac OS X) can refer to the LRZ documentation about globus-url-copy if interested in GridFTP, or to to the manual pages of sftp (or gsisftp) from within a shell, in the other case. For example, from the terminal on the user's local workstation, in order to copy the input file file.input to the VM whose ID is 928 and started as user regularuser:

globus-url-copy -vb file.input gsi
Source: file:///home/mylocalaccount/
Dest:   gsi

         3097 bytes         1.00 MB/sec avg         1.00 MB/sec inst


sftp's password:
Connected to
sftp> cd stagein/928
sftp> put file.input
Uploading file.input to /media/stagein/928/file.input
file.input                         100% 3097     3.0KB/s   00:00

The gsisftp command works pretty much in the same way of sftp, except for the authentication, which is based on certificates rather than passwords and/or keys.

The users who prefer a graphical tool for GridFTP could employ Globus Online. The endpoint is lrz#ONE while the path is /media/stagein/<VM numerical ID>, as shown in the picture (the right panel is the one of

Upload via Globus Online

On this topic there is also a dedicated tutorial .

For those willing to use sftp, a well-established GUI is Filezilla . The following picture shows a sftp session. In the top bar, the address to use is, username and password are known to the user, and port is 22. Please remember to navigate to the scratch folder on the right part (remote) of the middle pane, selecting the number corresponding to the ID of the desired VM. A right click on the local file (left part of the middle pane) shows the Upload option.

Upload via SFTP with Filezilla

Are there any VM setup examples?

We prepared three images, Ubuntu (id 66, hosting Ubuntu 16.04 LTS), Debian  (id 1, hosting Debian 8.4), and oSUSE (id 3, hosting OpenSUSE 13.2), with a basic (i.e., command line only), yet fully functional, operating system. The disk size is limited to 5 GB, however the same operating systems are also available on a 20 GB virtual disk. The names are Ubuntu_20G (id 130), Debian_20G (id 135) and oSUSE_20G (id 136), respectively. The three images should be employed together with the respective templates,Ubuntu (id 54), Debian (id 27) and oSUSE (id 2). Please refer to them as a good example on how to setup a VM, even if you wish to use your own image. The Cloud for Dummies - a Short Tutorial will guide you through the necessary steps to run your first VM. The configuration we envisaged shows all the advanced features described in the previous paragraphs:

  • all the disks use the virtio driver, in particular, the boot disk is explicitly assigned to vda, so that it is clear for ONE which disk to use. If nothing is specified, then the boot disk is the first one added to the template;
  • the swap partition is not included in the OS image, but it is a volatile disk. This allows to save space and time, since it is created on the fly at boot time. Please note that it has been assigned explicitly to vdb (in the template) and, inside the guest, it has been added to /etc/fstab as /dev/vdb instead of using the disk unique ID (UUID), since it changes at every deployment;
  • there is also a volatile disk for scratch data, just as an example;
  • the network profile is not assigned, since each group is given a virtual network (for isolation purposes) and there is not a "catch all" solution. Please clone the template and update it assigning a virtual network or deploy this template without any virtual network card attached (it is not possible to edit the template) and then add a NIC later on ;
  • the template does not include any virtual network since each group of users has its own. The template should be cloned and then the virtual network added. Alternatively, it is always possible to instantiate the template as it is and attach the VM(s) to a virtual network later on, during the running phase;
  • inside the VM, the guest OS has been configured with only one network interface, which expects the network configuration from a DHCP server, inlcuding the hostname assignment. Please use it as a reference for the network setup explained in the related section;
  • a basic firewall has been setup in the VM, by means of iptables. Outgoing traffic is always allowed, forwarding is disabled while incoming connections are filtered except for port 22 (SSH service). On OpenSUSE, please use the yast2 utility (based on thencurses text user interface) or the rcSuSEfirewall2 command (together with the configuration files /etc/sysconfig/SuSEfirewall2 and /etc/sysconfig/SuSEfirewall2.d/) to add more ports. On Debian, please add the additional iptables-like rules directly to the file/etc/iptables/rules.v4 (between -A INPUT -p tcp -m tcp --dport 22 -m state --state NEW -j ACCEPT and -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT, it is important, otherwise the firewall will not behave as expected) and then apply the changes issuing as root iptables-restore < /etc/iptables/rules.v4;
  • the images work with deferred disk snapshots, that is the operation implying the shutdown of the VM via the ONE inteface. This is very useful to customise the image and then save it in a consistent way. Once done with the modifications, just select a deferred disk snapshot and then the shutdown command for that VM from the Virtual machines view (under the submenu identified by the red trash bin). During the epilog phase the new image will be created;
  • the templates include the HOSTSHARE tag, so the stage-in folder is set up. The Debian guest mounts it at boot time in /mnt (see the entry in /etec/fstab) while this is not possible in the OpenSUSE VM: here the folder should be mounted manually, simply typing mount -t 9p -o trans=virtio,version=9p2000.L hostonly /mnt;
  • only the root user is present: please clone the template and add your SSH key to the Context section to have access.

As already said, our suggestion is to get familiar with VM management using these images and to clone the template in order to use it as a baseline for the VMs based on your own images.

You can follow our tutorial Cloud for Dummies - a Short Tutorial to set up your first VM.

Are there any other images to work with?

We are being asked why we don't provide more images for other operating systems, like Windows 2008, Windows Server, Red Hat, Scientific Linux, and, and, and. Everyone has his/her favorite operating system and the Cloud gives you the chance to work with it. However, we are only a very small crew to operate and maintain the LRZ Compute Cloud and we simply do not have the manpower to maintain a plethora of images. Just think: we have to keep them up-to-date, too, with security updates, new versions, and perhaps in various sizes. This is just not possible for us. In order to be able to offer you an LRZ Cloud at all, we selected the IaaS way: we provide the infrastructure and give you the freedom to use whatever image you chose. LRZ is responsible for the infrastructure, but you are responsible for your VM, including providing the image for it. This gives you a lot of freedom but also additional responsibility. The images we provide are just for your convenience to go the first steps in the Cloud, but soon you will have outgrown them and then it is time to prepare your own image.

You can download images from various places, but, please, only use a trustworthy source (or hackers will take over your VM in a jiffy!). You can also use a standard installation medium, like a CDROM (image), to install your OS onto an otherwise pristine virtual disk. If your OS does not support the virtio devices, then use others, like SCSI hard disks (sda) or standard networking cards (like rtl8139). These other drivers have a higher compatibility, especially with older installation media, but their performance is not as good as the virtio devices. But: better slow than not at all!

Needless to say: if you use a commercial product, like Windows, you need to have a license for it!

How can I check the quota usage?

The physical resources assigned to a project are limited, usually on a group basis (unless differently specified by the service administrator). In order to check the usage, please open the drop-down menu on the top right corner and named after the account's name, as shown in the following picture. The button is available in any page. From there, just click on Settings.

OpenNebula user settings menu

A new tab will open. Select Quotas (top row, rightmost button) and the Group Quotas. A page similar to the following one will appear.

OpenNebula group quotas

For each resource category (number of CPUs, space allocation for volatile disks, storage space on the image datastore ...) the current usage and the upper limit are displayed. If the latter is a dash (), then no restrictions are applied.

The SSH key(s) injection was successful, but I can't still login

The SSH key(s) injection procedure is only responsible to deliver the content to the /root/.ssh/authorized_keys, but it is up to the owner of the disk image to take care of the SSH service. First of all, it should be running. Please check with:

  • chkconfig sshd --list (or ssh, depending on the distribution), if using SystemV: the service should be flagged as on;
  • systemctl status sshd (or ssh, depending on the distribution), if using systemd: the service should be flagged as active (running);
  • initctl status sshd (or ssh, depending on the distribution), if using upstart: the service should be flagged as start/running).

If the SSH daemon is not running, then start it, replacing the command status with start (for SystemV the correct procedure is /etc/init.d/sshd start or service start sshd). Probably, it is also a good idea to add it to the boot sequence, if not yet done. The daemon should start automatically at boot time. Use the keyword enable instead of status (chkconfig sshd on for SystemV).

If SSH is available but still not working, then the configuration file should be inspected, usually /etc/ssh/sshd_config. Here, the options to enable SSH keys authentication are:

  • PubkeyAuthentication yes
  • AuthorizedKeysFile .ssh/authorized_keys
  • PermitRootLogin yes

These settings should be equal to the default settings of sshd.

Another problem that could prevent SSH key based authentication is related to SELinux. When writing the authorized_keys file during the injection phase, the security context is altered and the file label is not the one expected by the service. Using ls and its option -Z to show the security context information associated to the .ssh folder in root's home, the correct settings are as follow:

# ls -lZa .ssh/
drwxr-xr-x. root root system_u:object_r:ssh_home_t:s0  .
dr-xr-x---. root root system_u:object_r:admin_home_t:s0 ..
-rw-r--r--. root root system_u:object_r:ssh_home_t:s0  authorized_keys

If something different appears, especially for what concerns the authorized_keys file, it is possible to restore the right situation typing as root restorecon -v -R /root/.ssh. Since the goal is to enable passwordless root login, it seems quite unlikely that the user is able to run the command, so the suggestion in this case is to install and enable at boot time the daemon restorecond. It will set the right context for the files and folders listed in its configuration file, /etc/selinux/restorecond.conf (which should contain the entry /root/.ssh/*).

I have my own OS disk image, how can I configure cloud-init?

The package for cloud-init is available for many Linux distributions, usually in the default repository. Just install it via yum, zypper, apt-get or the available package manager. Usually the package name is simply cloud-init. Please beware that at least version 0.7.3 is required in order to cope with the ONE contextualisation infos. Please check here for more details.

The main configuration file is /etc/cloud/cloud.cfg and it lists the modules that are available. Their role is to perform a certain action, such as write the authorised SSH keys into root's home or create users. The configuration file lists the modules that should be executed, specifying also at which stage of the boot sequence. The images we provide are shipped with cloud-init, configured only to handle the SSH service, namely, creating new host keys and injecting the user provided SSH keys into root's home when the VM is deployed (i.e., it boots for the first time). In more detail, this is the content of /etc/cloud/cloud.cfg

disable_root: false
preserve_hostname: true
syslog_fix_perms: root:root

# The modules that run in the 'init' stage
 - ssh

# System and/or distro specific settings
   distro: opensuse
      cloud_dir: /var/lib/cloud/
      templates_dir: /etc/cloud/templates/
   ssh_svcname: sshd

The file is suitable for different Linux distributions, it is only necessary to adapt the value of the distro parameter.

cloud-init should also be instructed to read the information eventually contained in the ONE template. This is done adding a file named, for example, 06_one.cfg to the folder /etc/cloud/cloud.cfg.d whose content is

disable_ec2_metadata: True
datasource_list: ['OpenNebula']
    dsmode: local
    parseuser: nobody

It is worth noticing that OpenNebula is listed as a source and that the content of the template triggers an action (i.e., the dump of the user provided SSH keys into the disk image) when the local file system is available (dsmode: local). This means that the SSH keys are written even if the VM is not configured since the beginning with a virtual network card.

It is possible to check the behaviour of cloud-init typing cloud-init -d init --local. This, as previously said, this will generate new SSH host keys and write the SSH user keys provided by the ONE contextualisation to the file /root.ssh/authorized_keys. In order to repeat the procedure, the content of the folder /var/lib/cloud has to be removed, otherwise cloud-init will not perform any action. The reason is that normally cloud-init acts just once, the first time the VM boots.

Finally, remember that at least the service cloud-init-local should be added to the boot sequence (after cleaning the folder /var/lib/cloud and eventually the file /root.ssh/authorized_keys)

I can't use the sliders!

If you have AdBlockPlus (ABP) enabled for the LRZ Compute Cloud, then it might block your ability to use the sliders. Please add our site to the exception list of ABP or disable AdBlockPlus in order to be able to use this site.

The mouse in the VNC viewer window is out of sync

Here we mean the following problem: the VM has a graphical interface and the user can see 2 mouse pointers when a VNC session is opened through the web interface. One mouse pointer is referred to the VM graphical environment, the other one is the regular mouse pointer of the user's client. The two do not overlap but they move together. The root of the issue is a mismatch between the hypervisor, which sends the relative movements of the mouse pointer, and the VNC server, which does not know the starting coordinates of the mouse pointer. The result is the VM's pointer behaving like a "shadow" of the real mouse pointer. In reality the latter should clasp the former. The solution consists in defining, and using inside the VM's desktop environment, an input device that supports absolute positioning. An example is a Tablet device connected to the VM by means of an USB bus. This can be done in the VM's template, in the Input/Output section. Please refer to the manual for more details and for a screenshot.

I don't know the IP addresses my VMs will be given, how can I set up proper firewall rules?

All the VMs connected to a certain Virtual Network (MWN_access or Internet_access) in the LRZ Compute Cloud share the same IP subnet. Nonetheless, different groups, even on the same IP subnet, are completely isolated from each other, in a private VLAN way (filtering is applied to MAC addresses, so, at a lower layer with respect to the IP level). In other words, VMs belonging to different groups can not talk each other, even if their virtual network cards are attached to the same subnet (MWN_access or Internet_access). The user does not have to take care of firewall rules for the local network (LAN) of the VM, if the purpose is protection from people outside the group but on the same IP subnet. Once again, a VM in the LRZ Compute Cloud:

  • can be reached by other VMs belonging to users in the same group
  • can not be reached by other VMs launched by users belonging to different groups, even if in the same network (thanks to the private VLAN-like behaviour)
  • can be reached by the DHCP server (this can not be disabled, since it is fundamental for the setup of the network connection of the VM)
  • can be reached by all the hosts outside the local network and coming in through the default gateway. Please be advised that this means all hosts in MWN for the MWN_access virtual network cards and all possible hosts in the Internet for the Internet_access virtual network cards). This is needed to provide the VMs with plain Internet connectivity, but, if this feature is redundant, it can also be disabled simply setting the LAN mode flag (beware that Internet connectivity is cut, including the access to the default LRZ DNS servers: no name resolution is available!), or applying proper firewall rules to the guest OS.

As a conclusion, a proper firewall should be set up, but it should be focused on the hosts outside the local network. The inner cross talk among different groups is already taken care of by default.

Is it possible to expose the CPU of the worker node to the guest OS inside the VM?

The hypervisor shows the VM an emulated (generic) CPU. For example, on a linux guest OS, the command cat /proc/cpuinfo shows something along the lines:

processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 2
model name	: QEMU Virtual CPU version 1.4.2
stepping	: 3

In general terms, the cluster of the worker nodes could be made up of machines with different CPU architectures, so, this choice, guarantees that the VMs show the same behaviour no matter which host they are dispatched to by the scheduler. However, sometimes, it could be useful to see from inside the VM the actual CPU of the host, with most of its flags. In order to do that it is necessary to update the template of the VM, in particular the section named Other (green square in the following picture).

OpenNebula Other edit

The idea is to pass raw data directly to the hypervisor, so the option kvm should be picked from the TYPE drop-down list in the RAW data section. The DATA input field should be filled with the text marked in red: <cpu mode='host-passthrough'></cpu>.

Please beware of the single quote (') around the keyword host-passthrough. When the VM will boot, the command cat /proc/cpuinfo will report:

processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz
stepping	: 5

together with more CPU flags.

I'm using a desktop environment (or a graphical interface) in my VM and I access it via VNC. The graphic window is somehow corrupted, with fancy colors and/or characters. What can I do?

By default, KVM emuletes a Cirrus Logic GD 5446 graphic card with 16 MB of memory. Please check that your Guest OS supports it, otherwise install the related drivers (or kernel modules).

Another solution could be to force the hypervisor to emulate another device, for example, the VMware SVGA II Adapter, which is more oriented to virtualised environments. Again, please check that your guest OS supports it. The ONE VM template should be modified in order to instruct KVM. Please open the section named Other (green square in the following picture)

OpenNebula Other edit

and add the text <devices><video><model type='vmvga' vram='32960'></model></video></devices> to the RAW data section. The string should be entered into the DATA input box, while the TYPE menu should read kvm. These instructions will be passed directly to the hypervisor asking to emulate the VMware video adapter (see model type='vmvga') with 32 MB of video RAM (vram='32960' is the amount of memory dedicated to the video adapter, expressed in KB). Sometimes it could be beneficial just to increase the amount of video memory, keeping the Cirrus board. In this case, the raw data string would be <devices><video><model type='cirrus' vram='32960'></model></video></devices>, or any other value for the graphic memory.

I don't have running VMs but ONE complains that my group quota has exceeded

Please read the discussion in the section dedicated to the resource limits. Operations such as stop and undeploy do not refill the group quota (i.e., add back the CPUs booked by the VM), only shutdown and delete can do that. If the user whishes to keep the VMs in a paused state but minimise the impact of this behaviour, then the VMs in a sleep state should be resized

Is it possible to have a persistent IP?

Yes, it is possible to bind a certain MAC ot a given IP. A virtual network containing the MAC address(es) is created ad-hoc for the user so that he/she can assign them to the NIC(s) of his/her VM(s). The DHCP server is then instructed to issue always the same IP(s) to this MAC(s). All the details are explained here.

If all you need is a static host name, LRZ also offers a dynamic DNS service. You can go to, log in with your LRZ account and create a host name for your virtual machine. In the VM you need to create a script that queries the VM's IP address during boot and forwards it to the LRZ dyndns service. Details can be found on the service's page.

Is it possible to give a certain user(s) administrative privileges on a group?

Upon request it is possible to enlarge the privileges of certain account(s) in order to:

  • use/delete/modify all the images of other group's members;
  • use/delete/modify all the templates of other group's members;
  • open the VNC window to the VMs of other group's members.

The target group is the main group of the account being given extended rights. Regarding the VNC window, no console access can be given unless the password is know to the owner of the target account.

There are so many actions (stop, undeploy ...), what are they good for?

In this section we discuss, with some practical examples, when to issue the commands stated here. Remember that the corresponding action should be selected from the ONE interface rather than typed inside the VM.

  • stop: the resources are freed and the VM's RAM is preserved. This is the right status to save the budget (no resources are accounted since nothing is being used) while preserving the content of the VM's disks. Unfortunately, since the content of the RAM is restored when the VM is resumed, the IP the guest OS believes is assigned to the NIC(s) could be different from what issued by the DHCP, leading to a broken network connectivity. The solution consists in manually (i.e., logging in into the VM through VNC with username and password) forcing the renewal of the DHCP lease (on Linux, restarting the DHCP client daemon or the network manager);
  • undeploy: the resources are freed without dumping the content of the RAM. This is the right status to save the budget (no resources are accounted since nothing is being used) while preserving the content of the VM's disks. At resume time, the VM will boot from scratch, so the DHCP lease is renewed;
  • shut down: remove the VM from ONE because no longer needed. The VM can be recreated again, starting from its template. This is the right thing to do to trigger a deferred snapshot;
  • delete: remove immediately all the resources used by a VM, including the disks (deferred snapshot operation is cancelled). This should be used only if the VM is in the failed state. Please note that there is no possibility for us to recover the content of your non-persistent disk(s).

How to use the "LAN mode" flag of a NIC

The NIC template has an option named LAN mode. If it is ticked, all IP packets coming from the default gateway (including the DNS) are dropped. Only traffic to/from the DHCP server (for configuration reason) and other VMs belonging to the same group is allowed. The VM can be reached only via VNC (if configured) or through another VM in the same group acting as an access gateway. The typical use case is a small cluster on the cloud whose worker nodes should be isolated (i.e., should have the LAN mode flag). Here we discuss some tips on setting up a VM acting as an access gateway for other VMs in the same group isolated by means of the LAN mode mechanism:

  • the LAN mode should be unselected, otherwise the gateway would be isolated too, missing its goal. The idea of the LAN mode flag is to isolate a VM from the outside world, not from other VMs in the same group. Moreover, it does not define a subgroup, hence, the access gateway does not need it;
  • only one NIC in the same virtual network (usually MWN_access_<group id>) shared with the isolated VMs is needed. In fact, two NICs in the same subnet are usually confusing for the guest OS since there are two IP gateway entries in the routing table (the DHCP server provides one for each defined NIC, even if the IP belongs to the same network) without a clear priority. Moreover, the communication to the isolated VMs through the common virtual network is always guaranteed, as explained at the beginning;
  • supposing that the MWN_access_<group id> virtual network is used as a backbone for the access gateway and the isolated VMs and that the gateway has one NIC (without the flag, of course), then the gateway serves the whole MWN. Remember: the access gateway is reachable from MWN (the NIC does not have the LAN mode flag) and can communicate with the isolated VMs (despite they have the flag) since they all belong to the same group. On the other hand, the isolated VMs are on their own because of the flag. If access from Internet is required too, then the access gateway should be equipped with a second NIC, attached to the Internet_access virtual network, taking care of assigning priorities to the interfaces. In such a scenario, activating or not the LAN mode on the access gateway's NIC connected to the MWN_access virtual network is quite irrelevant: the main function of the access gateway is exerted through the public IP;
  • probably it is a good idea to reserve an IP for the access gateway, at least on the virtual network that is shared with the isolated VMs. This simplifies the setup and the access gateway can be stopped or undeployed as needed keeping the same IP.

What happens when the budget runs out?

Long story short, the running VMs are undeployed. In more details:

    • the physical resources in use (CPU cores and RAM memory) are freed, i.e., the VM is no longer running
    • the assigned IPs are freed, unless a reservation has been made;
    • the content of the disk(s) is preserved, including the volatile drive(s). They are not moved back to the datastore, they remain in the system store, ready to be accessed once the VM(s) are resumed;
    • on the other hand, the content of the VMs' RAM is not saved. When (and if) the VMs are resumed, the guest OS reboots.

The VMs in other states (stopped, powered off, failed) will maintain their condition, without any possibility of being resumed till the budget is adjusted. Owners will eventually be contacted by the administrators.

The decision to use undeploy rather than stop is due to the fact that the latter action preserves the content of the RAM, including the IP address(es) assigned to the NIC(s). It is very likely that the DHCP will assign new IP(s) at resume time. How this situation is handled depends heavily on the guest OS, so, for uniformity, we opted for a more controlled solution (undeploy and reboot when resumed).

Please note that all the members of the group will receive a notification via mail when the budget usage reaches 25%, 50%, and 75% of the total. An additional warning is issued when the 90% usage is surpassed. We kindly invite our users to contact us at this point to discuss an eventual extension of the budget. An additional message is sent once the budget has been used up to 100%. The undeploy operation will start automatically immediately after. Owners of the VMs will receive a final email when the system tries to undeploy the VMs, containing the numerical ID of the resources being put on hold.

The resume after a suspend or stop operation fails

A suspend or a stop operation envisages a dump of the content of the VM's RAM into a file. At resume time, the RAM is restored. For various reasons (the dump is corrupted, the worker node has a different memory mapping ...) this operation could fail and the VM is stuck in the suspended or stopped state. Please, do not delete the VM if the content of its disk(s) is important to you. Please contact us so that we can try to boot the VM manually, skipping RAM restoration. The VM should then boot from scratch, the content of the RAM is lost, but its disks are intact.

After resuming from a suspend (or stop) operation the network connectivity is lost

This is something to be expected, as explained here (see the description of suspend and/or stop). At resume time, the VM could be assigned a new IP, but the content of the RAM is restored (as envisaged after suspend or stop, the old address could persist in memory. If the guest OS is not capable of realising that the network configuration changed, the solution consists in logging in directly into the VM via a VNC session (if configured) and force the renewal of the DHCP lease. If VNC is not available, then a reboot should solve the situation.

The attach NIC operation (after a detach) fails

If the guest OS is (too) old, it does not support this operation. Unfortunately a simple reboot is not enough. The VM should be deployed again on the worker node. This is what happens when a VM is resumed. So, after detaching a NIC (and realising that the attach attempt fails), just stop or undeploy the VM and resume it again. Now it will be possible to attach the new NIC. Please be aware of the problems that you may incur in case of suspend or stop.

Is it possible to resize an image?

Yes, it is, though it is not completely a self service. In principle it is possible to both shrink and enlarge an image, but only the latter has a higher chance of success. There are 3 steps involved:

  • enlarge the container (i.e., the qcow2 or raw disk image), appending space at its end;
  • modify the partition table to take into account the padding space;
  • resize the filesystem on the image.

The first operation can not be carried out by the user, so please contact us specifying the image that should be enlarged and how much should be added. The user can perform the other two steps attaching the new image (with the persistent flag on, since changes are not temporary) to a running VM.

From now on, we will explain how to use a Linux VM to enlarge an image attached to the vdd device. Linux is suitable also to work on NTFS partitions (thanks to the ntfs-3g utilities), however, a Windows VM can be used as well to complete what will be discussed.

Few disclaimers:

  • what we give here is just an example to show what to expect, but things may differ, even a lot, from case to case;
  • be sure to make a backup of the original image, cloning it, if needed. In the end you are the sole responsible when working on your data. Nonetheless, if something goes wrong, please contact us as soon as possible;
  • in order to accomplish this task, some basic knowledge (such as disk partition, partition table, file system ...) is needed;
  • the operation is reasonably easy when the last partition on the disk has to be enlarged, otherwise it is better to create the disk from scratch and move the content over (or re-install the OS, if it is an image used to boot a VM). Changing a partition surrounded by other ones is not trivial and can not be achieved using this simple method.

The first thing to do is to modify the partition table so that it takes into account the added space. Let's start having a look at the partition table of the enlarged image. The tool we are going to use is fdisk. Since the image is available on vdd, the unmodified partition table looks like

# fdisk /dev/vdd

Command (m for help): p

Disk /dev/vdd: 21.4 GB, 21367461888 bytes
16 heads, 63 sectors/track, 41402 cylinders, total 41733324 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0003acf6

   Device Boot      Start         End      Blocks   Id  System
/dev/vdd1   *        2048      194559       96256   83  Linux
/dev/vdd2          194560    10274815     5040128   83  Linux

What we did so far is to start fdisk in edit mode and print (the command p) the partition table.

The container is 21.4 GB in total (see the first row of the output), but the partition table stops at 5 GB (see the number of blocks of the second partition of vdd, /dev/vdd2), its original size.

In order not to disrupt the content of the disk (this is the whole point), it is possible to append the space only to the last partition, i.e., the one with the highest starting block (don't be fooled by the partition number, always check the blocks). In our case, it's vdd2.

Changing a partition can sound dangerous: it should be deleted and then recreated specifying the new ending block. It is a safe operation, if performed on the last partition, since we're simply moving a boundary.

Let's start removing the partition number 2 (i.e., vdd2):

Command (m for help): d
Partition number (1-4): 2

Command (m for help):

Then we have to add it again, using the command n (for new partition)

Command (m for help): n
Partition type:
   p   primary (1 primary, 0 extended, 3 free)
   e   extended

Since the partition we want to add back is 2 (between 1 and 4), the right option in this case is p for primary

Select (default p): p
Partition number (1-4, default 2):

Type 2, afterwards fdisk wants to know the starting point of the partition

Partition number (1-4, default 2): 2
First sector (194560-41733323, default 194560):

Just pick the default (pressing enter on the keyboard), which also corresponds to the start of the old partition

First sector (194560-41733323, default 194560):
Using default value 194560
Last sector, +sectors or +size{K,M,G} (194560-41733323, default 41733323):

Now it is time to specify the end of the new partition. Since the space padded has to be used, just pick the end of the disk, i.e., the suggested default value, hitting again enter

Last sector, +sectors or +size{K,M,G} (194560-41733323, default 41733323):
Using default value 41733323

Command (m for help):

Print again the partition table, using p

Command (m for help): p

Disk /dev/vdd: 21.4 GB, 21367461888 bytes
16 heads, 63 sectors/track, 41402 cylinders, total 41733324 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0003acf6

   Device Boot      Start         End      Blocks   Id  System
/dev/vdd1   *        2048      194559       96256   83  Linux
/dev/vdd2          194560    41733323    20769382   83  Linux

and notice how the ending block of the partition changed, together with the number of blocks, showing now 20 GB, as expected. Please also check that the column Id, that is the partition type did not change, otherwise use t and the original partition code associated to restore the situation as before.

Last, important thing to do is to write the changes, using the key w

command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

This is all for fdisk, the only thing left to do is to resize the partition so that also the file system reflects the changes. You should know the file system type that has been used to format the partition, otherwise, quickly mount it and check with df -T -h

# mount /dev/vdd2 /media/
# df -T -h
Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/vdd2      ext4      4.7G  1.6G  2.9G  36% /media

It is clear that the partition is ext4 and the file system is not aware yet of all the changes

Unmount the partition

# umount /media/

then check and resize it, for example with e2fsck and resize2fs (suitable for native Linux partitions, otherwise refer to the most appropriate tool, i.e., ntfsck and ntfsresize).

# e2fsck -f /dev/vdd2
e2fsck 1.42.9 (4-Feb-2014)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/vdd2: 97182/315120 files (0.1% non-contiguous), 455904/1260032 blocks

# resize2fs /dev/vdd2
resize2fs 1.42.9 (4-Feb-2014)
Resizing the filesystem on /dev/vdd2 to 5192345 (4k) blocks.
The filesystem on /dev/vdd2 is now 5192345 blocks long.

Once mounted again, we finally see that the partition has been finally enlarged

# mount /dev/vdd2 /media/

# df -T -h
Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/vdd2      ext4       20G  1.6G   17G   9% /media

In fact, the size is now 20 GB, as desired. Please note that a btrfs partition can be resized online (i.e., without unmount). Supposing that the btrfs partition is mounted in /media, the command would be btrfs filesystem resize max /media, where max tells the utility to use all the available space.

The partition can be unmounted, the disk detached (from the GUI) and the persistent flag switched off (from the Image menu).

Which accounts can be enabled for the Compute Cloud?

The LRZ Compute Cloud strongly relies on the LRZ user administration system, which you may know via its web user interface, the LRZ ID Portal. A key prerequisite to use an account to access the LRZ Compute Cloud is that it is part of a project (also called group) managed by LRZ (though it can belong to another institution). Internal projects created for staff members of the institutions we serve (LMU, TUM, FH, ...) are not suitable. The very practical reason is that all the accounts belonging to these projects are known to us, but they are members of a "catch-all" container, so completely opaque. The consequence of this situation is that the entire network security isolation concept is ineffective. If your account is imported "as is", then your VMs would share network connectivity with VMs belonging to other cloud users completely unrelated to you (since you are all members of the same "catch-all" group, as explained before). In fact, decisions on whether VMs are allowed to reach each other depend on the group of the owners. Moreover, budget and resource limits are assigned on a group basis. Again, you would find yourself competing for resources with unrelated researchers.

When requesting access, please feel free to suggest the account you would like to use, but if we verify that the situation explained above applies (i.e., membership in a non-LRZ project), then we kindly ask you to apply for a new project. The whole procedure is rather easy, it consists of filling in a form that should be returned to your "Betreuer", printed on A4 paper, and with of all required fields completed, especially date and signature. The information requested is quite straightforward, it consists of:

  • the personal data of the master user, to whom the possibility to create new accounts (or close old ones) is delegated. She/he will be the link between LRZ and the members of the new group;
  • the desired service (in our case, the LRZ Compute Cloud: check "other" and add "Compute Cloud");
  • the signature of the head of your institution.

In other words, creating a new project is like creating a new Unix or Windows group. The only difference is that this process should be started by the user and not by us (the service administrators). All in all, we are going to authorise the master user to perform some administrative tasks, so a sort of preliminary vetting procedure is required.

All cloud accounts are added to a mailing list which is used to announce new features, scheduled downtimes, maintenances, and problems affecting our infrastructure. Moreover, it is an important tool for use to get in touch with all people we are currently serving. The number of messages is rather limited, nonetheless, please be aware that we can not remove an email address while the associated account is still cloud enabled. The only way to be excluded from the mailing list is to remove the account from the cloud platform.

Last but not least, an ordered (and consistent) user management approach is necessary for detailed accounting, statistics collection, and automatic procedures. Please do not ask to import accounts belonging to projects that are not cloud enabled, since this would create an exception.

I already have an account, how can I get access to the cloud?

Supposing that the account is suitable for the usage with the LRZ Compute Cloud (namely, it is part of a LRZ project), there are 2 requisites to be finally able to log in:

  • a valid password
  • the LRZ Compute Cloud Platform flag

Please note that both requirements have to be fulfilled in order to get access, however they're independent. In other words, it does not matter the order in which the two operations (change the password, assign the flag) are carried out, especially because two different subjects are involved).

Password management

This is the LRZ Compute Cloud account's password, it is up to the account's owner to

  • change the initial password (Startpasswort) received by the master user;
  • renew the password periodically, according to the LRZ policies.

In case the password has been forgotten, it can always be reset by the master user to an initial password, which has then to be updated. Please be also aware that the initial password (Startpasswort) grants no access.

The tool for password management is the LRZ ID Portal. Log in using your known password (the initial one is of course valid) and

  • on the left panel click on the Self Services button, if present;
  • always on the left pane, under Account (Kennung), click on modify password (Passwort aendern);
  • on the right part of the screen, a list of accounts owned by the user are shown. Please check that the status of the account meant to be used on the LRZ Compute Cloud reads valid (gueltig), otherwise click on the Select (Waehlen) button. A new page will open, in order to let you change the current password. Please type the old one (once) and the new one (twice), then click on Modify password (Passwort aendern).

LRZ Compute Cloud platform

A user account is able to access the LRZ Compute Cloud if it has been assigned to the corresponding platform. This can be done by the master user. The master user should connect to the LRZ ID Portal and

  • on the left panel click on the Master User Services (Master User Dienste) button, if present;
  • always on the left pane, under Project (Projekt), click on view / modify (anzeigen/bearbeiten);
  • on the right part of the screen, a list of projects managed by the master user are shown. Please click on the Select (Waehlen) button near the name of the project enabled for the cloud.

Something similar to the following picture should appear at the bottom of the page, or click on List of accounts (Kennungliste) on the top right corner of the page.

ID Portal project kontingen

Each project is assigned an Account Quota (Kontingent), i.e., the maximum number of account of the project (blue square in the previous picture). Cloud projects are also given a Cloud Quota (Cloud Kontingent), i.e., the maximum number of accounts that can be assigned to the cloud platform (yellow square in the previous picture). The master user should verify that the cloud quota is present and there are enough free tokens. If it is not the case, please contact us.

In the upper part of the page, the list of accounts and available platform is reported, as per the following screenshot.

ID Portal user platform

As suggested by the description, just put a X in the LRZ Cloud column (for each account that should be given access) and save it.

Important note: the LRZ Compute Cloud frontend and the ID Portal are synchronised every 15 minutes. After the account's activation please wait about a quarter of a hour before trying to login. If there are still problems, please contact us, do not try to remove the account or the flag and create it again. Moreover, as already specified, it is not important that the initial password is changed before or after the flag is assigned. The important thing is that the account has the flag and a valid password at login time.

How can I inject a script into a VM?

This entry addresses the need to execute a custom script in the VM at boot time, injecting the code on the fly. This is a two-fold task:

  • the VM template should contain the script;
  • cloud-init should be properly setup inside the VM in order to catch the code and execute it.

Regarding the template, two variables should be added to the contextualisation disk (highlighted in green in the following picture), simply entering them into the Context tab of the template (see the yellow square in the next picture), in particular, in the section Custom vars (on the left panel).

userdata in the template

The first variable is called USER_DATA and it should contain the script to be injected, base64 encoded. This choice could seem a hinder, but it avoids to deal with characters that should be escaped, such as quotes, new lines ....

. The easiest way to obtain the string to be assigned to this variable is to:

  • write down the script into a file, let's say;
  • dump the script on the screen and convert it, typing cat | base64.

The alphanumerical string given as output should then be pasted into the template as USER_DATA. The script can be of any kind (bash, ruby, perl ...), as long as it begins with a shebang (#!). It is worth noticing that cloud-init does not understand only scripts, but also Upstart jobs, Cloud Config and include files. Please refer to the cloud-init documentation on user data scripts for all the details about the format to use.

The second variable to add is USERDATA_ENCODING, whose value is simply the string base64. If this is left behind, then cloud-init will not decode the injected data, so no code will be executed.

Please be aware that script injection can be also achieved via the EC2 interface. Of course, in this case, the template should not be customised with the two variables mentioned above. Please refer to this page for all the details.

The second component involved is cloud-init. It has to be installed on the VM and it has to be executed at boot time to customise the Guest OS. It is made up of different modules (code executing a certain task, i.e., creating a certain user or creating a SSH server key) executed at different stages of the boot sequence (i.e., when the file system is available, when the network has been set up ...). The module in charge of running user's scripts is scripts-user and in the config file (/etc/cloud/cloud-cfg, on the VM) it is in the section cloud_final_modules. This means that it is executed pretty late, but most of the VM's services will be available. We already detailed how to setup cloud-init from scratch, its config file should be updated with the following text

 - scripts-user

In principle the module can be moved up (i.e., in the cloud_init_modules section) to be executed earlier, but some expected features, such as the network connectivity, could not be available yet. Please note that the corresponding cloud-init service should be enabled in the boot sequence as well. How to do that, it is strictly dependent on the Linux distro in use (some use systemd, others use Unity and there is still the legacy SystemV around). Following our examplea, cloud-final should be in the boot sequence, minding that usually it depends on cloud-conf and on cloud-init-local, since the cloud-init's OpenNebula source has been configured as available together with the file system rather than the network (see the setup in the folder /etc/cloud/cloud.cfg.d explained above). Generally speaking it is a good idea to disable undesired features from the configuration file rather than switching off services.

Finally, a hint on how to debug cloud-init. It is executed every time the folder /var/lib/cloud is empty. Please remove its content before running the service again, otherwise it will do nothing. In order to run the local section of cloud-init, type as root cloud-init -d init --local (or start the service cloud-init-local). For the network section, type the same commands removing the local keyword. The final section of the configuration file is triggered by cloud-init -d modules --mode final or starting cloud-final.

SSH error: Remote Host Identification Has Changed

If using SSH to connect to your VM(s), sooner or later the following message could show up, preventing your from logging in.





Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
b2:f5:36:e2:23:47:26:7d:7f:77:d0:b6:20:69:34:51 [MD5].
Please contact your system administrator.
Add correct host key in /home/user/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/user/.ssh/known_hosts:11
Password authentication is disabled to avoid man-in-the-middle attacks.
Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
X11 forwarding is disabled to avoid man-in-the-middle attacks.

Permission denied (publickey,password).

Each SSH server, including the one running on your VM(s), has a SSH host key, whose public part is provided to the client. The client registers it together with the IP or the hostname of the target server during the first connection attempt (i.e., when typing ssh <IP or hostname of the VM>) in the file $HOME/.ssh/known_hosts. Each time the user connects again to the same server, the client checks that the destination's SSH host key and IP match the information stored earlier on. If the check fails, then the above warning is issued. Given that a cloud environment is highly dynamic and IPs assigned to VMs may (and will) change, the SSH client can be induced to believe that there is a problem because of the following:

  • when a VM is resumed after being stopped/undeployed the IP may change, while the SSH host key of the server remains the same. The old IP has been probably assigned to another VM, with a different SSH host key, hence there is a mismatch. The SSH client detects the change and complains about it;
  • when a VM is newly deployed, maybe using the same fixed IP and the same image of an old one, usually a new SSH host key is generated. Once again, this is noticed by the SSH client and the user is warned. Please note that when a SSH daemon is started on the VM, if no SSH host keys are present, new ones are generated. If cloud-init is configured to run at boot, it will generate a new SSH host key even if already present.

Given that no tampering attempt on the VM has been attempted, the solution to this problem is rather easy. It consists in removing the tuple IP address / hostname - SSH public host key from the file $HOME/.ssh/known_hosts. The error message even points you to the correct location, see Offending ECDSA key in /home/user/.ssh/known_hosts:11, mentioning line 11 of $HOME/.ssh/known_hosts.

A more permanent solution involves the customisation of the SSH client configuration file, $HOME/.ssh/config. Just append the following:

Host vm-* 141.40.2* 10.155.2*
StrictHostKeyChecking no
UserKnownHostsFile /dev/null

Basically, we wish to tell the client to skip the match between VM's IP (or host name) and the saved SSH host key. The first line (Host ...) is a regular expression identifying the hostnames/IPs which the rules specified in the following lines should apply to. Here we take into account both the default DNS hostnames of VMs (vm-* or their IP (141.40.2* 10.155.2*). The rules will apply when you will try to connect to your VMs via SSH specifying the IP or the host name. The second line tells the client not to perform any check on the server's SSH host key. Anyhow, the tuple sever's IP or host name - SSH host key will be registered anyway. Since no verification is going to take place (because of the StrictHostKeyChecking entry), then there is no point in saving the info in $HOME/.ssh/known_hosts. This is why the third line has been added, tricking the client into writing the info in /dev/null (i.e., discarding it).

The undeploy operation fails. My VM always goes back to the "running" state

The rationale behind undeploy is to have a clean shut down of the guest OS. For example, on a Linux VM, this means sending the kill signal to all processes and unmount all disks. In order to achieve that, the hypervisor sends an ACPI signal to the the VM. If the signal is not caught, or it is ignored, the VM keeps going on, unchanged. After the 5 minutes from the issuing of the undeploy command, the hypervisor check again the status of the VM and if it is found again, then the status switches back to running. This situation has mainly two drawbacks:

  • the VM still consumes budget, though the user may thing that it has been paused. Please note that it is the user's responsability to check that the undeploy mechanism works. The alternative would be, on our side, to forcefully undeploy the VM after the 5 minutes grace period, something we want to avoid since it could corrupt the file system(s) of the VM;
  • the IP(s) of the VM are returned to the pool. If the operation is not completed, the VM runs without an IP(s) associated to its MAC(s). The quick workaround is to detach and re-attach the NIC (being aware of this potential problem).

Usually Windows users do not have the possibility to tamper with the ACPI signal and the problem described above does not occur. On Linux, the setup could be different and the ACPI signals could be mishandled. If the undeploy operation is troublesome, please check that:

  • the acpid daemon has been installed and configured correctly (and it is running, of course)
  • the desktop manager eventually present on the VM responds correctly to the ACPI shutdown signal (usually it is the action related to “what to do when the power button is pressed" among the energy options of the control panel and it should be shutdown)

If the undeploy operations still does not work, then shutdown the VM from within the guest OS and then use the “hard” version of undeploy, which will terminate the VM and set it to the expected state. Please do not skip the first step (shutdown from within the VM), otherwise the VMs’ disks will be in an inconsistent state (i.e., force checked at next boot).

How can I set up a Linux VM to forward IP packets?

The main use case of this FAQ is the following: the VMs deployed in the LRZ Compute Cloud need to access a service hosted on a server whose access is IP-restricted by a firewall or another ACL (Access Control List) mechanism, such as TCP wrapper. In other words, the service, which should be reachable (i.e., there is network connectivity) from within the LRZ Compute Cloud's network and under user's control (i.e., the user can modify its configuration), can be used only by clients whose IP address or IP subnet is whitelisted. The issue here is twofold. On one side, we (the service provider) do not want to assign fixed IPs to each and every VM. On the other side, the service owner can not whitelist the whole IP subnet used by VMs in the LRZ Compute Cloud because the VMs of any cloud user could have potentially access. The solution, in this case consists in setting up one VM with a fixed (not mandatory, but convenient) IP acting as a gateway for other VMs belonging to the same group.

The gateway VM needs few resources (i.e., 0.5 CPU, 2 or 4 GB of RAM) and it can be based on one of images we provide. Here we will discuss how to set up a Linux gateway by means of iptables to reach a service (form now on target service) by means of two alternative approaches. Before starting, some assumptions:

  • the target service is in the MWN and its IP address is w.x.y.z;
  • the IP of the gateway VM has to be whitelisted in the firewall or ACL rules protecting the target service;
  • only one NIC is needed by the gateway VM, connected to the MWN_access_<group id> virtual network (or to MWN_reservation_<user id>, in case of a fixed IP is assigned);
  • the gateway VM's NIC is eth0 and the IP is a.b.c.d;
  • we are going to use a Debian image, but instructions can be easily applied with few adjustments to any Linux flavour;

Port forwarding

This is useful in case the target service uses only one port and all the communications take place on that port. A typical example is SSH. This method, when applicable, is very efficient, since only the needed traffic is forwarded (the gateway has to keep track of all connections that are forwarded and limiting them allows to reduce the resources required). Some additional assumptions:

  • the port user by the target service is 22;
  • the port used by the gateway is 2222.

The goal can be achieved using two alternative tools, namely rinetd or iptables. We will start with the former, easier to configure.

rinetd is a daemon listening on a port of any interface of the local machine and forwarding packets to an address (local or remote) and a port specified in the configuration file. After installing it on the gateway VM, please edit the configuration file, usually /etc/rinetd.conf and add the following:

a.b.c.d	 2222	w.x.y.z    22

After restarting the service (/etc/init.d/rinetd restart), the packets coming on port 2222 of the gateway VM's interface whose IP is a.b.c.d will be forwarded to the port 22 on the remote host w.x.y.z. This is all, nothing else is needed, except an iptables rule to open the listening port on the gateway machine. Something like iptables -A INPUT -p tcp -m tcp -s a.b.c.d/21 --dport 2222 -m state --state NEW -j ACCEPT is enough (please note that this is not persistent, it should be added to the configuration file of iptables, such as /etc/iptables/rules.v4 in order to make it outlive the next reboot). Please note that we are filtering the IP of the machines allowed to access port 2222 (-s a.b.c.d/21), allowing only machines belonging to the same subnet as the gateway VM.

If rinetd is not available, the same result can be obtained in a more complicated way, at least from the configuration point of view, by means of iptables. In order to understand the meaning of the commands that are going to be entered, a short remark on how iptables works is necessary. iptables- is made up of different chains, grouped in tables (on the command line specified by the switch -t). The order in which they are evaluated, when they enter an interface is:

  • PREROUTING (belonging to table nat: it modifies the incoming packets before any routing decision is taken;
  • INPUT (belonging to table filter: evaluate incoming packets and eventually drop them;
  • FORWARD (belonging to table filter: route the packets to other hosts;
  • OUTPUT (belonging to table filter: evaluate outgoing packets and eventually drop them;
  • POSTROUTING (belonging to table nat: it modifies the outgoing packets after a routing decision is taken.

Now we will go through the commands line that have to be entered for each table. Just for clarity, the goal is to get packets arriving on a.b.c.d:2222 and send them to w.x.y.z:22. We also assume that the default policy of the chains in the filter table (INPUT and FORWARD, at least) is DROP, i.e., iptables -P [INPUT,FORWARD] DROP. This also means that ports used for services on the VM have to be explicitly opened. This is the way iptables is configured on the provided Linux images.

An important prerequisite for this method to work is that IP forwarding has to be enabled. This can be easily done on the fly, typing as root the following command on the shell: echo "1" > /proc/sys/net/ipv4/ip_forward or sysctl -w net.ipv4.ip_forward=1. To make it permanent, so that it survives a reboot, please edit the file /etc/sysctl.conf and add (or uncomment/modify, if already present) the line net.ipv4.ip_forward=1


First of all, let's start with the PREROUTING table, where we instruct iptables to change the destination port and address of incoming packets on port 2222 to port 22 and IP w.x.y.z, respectively (i.e. the target service's port and address):

iptables -A PREROUTING -t nat -i eth0 -s a.b.c.d/21 -p tcp --dport 2222 -j DNAT --to w.x.y.z:22

The switch -A stands for append to the chain, -t specify's the chain's table, -i is the interface where the incoming packets are arrivingi, -i limits the source IP subnet that is allowed to use the port forwarding, -p is the transport protocol used by the packets (TCP, but it could be UDP as well), --dport is the port on the gateway that is used for the forwarding, -j DNAT means perfom (jump to) a destination NAT and --to is the target of the destination NAT, that is the address/port tuple that should be set into the packet's header (in our case the target service IP and port, w.x.y.z:22). Please note that to avoid errors and to tighten the configuration, the interface is specified (-i eth0), though it is the only one available, and also the source IP subnet (-s a.b.c.d/21), since we want to avoid that the forwarding is used by any machine being able to reach the gateway VM (and there are many, since the LRZ Compute Cloud VM are visible in the MWN). Of course the notation -a.b.c.d/21 identifies only the VMs belonging to your group, as explained here, leaving out all other VMs of the LRZ Compute Cloud.

The second table that is traversed is INPUT. The gateway VM should be able to accept new packets coming to the port that we want to forward, 2222. The instruction to give is:

iptables -A INPUT -p tcp -m tcp -i eth0 -s a.b.c.d/21 --dport 2222 -m state --state NEW -j ACCEPT

We can note that the table (-t) is not specified, because by default it is filter and only new packets (-m state --state NEW) are accepted (-j ACCEPT) on the forwarded port (--dport 2222) if coming from the LRZ Compute Cloud's MWN subnet (-s a.b.c.d/21) and interface (-i eth0). The focus in only on new packets since we assume that there is already a rule to let through packets that are exchanged of a connection already established (i.e., started on a port that has been cleared, such as 2222). In other words, something like iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT should already be present.

The packets that arrived so far and matched the previous chains should be forwarded rather than processed locally, so some entries should be added to the FORWARD chain. Default forwarding should be avoided, as already explained: please be sure that the following is part of your iptables setup: iptables -P FORWARD DROP. Of course a rule should be present for the forwarding fo the packets from a.b.c.d:2222 (address:port of the gateway VM, to be forwarded) to w.x.y.z:22 (address:port of the target service):

iptables -A FORWARD -i eth0 -o eth0 -s a.b.c.d/21 -p tcp --syn -m conntrack --ctstate NEW  -d w.x.y.z --dport 22 -j ACCEPT

Since this deals only with new packets, as already explained for the INPUT chain, also the packets following up this connection should be accepted:

iptables -A FORWARD -i eth0 -o eth0  -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

Last, but not least, the POSTROUTING chain, where, after the routing decision (i.e., the packets are going to be sent to w.x.y.z:22) the source adress is changed to that of the gateway VM (a.b.c.d). This step is fundamental, since this is the only IP address of the entire subnet that has been cleared on the target service's side for its usage. If this is skipped, then the whole setup is useless.

iptables -t nat -A POSTROUTING -o eth0 -s a.b.c.d/21 -j SNAT --to a.b.c.d

Please note once again the location of the chain in the nat table (needed, since it is not the default one), the source subnet (-s) and the outgoing interface (-o eth0) for more clarity. The notation -j SNAT --to a.b.c.d specifies the value that the IP packet's source address should be changed to. The user should not worry about the reverse translation (i.e., from a.b.c.d to the original VM's IP) since it is implicit. Actually, iptables keeps track of the connection and is capable of doing that.

In conclusion, when dumping the chains of the filter table, the following entries should show up, among others:

# iptables -L -n -v
Chain INPUT (policy DROP 41 packets, 12547 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     tcp  --  eth0   *       a.b.c.d/21             tcp dpt:2222 state NEW
 1130 83260 ACCEPT     all  --  *      *              state RELATED,ESTABLISHED

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    1    60 ACCEPT     tcp  --  eth0   eth0    a.b.c.d/21           w.x.y.z              tcp dpt:22 flags:0x17/0x02 ctstate NEW
   25  5610 ACCEPT     all  --  eth0   eth0              ctstate RELATED,ESTABLISHED

Chain OUTPUT (policy ACCEPT 301 packets, 34545 bytes)
 pkts bytes target     prot opt in     out     source               destination

The same command, but for the chains in the nat table should look like:

# iptables -L -n -v -t nat
Chain PREROUTING (policy ACCEPT 444 packets, 49403 bytes)
 pkts bytes target     prot opt in     out     source               destination
    5   300 DNAT       tcp  --  eth0   *       a.b.c.d/21             tcp dpt:2222 to:w.x.y.z:22

Chain INPUT (policy ACCEPT 1 packets, 60 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 5 packets, 346 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    6   406 SNAT       all  --  *      eth0    a.b.c.d/21             to:a.b.c.d

Additional note: if, for any reason, the gatewy VM has 2 NICs, for example, the second one (eth1) provided with a public IP (i.j.k.l) and this has to be used to contact the target service, then the previous commands are modified in the following way:

iptables -A PREROUTING -t nat -i eth0 -s a.b.c.d/21 -p tcp --dport 2222 -j DNAT --to w.x.y.z:22
iptables -A INPUT -p tcp -m tcp -i eth0 -s a.b.c.d/21 --dport 2222 -m state --state NEW -j ACCEPT
iptables -A FORWARD -i eth0 -o eth1 -s a.b.c.d/21 -p tcp --syn -m conntrack --ctstate NEW  -d w.x.y.z --dport 22 -j ACCEPT
iptables -A FORWARD -i eth0 -o eth1  -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i eth1 -o eth0  -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -t nat -A POSTROUTING -o eth1 -s a.b.c.d/21 -j SNAT --to i.j.k.l

The packets arriving to the gateway VM on the forwarded port (2222) of the MWN interface (eth0) from the VMs belonging to the user's group in the MWN subnet a.b.c.d/21 have to be sent to the target service (w.x.y.z:22) via the gateway VM's public inteface (eth1) using its public IP (i.j.k.l).

We can note that:

  • the PREROUTING and INPUT rules are unchanged;
  • the forwarding rule for new packets matches the packets flowing from the MWN interface (-i eth0) to the public interface (-o eth1);
  • the rule for the related and established packet in the FORWARD chain has now been introduced for both directions, MWN to public (-i eth0 -o eth1) and public to MWN (-i eth1 -o eth0);
  • the POSTROUTING rule now changes the source address of the outgoing packets to the gateway VM's public IP (i.j.k.l) and the outgoing interface is the public one (eth1).

So far we reviewed the configuration of the gatewy VM, but what about the the client? On this side, things are really easy, just replace the target service's IP and port with the gateway VM's IP and forwarded port, respectively. This should be done wherever is needed: script, command line, configuration files ... Continuing our example, on all your VMs (except for the gateway), instead of using

ssh -p 22 myuser@w.x.y.z

which will result in the client hanging till the timeout is reached, just type

ssh -p 2222 myuser@a.b.c.d

Traffic forwarding

This method consists in instructing the gateway VM to forward all the packets coming to its interface from any port and change their source address into that of the gateway itself. In other words, the gateway VM also acts as a NAT (Network Address Translator). The scenario we are taking into consideration is the following:

  • the gateway VM receives packets on any port from other VMs belonging to the same group and placed in the MWN;
  • the source IP address of the packets is changed to the gateway VM's (NAT functionality);
  • the IP packets are forwarded to the target service (IP w.x.y.z);
  • all related packets coming from the target service are forwarded by the gateway back to the VM that initiated the communication, simply replacing the destination IP address (this is also part of the NAT functionality, reversing the operation performed in the second step is done automatically, since the kernel keeps track of the connections and the mappings).

It is easy to understand that this solution is convenient when many ports are involved, or the service to be accessed picks a port from a given range. The setup procedure envisages iptables, please refer to the previous section for a basic introduction on the tool. We also assume that the default policy of the chains in the filter table (INPUT and FORWARD, at least) is DROP, i.e., iptables -P [INPUT,FORWARD] DROP. This is the way iptables is configured on the provided Linux images. The following instructions should be carried out on the gateway VM.

First of all, it is mandatory to enable packet forwarding. This can be done on the fly, typing as root echo "1" > /proc/sys/net/ipv4/ip_forward or sysctl -w net.ipv4.ip_forward=1. To make it permanent, so that it survives a reboot, please edit the file /etc/sysctl.conf and add (or uncomment/modify, if already present) the line net.ipv4.ip_forward=1


The gateway can be turned into a NAT simply adding the following rule to the POSTROUTING chain of the nat table:

iptables -t nat -A POSTROUTING -s a.b.c.d/21 -o eth0 -j MASQUERADE

The keyword MASQUERADE triggers the NAT functionality (replace the source IP of the packet with that of the gateway), while -o eth0 specifies the interface to which the packets should be forwarded to (its IP address will be used to replace the original source). It is also a good practice to specify the source IP network (-s a.b.c.d/21) so that only requests coming from LRZ Compute Cloud VMs will be served.

The next step is to configure the FORWARD chain (reminder: the assumption is that the default policy is DROP):

iptables -A FORWARD -i eth0 -o eth0 -p tcp -s a.b.c.d/21 -d w.x.y.z -j ACCEPT
iptables -A FORWARD -i eth0 -o eth0 -p tcp -s w.x.y.z  -d a.b.c.d/21 -m state --state RELATED,ESTABLISHED -j ACCEPT

Please note that there is one entry per direction and both the interface and the source/destination IP addresses are specified (for consistency and to restrict the usage). Here we also specified the protocol, -p tcp. Of course this entry can be omitted if it is necessary to forward all traffic, not only TCP. Please remember that it is not possible to specify a list of protocols with the option -p. If many protocols should be enabled, then an entry for each one should be added. The possible values are those listed in the file /etc/protocols, plus the catch-all value all.

As per the INPUT table, dealing with the incoming packets, nothing should be changed nor added. In fact, the rules of this table applies to packet directly to the gateway VM itself (i.e., the IP destination address is a.b.c.d), while here we are interested in the packets that will be forwarded (i.e., IP destination address w.x.y.z). This also means that it is always possible to login into the gateway VM via SSH from any other VM in the LRZ Compute Cloud, given that this is allowed by a rule in the INPUT table, of course.

In conclusion, when dumping the chains of the filter table, the following entries should show up, among others:

# iptables -L -n -v
Chain INPUT (policy DROP 234 packets, 70948 bytes)
 pkts bytes target     prot opt in     out     source               destination
    4   244 ACCEPT     tcp  --  *      *              tcp dpt:22 state NEW
 2773  198K ACCEPT     all  --  *      *              state RELATED,ESTABLISHED

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
   58  8858 ACCEPT     tcp  --  eth0   eth0    w.x.y.z        a.b.c.d/21      state RELATED,ESTABLISHED
   74 10645 ACCEPT     tcp  --  eth0   eth0    a.b.c.d/21      w.x.y.z

Chain OUTPUT (policy ACCEPT 139 packets, 28295 bytes)
 pkts bytes target     prot opt in     out     source               destination

The same command, but for the chains in the nat table should look like:

# iptables -t nat -L -n -v
Chain PREROUTING (policy ACCEPT 2883 packets, 388K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain INPUT (policy ACCEPT 3 packets, 180 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 24 packets, 1802 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
   27  1982 MASQUERADE  all  --  *      *       a.b.c.d/21

Additional note: if, for any reason, the gatewy VM has 2 NICs, for example, the second one (eth1) provided with a public IP (i.j.k.l) and this has to be used to contact the target service, then the previous commands are modified in the following way:

iptables -t nat -A POSTROUTING -s a.b.c.d/21 -o eth1 -j MASQUERADE
iptables -A FORWARD -i eth0 -o eth1 -p tcp -s a.b.c.d/21 -d w.x.y.z -j ACCEPT
iptables -A FORWARD -i eth1 -o eth0 -p tcp -s w.x.y.z  -d a.b.c.d/21 -m state --state RELATED,ESTABLISHED -j ACCEPT

It is clear that the interface used for masquerading is now eth1.

On the other hand, the configuration for client VMs, namely VMs in the LRZ Compute Cloud that should use the ad-hoc gateway to reach the target resource, is a bit more complicated. In few words, an entry should be added to the routing table so that the gateway VM is used to contact the target instead of the default gateway. Here we will discuss the steps to achieve that on a VM with the IP m.n.o.p. Let's start to have a look at the routing table:

# ip route show
default via  dev eth0 m.n.o.p/21 dev eth0 proto kernel scope link src m.n.o.p 

Here we simply see that all IPs outside the local subnet (m.n.o.p/21) are reached via the default gateway. We already know that this would prevent to contact the target resource at w.x.y.z and this is the reason why the gateway VM (a.b.c.d) has been setup. The command to enter in order to route all requests from m.n.o.p to w.x.y.z through a.b.c.d is:

# ip route add w.x.y.z via a.b.c.d dev eth0 table main

What the command does is to add an entry to the main routing table (route ip add ... table main), i.e., the one used to go out of the local network, instructing the kernel to use the IP a.b.c.d (via a.b.c.d) to reach out to w.x.y.z. A new dump of the routing table will now give

# ip route show
default via  dev eth0 a.b.c.e/21 dev eth0 proto kernel scope link src m.n.o.p w.x.y.z via a.b.c.d dev eth0 

Please be aware that modifications to the routing tables entered directly on the command line are not persistent, i.e., they disappear after a reboot and they should be entered again. For some Linux distribution, a route can added permanently appending the following to the file /etc/sysconfig/network/routes, if already present:

w.x.y.z   a.b.c.d   -   eth0

that is <target ip> <gateway> - <NIC connected to the subnet where the gateway is reachable>.

On Debian and derivatives, such as Ubuntu, it is possible to use the post-up keyword in the network interfaces definition file (/etc/network/interfaces) to specify a command that should be executed after an interface is brought up:

allow-hotplug eth0
post-up ip route add w.x.y.z via a.b.c.d dev eth0 table main

A similar approach can be used in Red Hat and derivatives (i.e. SuSE) distros as well. The configuration file of the interface where the gateway VM is reachable, i.e., /etc/sysconfig/network/ifcfg-eth0, can contain the variable POST_UP_SCRIPT, pointing to a script that will be executed right after the interface is brought up. The line to add to the previously mentioned configuration file looks like:


where compat:suse: is the script type (the guest OS will expect a bash script) while /etc/sysconfig/network/scripts/addedroute is a script that should be created by the user, which content could be as simple as:

ip route add w.x.y.z via a.b.c.d table main

The path /etc/sysconfig/network/scripts contains other similar network scripts and the file addedroute should be executable (i.e., chmod a+x /etc/sysconfig/network/scripts/addedroute).

I would like to acknowledge the LRZ in my publication, is there a standard sentence?

In case you are happy with the LRZ Compute Cloud and you would like to acknowledge the LRZ in your publication, please use the following sentence

We thank the Leibniz Supercomputing Centre (LRZ) of the Bavarian Academy of Sciences and Humanities (BAdW) for the support and provisioning of computing infrastructure essential to this publication.

or, in German:

Wir danken dem Leibniz-Rechenzentrum der Bayerischen Akademie der Wissenschaften für die Unterstützung und Bereitstellung der Recheninfrastruktur, die für diese Publikation wesentlich war.

My VMs are stuck in pending state, what's happening?

The first, and most obvious, reason is that the budget has been used up. Warning emails are sent at regular intervals, especially when the usage hits the 90% and the 100% of the total, however, it is always possible to check the current usage. For example, it is shown in the dashboard, right after logging in or clicking on Dashboard from the main menu on the left. Please refer to the BUDGET USAGE box on the top row (right side), as shown in the following picture.


Otherwise, please check the Budgeting item in the main menu.

If this is not the case, i.e., the budget consumption is less than 100 %, then there is a high chance that there are no enough resources to accommodate your VM(s) because of the high demand of the infrastructure. In other words, there are no physical nodes with enough free cores to fulfill the number of CPUs requested by the template. In this case, there is a pop up in the detailed view of the VM. Here we explain how to check and what to do to mitigate this situation.

Let's start from the general VMs view, where the active and paused instances are shown.

Pending VM

Please note the VM in the PENDING state. In order to check if the deployment is being hindered by the lack of free resources, please click on the row associated to the VM (highlighted in blue in the previous screenshot) to open the detailed view.

Pending VM - detailed view

If there are not enough free resources for the VM, then a pop-up window will appear in the lower left corner (see the grey box in the corresponding position in the previous picture).

The VM stays in the pending state till a node with enough capacity will be available. Unfortunately this does not depend on us and it is hard to predict when the load is going to decrease. However, once the scheduler can accommodate the VM, the status will change from Pending to Booting and then to Running. If you have enough patience and time to wait, no further actions are required. The only mitigation action that is possible at this point consists in downsizing the VM (i.e., reduce the amount of requested resources), if this is applicable to your case. If you decide to resize the VM, please put in on hold picking the corresponding action, as shown in the following picture (blue square).

Pending VM - put on hold

Right after, please switch to the Capacity tab, where you can see that 8 CPUs are requested (yellow square in the following screenshot).

Pending VM - resize

Click on the green Resize button and decrease the number of CPUs , as already explained here. Once done, Release the VM (next picture, blue square), so that it is taken again into consideration by the scheduler.

Pending VM - release

Please also note the new value for the requested CPUs. If the VM can not go through yet, please try to modify the amount of memory requested or further reduce the number of CPUs (if applicable).

I would like to send system emails (i.e., cron job) from my VM, how can i do that?

Here we will explain how to configure postifx on a VM so that system emails, for example, related to cron jobs, can be sent. In particular, an external SMTP server which requires authentication, namely, a relay, will be used. Any SMTP server can be picked, however, we will discuss the case of which can be accessed by means of your personal username and password as defined in the LRZ ID portal (if a mail box is associated to your account).

First of all, install postfix on the VM via the package manager of your guest OS, then a file with the username and the password to access the relay should be prepared for postfix. The credentials are not in clear text, they're hashed:

  • create the text file /etc/postfix/sasl_passwd with the following content: [<your mail server>]:<port number> <username>:<password>. Please note that the square brackets are needed (they prevent to look for the MX record), while the port number is optional. If you want to use LRZ resources, then specify as mail server and replace username and password with your LRZ account credentials (if a mail box is associated to it);
  • hash the file typing at the prompt postmap /etc/postfix/sasl_passwd as root. A new file will appear, sasl_passwd.db. Place it in /etc/postfix and remove the clear text file sasl_passwd.

The configuration file of postfix, /etc/postfix/, should also be updated:

  • smtp_sasl_auth_enable = yes: tell postfix to use username and password for the authentication against the relay;
  • smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd: tell postfix where to find the credentials. Please note that postfix will read sasl_passwd.db and not sasl_passwd (the clear text file, which has to be removed). The suffix .db is implicit in the hash: specification;
  • smtp_use_tls = yes: use TLS for the communication with the relay. This is not mandatory, it depends on the relay. If unsure, just test it, connecting via telnet to the relay, i.e. telnet 25 (don't forget the port!), type at the prompt EHLO <relay address>, i.e., EHLO, hit enter and look for STARTTLS in the answer. Type quit to return to the shell. If STARTTLS was present, then the smtp_use_tls option should be set to yes;
  • smtpd_tls_CApath = /etc/ssl/certs: where postfix should find the CA certificates for the communication with the relay. The location provided is the default one for the openssl certs (usually installed by the corresponding package);
  • smtp_sasl_security_options = noanonymous: empty passwords not allowed.

There are also other obvious items, such as:

  • myhostname = the fully qualified domain name of the VM;
  • relayhost = [<your mail server>]:<port number<: the relay. Port is optional and the square brackets prevent to look for the MX record;
  • the file /etc/mailname should contain the fully qualified domain name of your VM (see myhostname);

It could happen that the relay complains about the fact that the address root@<fully qualified domain name of your VM> does not belong to your user (i.e., your email address) on the relay itself. This is legitimate, you are logged in in the relay as username and you're trying to send an email as someone else. In order to overcome this:

  • create the file /etc/postfix/sender_canonical with the following content: root$<fully qualified name of your VM> <your email address on the relay>. Add as many mappings as needed (i.e., if you need to send an email as user user1, then add also user1@<fully qualified name of your VM> <your email address on the relay>), one per line;
  • hash the previous file: postmap /etc/postfix/sender_canonical. The file /etc/postfix/sender_canonical.db will be created. You can remove /etc/postfix/sender_canonical, though it is not crucial;
  • edit /etc/postfix/ and add sender_canonical_maps = hash:/etc/postfix/sender_canonical. Please note that postfix will read sender_canoncial.db and not sender_canonical (the clear text file, which can be removed). The suffix .db is implicit in the hash: specification.

Once done with the customisations, please remember to restart postfix. A more in-depth view is provided here or in this guide.