vm's and bridges and proxies

oh my!

We bought a rack server for the office, to help replace our aging systems with something a little more civilised. I spent a lot of time this week trying to figure out how to configure it best.

I wanted to install services and servers on the machine in such a way that I could easily move them onto a new machine if things get too busy. For this, I chose to use the QEMU virtual machine emulator. Some people might think that VMWare would be a better choice, but I did some research on it and couldn’t find any compelling reason why I should choose VMWare over QEMU.

To have the system networked properly in the LAN, I wanted to be able to address each vm using a separate IP number. To do this, I had to set up QEMU to use eth0 as a bridge. So, I had this in the host’s /etc/rc.local.

echo 1024 > /proc/sys/dev/rtc/max-user-freq
modprobe kqemu
modprobe tun
/etc/init/iptables down

/sbin/ifdown eth0
/sbin/ifconfig eth0 0.0.0.0 up
/usr/sbin/brctl addbr br0
/usr/sbin/brctl addif br0 eth0
/usr/sbin/brctl stp br0 off
/sbin/ifconfig br0 192.168.2.1 netmask 255.255.0.0 up
/sbin/route add default gw 192.168.1.254

and this was in /etc/qemu-ifup

#!/bin/sh
/sbin/ifdown eth0
/sbin/ifconfig eth0 0.0.0.0 up
/sbin/ifconfig $1 0.0.0.0 promisc up
/usr/sbin/brctl addif br0 $1
/sbin/route del default
/sbin/route add default gw 192.168.1.254

Note that I’ve used $1 instead of tap0 (which is shown in some examples) – this is because when you start up your QEMU instances, each one should use a different tap device.

When loading the QEMU instance, be sure to give each one a different MAC address. Otherwise strange stuff will happen.

xhost +local:root
su -c "qemu -boot c -hda vmServices.img -localtime -net nic,macaddr=52:54:00:00:00:01 -net tap -m 192 -usb -soundhw sb16 &"

In the above case, I’m loading a QEMU instance saved as “vmServices.img”, and have given it a MAC address 52:54:00:00:00:01. The default address is 52:54:00:12:34:56. Be sure to override that.

You’ll have noticed that I turned off iptables in the host’s /etc/rc.local. I’m not an expert at that stuff so that was the simplest solution to enable networking without problems. Be sure to also do it in the client’s /etc/rc.local files as well – otherwise you may have problems accessing hosted web servers, for example.

When the client is loaded up, assign a static IP address to it. I choose static IPs for these servers because they’re not client machines, and I need to be able to consistently access the right one from an external request.

Now you have your network up and running properly, with separate IP addresses for each vm.

The next step is to route incoming web traffic to the right machines.

Let’s say that you want a worker outside the office to access dotproject.youroffice.com, and you want a client to see his test server using blah.com.test.youroffice.com. The problem is that you are using a standard DSL connection, only have one static IP, and the dotproject and test web servers are held in separate VMs on the machine.

In this case, the solution is to use mod_proxy to route to the right machine.

So, you set up a rudimentary virtual hosted webserver on the host machine. The first virtual host should be something generic which perhaps just reports the status of the host. After that, we add the magic:

<VirtualHost *>
  ServerName dotproject.youroffice.com
  ProxyPreserveHost On
  ProxyPass / http://192.168.2.2/
  ProxyPassReverse / http://192.168.2.2/
</VirtualHost>
<VirtualHost *>
  ServerName test.youroffice.com
  ServerAlias *.test.youroffice.com
  ProxyPreserveHost On
  ProxyPass / http://192.168.2.3/
  ProxyPassReverse / http://192.168.2.3/
</VirtualHost>

From an external browser’s perspective, both web servers are running on the same machine, but internally, we can see that there are three involved – a proxy router, and the two separate virtual machines’ web servers.

There may be more-correct ways of doing the above, but this works for me.

3 Comments.

  1. Thanks for the how-to. I must look into QEMU more. Did you consider Xen at any stage?

  2. It didn’t even occur to me. I had used QEMU a few times for my own use – for installing winXP under Linux on my laptop for testing IE7, for example. VMWare is used by a few people I know in a “partner” company. I remember looking into Xen a while back but it was not clear how to use it.

    Maybe that has changed, but I tend to stick with things that I know or have seen used. A bit conservative of me, maybe, but at least I am fairly sure I’ll know what’s going on during the installation and maintenance!

  3. Same with me so, I’ve used Xen a couple of times so its a case of sticking to what I know aswell. It does 99% of what I need. Running windows on Xen however is a PITA as it requires a cpu with virtualisation support. I might look to QEMU for a solution to that however.

    Thanks again for the heads-up.

%d bloggers like this: