klog

oh my!

We bought a rack server for the office, to help replace our aging systems with something a little more civilised. I spent a lot of time this week trying to figure out how to configure it best.

I wanted to install services and servers on the machine in such a way that I could easily move them onto a new machine if things get too busy. For this, I chose to use the QEMU virtual machine emulator. Some people might think that VMWare would be a better choice, but I did some research on it and couldn’t find any compelling reason why I should choose VMWare over QEMU.

To have the system networked properly in the LAN, I wanted to be able to address each vm using a separate IP number. To do this, I had to set up QEMU to use eth0 as a bridge. So, I had this in the host’s /etc/rc.local.

echo 1024 > /proc/sys/dev/rtc/max-user-freq
modprobe kqemu
modprobe tun
/etc/init/iptables down

/sbin/ifdown eth0
/sbin/ifconfig eth0 0.0.0.0 up
/usr/sbin/brctl addbr br0
/usr/sbin/brctl addif br0 eth0
/usr/sbin/brctl stp br0 off
/sbin/ifconfig br0 192.168.2.1 netmask 255.255.0.0 up
/sbin/route add default gw 192.168.1.254

and this was in /etc/qemu-ifup

#!/bin/sh
/sbin/ifdown eth0
/sbin/ifconfig eth0 0.0.0.0 up
/sbin/ifconfig $1 0.0.0.0 promisc up
/usr/sbin/brctl addif br0 $1
/sbin/route del default
/sbin/route add default gw 192.168.1.254

Note that I’ve used $1 instead of tap0 (which is shown in some examples) - this is because when you start up your QEMU instances, each one should use a different tap device.

When loading the QEMU instance, be sure to give each one a different MAC address. Otherwise strange stuff will happen.

xhost +local:root
su -c "qemu -boot c -hda vmServices.img -localtime -net nic,macaddr=52:54:00:00:00:01 -net tap -m 192 -usb -soundhw sb16 &"

In the above case, I’m loading a QEMU instance saved as “vmServices.img”, and have given it a MAC address 52:54:00:00:00:01. The default address is 52:54:00:12:34:56. Be sure to override that.

You’ll have noticed that I turned off iptables in the host’s /etc/rc.local. I’m not an expert at that stuff so that was the simplest solution to enable networking without problems. Be sure to also do it in the client’s /etc/rc.local files as well - otherwise you may have problems accessing hosted web servers, for example.

When the client is loaded up, assign a static IP address to it. I choose static IPs for these servers because they’re not client machines, and I need to be able to consistently access the right one from an external request.

Now you have your network up and running properly, with separate IP addresses for each vm.

The next step is to route incoming web traffic to the right machines.

Let’s say that you want a worker outside the office to access dotproject.youroffice.com, and you want a client to see his test server using blah.com.test.youroffice.com. The problem is that you are using a standard DSL connection, only have one static IP, and the dotproject and test web servers are held in separate VMs on the machine.

In this case, the solution is to use mod_proxy to route to the right machine.

So, you set up a rudimentary virtual hosted webserver on the host machine. The first virtual host should be something generic which perhaps just reports the status of the host. After that, we add the magic:

<VirtualHost *>
  ServerName dotproject.youroffice.com
  ProxyPreserveHost On
  ProxyPass / http://192.168.2.2/
  ProxyPassReverse / http://192.168.2.2/
</VirtualHost>
<VirtualHost *>
  ServerName test.youroffice.com
  ServerAlias *.test.youroffice.com
  ProxyPreserveHost On
  ProxyPass / http://192.168.2.3/
  ProxyPassReverse / http://192.168.2.3/
</VirtualHost>

From an external browser’s perspective, both web servers are running on the same machine, but internally, we can see that there are three involved - a proxy router, and the two separate virtual machines’ web servers.

There may be more-correct ways of doing the above, but this works for me.

Spent a while looking for info on this online, but couldn’t find it.

We have a windows machine which has a few printers on it. I want to connect to it from Linux (Fedora 8).

First, share the printer via windows share. Give it a name like OKIB2200.

Now, open system-config-printer

Choose “New Printer”, then “Windows Printer via SAMBA”. If that option does not appear, then close out of system-config-printer and install samba (yum install samba*) then repeat the above.

Enter the appropriate smb address. In my case, it was smb://mshome/192.168.1.252/OKIB2200.

Click “Verify” to make sure it’s accessible (if not, check spelling in the URL. if still not, I dunno…).

For the Make, choose Generic

For the Model, choose PCL 4 (laserjet).

test, and you’re done.

Just thought I’d regale ye with two errors from my past.

The first involves SSH. SSH rules. It is /so/ handy to be able to work on many machines simultaneously. However, there is a single fundamental flaw to this method of working.

[root@localhost ~]# poweroff

I turn my laptop off every night. As I usually have my fingers on the keyboard, I usually turn it off using the above command.

One night, I turned my machine off, then went upstairs to bed. I realised I’d forgotten to lock the back door, and came down to do so.

My laptop was still humming away.

I don’t think you can imagine the feeling of dread that came over me. A cold sweat welled up, and I could feel the hair on the back of my neck raising. I realised I must have typed poweroff in an active SSH connection. I checked, and it was our live production server, with a hundred or more active websites on it.

The next hour or two was frantic, as I tried to get in contact with the hosting provider (which has a supposed “24-7″ phone line which is only ever answered from 9-5).

Luckily, it was Sunday, so no-one noticed before the provider’s support techies finally turned up for work in the morning and turned it back on.

As soon as I knew the machine was on, I logged into it and added this line to the ~/.bashrc file:

alias poweroff="echo No! Step away from the computer and think about what you\'re doing"

The second error is probably a common one.

It /sounds/ like a good idea to set up an error log which emails you if an error ever occurs on your server.

Don’t! Or at least, read on and find out how to do it properly.

I made the mistake of setting up an error catcher which would email me as soon as an error occurred. The reasoning was that I’m usually online, so I could catch the error quickly and fix it before the client even noticed the error was happening.

Unfortunately, one day I made an upgrade to one piece of code which adversely affected another piece that was almost unrelated (so it didn’t occur to me to check it - admit it, you’ve all made this mistake).

I went home and was offline for the rest of the day. The following day, I came in to find the office phone’s answering machine was blinking. Apparently, that client’s site had “stopped responding”.

I thought it would be something simple, so tried to log in. The system was /slow/. After a few minutes of painful testing (even SSH is very slow if your load is high enough!), I found the problem - about 5 million emails in the email queue, and qmail was going CPU mad trying to deal with it all.

It turns out that any time anyone accessed the client’s site, it would trigger a recursive chain of events which each caused an error. Before we could even start on solving the problem, we needed to turn off web access to the machine! Not a good thing, when your business is the web.

The next few hours days were spent clearing the emails and scouring for any code which sends an automatic error email…

The solution to the above? Log your errors to the syslog instead of via email. If you still want an email sent out, then set it up using a cron job which emails the contents of the log file then clear it.

Moral of the story is - there is always a better way to do things. Usually, though, it will not occur to you until after the damage is done.