klog

Yesterday we had to move about 300 domains from one machine to another. We bought a new machine recently and are taking this opportunity to move from Qmail (difficult to use, in my opinion) towards Postfix.

After doing one or two by hand, i decided that’s stupid - why not just automate the whole thing.

So I whipped up a script that reads details from vqadmin and uses those details to create postfix emails using mailadmin.

Please note that this does not handle forwards and other weirdness - just plain old email accounts. After running it, you need to check the accounts for forwards. I /could/ adapt it, but am too lazy.

<?php

$vqadmin_url='http://1.2.3.4/cgi-bin/vqadmin/vqadmin.cgi';
$vqadmin_auth='username:password';
$postfixmailadmin_url='http://5.6.7.8/mailadmin/';
$postfixmailadmin_username='your@email.address';
$postfixmailadmin_password='password';

ob_start();

// { get all domains
	$ch=curl_init($vqadmin_url);
	curl_setopt($ch, CURLOPT_USERPWD,$vqadmin_auth);
	curl_setopt($ch, CURLOPT_POST, true );
	curl_setopt($ch, CURLOPT_POSTFIELDS, 'nav=list_domains' );
	curl_setopt($ch, CURLOPT_RETURNTRANSFER, true );
	curl_exec($ch);
	$page=join('',explode("\n",curl_multi_getcontent($ch)));
	preg_match_all('#<a href=vqadmin.cgi\?nav=view_domain&dname=([^>]*)>[^<]*</a>#',$page,$domains);
	curl_close($ch);
// }

$numdomains=0; $numemails=0;
foreach($domains[1] as $domain){
	$numdomains++;
	echo '<h2>'.$numdomains.' - '.$domain.'</h2>';
	// { get password page
		$ch=curl_init($vqadmin_url);
		curl_setopt($ch, CURLOPT_USERPWD,$vqadmin_auth);
		curl_setopt($ch, CURLOPT_POST, true );
		curl_setopt($ch, CURLOPT_POSTFIELDS, 'dname='.$domain.'&Submit=Show Users&nav=show_users' );
		curl_setopt($ch, CURLOPT_RETURNTRANSFER, true );
		curl_exec($ch);
		$page=join('',explode("\n",curl_multi_getcontent($ch)));
		curl_close($ch);
	// }
	// { extract passwords
		preg_match_all('#<tr><td><FONT face=Verdana color="\#FFFFFF"><a href=[^>]*>[^<]*</a></FONT></td><td align=middle><FONT face=Verdana color="\#FFFFFF">[^<]*</FONT></td><td align=middle><FONT face=Verdana color="\#FFFFFF">[^<]*</FONT></td><td align=middle><FONT face=Verdana color="\#FFFFFF">[^<]*</FONT></td><td align=middle><FONT face=Verdana color="\#FFFFFF">[^<]*</FONT></td><td align=middle><FONT face=Verdana color=\#FFFFFF>(<B>)?[^<]*(</B>)?</FONT></td><td align=middle><FONT face=Verdana color=\#FFFFFF>[^<]*</font></td></tr>#',$page,$matches);
		$matches=$matches[0];
		$emails=array();
		foreach($matches as $match){
			$username=preg_replace('#<tr><td><FONT face=Verdana color="\#FFFFFF"><a href=[^>]*>([^<]*)<.*#','$1',$match);
			$password=preg_replace('#<tr><td><FONT face=Verdana color="\#FFFFFF"><a href=[^>]*>[^<]*</a></FONT></td><td align=middle><FONT face=Verdana color="\#FFFFFF">([^<]*)<.*#','$1',$match);
			$emails[]=array('username'=>$username,'password'=>$password);
		}
	// }
	// { log into postfix mailadmin
		$ch=curl_init($postfixmailadmin_url.'login.php');
		curl_setopt($ch, CURLOPT_POST, true );
		curl_setopt($ch, CURLOPT_POSTFIELDS, 'fUsername='.$postfixmailadmin_username.'&fPassword='.urlencode($postfixmailadmin_password).'&lang=en&submit=Login');
		curl_setopt($ch, CURLOPT_RETURNTRANSFER, true );
		curl_setopt($ch, CURLOPT_AUTOREFERER, true );
		curl_setopt($ch, CURLOPT_COOKIEJAR, 'tmp/cookies.txt');
		curl_exec($ch);
		curl_close($ch);
	// }
	// { create domain
		$ch=curl_init($postfixmailadmin_url.'create-domain.php');
		curl_setopt($ch, CURLOPT_COOKIEFILE, 'tmp/cookies.txt');
		curl_setopt($ch, CURLOPT_POST, true );
		curl_setopt($ch, CURLOPT_POSTFIELDS, 'fDomain='.$domain.'&fAliases=25&fMailboxes=25&submit=Add Domain');
		curl_setopt($ch, CURLOPT_RETURNTRANSFER, true );
		curl_exec($ch);
		curl_close($ch);
	// }
	// { create email accounts
		foreach($emails as $email){
			$ch=curl_init($postfixmailadmin_url.'create-mailbox.php');
			curl_setopt($ch, CURLOPT_COOKIEFILE, 'tmp/cookies.txt');
			curl_setopt($ch, CURLOPT_POST, true );
			curl_setopt($ch, CURLOPT_POSTFIELDS, 'fDomain='.$domain.'&fUsername='.$email['username'].'&fPassword='.$email['password'].'&fPassword2='.$email['password'].'&fActive=on&fMail=on&submit=Add Mailbox');
			curl_setopt($ch, CURLOPT_RETURNTRANSFER, true );
			curl_exec($ch);
			$numemails++;
			echo $numemails.':'.$email['username'].'@'.$domain.' ';
			curl_close($ch);
		}
	// }
	flush();
	ob_flush();
}

Make sure to up the timeout length of your PHP scripts - it took me about 10 minutes to transfer 1607 emails.

Also, be aware that this does not transfer the /contents/ of the email accounts - just recreates them on the other server - I don’t even want to touch that particular problem…

Every now and then, I get a call from a client who is puzzled why their site is running slow. I would look at their page and see an innocuous image inserted into a paragraph. When I examine the image, though, I see that the client has artificially resized the image using HTML.

One recent example showed on-screen as a 300px-wide image. When I examined it, it was actually 3000px wide (approx). As explained to the client, this had the effect of forcing the browser to use about 100 times more RAM (not counting the overhead of the transformation to 300px-wide), and the download was slower as well.

One solution to all this is to teach all clients how to resize images before they upload them. I did that in this case. But it’s not the easiest solution, and people forget how to do things.

Another solution was proposed by Ken, and that is to parse any submitted HTML for images and check that the size they claim to be is actually correct. he said that he’d had the idea ages ago but never implemented it. I think its time has come, so let’s do it.

There are four ways that images can get resized. through HTML parameters, inline CSS, selector-based CSS and JavaScript. We will address the first two, as the others would be too complex to solve in a small application.

How this will work is that resized images, if detected, will be adjusted in the HTML so their ’src’ parameter points to a pre-created resized version of the image. The entire script is run when the HTML is submitted into a CMS, before the HTML is placed in the database or published to a file.

First, we need to detect image sources and their assigned sizes.

Here is some sample HTML with images from this site.

<p><img src="http://verens.com/wp-content/themes/mandigo-14/images/green/head.jpg" width="76" height="24" /></p>
<p><img src="/wp-content/themes/mandigo-14/images/green/head.jpg" style="width:76px;height:24px" /></p>

What we want is a function which, when fed that HTML, returns HTML which is modified such that images with incorrect widths and heights have their srcs modified to point to a pre-resized version, which is created using ImageMagick.

Here it is:

define('WORKDIR_IMAGERESIZES',$_SERVER['DOCUMENT_ROOT'].'/demos/html_imageresizer/f/');
define('WORKURL_IMAGERESIZES','/demos/html_imageresizer/f/');
function html_fixImageResizes($src){
	// checks for image resizes done with HTML parameters or inline CSS
	//   and redirects those images to pre-resized versions held elsewhere

	preg_match_all('/<img [^>]*>/im',$src,$matches);
	if(!count($matches))return $src;
	foreach($matches[0] as $match){
		$width=0;
		$height=0;
		if(preg_match('/width="[0-9]*"/i',$match) && preg_match('/height="[0-9]*"/i',$match)){
			$width=preg_replace('/.*width="([0-9]*)".*/i','\1',$match);
			$height=preg_replace('/.*height="([0-9]*)".*/i','\1',$match);
		}
		else if(preg_match('/style="[^"]*width: *[0-9]*px/i',$match) && preg_match('/style="[^"]*height: *[0-9]*px/i',$match)){
			$width=preg_replace('/.*style="[^"]*width: *([0-9]*)px.*/i','\1',$match);
			$height=preg_replace('/.*style="[^"]*height: *([0-9]*)px.*/i','\1',$match);
		}
		if(!$width || !$height)continue;
		$imgsrc=preg_replace('/.*src="([^"]*)".*/i','\1',$match);

		// get absolute address of img (naive, but will work for most cases)
		if(!preg_match('/^http/i',$imgsrc))$imgsrc=preg_replace('#^/*#','http://'.$_SERVER['HTTP_HOST'].'/',$imgsrc);

		list($x,$y)=getimagesize($imgsrc);
		if(!$x || !$y || ($x==$width && $y==$height))continue;

		// create address of resized image and update HTML
		$dir=md5($imgsrc);
		$newURL=WORKURL_IMAGERESIZES.$dir.'/'.$width.'x'.$height.'.png';
		$newImgHTML=preg_replace('/(.*src=")[^"]*(".*)/i',"$1$newURL$2",$match);
		$src=str_replace($match,$newImgHTML,$src);

		// create cached image
		$imgdir=WORKDIR_IMAGERESIZES.$dir;
		@mkdir($imgdir);
		$imgfile=$imgdir.'/'.$width.'x'.$height.'.png';
		if(file_exists($imgfile))continue;
		$str='convert "'.addslashes($imgsrc).'" -geometry '.$width.'x'.$height.' "'.$imgfile.'"';
		exec($str);
	}

	return $src;
}

The return string from calling that function with the above HTML is this:

<p><img src="/demos/html_imageresizer/f/6bf7dd2b8232448e85d7fa9cd1009b44/76x24.png" width="76" height="24" /></p>

<p><img src="/demos/html_imageresizer/f/6bf7dd2b8232448e85d7fa9cd1009b44/76x24.png" style="width:76px;height:24px" /></p>

Here is an example of it running, and here is the source of that demo.

Almost every time someone says to me that something I was working on is broken, the answer is “clear your cache”, and that automagically fixes everything. However, that’s not an ideal solution - ideally, the problem would never happen in the first place.

So, why does the problem happen?

Let’s say that there is a HTML file which calls a JS function like this: showImage(); - the JS function is included from the external file /j/images.js.

Browsers are usually set to cache .js files, and that’s the correct behaviour for the most part. Unfortunately, when a file needs to be fixed, it can cause problems.

For example, let’s say that I’ve corrected the function name to match my usual naming scheme - images_show();. I change the reference in both places. The browser reads the new HTML file from the net, but loads the JavaScript from the cache - suddenly there’s a mismatch which causes a problem.

So, how to get around this?

The solution I’m using at the moment involves a little bit of mod_rewrite and PHP.

Sticking with the contrived example, let’s rewrite /j/images.js so it is accessible from /j/images (using /.htaccess):

ExpiresActive On
ExpiresDefault A259200
RewriteEngine on
RewriteRule ^j/images$ /j/images.js [L]

Now, we add a little magic. We want to change the URL if the file has changed. The only way to know this is to look at the modified date of the file.

In your PHP, you could do it like this:

<script type="text/javascript" src="/j/images/<php? echo md5(`ls -l j/images.js`); ?>"></script>

and then change the .htaccess file to allow that:

ExpiresActive On
ExpiresDefault A259200
RewriteEngine on
RewriteRule ^j/images/(.*)$ /j/images.js [L]

Now, if no file changes happen, then the MD5 hash (and therefore the URL) will be cacheable, and if the file changes, then the URL will automatically change as well.

…and that’s not all!

I like to aggregate my JavaScript files to reduce the network pain felt by the browser. In my CMS, it’s done with a /j/js.php file. Here’s a short excerpt:

<?php
$js=file_get_contents('jquery-1.2.3.min.js');
$js.=file_get_contents('js.js');
$js.=file_get_contents('tabs.js');
$js.=file_get_contents('addrow.js');
$js.=file_get_contents('formhide.js');
/* more files */

header('Cache-Control: max-age=2592000');
header('Expires-Active: On');
header('Expires: Fri, 1 Jan 2500 01:01:01 GMT');
header('Pragma:');
header('Content-type: text/javascript; charset=utf-8');

echo $js;

That’s then pointed to with this line in my .htaccess:

RewriteRule ^js/(.*)$ /j/js.php [L]

And it’s referenced in the browser like this:

echo '<script type="text/javascript" src="/js/'.md5(`ls -l j`).'"></script>';

Simple, innit! That simple trick now keeps track of a number of files, and the browser knows immediately if there are any changes.

BTW: The same trick can be used with images, css, and any number of other “static” objects.