Category Archives: linux

FB and G+, the new OS-wars

Back when Linux was a gangly youth, there was a great excitement every time you did an update. Unlike Windows and Mac with their expensive three-year-nothing-then-a-service-pack-with-cool-stuff routines, in Linux, there were so many cool free packages around, and /every one/ of them would have something new almost every week.

So, I spent quite a lot of time compiling and recompiling, everything from wget up to the KDE behemoth. And I’d be reading the weekly release notes as well to see if there was any new trick out that week.

I can’t tell you how excited I was when I first installed a copy of Linux where I didn’t have to configure X11 using a text-mode Xconfigureateur!

Or when I recompiled my kernel the first time, or when I recovered from a bad update straight from the GRUB command-line. Or the first time I ejected a CD from across the room (logged in remotely), or freaking out the wife by playing some music on her laptop from a different room.

In the last few years, Linux development has matured, though, so there’s not the same edge-of-the-seat excitement that there used to be, but I think there’s still hope for the techadrenaline junkies out there, because Facebook and Google+ are the new cool, and there’s a /ton/ of stuff that can be done with that!

My eyes are on how FB and G+ evolve. now that FB has competition, I expect some /really/ cool stuff is going to come out of the woodwork.

multi-tenanted CMS architecture

Last week, I did a talk at the Dublin Google buildings titled “Multi-tenanted CMS Architecture using PHP”.

Here are the slides that I used:

While talking with Google’s Brian Brazil, he explained that it is actually more efficient to use one database and many separate tables, than to separate each installation into a separate database, so one point I made (that KV-WebME uses separate databases per site) will change in the future.

I think the talk went down well, by the number of questions afterwards.

Last year, I gave a similar talk, and made the mistake of including way too much PHP in it – I had assumed that the audience would be composed of PHP developers. This year, there is just one slide of PHP, and that’s just to illustrate one possible way to build a proxy config.

Lesson’s learned for this time:

  • Talk slower. When I’m explaining something, I tend to try to get as much in as possible, so speak very fast. This makes it hard to hear what I’m saying.
  • More pictures, less words!
  • Stats. Some of the questions were around how efficient certain parts of the method were – particularly on the overhead of piping a file through a script as opposed to simply delivering it via Apache. I need to come up with numbers for that.

Overall, I was happy with this presentation.

script for shrinking videos

For this tip, you’ll need mencoder, ffmpeg and php installed. Only tested on Linux.

I’ve a load of videos, because I don’t like deleting anything in case I want to watch it again later (hoarding!).

I’ve three 1TB drives stuffed full of what I’ve collected over the last 20 years, but still I keep running out of room.

So, choice – buy more drives, or shrink what I have? I chose to shrink what I have, because I’m totally happy with VHS-quality video in stereo – I don’t need to watch fan-subbed Bleach in HD in 5.1 surround sound!

So, I wrote a script which will search the current directory (and its sub-directories) for all videos, will figure out their bit-rates, and will shrink them if they’re too large.

The script also reduces the videos so they match a maximum width and height.

It’s written in such a way that it will start with the most wasteful files (those that are taking up the most room), and work from that to the least wasteful.

It’s also written to only do 100 videos in a run. There’s no point shrinking absolutely every video you have, as most will probably save only a few kilobytes. The script does 100 of the most wasteful ones.

Of course, you could always run it again and again afterwards, and it should choose different videos each time to shrink.

How to use: copy the file into the root directory of your video repository. Go to that directory. Type “php shrink.php” into the console. Now go do something for a few days. Simple as that!

<?php

$maxwidth=624;
$maxheight=352;

function list_files($dir, $list) {
  $files=new DirectoryIterator($dir);
  foreach ($files as $f) {
    if ($f->isDot()) {
      continue;
    }
    $fname=$f->getPathname();
    if (is_dir($fname)) {
      $list=list_files($fname, $list);
      continue;
    }
    if (in_array(
      strtolower(preg_replace('/.*\./', '', $fname)),
      array('avi', 'mpg', 'mp4')
    )) {
      if (preg_match('/-shrink.avi$/', $fname)) {
        unlink($fname);
        continue;
      }
      $arr2=array();
      // { get duration of video
        exec('ffmpeg -i "'.$fname.'" 2>&1 | grep Duration', $arr2);
        $bits=explode(':', preg_replace('/.*Duration: ([^,]*),.*/', '\1', $arr2[0]));
        if (count($bits)!=3) {
          echo 'failed to get duration of "'.$fname.'"'."\n";
        }
        $duration=(int)preg_replace('/^0/', '', $bits[2])
          +((int)preg_replace('/^0/', '', $bits[1]))*60
          +((int)preg_replace('/^0/', '', $bits[0]))*3600;
      // }
      // { get width/height of video
        exec('ffmpeg -i "'.$fname.'" 2>&1 | grep "Video:"', $arr2);
        $bits=explode('x', preg_replace('/.*\s([0-9]+x[0-9]+)[\s,].*/', '\1', $arr2[1]));
        if (count($bits)!=2) {
          echo 'failed to get dimensions of "'.$fname.'"'."\n";
          continue;
        }
        $width=$bits[0];
        $height=$bits[1];
      // }
      $fsize=filesize($fname);
      $bps=$fsize/$duration;
      echo '.';
      $list[]=array($bps, $fsize, $width, $height, $duration, $fname);
    }
  }
  return $list;
}

echo "\ngetting file details\n";
$list=list_files('.', array());

echo "\nsorting by bits per second\n";
for ($i=0; $i<count($list)-1; ++$i) {
  echo '.';
  for ($j=$i+1; $j<count($list); ++$j) {
    if ($list[$j][0]>$list[$i][0]) {
      $tmp=$list[$j];
      $list[$j]=$list[$i];
      $list[$i]=$tmp;
    }
  }
}

echo "\nshrinking\n";
for ($i=0; $i<100&&$i<count($list); ++$i) {
  $fname=$list[$i][5];
  echo $fname."\nold size: ".$list[$i][1]."\n";
  @unlink('divx2pass.log');
  $resize='';
  if ($list[$i][2]>$maxwidth || $list[$i][3]>$maxheight) {
    $rx=$maxwidth/$list[$i][2];
    $ry=$maxheight/$list[$i][3];
    $ratio=$rx<$ry?$rx:$ry;
    $width=(int)($list[$i][2]*$ratio);
    $height=(int)($list[$i][3]*$ratio);
    $resize='-vf scale='.$width.':'.$height;
  }
  exec('mencoder "'.$fname.'" -ovc xvid -xvidencopts bvhq=1:chroma_opt:quan'
    .'t_type=mpeg:bitrate=400:pass=1 '.$resize.' -alang en -oac copy -o "'
    .$fname.'-shrink.avi" 2>&1');
  exec('mencoder "'.$fname.'" -ovc xvid -xvidencopts bvhq=1:chroma_opt:quan'
    .'t_type=mpeg:bitrate=400:pass=2 '.$resize.' -alang en -oac mp3lame '
    .' -o "'.$fname.'-shrink.avi" 2>&1');
  echo 'new size: '.filesize($fname.'-shrink.avi')."\n\n";
  if (filesize($fname.'-shrink.avi')<$list[$i][1]) {
    unlink($fname);
    rename($fname.'-shrink.avi', preg_replace('/\.[^\.]*$/', '.avi', $fname));
  }
  else {
    unlink($fname.'-shrink.avi');
  }
}

SaorFM, first few commits

Last Sunday, I announced that Conor and myself were starting a new file manager called SaorFM.

We started out slow, because we want to have absolutely everything in this to be checked by PHPUnit (for console-style testing) or Selenium (for GUI testing).

We’ve got enough written that you can do the following:

  • Write and save a config file.
  • Rename/Move files.
  • Delete files/directories.
  • List the contents of a directory.

All of the above have automated tests, so we’re fairly sure they’re correct. The main class’s source is readable here, and the test’s source is readable here.

Conor is currently adding support for SaorFM into his own CMS, Furasta. I’ll be working on adding it to my own tomorrow (got relatives coming over today – busy busy!).

We need to add some more functionality to make this basically complete:

  • Move directories.
  • Upload files.

After that, things get interesting, as we’ll be adding a plugin architecture.

Tomorrow, I’ll be finishing up the basics, and will write some simple jQuery plugins to select files, select directories, upload files, etc.

Soon, we’ll be catching up on KFM in functionality ;-) .

SaorFM

Last night, I spent four hours chatting with Conor MacAoidh.

We’re both the authors of CMSes, and both need file managers.

I’m the original creator of KFM, but recently, I’ve been getting annoyed at it. The project has grown too large to be easily managed, and it’s slow to start up because of the amount of database configuration involved.

We discussed this, and came up with a plan, which coincides with what I wanted to do for KFM2, but is probably much better.

We are going to reboot the whole thing – write a complete new file manager from scratch. It will only use code from the original KFM2 if the code is demonstrably better than any alternative we come up with.

The project will be properly documented, will have 100% test coverage, and will be completely free.

It will come in a number of separate parts, but only one, the core, will be absolutely needed.

The core of the engine is the bit which handles the actual file management. It will be designed to load in only two or three files for the most part, and as fast as possible.

Communication with the core will be done by either including the core as part of your own CMS, or by interacting with it via RPC.

The RPC will be very important – you send a command such as /rpc.php?action=move&from=/my-files/test1.jpg&to=/images/me.jpg, and results will be returned as JSON.

We decided on the name SaorFM. While this may be slightly confusing for non-Irish-speakers (“Saor”, pronounced half-way between “sair” and “seer”, means “Free”), we feel this is not very important. After all, Ubuntu is a household name, and that’s Bantu.

The main site will be SaorFM.org, the blog will be here, and downloads, issue tracker and SVN can be found here.

We’re still deciding on how to go about things, so there are no downloads yet. The decision to do this was made literally last night.

Having a co-developer on board from the absolute start will encourage me to get my arse in gear on this – if Conor does something cool, I have to beat that. And vice-versa, hopefully!

I’m starting the project off at the moment by working on a description of what it’s all about, and then will start writing some starter tests. This will use “test-driven development”, so every single line of code in this project will be repeatedly tested throughout the development.

We’re planning a load of features, such as desktop-/system- integration for Linux, Mac and Windows, and having totally external UI systems. We even considered going mad and creating a bump-top-like UI for it.

It’s taken me almost a year since planning KFM2 and getting to this.

Part of the reason for the delay is that this is so far removed from the current KFM, that I really didn’t know how to bring KFM1 up to the specification I wanted to reach.

Starting from absolute scratch with a brand new name is the right thing to do, I think.

audio alerts for php errors

I tend to keep a log tailing in one console while working in another.

tail is a Linux program that displays the last few lines of a file. If you run tail -f /var/log/httpd/error_log as root, or as your normal user if you’ve set the right permissions, then you will see any errors as they are added to the log.

However, I also tend to get immersed in my coding and not notice any errors until they cause a visual problem.

What I needed was a program that would watch my log, and beep to get my attention if an error was added.

This article explains how to set that up in Fedora.

First, install the program swatch:

[root@ryuk ~]# yum install swatch

Now create /etc/init.d/swatch, which is the startup/shutdown script for the logger:

#!/bin/sh

case "$1" in
'start')
                /usr/bin/swatch --daemon --config-file=/etc/swatch-httpd-errors --tail-file=/var/log/httpd/error_log --pid-file=/var/run/swatch-httpd-errors.pid
                ;;
'stop')
                PID=`cat /var/run/swatch-httpd-errors.pid`
                kill -9 $PID
                ;;
*)
                echo "Usage: $0 { start | stop }"
                ;;
esac
exit 0

Notice the two file locations in the /usr/bin/swatch command – the configuration file location, and the log file to be tailed.

Make the file executable:

[root@ryuk ~]# chmod +x /etc/init.d/swatch

Now create the configuration file, /etc/swatch-httpd-errors:

watchfor /PHP Parse error|PHP Fatal error/
        exec mplayer /home/kae/sounds/beep-8.wav > /dev/null 2> /dev/null

You can have swatch do basically anything, from sending an email, to flashing lights in your face if you have them connected to the computer. All I wanted was a little beep.

Change the exec command to whatever your want.

I started experimenting by having Arnold Schwarzenegger yell “What the hell are you doing??” at me, but that could get annoying for other people. In the end, I changed it to a beep – a very /short/ beep. More of a tick than a beep.

Now start up the daemon:

[root@ryuk ~]# /etc/init.d/swatch start

Create a php script with an error in it, and view it in your browser. You should get a satisfying beep.

If you want to have this run automatically when the laptop boots, add the above command to the /etc/rc.local file.

php.ie slowly upgrading

It’s been a while since I wrote anything vaguely technical. I guess it’s because I like to write only when there’s something new to say, and usually only if I have some new code to give away.

No new code today, but I can describe the recent work on php.ie (I’m the secretary of the Irish PHP Users’ Group).

So firstly, it was basically a static/brochure site for about a year, until we installed WebME (written by me!) as the CMS and created a skin for it so there’s only a tiny design difference. If you want to try out WebME, then download the SVN version from the google code site, or create a test site here (uses a really old version of WebME – you’re better off using the SVN version).

Then, I started rewriting the right panel. Beforehand, it displayed recent twitter messages, but they’re not often put out so it was a bit of a wasted space.

The panel now uses a WebME widget which displays recent Twitter messages, emails from the mailing list, and posts from the forum.

Over the next few days, I’ll be adding a new News section to the site, and the message widget will be able to show new articles from planet php.ie and new jobs from the jobs page.

I’m currently reading through Ken’s linux.ie todo list to see what I can appropriate for php.ie for its ongoing development.

Big thanks go to Michele and the team at blacknight for hosting the site.

Oh! Just a reminder, buy my book! JQuery 1.3 with PHP – hasn’t been reviewed by anyone yet, as far as I know, but my own opinion is that it is worth having on your shelf if you are a PHP developer that wants to step into jQuery.

jQuery 1.3 With PHP: chp7, image manipulation

I’ve submitted chapter 7 of jQuery 1.3 with PHP to Packt, which involves image manipulation. The editor I build in that chapter allows you to non-destructively manipulate an image in the most commonly needed ways – resize, rotate and crop.

The idea is that when you upload an image, it’s usually not yet right to include in a website. People tend to upload massive photos (3000×2000 or so) and resize it down using the <img> attributes, which is the wrong way to do it (see here for a solution to that particular problem).

However, once you’ve resized an image down, you can’t change it back.

The solution is to make copies of the image with your manipulations applied to them. That lets you make multiple versions of the same image.

But again, there’s the problem that if you change your mind about the original image, you can replace that, but the others all need to be redone.

Yet another problem is that unless you’ve kept records, there’s no link from the new images back to the original, so if you need to make space on the server by removing files that might not be in use anymore, you can’t be certain what thumbnails are still relevant and what are not.

The solution is to not actually create copies, but to record the changes applied to the original image in the URL of your manipulated image. When the server is asked for those URLs, it runs the manipulations on the original image and gives you back the new image. Of course, we will cache the new manipulations so this doesn’t need to be done every time, but this now allows you to replace the original image, then clear the cache, forcing all the variants of it to refresh, without you needing to do it again.

What this means, is that when the manipulation is recorded in the URL, no new copy is actually created until some browser actually requests it. This means that you can periodically clear your old thumbnails, safe in the knowledge that if they were still relevant, they will be recreated from the archived originals automatically the next time they’re needed.

Enough talk.

demo

This is, as usual, just a demo. It’s not designed to be impressive, but just to show how to do manipulations.

A permanent URL is provided, in the same manner as google maps – it updates itself as you manipulate it. To see the actual cropped, resized and rotated image, try opening that permanent URL in a different tab.

The GUI here is a means to an end (the permanent URL). In your own CMS, you might force a certain width/height for uploaded images automatically using the URL for example.

The example image I manipulated in the chapter is the IMG_0134.JPG in chp7. Try rotating, resizing, and cropping (drag your mouse on the image).

example:

Little thing to note about that URL – there’s no query indicator. Standards-compliant browsers treat the ‘?’ mark in a URL as an indication that the result should never be cached, but that would be pretty expensive for the server, so we remove that. The get.php file rebuilds the query from the $_SERVER['REQUEST_URI'] and sends out the manipulated image with caching headers.

The graphics manipulation was done using Imagick, which is a PECL extension that allows PHP to run ImageMagick functions internally.

In Fedora, you can install Imagick with yum install php-pecl-imagick. In CentOS, follow these instructions.

CentOS users might have a problem with the latest Imagick (2.2.2), so change the pecl install imagick line to pecl install imagick-2.2.1.

download the demo

Make sure to edit the images_libs.php file to point to your own image repositories. The images don’t need to be in a web-accessible directory.

I’ve a bit of work to do now, and then I’ll be applying the tricks in this chapter to our own CMS, WebME.

hosting multiple sites from the same CMS engine

I posted about webwork.ie‘s free website engine a few days ago. In the comments, Mickster asked how he would go about doing it himself.

I haven’t studied how other engines do it, but here’s how I do it.

First off, some benefits to sharing the CMS across separate sites:

  • Reduced resource usage. Because a single PHP engine is used, it is easier to optimise using RAM-disks, PHP caching, and other optimisations.
  • Easier upgrades – upgrade one system, upgrade them all!
  • Easier bug-fixing. One of the banes of my life is discovering a bug in a CMS that you know exists in at least ten other instances of that CMS. Now, you fix the bug in one place, and it’s fixed everywhere!

Convinced yet? Of course you are. Here’s how you do it.

First, separate all site-specific files from your engine files. To do this, you need to strictly keep your uploaded files and site-specific CSS in separate from your executable PHP. After doing this step, you should be able to clearly point out directories in your CMS and say “those are the engine’s executables and other resources, these are the design files and other uploadables which are site-specific.”

Second, all site-specific resources should be served through a script. The reason for this is that we are going to be moving the files away from the perceived directory that they’re in. Instead of, say, a web root of /home/webme/public_html/ with an image located at /home/webme/public_html/i/image.jpg, you might have the image located at /home/webme/sites/webme.eu/i/image.jpg, and another image from a different site located at /home/webme/sites/an-other-site.com/i/image.jpg. The choice of which image to show when /i/image.jpg is referenced depends then on the domain-name the browser requested the file from.

Then you need to over-ride the config at the time of its request. In the WebME CMS, the config file is located at /.private/config.php. What we want is to replace that with a “proxy” config file which, when included, checks the domain name and includes the real config file.

So how does the proxy config know where the real one is located? the way I handle this is to have a directory named after the site, located in /home/webme/sites/$NAME. If, for example, blah.webme.eu or www.blah.webme.eu is requested, you strip off any leading www. strings and trailing webme.eu strings with a regexp, and check to see if the config (/home/webme/sites/blah/config.php) exists. If so, include it.

To handle aliases, for example, if you wanted www.blah.com to load the blah.webme.eu site, you create a directory /home/webme/sites.aliases/$NAME, where $NAME (blah.com) is a symbolic link pointing to the correct directory.

It’s not a difficult trick, but it works.

Here’s my proxy config file:

Here is a copy of the proxy file that I use in webme.eu’s multi-user engine.

<?php
$host=preg_replace('/^www\.|\.webme.eu$/','',strtolower($_SERVER['HTTP_HOST']));
$cfile='/home/webme/sites/'.$host.'/config.php';
if(!file_exists($cfile)){
        $cfile='/home/webme/sites.aliases/'.$host.'/config.php';
        if(!file_exists($cfile)){
                header('Location: http://webme.eu/');
                exit;
        }
}
require $cfile;
define('CONFIG_FILE',$cfile);

running Thief in Wine

thief1

WOOT! Finally got Thief: The Dark Project running in Linux

One of the reasons I finally removed my Windows partition was that it had become absolutely useless to me.

The only reasons I’ve used Windows at all in the last few years are to play games or to test something in IE. Absolutely nothing else. And these days, most of the games I like to play simply do not work on Windows anymore. Vista, for example, refused to play Thief 2, and I haven’t been able to play Thief 1 since XP. For IE, I have another machine in the office in work, so there’s absolutely no reason for me to run Windows.

So anyway, I removed the partition with the reasoning that if Windows can’t play the games, and Wine can, then Wine is better. Ironic, really.

Thief is one of those game series that is just so brilliant that you need to buy it a few times, because you keep lending your copy to someone else and forgetting exactly who (and they like it so much they simply “forget” to give it back). Dungeon Keeper is another brilliant series like that.

Thief finally works in Wine 1.1.12, which was freshly released via Yum on the Fedora network just today. This is great news! The game did not work in Wine 1.1.11, when I tested last week. This week it’s perfect.

I fired up my previously failing installation, and after a few changes, was able to get it running.

Christos, in the bug tracker, says he re-installed the game and codecs to get it working. I didn’t have to reinstall anything. The only changes I had to make are:

  • use winecfg to change the emulation to Win98, and the graphics to use a virtual desktop. My laptop is widescreen, and Thief starts in 640×480 mode – for some stupid reason, x.org chops off the bottom of the screen when you request 640×480, and makes it 640×400. This makes Thief unplayable, so you need to use a virtual desktop.
  • write a small script to run the game in single-CPU mode – I created a file bin/thief containing this:
    #!/bin/sh
    
    cd ~/.wine/drive_c/Games/Thief/
    schedtool -a 0x2 -e wine THIEF.EXE

That was it – when I run thief from the console I get to play the game in full glory. It’s even better than the first time because this time my machine vastly beats the minimum requirements!