Less short: I’m trying to get work out the door, get a good run at some personal projects, pass grade 2 piano, get organised, and generally improve my lot.

None of this is working. I think the “get organised” bit is the most important, as it will help the [...]]]> Short run-down of what I’m doing lately: nothing.

Less short: I’m trying to get work out the door, get a good run at some personal projects, pass grade 2 piano, get organised, and generally improve my lot.

None of this is working. I think the “get organised” bit is the most important, as it will help the rest of it fall into place.

I usually only post about web-development-related topics here, as that’s the only subject where I feel I can contribute something new and interesting, so I tend to not talk about other stuff. But sometimes, rattling off the current state of the head is good for clearing it.

In work, I can’t really complain – we have a number of largish projects which are slowly creeping towards completion. The hardest thing about them is getting information from the clients, and then a week or two later being told that half the information is not required. I guess my main complaint at work is the inexorably slow completion rate.

On the personal projects side:

There are still a number of small bugs in KFM 1.4, and either I don’t have the time to get to them, or there is no enough information to recreate the bug and the submitter doesn’t give me access to their copy so I can’t see it from their side.

KFM 2 has been halted for a while – the idea is huge, but I simply don’t have the time, and no-one is clambering for it. I’ll get to it when I have time, but I might have to approach it by evolving KFM 1.x into meeting what I wanted, instead of the original goal of building KFM 2 from scratch.

I started a new project, OddJobs4Locals two weeks back, and got a good two-day run at it, then time got ahead of me again. I think this will be a good one, when I can complete it. Useful for students, people with a little spare time, or simply people that just want to make a little extra cash. Not yet working, but it will be soon, I hope… This is doubly interesting to me, as it is done purely through AJAX, so it will be easy to do a smart-phone client or a desktop client when the time comes.

I’m in the back/forth stage of working with Packt publishing to see if they want me to do a second book (the first one has no bad reviews at all). We’ve mostly agreed on a table of contents, and I’m just trying to get the time to combine a few of the smaller chapters together.

On the piano, I’ve been ready for the grade 2 exam since November, and am still waiting to see if there will be an exam near me any time soon – I hate the effort that goes into travelling (I have a family, and no car). I was hoping to do a grade every 6 months. It looks like this might not be possible, despite me being ready for it… The tunes I’m doing for it are Beethoven’s Sonatina in G Major, a waltz by Bela Bartok, and Boys And Girls Come Out To Samba, by Terence Greaves – by the way, I don’t like those videos; there are no dynamics in any of them, and I can hear a number of mistakes as well. No video apparently of the Terence Greaves one.

As for organisation… well I guess I’d better start working with Mantis again.

My lot will have to wait – I’ve a load of work to get done before it can improve.

Meh. Depression taking hold again.

To do this, first create the element:

Notice [...]]]> As a response to a reported bug where Chrome was taking ages to load up a flash multiple-file uploader, I’ve updated KFM to use HTML5’s multiple-file input box where possible.

To do this, first create the element:

  var input=document.createElement('input');
  input.type='file'; // use old-style JavaScript method to make sure all browsers respect it
  input.name='kfm_file';

Notice that we’re not using setAttribute to set the type and name – that’s a DOM method which works in most browsers but not (of course…) in Internet Explorer 6, where it has bugs.

And now, we tell the input to use the multiple-upload method. We use .setAttribute in this case because we only expect newer browsers to succeed with it.

  input.setAttribute('multiple','multiple');
  if(input.multiple)input.name='kfm_file[]';

In the second line, we check to see if the element is now marked as a multiple-uploader (most current browsers will not succeed in this), and if it does, then rename the input element by adding a [] to the end. If this is not done, then the server will only see the first file which is uploaded.

That’s the client-side done. This will only be visible in newer browsers such as Chrome, Safari 4, Firefox 3.6. I expect Internet Explorer will eventually catch up by 2020 or so.

If you’re doing this in pure HTML, then I suppose this would be good enough for you:

<input type="file" multiple="multiple" name="file[]" />

In this case, you must put the [] in the name in all cases.

On the server-side, you need to write your upload receiver to expect either a single element, or an array.

For some really goddamned stupid reason, when multiple files are uploaded to PHP, the results are interlaced in a really crappy and awkward manner (I don’t like it).

Instead of something logical and easy to use, like this:

array(
  [0] => array(
    'name' => 'file1.txt',
    'tmp_name' => '/tmp/abcdef'
    ....
  ),
  [1] => array(
    'name' => 'file2.txt',
    'tmp_name' => '/tmp/ghijkl'
    ....
  )
);

You get this…

array(
  'name' => array(
    [0] => 'file1.txt',
    [1] => 'file2.txt'
  ),
  'tmp_name' => array(
    [0] => '/tmp/abcdef',
    [1] => '/tmp/ghijkl'
  ),
  ...
);

While that looks at first glance to be easy to use, it’s not. You can’t do a simple “foreach($_FILES['kfm_file'] as $file)” and expect the above to be usable at all…

So, the first thing I do, is to check for the $_FILES['kfm_file'], and convert it into the first form above, which is very easy to work with:

$files=array();
$fdata=$_FILES['kfm_file'];
if(is_array($fdata['name'])){
 for($i=0;$i<count($fdata['name']);++$i){
  $files[]=array(
   'name'    =>$fdata['name'][$i],
   'tmp_name'=>$fdata['tmp_name'][$i],
  );
 }
}
else $files[]=$fdata;

In my own case, I’m only interested in the name and tmp_name variables, so that’s all I set up.

Now you can do a foreach on $files and treat them all individually.

foreach($files as $file){
  // uploaded location of file is $file['tmp_name']
  // original filename of file is $file['file']
}

If you want to see this in KFM, have a look at the nightly-updated demo tomorrow, or download from SVN right now.

oh – and buy my book!

here is the demo

You can see from the source how short it is – all it does is load up the jsTree library (and jQuery), then it connects directly to KFM2 to get data about the [...]]]> As an example of a front-end widget, I’ve whipped up a demo of file-tree navigation using the excellent jsTree widget.

here is the demo

You can see from the source how short it is – all it does is load up the jsTree library (and jQuery), then it connects directly to KFM2 to get data about the files on the server.

in fact… it’s so short I’ll just paste the demo into this page as well:

You should see a list of files just above this paragraph.

No modification of jsTree was necessary, as it handily gives two event triggers which can be used to translate the request data to KFM’s request format, and the received data back into jsTree’s data format.

Here is the code used to attach the tree to the #tree_div element in this page:

$("#tree_div").tree({
	data : {
		type : "json",
		async : true,
		opts : {
			async : true,
			method : "POST",
			url : "/demos/kfm2/trunk/?p=get_file_listing" // KFM LOCATION
		}
	},
	callback : {
		'beforedata':function(node){
			var id=$(node).attr('id');
			if(!id)return {'d':'/'};
			return {'d':$(node).attr('id').replace(/^kfm2_node_/,'')};
		},
		'ondata':function(res){
			var data=[],i;
			for(i in res.files)data.push({'data':i,'attributes':{'id':'kfm2_node_'+res.d+'/'+i}});
			for(i in res.directories)data.push({'data':i,'state':'closed','attributes':{'id':'kfm2_node_'+res.d+'/'+i}});
			return data;
		}
	}
});

This is just an example of what can be built quickly with the new RPC-based KFM.

Further and more useful examples will come soon.

Anyway, I’ve put together the basics enough that there is one single plugin, which gets a directory listing, and echoes it to the screen in JSON.

You [...]]]> I know, I promised this last week – I’m afraid I’ve been up to my eyes in work and just exhausted at home, so all I can do is apologise.

Anyway, I’ve put together the basics enough that there is one single plugin, which gets a directory listing, and echoes it to the screen in JSON.

You can get a copy of the basics using the following SVN command:

svn checkout http://kfm.googlecode.com/svn/kfm2/ kfm2

In that, there are two directories – trunk and plugins. The trunk is what will become the new KFM core, and the items in plugins will be server-side plugins that can be used by copying them into trunk/plugins/ (or do as I do and ln -s ../../plugins/* from the trunk/plugins/ to create symlinks).

When you load up KFM2 initially, it will do an extremely basic install, with no plugins enabled. Follow the instructions on-screen to get set up.

The config.php contains just two items at the moment. I want to keep this as absolutely clean as possible, so the items in there are the only essentials – all other configuration will be done using KFM’s admin section.

• KFM_USERDIR – this is the directory that is to be managed by KFM. Just like KFM1, a directory called .files will be created in there that holds config stuff, such as the list of enabled plugins, etc.
• KFM_ADMIN_PASSWORD – this password allows access to KFM’s admin section, where you can currently just enable/disable plugins. By default, it is blank, and you will need to change it before you can access the admin section.

There is no JavaScript yet in this, so it’s tricky to see exactly what it’s supposed to do.

I’ve set up a basic demo here, but loading that directly will just get you a login screen.

KFM2 is designed to be interacted with purely through RPC (except for configuration which requires browser access). The demo has the first plugin, “get_file_listing” enabled, which we can use to see what happens.

/?p=get_file_listing – this example calls KFM2 and tells it to run the get_file_listing plugin. by default, it will return the root directory of your KFM_USERDIR in JSON format:

{
  "d":"",
  "files":{
    "file2":{
      "size":0,
      "mtime":1255875795
    },
    "file1":{
      "size":0,
      "mtime":1255875794
    },
    "file3":{
      "size":0,
      "mtime":1255875796
    }
  },
  "directories":{
    "dir1":{
      "mtime":1255875809
    }
  }
}

The d parameter there is the directory that was requested (by default, it’s blank), which is internally added to the KFM_USERDIR to get the actual local directory.

The results returned are separated into two arrays, of files and directories. My guess is that most required uses on the client-side will be specific to either directories or files, so this saves you having to separate them yourself.

Changing the requested directory is simple – just add a d parameter to the request URL:

/?p=get_file_listing – this example requests the contents of /dir1.

How to write your own plugin

The system as it is will let you write plugins that handle most of the things that you might want to do – upload files, create/move/delete files or directories, download files, etc.

I’ll get around to writing these anyway, but if you feel like writing them, then feel free! Just copy how the get_file_listing plugin works.

So, here’s how a basic plugin works.

In KFM2’s plugin’s directory, create a directory named after the plugin. Please use just lowercase letters and underscores. Examples: “upload”, “get_file”, “rename_file”, “zip_directory”, etc.

In that directory, create a plugin.php file, which describes the plugin:

<?php
$plugin=array(
  'name'=>'Get File Listing',
  'description'=>'Retrieves a list of files and directories within a given directory',
  'version'=>0
);

Use an integer for the version number. It will probably never need to be used, but if it is, it is easier to compare two integers than two of the usual x.y.z version numbers.

When you log into your admin area, you’ll see the details above shown in your plugin management module.

For most plugin cases, you will also need a file named after the plugin itself. Until the events code is written, this is all cases.

In our example, we create a file named plugins/get_file_listing/get_file_listing.php. It should contain a function also named after the plugin:

function get_file_listing(){
  // place plugin code here
}

In the URL above, I had /?p=get_file_listing – what KFM then did was to load up the get_file_listing/get_file_listing.php file and run get_file_listing(). That function then used whatever other things were mentioned in the $_REQUEST array to carry out its work.

That’s pretty much it for now. That should be enough to create most basic plugins.

For the next KFM article, I’ll write up another server-side plugin, and will add in some client-side code so you can see how it would actually be used in a real project. It will demonstrate how JavaScript and KFM will talk to each other.

If anyone writes up any plugins that they want included in the SVN download, please email them to me (kae@verens.com).

Requested plugins

Here’s the list of plugins that have already been identified as needing to be written.

Server-side plugins that can be written right now:

• file upload
• delete file / directory
• rename file / directory (also counts as “move”)
• download file
• create zip of directory
• unzip a zip file
• manipulate an online image

Plugins that require more work in the core:

• connect to remote file-systems (FTP, NFS, other KFMs, etc)
• user-based security
• different KFM_USERDIR based on the logged-in user, or even the site’s hostname
• plugin to resize images that were uploaded so they’re kept to a certain max size
• quotas (can /probably/ be done at the moment… but not sure)
• databases

The main problem with 1.x was that it was monolithic – there was one single codebase, which encompassed both the server-side and the client-side. The client-side code was useless without [...]]]> As ye all know now, KFM1.x is finished. I’m not interested in struggling with it anymore. I’ve started a new project which will exceed KFM1.x’s capabilities.

The main problem with 1.x was that it was monolithic – there was one single codebase, which encompassed both the server-side and the client-side. The client-side code was useless without the server-side, and the server-side code had hooks specifically designed to be used by the client-side.

While it was possible to build a different client-side GUI that would hook into the server-side, it would be a large undertaking, and would involve changing a lot of the PHP – especially as the PHP wrote the “boot” JavaScript in the first place…

KFM2 is designed from the beginning to have a different architecture based specifically on plugins. The core code is a small plugin manager (mostly written already), written in PHP, but it will not be necessary that the client-side is in JavaScript – or even that there is any interaction with a browser at all!

The idea is this – if you want to do something incredibly simple, such as select an existing directory that will be used in your CMS to create an image gallery out-of (let’s say you need to choose from a number of existing directories full of images), then it is a total waste of bandwidth and CPU to load up a full-on file-manager.

In this case, and it’s the first I’ll be building to show how all this works, what should happen is that the browser will show something simple like this:

…and when that’s clicked, a small popup will appear which lets you choose the directory you want, in a tree-like manner – you open the branches until you find the one you want, then you click it. Afterwards, the form will change:

From the technical point of view, there’re a few things to explain about how this would be done.

jQuery would be used to attach an event to the button which would popup the selection window. This selection window will use Ajax to speak with the KFM2 server and retrieve the list of directories, one level at a time as the branches are expanded. When one is selected, a hidden input will be set to the name of the directory.

This sounds incredibly simple, and it is. What’s important, though, is that no-one is doing this at the moment!. Right now, if you go search for existing plugins to do this kind of thing, you’ll find either small widgets with both the server and client-side code ready for you, or full-on applications (such as KFM1.x) which can do the job, but which are overkill.

The problem with the applications is obvious – too much stuff going on, making it slow and cumbersome.

The problem with the small widgets is that each one is complete in itself, and therefore a lot of the functionality is repeated. This is a bad idea – what if you need to make a site-wide change? You’d need to go through every widget and change it!

What I’m proposing to make of KFM2 is that it will mostly be a server-side application, but which has a nice easy-to-use API which can be hooked into by lots of different jQuery widgets (or Mootools, or whatever you want, really – write it in Python or Delphi if you want…).

So, my first widget will be the one described above – a directory selector.

I’d like to know what other things people need to do with files online, which are simple and they’d prefer not to have to load a whole file-manager to do.

I’ll release the first widget and a demo of it some time over the next week. I intend to write a whole load of widgets, culminating in a full-on GUI similar to the present KFM1.x one, all hooking into the new engine.

KFM 1.4.2 has been released. It’s a minor maintenance release, but it adds support for CKeditor. TinyMCE and FCKeditor have been checked as well.

demo using CKeditor

This release also adds a configurable restriction, where images larger than a certain resolution will simply not be allowed in the repository.

Very large images can cause servers to crash when [...]]]>

KFM 1.4.2 has been released. It’s a minor maintenance release, but it adds support for CKeditor. TinyMCE and FCKeditor have been checked as well.

demo using CKeditor

This release also adds a configurable restriction, where images larger than a certain resolution will simply not be allowed in the repository.

Very large images can cause servers to crash when the system tries to generate thumbnails, because of the large amount of RAM needed. For example, a 3000×2000 image will use up at least 24MB just to hold the thing in memory, and much more to handle the manipulation. 24MB is a large amount of RAM to a web app.

There are also some code additions which are not used by the KFM GUI, but can be used by tools that hook into the KFM engine. These allow uploaded files to be placed in directories specified at the time of upload, or to replace existing files, or even to take an uploaded image and apply HSL transformations to it (when ImageMagick is installed). These are not documented, but anyone that needs this functionality will be a good enough programmer to read the source and find it (or email me ).

Download KFM 1.4.2 here

From my records, I can see that KFM has been installed on over 9000 separate domains. That is quite a lot of sites, and says quite loudly how important it [...]]]> KFM 1.x has reached the end of its development. It has fulfilled its original purpose; to improve on the default file-manager for FCKeditor, and well-exceeded it.

From my records, I can see that KFM has been installed on over 9000 separate domains. That is quite a lot of sites, and says quite loudly how important it is that KFM2 is at least as good as KFM1 (we all know about the KDE4.0 fiasco…).

KFM2 will blow KFM1 well out of the water. It will do everything that KFM1 does, but much quicker, and with a much less monolithic architecture, allowing it to be hugely flexible. I am planning on creating a download section similar to jQuery UI’s, where you can choose the components that you want in the system, and it will be built up and configured for you, ready for download.

See, KFM 2 will be modular. If you just want a simple upload/download facility, then you will only need the core, and plugins to handle uploads and downloads. If you want the whole shebang, you will find plugins for various ways of selecting files and directories, full-on graphical UIs such as the existing KFM1 forces on the user, plugins for hooking into external file systems using FTP or whatever other method people can imagine, plugins for multi-media, file editing, databases, user authentication, search, RSS, and so on.

Basically everything that KFM1 already does, but with the option to easily remove/add bits that you want. Make it as fast or as “bling” as you want.

KFM2 will have a number of new things which are not available in any other online file-systems, that I know of.

new file-system things

The file system will be modular. You will be able to attach symbolic links to any part of the file system, linking to other external systems.

As an example, let’s look at a URL: http://example.com/kfm/get/images/site1/logo.jpg.

One of the new ideas is that this will not necessarily be located on the server named example.com.

If you are running a very busy website, you might prefer to delegate image management to a different machine deeper in your network, so you add a symbolic link /images/ linking to the image-management machine. This causes the server to issue a redirect to the browser, redirecting it to http://images.example.com/kfm/get/site1/logo.jpg.

Now, imagine that the image management server doesn’t host the logo.jpg file on itself. Maybe it’s one of a number of images.example.com servers in a load-balancing cluster, and the file is actually located on a file-server accessible only by FTP. So, we add a symbolic link in the server’s KFM, telling it how to connect to the FTP server. The image server does this, gets the required file, and sends it to the browser.

This sounds a bit complex, but at least it’s possible with the new system. The old system simply would not be able to handle that at all.

API

At the moment, if you want to select a file or directory, you fire up KFM1, wait 15 seconds for it to finish booting itself up, and select the file or directory. That’s way too long. Really, you should only need to load up a tiny widget that’s designed specifically for picking a file.

The new system will have a simple core with an API that can be connected to by little JavaScript widgets. As an example, let’s say you just want to get the file-listing for the foo directory.

At the moment, you need to go through the whole business of booting up KFM’s GUI, navigating to the right directory, then waiting for the server to connect to the database, generate thumbnails and other unnecessary stuff.

You should just have to do something simple like load up http://example.com/kfm/rpc.php?a=get_file_listing&v=/foo through jQuery’s $.get() function. The RPC script does exactly what was asked, and nothing more.

This should allow us to write some incredibly fast and tiny scripts for all the file manipulation things you could possibly want.

Think of it as the “gnu” way of doing KFM – many small tools, each of which is designed to do one task well.

Of course, you will still be able to use the original GUI with this, after it’s been re-engineered to hook into the new RPC system.

Or, you can write your own – the RPC’s API will be well-documented, and will use simple HTTP parameters, so you can write your widgets in whatever language you want – JavaScript is what I will be writing in, but there’s no reason why you’re confined to that – write a GUI in Python, C, Flash – whatever you’re comfortable with.

plugins

I think the new plugin architecture will make this profoundly extensible. I’m writing the core to be as flexible and stable as possible. You will be able to write a plugin to just about anything:

• authenticate a user before serving a file.
• change a user’s root directory based on the server-name or the user’s authentication.
• the entire system is held on a different machine accessible only by FTP? fine – let’s change how files are accessed.
• record all files and directories in a database, allowing searches to be done.
• manipulate files online – edit text files, rotate or crop or resize images, move or rename or delete, etc. standard KFM1 stuff, these.
• create a log plugin, so all access is recorded.
• create a quota plugin, for denying uploads or downloads when the quota is exceeded.

It was possible for us (Benjamin and myself) to write all of these into KFM1, but as we are only two people, and very busy people at that, it’s never been the absolute top priority to get all these things done.

But, when KFM2 is released, you won’t have to wait for us anymore. You will be able to write plugins, or download plugins created by other people, which do all of the above and I’m sure there will be plugins that I simply have not imagined yet, that inventive people will come up with (Conor, I’m sure you’ll be one of those!)

languages

Languages have always been one of the cool things about KFM. Version 1.3 was released in about 13 languages in total. Unfortunately, the code got too complex in 1.4 to easily add more, but KFM2 is starting from scratch with the learning-curve of KFM1 well behind me. I’ve already made most of the big mistakes I’m likely to make, so from now on, things should be much easier.

How languages will work is that a website will be created which allows people to help out open-source projects, not just KFM by providing translations of words and sentences.

The programmer will create a screen-shot showing the text in use, and will provide the untranslated text in a format such as .po. Translators will translate the text on-line, and the translation will be available to download by the programmer as soon as it’s done.

To ensure good translation, each line will be presented to a number of different translators. The translations which agree the most with each other, will be considered the “right” one.

Using this, it will be possible for KFM to organically add languages with no interaction from myself or any other programmers. When first loaded, a plugin would try to translate messages into the user’s language. Finding that it doesn’t have the language in its database, it will try downloading it from the language translation website. If it’s not available, a request will be added that it be created. When translators that speak that language come to the website, they will be automatically given the KFM text to translate to that language. The next time the plugin checks, the translation might be done, and it will be added to the KFM instance’s local language database.

Of course, people don’t just do this stuff for the hell of it. It’s more fun if there’s a competition or reward. It would be nice if I could get a few paying customers to add their own translation requests to the website – that way I could offer a reward to the best translators (those that do the most work, or are the most consistent).

The competition aspect comes in, where you give people points for every translation they do that is matched by someone else’s translation of the same text.

Google does a similar thing, for example, where it shows images to people and asks them to come up with keywords describing the image – the commonest keywords are considered to be correct. Same trick, different target.

Anyway! Without further ado, I have a core to design, and tests to create. I was going to talk about testing with PHPUnit, automatic upgrades, and the like, but to hell with ye all – I’ve work to do

Benjamin ter Kuile, though, has! Today, we release KFM 1.4

In the time that I’ve been busy with other stuff, and procrastinating on KFM, Benjamin has been working diligently, adding features and making the core files [...]]]> It’s been almost a full year since the last release, 1.3.1, and I haven’t done much on KFM in that time.

Benjamin ter Kuile, though, has! Today, we release KFM 1.4

In the time that I’ve been busy with other stuff, and procrastinating on KFM, Benjamin has been working diligently, adding features and making the core files more stable.

Here is a short list of improvements, gleaned from the SVN logs over the last year. I’m sure I’m missing a lot.

• jQuery upgraded to 1.3.2 (faster, better, stronger)
• you can edit multiple text files simultaneously
• context menu code improved to help it work in Chrome, Opera and Safari
• file icons are now created as a montage, improving display speed
• a lot of functions converted to “lazyload” – improves startup speed
• images are also lazyloaded – if an icon is not visible on the screen, its thumbnail is not loaded
• lots of bug-fixes

Not many new features, as KFM was basically complete.

1.4 is the last 1.x version which will be released. We’re done with the code-base, and ready to start on a much more ambitious project.

We’re going to rebuilt the entire system from scratch in a modular way, adding functionality that would be very difficult to shoehorn into the present system. The new version will be called KFM 2, and I’m really excited about it. I will be starting the coding for it very soon, as soon as I’ve finished with the release of my book, jQuery 1.3 with PHP.

For those of you that are already using 1.3; 1.4 is /faster/, but 1.3 already completes the initial spec, so only upgrade if you find that 1.3 is slow. New users should install 1.4.

I’ve been sent a mockup for the book’s cover. The suggested title of the book is “jQuery 1.3 with PHP”. The working title was “PHP and jQuery”. Which do you prefer?

The book has been completed, in that all the chapters are written, and it’s in the final edit phase at the moment. This involves Packt [...]]]>

I’ve been sent a mockup for the book’s cover. The suggested title of the book is “jQuery 1.3 with PHP”. The working title was “PHP and jQuery”. Which do you prefer?

The book has been completed, in that all the chapters are written, and it’s in the final edit phase at the moment. This involves Packt having a technical editor try everything in the book just to iron out any kinks. It’s already been gone over by three other reviewers, and the only problem appears to have been with the File Management chapter, where the web-server was IIS on Windows. That should be solved by the time the book comes out.

I’ve learned a lot while writing this book. A major point that keeps raising its head is that I keep using colloquialisms and aphorisms (ha! “raising its head”), and those are not globally understood. Another is that I keep using British spelling, but it’s expected that most readers will be American.

From a coding point of view, I tend to write compact code with comments only appearing where something is obviously confusing, but I’ve tried to put proper comments in the book whenever any reviewer asked a question about the code.

Anyway – I expect it will be in PDF form in only a few weeks! I’m looking forward to hearing what people think of it.

On a funny note, I was working on something in work recently, and was trying to figure the best way to do it, when I suddenly remembered I’d written a whole chapter on it, so went and read what I’d written! I’ll be keeping a copy of the book on my own shelf

This kind of thing is always happening to me – I would need to solve some problem (hooking an OKI B2200 printer to Linux over Samba, for example), go searching for the answer, and find that I’d written the solution for it a year or two previously…

By the way, KFM 1.4 will be released next week. It will be the last 1.x version. We (Benjamin and myself) are starting a total rewrite after that, which will become KFM 2. It’s going to be massive!

EDIT: 2009-09-18 Wow, that was quick! The book is already available to pre-book

In the end (after a day of considering this and that and [...]]]> I debugged a problem for a friend of mine over the last day. He had installed bbPress on a server which also had WordPress on it, and his WordPress site just suddenly stopped serving comments and pages, and started killing the database server instead.

In the end (after a day of considering this and that and a lot of head-scratching), it turns out that the bbPress installation had been done using the same table prefix as the WordPress one.

See, when a web app is installed, it usually installs a load of database tables. The way to avoid these tables overwriting each other is to “namespace” them. The most common method is to simply place a unique prefix in front of the table names.

For example, WordPress and bbPress both have tables for posts. The default installation is to name them wp_posts and bb_posts, respectively, using wp_ and bb_ as prefixes. KFM does that as well, using kfm_ by default.

In the current case, what happened is that the bbPress installation was done using wp_ as the prefix, and some of the WordPress tables were then changed to match what bbPress expected.

In the case of wp_posts, the field post_status was changed from a varchar holding values such as “publish” or “draft”, to a smallint holding just ‘1′ or ‘0′. This had the immediate effect of unpublishing over 6000 articles on the blog. (aside: oi, WordPress – why are you not using enums?)

A few hours, spread over a day or so, were spent figuring that out then fixing that.

So, what actually happened? Why was wp_ used as the prefix?

Let’s look at the WordPress installation.

When you install a copy of WordPress, the first thing you do is fill in the database details. This is the field for the prefix:

<input id="prefix" type="text" size="25" value="wp_" name="prefix"/>

Notice the name, “prefix”. A lot of browsers these days include auto-fill, which is sometimes useful. When encountering a form, the browser would fill in the details that it already knows, such as email address, etc.

The problem, though, is that sometimes, autofill might fill in details that really should be left blank or at their default states. An example of this is in phpBB, the popular forum software – whenever an admin goes to edit a user’s details, autofill in Firefox tends to fill in the “password” field in the user form, but it really shouldn’t do that, and can potentially break a user’s account.

This is actually quite easy to fix, though, in the installer – the installation program just needs to change to use a different name for its input box:

<input id="prefix" type="text" size="25" value="wp_" name="wp_prefix"/>

Now the value should only be autofilled by values recorded from other WordPress installations, and should not affect future installations of other software.

The thing is, this is not what broke the system I was fixing.

Let’s look at the bbPress installation.

When writing bbPress, more thought appears to have gone into the installation. The only fields visible here are database name, user, and password. These are the most commonly changed details, so are shown prominently.

There is an “advanced” checkbox. When clicked, this shows fields that should only be changed if you are sure of what they do:

The fields shown here rarely need to be changed. They include database host, collation, character set and table prefix.

The default character set is UTF8, and I can’t imagine why anyone would need anything other than that (that also covers collation). My own opinion is that these ones shouldn’t even be in advanced – they should be hard-coded.

There are not many reasons why the table prefix would ever need to be changed, but in the case I was troubleshooting, the database host was held on a separate server to the site itself.

So, because the database host needed to be changed, that explains why the admin opened the Advanced section in the first place.

If the prefix input box was named “prefix”, this would be the end of the story – I might have assumed that the problem was autofill (as I did for a while), and suggest the above solution to them.

But, the authors of bbPress have obviously come across the autofill problem themselves, because they named their prefix button “bb_table_prefix”, exactly as I suggest every installer should do (now do it with WordPress, guys!):

<input [...] name="bb_table_prefix" class="text" value="bb_" tabindex="8" />

So, the problem is not the software.

In this case, I believe it’s human error. What follows is my own guess at what happened, in hind-sight and without the benefit of watching the proceedings in person.

I think what happened is that our admin’s techie had left some notes about the settings used for the WordPress installation, including database username, password, etc. Then the admin had gone to install bbPress, in the correct assumption that installation should be straightforward and compatible with WordPress.

However, I believe the problem is that the techie had left a note about table prefix, but had not explained to the admin that all the other settings must be as written but the prefix must not.

The admin had simply filled in what was written, and bbPress did what it does, and changed WordPress’s database tables.

So, who’s at fault?

I believe the problem here was a misunderstanding – to all of us who have done a million installations of a thousand different software applications, it’s very obvious what the table prefix is, and what it does. However, I don’t believe I’ve ever seen it written down, or the dangers of using an existing value.

Let’s look at the bbPress installation again, this time with the ‘?’ expanded:

It says “If you are running multiple bbPress sites in a single database, you will probably want to change this.” While literally correct, this actually doesn’t explain what the prefix is or why it should be changed. And most importantly, it does not say “This value must be unique in your database, or it may overwrite your existing database tables.” The help text in bbPress’s case does not actually explain the danger of using existing values for the prefix.

The techie is also at fault. If leaving notes for someone, always make sure to explain anything which needs explaining. In this case, it should be something like “If installing new software, use these database access details, but make sure that you do not use these database table prefixes.” In fact, the table prefix should not have been in the notes at all – if the value is a default, then don’t write it down.

The admin is also at fault, for following the notes to the letter and filling in the prefix where the default value was actually perfect. I won’t fault him for installing software when the techie wasn’t at hand to point out mistakes, as there really shouldn’t have been any – the prefix error was actually quite easy to make; it just happened to have devastating effects.

However, these are all minor faults. They’re all easy mistakes to make, and it was the combination of all three that caused the error.

Having said all that, I realise that I have some installers to check now to make sure I follow my own advice!