27 Sep

KFM 1.4 released

It’s been almost a full year since the last release, 1.3.1, and I haven’t done much on KFM in that time.

Benjamin ter Kuile, though, has! Today, we release KFM 1.4

In the time that I’ve been busy with other stuff, and procrastinating on KFM, Benjamin has been working diligently, adding features and making the core files more stable.

Here is a short list of improvements, gleaned from the SVN logs over the last year. I’m sure I’m missing a lot.

  • jQuery upgraded to 1.3.2 (faster, better, stronger)
  • you can edit multiple text files simultaneously
  • context menu code improved to help it work in Chrome, Opera and Safari
  • file icons are now created as a montage, improving display speed
  • a lot of functions converted to “lazyload” – improves startup speed
  • images are also lazyloaded – if an icon is not visible on the screen, its thumbnail is not loaded
  • lots of bug-fixes

Not many new features, as KFM was basically complete.

1.4 is the last 1.x version which will be released. We’re done with the code-base, and ready to start on a much more ambitious project.

We’re going to rebuilt the entire system from scratch in a modular way, adding functionality that would be very difficult to shoehorn into the present system. The new version will be called KFM 2, and I’m really excited about it. I will be starting the coding for it very soon, as soon as I’ve finished with the release of my book, jQuery 1.3 with PHP.

For those of you that are already using 1.3; 1.4 is /faster/, but 1.3 already completes the initial spec, so only upgrade if you find that 1.3 is slow. New users should install 1.4.

18 Sep

jQuery 1.3 With PHP: cover mockup

6989_MockupCover

I’ve been sent a mockup for the book’s cover. The suggested title of the book is “jQuery 1.3 with PHP”. The working title was “PHP and jQuery”. Which do you prefer?

The book has been completed, in that all the chapters are written, and it’s in the final edit phase at the moment. This involves Packt having a technical editor try everything in the book just to iron out any kinks. It’s already been gone over by three other reviewers, and the only problem appears to have been with the File Management chapter, where the web-server was IIS on Windows. That should be solved by the time the book comes out.

I’ve learned a lot while writing this book. A major point that keeps raising its head is that I keep using colloquialisms and aphorisms (ha! “raising its head”), and those are not globally understood. Another is that I keep using British spelling, but it’s expected that most readers will be American.

From a coding point of view, I tend to write compact code with comments only appearing where something is obviously confusing, but I’ve tried to put proper comments in the book whenever any reviewer asked a question about the code.

Anyway – I expect it will be in PDF form in only a few weeks! I’m looking forward to hearing what people think of it.

On a funny note, I was working on something in work recently, and was trying to figure the best way to do it, when I suddenly remembered I’d written a whole chapter on it, so went and read what I’d written! I’ll be keeping a copy of the book on my own shelf 😉

This kind of thing is always happening to me – I would need to solve some problem (hooking an OKI B2200 printer to Linux over Samba, for example), go searching for the answer, and find that I’d written the solution for it a year or two previously…

By the way, KFM 1.4 will be released next week. It will be the last 1.x version. We (Benjamin and myself) are starting a total rewrite after that, which will become KFM 2. It’s going to be massive!

EDIT: 2009-09-18 Wow, that was quick! The book is already available to pre-book

04 Aug

a tale of prefixes

I debugged a problem for a friend of mine over the last day. He had installed bbPress on a server which also had WordPress on it, and his WordPress site just suddenly stopped serving comments and pages, and started killing the database server instead.

In the end (after a day of considering this and that and a lot of head-scratching), it turns out that the bbPress installation had been done using the same table prefix as the WordPress one.

See, when a web app is installed, it usually installs a load of database tables. The way to avoid these tables overwriting each other is to “namespace” them. The most common method is to simply place a unique prefix in front of the table names.

For example, WordPress and bbPress both have tables for posts. The default installation is to name them wp_posts and bb_posts, respectively, using wp_ and bb_ as prefixes. KFM does that as well, using kfm_ by default.

In the current case, what happened is that the bbPress installation was done using wp_ as the prefix, and some of the WordPress tables were then changed to match what bbPress expected.

In the case of wp_posts, the field post_status was changed from a varchar holding values such as “publish” or “draft”, to a smallint holding just ‘1’ or ‘0’. This had the immediate effect of unpublishing over 6000 articles on the blog. (aside: oi, WordPress – why are you not using enums?)

A few hours, spread over a day or so, were spent figuring that out then fixing that.

So, what actually happened? Why was wp_ used as the prefix?

Let’s look at the WordPress installation.

wordpress

When you install a copy of WordPress, the first thing you do is fill in the database details. This is the field for the prefix:

<input id="prefix" type="text" size="25" value="wp_" name="prefix"/>

Notice the name, “prefix”. A lot of browsers these days include auto-fill, which is sometimes useful. When encountering a form, the browser would fill in the details that it already knows, such as email address, etc.

The problem, though, is that sometimes, autofill might fill in details that really should be left blank or at their default states. An example of this is in phpBB, the popular forum software – whenever an admin goes to edit a user’s details, autofill in Firefox tends to fill in the “password” field in the user form, but it really shouldn’t do that, and can potentially break a user’s account.

This is actually quite easy to fix, though, in the installer – the installation program just needs to change to use a different name for its input box:

<input id="prefix" type="text" size="25" value="wp_" name="wp_prefix"/>

Now the value should only be autofilled by values recorded from other WordPress installations, and should not affect future installations of other software.

The thing is, this is not what broke the system I was fixing.

Let’s look at the bbPress installation.

bbpress

When writing bbPress, more thought appears to have gone into the installation. The only fields visible here are database name, user, and password. These are the most commonly changed details, so are shown prominently.

There is an “advanced” checkbox. When clicked, this shows fields that should only be changed if you are sure of what they do:

bbpress2

The fields shown here rarely need to be changed. They include database host, collation, character set and table prefix.

The default character set is UTF8, and I can’t imagine why anyone would need anything other than that (that also covers collation). My own opinion is that these ones shouldn’t even be in advanced – they should be hard-coded.

There are not many reasons why the table prefix would ever need to be changed, but in the case I was troubleshooting, the database host was held on a separate server to the site itself.

So, because the database host needed to be changed, that explains why the admin opened the Advanced section in the first place.

If the prefix input box was named “prefix”, this would be the end of the story – I might have assumed that the problem was autofill (as I did for a while), and suggest the above solution to them.

But, the authors of bbPress have obviously come across the autofill problem themselves, because they named their prefix button “bb_table_prefix”, exactly as I suggest every installer should do (now do it with WordPress, guys!):

<input [...] name="bb_table_prefix" class="text" value="bb_" tabindex="8" />

So, the problem is not the software.

In this case, I believe it’s human error. What follows is my own guess at what happened, in hind-sight and without the benefit of watching the proceedings in person.

I think what happened is that our admin’s techie had left some notes about the settings used for the WordPress installation, including database username, password, etc. Then the admin had gone to install bbPress, in the correct assumption that installation should be straightforward and compatible with WordPress.

However, I believe the problem is that the techie had left a note about table prefix, but had not explained to the admin that all the other settings must be as written but the prefix must not.

The admin had simply filled in what was written, and bbPress did what it does, and changed WordPress’s database tables.

So, who’s at fault?

I believe the problem here was a misunderstanding – to all of us who have done a million installations of a thousand different software applications, it’s very obvious what the table prefix is, and what it does. However, I don’t believe I’ve ever seen it written down, or the dangers of using an existing value.

Let’s look at the bbPress installation again, this time with the ‘?’ expanded:

bbpress3

It says “If you are running multiple bbPress sites in a single database, you will probably want to change this.” While literally correct, this actually doesn’t explain what the prefix is or why it should be changed. And most importantly, it does not say “This value must be unique in your database, or it may overwrite your existing database tables.” The help text in bbPress’s case does not actually explain the danger of using existing values for the prefix.

The techie is also at fault. If leaving notes for someone, always make sure to explain anything which needs explaining. In this case, it should be something like “If installing new software, use these database access details, but make sure that you do not use these database table prefixes.” In fact, the table prefix should not have been in the notes at all – if the value is a default, then don’t write it down.

The admin is also at fault, for following the notes to the letter and filling in the prefix where the default value was actually perfect. I won’t fault him for installing software when the techie wasn’t at hand to point out mistakes, as there really shouldn’t have been any – the prefix error was actually quite easy to make; it just happened to have devastating effects.

However, these are all minor faults. They’re all easy mistakes to make, and it was the combination of all three that caused the error.

Having said all that, I realise that I have some installers to check now to make sure I follow my own advice!

17 Apr

update on Me

Quick update of what’s happening in the world of Kae.

I’ve been busy recently with a new project in the office, making an ultra-WebME system which can be used by companies who are not web designers, to offer their clients “pre-packaged” websites. I may elaborate on that in a few weeks once the pilot study is out of the way.

I haven’t touched the KFM file manager project in ages, and recently realised that it’s because I’m just not happy with it anymore. It’s monstrously complex now, and just not flexible enough for the changes I want to build into it. So, I’m seriously thinking of rewriting it completely, or building a new file manager project to compete with it.

WebME is progressing quickly. The plugins architecture is damned easy to use, if I do say so myself. So much so that Conor wrote a mailing-list plugin before I even got to announce the plugin system. To try it out, get the Subversion copy of WebME, and also the subversion copy of the existing plugins (svn checkout http://webworks-webme.googlecode.com/svn/ww.plugins/) and create a symbolic link from ‘ww.plugins’ in the root of your WebME installation to the checked-out ww.plugins directory. Then in the /ww.admin section, go to Site Options > Plugins and try some stuff. I’ll be writing a proper tutorial on how to create a plugin. Probably today.

I started learning piano at the beginning of last month and am proud to say that I appear to be learning at a prodigious rate. This is probably because I can already play well on a number of other instruments. Last tune I learned was Ecossaise in E Flat by Beethoven (WoO 86), played here by Matt Baker:

Before that was I’ll Take You Home Kathleen, played here by Dave Seddon on steel guitar (couldn’t find a piano version that wasn’t ruined by people singing over the music 😉 ):

I’m still in quite a lot of pain since last week’s operation, but hopefully it will ease off over the weekend, as I have the next chapter of my book to submit.

10 Jan

webme: admins, frontend editing, KFM, and comments

I’ve done a bit more updating of the WebME code.

I realised I’d forgotten to add some code to create the first admin. That’s fixed now – just go to the admin area. If there are no existing admins, then the first attempt to login will create a valid user account.

You can insert images and stuff now through the FCKeditor in the Pages section. The file manager used is KFM.

When you create/edit a page, see the Advanced tab, where there are a number of options. You can set a page to be hidden from the navigation menu, you can set any page to be the front page, and you can also set a page to allow public comments.

The public comments thing is interesting, I think – I wrote it using a combination of AJAX and some verification tricks. On all the sites that it exists, I have not seen a single spam comment (but now that the challenge is out there…).

If you have logged in as someone who has Admin rights, then you can edit pages directly from the front end just by right-clicking on the page. Choose “Edit Page” and you’ll be able to write directly into the page. Right-click outside the editable area when you’re done, and choose “Save Changes and Reload Page”.

As usual, you can grab the code here: Webworks WebME SVN checkout. I think we’re just about at the point where a proper package will be useful. That will hopefully be provided tomorrow, depending on time.

And now for some news – I’ve been discussing making this into a free service with John (webworks‘ head honcho), and we’re going to go ahead with it. The idea is that you will create your own website on our hosting platform, get some basic services free, and will pay for extras if you want them (domain names, SMS services, etc), but there will be /no/ pressure – if you just want a simple website with an easy-to-use CMS, then that will be available for free. The great thing about this is that the system will be upgraded very often, with new services coming online as they’re finished, and you, as users, will have input into what gets worked on.

14 Nov

webworks to opensource its tools

I had a nice chat with John earlier. We‘re considering releasing our tools under an open source license.

We’re doing this for a number of reasons. I think that chief among them is the hope that with many developers (or at least, with more than just myself), we should be able to keep on top of all the modern net tricks that keep appearing – RSS, Ajax, etc. So far, I’ve been doing that myself, but as our CMS, WebME (Web Management Engine) has grown, the effort to keep it uptodate has become trickier.

So, we are planning on releasing the code as open-source while I am still on top of it, so I can help other developers join in on the coding while still working away on it myself.

I’m very excited about this.

A few years ago, we tried this very thing, open-sourcing WebME, but no-one took us up on the offer, and we brought it back in-house.

However; since then, I released KFM, which has developed into a pretty popular open-source project, and I feel that releasing WebME again will work based on whatever OS credit I might have earned from that project.

We are hoping to release our newsletter-management service as well, depending on how WebME does.

I’m very interested to see what other people think of this. I know most of you will not have seen WebME’s admin area, but I think it’s one of the most user-friendly ones out there (even if I say so myself), and will be a boon for developers who want a simple CMS for simple sites – page-based sites with galleries, simple online stores, etc.

10 Nov

lazy loading functions in javascript

Lazy loading is the act of loading an asset (JavaScript, image, whatever) just before it’s needed, instead of at the beginning of the page load. This allows you to save bandwidth in cases where the asset will not be required at all, and also allows you to load up the page quicker because there is less JavaScript for the browser to interpret before it shows the screen.

A while ago I wrote lazy-loading for images into KFM. This time, it’s time to do it for JavaScript.

Here are two examples of lazy-loading methods for JavaScript. I wasn’t happy with either. In both cases, when a lazy-loaded function is to be called, the scripter must be careful about calling the function in case it’s not yet loaded, and the calling of the function is different to how you would do it in a normal script without lazy-loading applied.

What I wanted was a way to simply call a script (do_something('blah');) and have the script automatically load the function and when it’s loaded, run the function call again.

The way to do this is to replace the function do_something() with a stub. This stub loads up the real code, replaces itself with the code, then calls the replacement code with the original parameters (there, that’s the pattern in a nutshell).

So, to start with, your stub looks like this:

window.do_something=function(){
  var ps=arguments;
  x_kfm_getJsFunction("do_something",function(js){
    lazyload_replace_stub("do_something",js,ps);
  });
};

In the above, x_kfm_getJsFunction() is a simple Ajax function which calls kfm_getJsFunction() on the server-side with the first parameter, and when the result is returned, calls the second parameter as a callback, sending it the result. I won’t give the source for x_kfm_getJsFunction() as it’s framework-dependent. Each framework (prototype, mootools, dojo, etc.) has its own way of doing exactly that. My method uses a version of Sajax which I forked a long time ago.

On the server-side, the code is simple. Just keep each function in its own file in a directory called ‘functions’.

function kfm_getJsFunction($name){
  if(preg_replace('/[0-9a-zA-Z_.]/g','',$name)!='')return 'alert("no hacking!");';
  $js=file_get_contents('j/functions/'.$name.'.js');
  if(!$js)return 'alert("could not find function \''.addslashes($name).'\' on the server-side!");';
  return $js;
}

And when that’s retrieved, here’s the function that replaces the original function:

function lazyload_replace_stub(fname,js,ps){ // replace stub with function, then call function with original parameters
  eval(js);
  (eval(fname))(ps[0],ps[1],ps[2],ps[3]); // hacky method... replace when a better idea comes along
}

There final line calls the new function with an assumed four parameters. JavaScript is fine with this, so the function works. If there are more than four parameters you need to either change that line, or rethink your function’s parameters.

Now obviously, the do_something() code I pasted earlier is too large to be useful. I mean, we’re trying to save bandwidth, and here I advocate a large lump of code which pulls in a possibly smaller lump! That would be silly.

So, instead, what you do is to move all the functions you want lazy-loaded into their own files in your functions directory, and have the client generate the necessary lazy-load functions itself.

var llStubs=[];

llStubs.push('do_something'); // add do_something to the list of lazy-load functions
llStubs.push('another_function');
function non_lazyloaded_function(var1, var2){
...
}

var i,funcs=[],fname;
for(i=0;i<llStubs.length;++i){
  fname=llStubs[i];
  funcs.push('window.'+fname+'=function(){var ps=arguments;x_kfm_getJsFunction("'+fname+'",function(js){lazyload_replace_stub("'+fname+'",js,ps);});};');
}
eval(funcs.join("\n"));
funcs=null;
i=null;
fname=null;

I think that’s about it – you have the idea of it now.

So, some cases where you shouldn’t use lazy-loading.

  • When the function is needed upon load of the page. For example, if the function is part of your onload() initialisation. You can test this by adding alert(fname); to the lazyload_replace_stub() function – if anything is alerted upon loading the page, then you are lazy-loading functions unnecessarily.
  • When the function is called repeatedly in a rapid-fire manner. For example, if you have drag/drop code, the “ondrag” event should not be lazy-loaded, as it is called many times a second.
  • When the function returns a result. As noted, lazy-load is asynchronous, which means you cannot lazy-load a function which returns a value.
  • When the function performs an action which is needed somewhere later in the calling function. Obviously, an asynchronous call will take place at least 1ms after the call was made, and that’s well after the calling function would be finished.

In the cases above where the asynchronous nature of lazy-loading is the problem, you could simply change your AJAX method to use synchronous calling, but that makes your client feel sluggish.

One more thing – as this is supposed to save bandwidth, if you need 10 functions, you should bunch the requests into one single AJAX request instead of making 10 requests. My Sajax fork (Kaejax – yeah, I know) does that automatically. I don’t know if any others do it.

And another thing – functions that are to be lazy-loaded should be opened with window.function_name=function(){ instead of function function_name(){ in the retrieved code. This is to avoid any scoping problems JavaScript might throw up.

13 Oct

security hole for files with a dot at the end

I received an email this morning saying that KFM has a security hole – if a user creates a file named “test.php.” (note the ‘.’ at the end), then it is run as if it was “test.php”, even if you explicitly banned the .php extension in your settings.

I immediately added a line of code to ban all filenames which end in ‘.’, released a new version of 1.3 (available on the front page of the site) and corrected 1.4 in SVN.

After thinking about it, I realised that the security problem is not as serious as it may seem (for KFM – not in general). It’s definitely a problem, but in order to use it, you need to have access to a KFM instance in the first place. As securing KFM is not difficult, I think the problem may be contained.

But I digress – this appears to be a problem in Apache. To test it, I checked if renaming a Perl CGI file from .cgi to .cgi. would work, and it did.

This is a little disturbing, as it does not appear to be documented anywhere, so there is no way that a developer would know to avoid this security hole.

So, if you write programs that allow your users to upload or rename files online, make sure that the filename does not have a ‘.’ at the end!

edit: OMG! I was reading the Apache source to try spot the problem, and found the area where it happens – it’s in the file “http/mod_mime.c”. The function “find_ct()” extracts the extension for the server to use. Unfortunately, it ignores all extensions it does not understand, so it’s not just a case of “test.php.” being parsed as ‘.php’, but also “test.php.fdabsfgdsahfj” and other similar rubbish files! This is a serious problem.

There are a number of solutions to this:

  1. Possibly the correct solution: Keep your downloadables outside the web-accessible area and force the download through a PHP script. Doesn’t matter what extension the file has then.
  2. Tricky but easier to make portable: Write your own extension identifier using the httpd source as a guide, so you know what Apache will identify the file as (annoyingly complex, possibly, but I’ll need to do it…)
  3. Easiest, but most annoying for users: Only allow one ‘.’ per filename.
  4. More difficult, but possibly also correct: Convince Apache that this needs to be fixed, then upgrade immediately when the fix is available.

further update: An easy solution. This problem rears its head when PHP is identified in your httpd.conf using this:

AddHandler php5-script .php

the solution is to change the above to specify the extension must be at the end.

<FilesMatch \.php$>
  SetHandler php5-script
</FilesMatch>
10 Oct

kfm progress

Didn’t do much on KFM this week. I’ve spent most of the time in work, sleeping, or trying to convince the kids that 3am is not a good time to be awake.

This morning (got up at 3.30…) I implemented lazy-loading for icon thumbnails.

What this means, is that in a folder which has 500 icons in it, if only 25 are visible without scrolling, then only 25 are actually downloaded. This helps speed up your usage in a few ways.

  • if you download an icon which is not immediately visible, then you are slowing down the showing of an icon which is right in front of you.
  • Sometimes you only want to see the first few icons of a directory anyway – let’s say they’re photos and the folder title is something like “DCIM1010” (you know what I mean), then its easy to see if they’re the right photos by viewing the first few.
  • If you’re downloading 500 icons at one time, that means not many other HTTP actions can be taken until those have been taken care of… – so if you wanted to rename one of the visible files, for example, then you’ll just have to wait for the other hundreds of thumbs do download first!

What I’m supposed to be working on is the new translation system, but that’s very hard. There are some little problems I haven’t cleared in my head yet. Hopefully the solution will come to me before the end of the day so I can get it finished this weekend.

04 Oct

KFM work restarted, translation plans

After a long break, I’m back working on KFM. I’ve done an average of maybe 2 hours every day this week, cleaning up code and preparing for future plans.

What I’m working on at the moment is a rewrite of the translations code. There are a few things wrong with the way that it works at the moment:

  • JavaScript and PHP language strings are currently separate – en.php and en.js for example.
  • Whenever KFM is loaded, the entire JS language file is loaded, even if very little of it will actually be used.
  • There is no easy way apart from hacking KFM to add translations for user-contributed plugins.

The solution we (Benjamin and myself) have come up with is to use a database solution, with an import facility for plugins. The idea is a complete rewrite of how we do it.

Currently, translations are recorded in a JS object something like this:

kfm.lang={
  Errors                  : "Ошибки",
  LastModified            : "последний раз изменен"
};

The above is easy to use in KFM, in that the strings can be accessed using normal object notation, kfm.lang.Errors for example. However, it should be easier.

What we will be doing is to convert all language files so they are written using the English string as the key instead of a code such as “LastModified“. And, using the common nomenclature of other languages, the translation will be done such as: alert(__("What do you want to rename %1 to?","default",filename));, where the second parameter is the language context (are we talking about files, people, dogs, etc?).

To help improve the speed at which KFM opens, only the most popular strings will be preloaded. If an unknown string is requested, it will be loaded through AJAX.

How that will work is that all translations will be asynchronous. When you request a translation, a span element will be returned containing the translation. If the translation is already known or cached by the client, then the element will be pre-filled. Otherwise, an AJAX request will be fired off, and when it returns, the element will be populated.

The popularity of the strings will be determined by the number of requests for it.

On the server side, when a request comes in, KFM will search its database looking for the string. If it is found, it is returned (and its ‘requests’ field incremented). If it is not found, then all installed plugins will be searched for language files, which will be imported into the database then returned. If the string is not found anywhere, then the plain English is recorded in the database, and a note is added that the translation was not found.

I’m considering whether or not to add in a bit which periodically reports back to the main KFM server a list of missing translations. Also, a periodic report of popular strings and even of unused strings, allowing us to prune the files for future releases.

If you want to try out the work, then download a copy of the trunk via SVN: svn checkout http://kfm.googlecode.com/svn/trunk/trunk/ kfm