Monthly Archives: November 2008

webme, step 2: upgrading and user accounts

Last time, we wrote a basic installer and nothing else. This time, we’ll create user/admin accounts and discuss how upgrading can be done easily.

With user accounts, it is tempting to create two separate tables of users – one of admins and another of ordinary users, but that’s redundant. The same information is saved in both cases, so it makes more sense to have one table dedicated to users as a whole and some way of distinguishing the ordinary users from the administrators.

The way we’ll handle this is through use of Roles. The simplest being to have a table of users and a table of groups, and if a user is a member of the “admins” groups, then they have admin privileges. This is an example of ACL, which can get very complex if you dig into it. I’d recommend sticking to just users and groups unless there is a really good reason to get more in-depth than that.

So, the next thing we need in the CMS is the ability to set up one admin from the installer, and the tables needed to support it.

Logging in will be done via email and password.

Up until recently, the most common method was logging in via username and password, but that has a disadvantage – when you forget your password, it’s also possible that you forgot which email address you registered with, or you no longer have access to the email address.

Using your email address as the login, you are reminded every time you login what email address the new password will be sent to if you forget the old one. And if you know you’ve lost access to the old email address, you can take measures to change the login email before it becomes impossible to do so.

Adapting the installer to ask for these details was not difficult so I won’t detail it.

Upgrades are an interesting problem.

In a non-database environment, upgrades consist of merely unzipping the new version over the old one (or running “svn up” in the root of the site).

Upgrades become much trickier when databases are involved. Upgrading the files might cause an inconsistency, where the scripts may reference database tables that don’t exist or have been changed.

One project I worked on got around this by having a directory which held database patch scripts which an upgrader should run before the site should be used again. That was messy.

Other projects get around this by requiring you to upgrade your files, and then go to a URL (/upgrade.php for example) which will do the database upgrade for you. Better, but still messy.

I think the solution is to have a database version number kept in a file which is upgraded (the front index.php for example), and another version number kept in the site config file, which is not upgraded. When someone visits the site index, it’s a non-expensive process to simply compare those numbers, and if they are different, then an upgrade is automatically performed, and the config file is adjusted with the new version number.

In that way, upgrading is simply a matter of unzipping the new files, and the next time anyone visits the site, it is upgraded automatically.

Anyway – that’s enough for this step. You still can’t create any pages, but we’re almost ready for it.

As usual, you can get this code from the google Webworks-Webme repository. No zipped package available yet, although the next article might be the first which warrants it (we’ll create an actual page and discuss templates).

hooking up with an SD21

I bought some bits and pieces for the robot project. After a lot of consideration, I chose to go the servo route instead of the DC motor route. The difference is that with DC motors, you just turn them on and off they go, but with servos, you can tell the motors how far to go, and at what speed. I like the idea of having that fine control (which would probably be needed if I’m picking up twigs etc).

Unfortunately, the LPT port is pretty rare these days. In the first incarnation of my robot, I used the LPT port to directly turn motors on or off, using a home-built h-bridge to tell the motors to go forward or backward depending on what pins of the LPT I turned on.

I opted instead to use USB and I2C. I2C is a cool little standard – it lets you chain a load of devices into one connection. There is no direct I2C port on the outside of a laptop though, so i bought a USB-I2C convertor.

I found a shop online that sold all the necessary parts. Robot Electronics appears to be the shop-front for Devantech, who make a load of brilliant gadgets such as ultra-sonic rangers (not related to power rangers), on-chip compasses, and what I was interested in – servo controllers.

I bought the SD21 servo controller (controls up to 21 servos), a USB-I2C interface module so I could talk to the SD21, and 2 HS-55 Micro Lite servos so the other stuff actually did something, as well as some leads to connect the interface module to the motor controller.

By the way, the word ‘micro’ is a very apt description – they’re tiny! The image on the right shows the two of them with my mouse in the background.

The stuff finally arrived. Next was the tricky part – figuring out how they all go together.

Using the tech-specs for the usb-i2c and the sd21, it was easy to figure out. The only tricky part was that the USB-I2C module has 5 pins on it, although I2C actually only uses 4. in my photo above-left, the white cable is not used at all (it’s a test pin used by Devantech). So hook the USB-I2C and SD21 together connecting to one of the I2C connectors on the board.

in my case, I got a 5-way and a 4-way cable and just stuck the ends together. Connect black to black, red to red. But the blue in each cable connects to the yellow in the other. Then stick them onto the boards.

Power is provided by hooking 7.2v into the green block. I used 6 1.2v rechargeable batteries. Actually, 4 of them are 1.25v, but the board didn’t complain much. Be very careful that you hook them into the SD21 the right way around! I put the power in the wrong way once and had to yank them apart when I smelled burning parts! Luckily, there wasn’t anything wrong afterwards, but beware!

After all that, the hard part was figuring out how to actually talk to the thing. It took me some reading to figure out the obvious… you don’t speak directly to the SD21. You speak to the USB-I2C and it then talks to the SD21.

Here’s a small C program that turns the motors first in one direction, and then in the other. BTW: it’s been a century since I wrote C, so if I’ve done anything obviously wrong (especially that int to bytes conversion bit…) please tell me.

#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
#include <termios.h>

struct termios options;

int open_port(void) {
  // this function taken from http://www.robot-electronics.co.uk/forum/viewtopic.php?f=5&t=165
  int fd; // File descriptor for the port
  fd = open("/dev/ttyUSB0", O_RDWR | O_NOCTTY);
  if (fd == -1) {
    perror("open_port: Unable to open /dev/ttyUSB0 - "); // Could not open the port.
  }
  else {
    fcntl(fd, F_SETFL, 0);

    // Get the current options for the port...
    tcgetattr(fd, &options);

    // Set the baud rates to 19200...
    cfsetispeed(&options, B19200);

    // Enable the receiver and set local mode...
    options.c_cflag |= (CLOCAL | CREAD);

    // Set no parity bit
    options.c_cflag &= ~PARENB;

    // Set 2 stop bits
    options.c_cflag &= ~CSTOPB;

    // Set the character size
    options.c_cflag &= ~CSIZE;
    options.c_cflag |= CS8;

    // Set the new options for the port...
    tcsetattr(fd, TCSANOW, &options);

    fcntl(fd, F_SETFL, FNDELAY);
  }
  return (fd);
}

void set_servo(unsigned char reg, int pos) {
  // sets servo 'reg' to position 'pos'
  unsigned char sbuf[7];
  unsigned char lbyte, hbyte;
  int fd;

  hbyte=(unsigned char) (pos >> 8);
  lbyte=(unsigned char) (pos);

  sbuf[0] = 0x55;  // ftdi mode (multibyte device)
  sbuf[1] = 0xC2;  // address of the i2c device
  sbuf[2] = reg*3; // each servo has three registers
  sbuf[3] = 0x03;  // bytes to send
  sbuf[4] = 0x00;  // set to full speed
  sbuf[5] = lbyte; // position low byte
  sbuf[6] = hbyte; // position high byte

  fd = open_port();
  write(fd, sbuf, 7);
  close(fd);
}

int main() {
  // take a step to the left
  set_servo(0, 800);
  set_servo(1, 800);

  sleep(2);

  // and then a jump to the right
  set_servo(0, 2200);
  set_servo(1, 2200);

  return 0;
}

webme, step 1: where to start? the beginning sounds good.

Okay – WebME…

Before I introduce the first bits of code for WebME, I want to try explain what it is, and why it is different to, say, Joomla, Mambo or PHPNuke.

As I explained in the Webworks blog, WebME grew up as a tool designed to do some very very common tasks on small websites.

This CMS was built for about 250 separate websites. Each of those sites wanted a simple task – create pages, and edit those pages. There are other tricks such as Forms and Online Stores, etc, but we’ll get to them. The core task for this CMS is to create a page, and allow editing of that page.

The larger, more well-known CMSes, on the other hand, are built to be modular, extensible platforms. It’s difficult to define a single “purpose” to those platforms. For WebME, it’s simple – “manage some webpages”. For the others… I don’t know; they’re too large to define simply.

It’s tricky to decide where to start with this release. I can’t just release the whole damned thing, as that would be overwhelming, and probably a bad idea, security-wise. Instead, I’m going to release a piece at a time.

Today’s piece will provide the bare minimum – you will be able to download WebME, and install it. That’s it. Nothing else. No admin area, and no editing of pages. In fact, not even any pages. You at the back, have patience!

For today, I’ve written the basics of the installer. You can download it from Google, via SVN (subversion). WebME won’t be available via zipped package until I’ve released enough of it to be useful for the average web developer.

Today’s download really is the bare minimum – it creates a config.php file, but nothing else. Tomorrow I’ll show how upgrades work in WebME, and will show the admin area.

Despite the apparent nothingness of today’s release, I hope people will download and try it out. Each day will produce a few new ideas.

Today’s idea is how to create an installer. Most (all?) projects start off any page by loading the config. If that config does not exist, then either the system is not installed, or there’s something wrong. We use that idea to allow a freshly downloaded copy of WebME to bring the browser straight into the installation script.

It’s a simple and obvious idea, but useful. Tomorrow’s idea will show how to handle upgrades in a “continuous integration” way (…ish). In particular, tomorrow’s trick is useful for handling database evolution.

webworks to opensource its tools

I had a nice chat with John earlier. We‘re considering releasing our tools under an open source license.

We’re doing this for a number of reasons. I think that chief among them is the hope that with many developers (or at least, with more than just myself), we should be able to keep on top of all the modern net tricks that keep appearing – RSS, Ajax, etc. So far, I’ve been doing that myself, but as our CMS, WebME (Web Management Engine) has grown, the effort to keep it uptodate has become trickier.

So, we are planning on releasing the code as open-source while I am still on top of it, so I can help other developers join in on the coding while still working away on it myself.

I’m very excited about this.

A few years ago, we tried this very thing, open-sourcing WebME, but no-one took us up on the offer, and we brought it back in-house.

However; since then, I released KFM, which has developed into a pretty popular open-source project, and I feel that releasing WebME again will work based on whatever OS credit I might have earned from that project.

We are hoping to release our newsletter-management service as well, depending on how WebME does.

I’m very interested to see what other people think of this. I know most of you will not have seen WebME’s admin area, but I think it’s one of the most user-friendly ones out there (even if I say so myself), and will be a boon for developers who want a simple CMS for simple sites – page-based sites with galleries, simple online stores, etc.

lazy loading functions in javascript

Lazy loading is the act of loading an asset (JavaScript, image, whatever) just before it’s needed, instead of at the beginning of the page load. This allows you to save bandwidth in cases where the asset will not be required at all, and also allows you to load up the page quicker because there is less JavaScript for the browser to interpret before it shows the screen.

A while ago I wrote lazy-loading for images into KFM. This time, it’s time to do it for JavaScript.

Here are two examples of lazy-loading methods for JavaScript. I wasn’t happy with either. In both cases, when a lazy-loaded function is to be called, the scripter must be careful about calling the function in case it’s not yet loaded, and the calling of the function is different to how you would do it in a normal script without lazy-loading applied.

What I wanted was a way to simply call a script (do_something('blah');) and have the script automatically load the function and when it’s loaded, run the function call again.

The way to do this is to replace the function do_something() with a stub. This stub loads up the real code, replaces itself with the code, then calls the replacement code with the original parameters (there, that’s the pattern in a nutshell).

So, to start with, your stub looks like this:

window.do_something=function(){
  var ps=arguments;
  x_kfm_getJsFunction("do_something",function(js){
    lazyload_replace_stub("do_something",js,ps);
  });
};

In the above, x_kfm_getJsFunction() is a simple Ajax function which calls kfm_getJsFunction() on the server-side with the first parameter, and when the result is returned, calls the second parameter as a callback, sending it the result. I won’t give the source for x_kfm_getJsFunction() as it’s framework-dependent. Each framework (prototype, mootools, dojo, etc.) has its own way of doing exactly that. My method uses a version of Sajax which I forked a long time ago.

On the server-side, the code is simple. Just keep each function in its own file in a directory called ‘functions’.

function kfm_getJsFunction($name){
  if(preg_replace('/[0-9a-zA-Z_.]/g','',$name)!='')return 'alert("no hacking!");';
  $js=file_get_contents('j/functions/'.$name.'.js');
  if(!$js)return 'alert("could not find function \''.addslashes($name).'\' on the server-side!");';
  return $js;
}

And when that’s retrieved, here’s the function that replaces the original function:

function lazyload_replace_stub(fname,js,ps){ // replace stub with function, then call function with original parameters
  eval(js);
  (eval(fname))(ps[0],ps[1],ps[2],ps[3]); // hacky method... replace when a better idea comes along
}

There final line calls the new function with an assumed four parameters. JavaScript is fine with this, so the function works. If there are more than four parameters you need to either change that line, or rethink your function’s parameters.

Now obviously, the do_something() code I pasted earlier is too large to be useful. I mean, we’re trying to save bandwidth, and here I advocate a large lump of code which pulls in a possibly smaller lump! That would be silly.

So, instead, what you do is to move all the functions you want lazy-loaded into their own files in your functions directory, and have the client generate the necessary lazy-load functions itself.

var llStubs=[];

llStubs.push('do_something'); // add do_something to the list of lazy-load functions
llStubs.push('another_function');
function non_lazyloaded_function(var1, var2){
...
}

var i,funcs=[],fname;
for(i=0;i<llStubs.length;++i){
  fname=llStubs[i];
  funcs.push('window.'+fname+'=function(){var ps=arguments;x_kfm_getJsFunction("'+fname+'",function(js){lazyload_replace_stub("'+fname+'",js,ps);});};');
}
eval(funcs.join("\n"));
funcs=null;
i=null;
fname=null;

I think that’s about it – you have the idea of it now.

So, some cases where you shouldn’t use lazy-loading.

  • When the function is needed upon load of the page. For example, if the function is part of your onload() initialisation. You can test this by adding alert(fname); to the lazyload_replace_stub() function – if anything is alerted upon loading the page, then you are lazy-loading functions unnecessarily.
  • When the function is called repeatedly in a rapid-fire manner. For example, if you have drag/drop code, the “ondrag” event should not be lazy-loaded, as it is called many times a second.
  • When the function returns a result. As noted, lazy-load is asynchronous, which means you cannot lazy-load a function which returns a value.
  • When the function performs an action which is needed somewhere later in the calling function. Obviously, an asynchronous call will take place at least 1ms after the call was made, and that’s well after the calling function would be finished.

In the cases above where the asynchronous nature of lazy-loading is the problem, you could simply change your AJAX method to use synchronous calling, but that makes your client feel sluggish.

One more thing – as this is supposed to save bandwidth, if you need 10 functions, you should bunch the requests into one single AJAX request instead of making 10 requests. My Sajax fork (Kaejax – yeah, I know) does that automatically. I don’t know if any others do it.

And another thing – functions that are to be lazy-loaded should be opened with window.function_name=function(){ instead of function function_name(){ in the retrieved code. This is to avoid any scoping problems JavaScript might throw up.