Morning all!
The WebME CMS has a new home, at webme.eu. You can download the CMS from the Downloads page there.
Tonight, I should have the first version of the site-builder ready. I’m calling it WebMEµ (webme-mu; web management engine – multi-user), inspired by Donncha’s http://mu.wordpress.org/ project. This will allow a load of separate CMSes to run from the same engine. Donncha – I love the feature list – #5 is a corker! (Ambiguity about how to pronounce its name).
OK – finally, there is something that you can download as a package, with a theme an’ all!
You can get the package from the WebME downloads page.
See what I did there? I linked to another site – that site is the new home of Webworks WebME, until we come up with a better domain name. Note that the webme.verens.com website is itself running with a copy of the CMS. Eating my own dog food, dontchaknow!
I’ve added the ability to choose which theme to use for the site. You can do this very simply – download a new theme (or create one yourself) from the downloads page, and unzip it in /ww.skins/. Then in the admin area, go to users and admins > site options and click the theme you want to use. Couldn’t be simpler!
Also, I’ve added in some search functionality. If the site detects a search GET request, and no page name is selected, then a search results page is shown. If there is no existing search results page already then one is generated automatically and added to the database (and hidden from navigation).
I’ve been through the installation process a few times now, and I think I’ve ironed out the obvious problems. Please give this a try. I’m going to take a break, fix a Linux problem for a friend, then work on the new multi-user version of WebME that I mentioned yesterday.
I’ve done a bit more updating of the WebME code.
I realised I’d forgotten to add some code to create the first admin. That’s fixed now – just go to the admin area. If there are no existing admins, then the first attempt to login will create a valid user account.
You can insert images and stuff now through the FCKeditor in the Pages section. The file manager used is KFM.
When you create/edit a page, see the Advanced tab, where there are a number of options. You can set a page to be hidden from the navigation menu, you can set any page to be the front page, and you can also set a page to allow public comments.
The public comments thing is interesting, I think – I wrote it using a combination of AJAX and some verification tricks. On all the sites that it exists, I have not seen a single spam comment (but now that the challenge is out there…).
If you have logged in as someone who has Admin rights, then you can edit pages directly from the front end just by right-clicking on the page. Choose “Edit Page” and you’ll be able to write directly into the page. Right-click outside the editable area when you’re done, and choose “Save Changes and Reload Page”.
As usual, you can grab the code here: Webworks WebME SVN checkout. I think we’re just about at the point where a proper package will be useful. That will hopefully be provided tomorrow, depending on time.
And now for some news – I’ve been discussing making this into a free service with John (webworks‘ head honcho), and we’re going to go ahead with it. The idea is that you will create your own website on our hosting platform, get some basic services free, and will pay for extras if you want them (domain names, SMS services, etc), but there will be /no/ pressure – if you just want a simple website with an easy-to-use CMS, then that will be available for free. The great thing about this is that the system will be upgraded very often, with new services coming online as they’re finished, and you, as users, will have input into what gets worked on.
Last time, we wrote a basic installer and nothing else. This time, we’ll create user/admin accounts and discuss how upgrading can be done easily.
With user accounts, it is tempting to create two separate tables of users – one of admins and another of ordinary users, but that’s redundant. The same information is saved in both cases, so it makes more sense to have one table dedicated to users as a whole and some way of distinguishing the ordinary users from the administrators.
The way we’ll handle this is through use of Roles. The simplest being to have a table of users and a table of groups, and if a user is a member of the “admins” groups, then they have admin privileges. This is an example of ACL, which can get very complex if you dig into it. I’d recommend sticking to just users and groups unless there is a really good reason to get more in-depth than that.
So, the next thing we need in the CMS is the ability to set up one admin from the installer, and the tables needed to support it.
Logging in will be done via email and password.
Up until recently, the most common method was logging in via username and password, but that has a disadvantage – when you forget your password, it’s also possible that you forgot which email address you registered with, or you no longer have access to the email address.
Using your email address as the login, you are reminded every time you login what email address the new password will be sent to if you forget the old one. And if you know you’ve lost access to the old email address, you can take measures to change the login email before it becomes impossible to do so.
Adapting the installer to ask for these details was not difficult so I won’t detail it.
Upgrades are an interesting problem.
In a non-database environment, upgrades consist of merely unzipping the new version over the old one (or running “svn up” in the root of the site).
Upgrades become much trickier when databases are involved. Upgrading the files might cause an inconsistency, where the scripts may reference database tables that don’t exist or have been changed.
One project I worked on got around this by having a directory which held database patch scripts which an upgrader should run before the site should be used again. That was messy.
Other projects get around this by requiring you to upgrade your files, and then go to a URL (/upgrade.php for example) which will do the database upgrade for you. Better, but still messy.
I think the solution is to have a database version number kept in a file which is upgraded (the front index.php for example), and another version number kept in the site config file, which is not upgraded. When someone visits the site index, it’s a non-expensive process to simply compare those numbers, and if they are different, then an upgrade is automatically performed, and the config file is adjusted with the new version number.
In that way, upgrading is simply a matter of unzipping the new files, and the next time anyone visits the site, it is upgraded automatically.
Anyway – that’s enough for this step. You still can’t create any pages, but we’re almost ready for it.
As usual, you can get this code from the google Webworks-Webme repository. No zipped package available yet, although the next article might be the first which warrants it (we’ll create an actual page and discuss templates).
Okay – WebME…
Before I introduce the first bits of code for WebME, I want to try explain what it is, and why it is different to, say, Joomla, Mambo or PHPNuke.
As I explained in the Webworks blog, WebME grew up as a tool designed to do some very very common tasks on small websites.
This CMS was built for about 250 separate websites. Each of those sites wanted a simple task – create pages, and edit those pages. There are other tricks such as Forms and Online Stores, etc, but we’ll get to them. The core task for this CMS is to create a page, and allow editing of that page.
The larger, more well-known CMSes, on the other hand, are built to be modular, extensible platforms. It’s difficult to define a single “purpose” to those platforms. For WebME, it’s simple – “manage some webpages”. For the others… I don’t know; they’re too large to define simply.
It’s tricky to decide where to start with this release. I can’t just release the whole damned thing, as that would be overwhelming, and probably a bad idea, security-wise. Instead, I’m going to release a piece at a time.
Today’s piece will provide the bare minimum – you will be able to download WebME, and install it. That’s it. Nothing else. No admin area, and no editing of pages. In fact, not even any pages. You at the back, have patience!
For today, I’ve written the basics of the installer. You can download it from Google, via SVN (subversion). WebME won’t be available via zipped package until I’ve released enough of it to be useful for the average web developer.
Today’s download really is the bare minimum – it creates a config.php file, but nothing else. Tomorrow I’ll show how upgrades work in WebME, and will show the admin area.
Despite the apparent nothingness of today’s release, I hope people will download and try it out. Each day will produce a few new ideas.
Today’s idea is how to create an installer. Most (all?) projects start off any page by loading the config. If that config does not exist, then either the system is not installed, or there’s something wrong. We use that idea to allow a freshly downloaded copy of WebME to bring the browser straight into the installation script.
It’s a simple and obvious idea, but useful. Tomorrow’s idea will show how to handle upgrades in a “continuous integration” way (…ish). In particular, tomorrow’s trick is useful for handling database evolution.
I had a nice chat with John earlier. We‘re considering releasing our tools under an open source license.
We’re doing this for a number of reasons. I think that chief among them is the hope that with many developers (or at least, with more than just myself), we should be able to keep on top of all the modern net tricks that keep appearing – RSS, Ajax, etc. So far, I’ve been doing that myself, but as our CMS, WebME (Web Management Engine) has grown, the effort to keep it uptodate has become trickier.
So, we are planning on releasing the code as open-source while I am still on top of it, so I can help other developers join in on the coding while still working away on it myself.
I’m very excited about this.
A few years ago, we tried this very thing, open-sourcing WebME, but no-one took us up on the offer, and we brought it back in-house.
However; since then, I released KFM, which has developed into a pretty popular open-source project, and I feel that releasing WebME again will work based on whatever OS credit I might have earned from that project.
We are hoping to release our newsletter-management service as well, depending on how WebME does.
I’m very interested to see what other people think of this. I know most of you will not have seen WebME’s admin area, but I think it’s one of the most user-friendly ones out there (even if I say so myself), and will be a boon for developers who want a simple CMS for simple sites – page-based sites with galleries, simple online stores, etc.
Just a quickie. I had to see if a selected country (selected by 3-letter code) was in Europe, but couldn’t find a handy function online for it.
So, here it is:
function inEurope($code){
return in_array($code,array(
'ALB','AND','ARM','AUT','AZE','BLR','BEL','BIH','BGR','HRV','CYP',
'CZE','DNK','EST','FIN','FRA','GEO','DEU','GRC','HUN','ISL','IRL',
'ITA','KAZ','LVA','LIE','LTU','LUX','MKD','MLT','MDA','MCO','MNE',
'NLD','NOR','POL','PRT','ROU','RUS','SMR','SRB','SVK','SVN','ESP',
'SWE','CHE','TUR','UKR','GBR','VAT'
));
}
That returns true or false depending on whether the submitted $code (an “ISO 3166-1 alpha-3” code) is in Europe or not.
I received an email this morning saying that KFM has a security hole – if a user creates a file named “test.php.” (note the ‘.’ at the end), then it is run as if it was “test.php”, even if you explicitly banned the .php extension in your settings.
I immediately added a line of code to ban all filenames which end in ‘.’, released a new version of 1.3 (available on the front page of the site) and corrected 1.4 in SVN.
After thinking about it, I realised that the security problem is not as serious as it may seem (for KFM – not in general). It’s definitely a problem, but in order to use it, you need to have access to a KFM instance in the first place. As securing KFM is not difficult, I think the problem may be contained.
But I digress – this appears to be a problem in Apache. To test it, I checked if renaming a Perl CGI file from .cgi to .cgi. would work, and it did.
This is a little disturbing, as it does not appear to be documented anywhere, so there is no way that a developer would know to avoid this security hole.
So, if you write programs that allow your users to upload or rename files online, make sure that the filename does not have a ‘.’ at the end!
edit: OMG! I was reading the Apache source to try spot the problem, and found the area where it happens – it’s in the file “http/mod_mime.c”. The function “find_ct()” extracts the extension for the server to use. Unfortunately, it ignores all extensions it does not understand, so it’s not just a case of “test.php.” being parsed as ‘.php’, but also “test.php.fdabsfgdsahfj” and other similar rubbish files! This is a serious problem.
There are a number of solutions to this:
further update: An easy solution. This problem rears its head when PHP is identified in your httpd.conf using this:
AddHandler php5-script .php
the solution is to change the above to specify the extension must be at the end.
<FilesMatch \.php$> SetHandler php5-script </FilesMatch>
Just a quick note. If you find that your customers can log into their postfix account ok, but can’t read their email, check the directory their email is in:
[root@postfix /]# cd /home/vmail/domainname.ie/ [root@postfix domainname.ie]# ls -l drwx------ 5 postfix postfix 4096 Sep 4 11:52 joe.smith drwx------ 9 postfix postfix 4096 Sep 9 09:30 jane.smyth drwx------ 9 postfix postfix 4096 Sep 9 21:09 Jane.Smyth
The problem appears to be that postfix authenticates using MySQL, which is case-insensitive by default, then creates a default email skeleton directory named after the login username if it doesn’t find one already.
You can see in the above example that our user has logged in using uppercase letters in the username, MySQL verified the user had a right to be there, then postfix created the user’s directory using the login credentials, even though a lower-case version of the directory name already existed…
Obviously, this incorrect account will not receive email – email will be sent to the correct one.
There are a few solutions:
Personally, i go for the first solution.
Yesterday we had to move about 300 domains from one machine to another. We bought a new machine recently and are taking this opportunity to move from Qmail (difficult to use, in my opinion) towards Postfix.
After doing one or two by hand, i decided that’s stupid – why not just automate the whole thing.
So I whipped up a script that reads details from vqadmin and uses those details to create postfix emails using mailadmin.
Please note that this does not handle forwards and other weirdness – just plain old email accounts. After running it, you need to check the accounts for forwards. I /could/ adapt it, but am too lazy.
<?php
$vqadmin_url='http://1.2.3.4/cgi-bin/vqadmin/vqadmin.cgi';
$vqadmin_auth='username:password';
$postfixmailadmin_url='http://5.6.7.8/mailadmin/';
$postfixmailadmin_username='your@email.address';
$postfixmailadmin_password='password';
ob_start();
// { get all domains
$ch=curl_init($vqadmin_url);
curl_setopt($ch, CURLOPT_USERPWD,$vqadmin_auth);
curl_setopt($ch, CURLOPT_POST, true );
curl_setopt($ch, CURLOPT_POSTFIELDS, 'nav=list_domains' );
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true );
curl_exec($ch);
$page=join('',explode("\n",curl_multi_getcontent($ch)));
preg_match_all('#<a href=vqadmin.cgi\?nav=view_domain&dname=([^>]*)>[^<]*</a>#',$page,$domains);
curl_close($ch);
// }
$numdomains=0; $numemails=0;
foreach($domains[1] as $domain){
$numdomains++;
echo '<h2>'.$numdomains.' - '.$domain.'</h2>';
// { get password page
$ch=curl_init($vqadmin_url);
curl_setopt($ch, CURLOPT_USERPWD,$vqadmin_auth);
curl_setopt($ch, CURLOPT_POST, true );
curl_setopt($ch, CURLOPT_POSTFIELDS, 'dname='.$domain.'&Submit=Show Users&nav=show_users' );
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true );
curl_exec($ch);
$page=join('',explode("\n",curl_multi_getcontent($ch)));
curl_close($ch);
// }
// { extract passwords
preg_match_all('#<tr><td><FONT face=Verdana color="\#FFFFFF"><a href=[^>]*>[^<]*</a></FONT></td><td align=middle><FONT face=Verdana color="\#FFFFFF">[^<]*</FONT></td><td align=middle><FONT face=Verdana color="\#FFFFFF">[^<]*</FONT></td><td align=middle><FONT face=Verdana color="\#FFFFFF">[^<]*</FONT></td><td align=middle><FONT face=Verdana color="\#FFFFFF">[^<]*</FONT></td><td align=middle><FONT face=Verdana color=\#FFFFFF>(<B>)?[^<]*(</B>)?</FONT></td><td align=middle><FONT face=Verdana color=\#FFFFFF>[^<]*</font></td></tr>#',$page,$matches);
$matches=$matches[0];
$emails=array();
foreach($matches as $match){
$username=preg_replace('#<tr><td><FONT face=Verdana color="\#FFFFFF"><a href=[^>]*>([^<]*)<.*#','$1',$match);
$password=preg_replace('#<tr><td><FONT face=Verdana color="\#FFFFFF"><a href=[^>]*>[^<]*</a></FONT></td><td align=middle><FONT face=Verdana color="\#FFFFFF">([^<]*)<.*#','$1',$match);
$emails[]=array('username'=>$username,'password'=>$password);
}
// }
// { log into postfix mailadmin
$ch=curl_init($postfixmailadmin_url.'login.php');
curl_setopt($ch, CURLOPT_POST, true );
curl_setopt($ch, CURLOPT_POSTFIELDS, 'fUsername='.$postfixmailadmin_username.'&fPassword='.urlencode($postfixmailadmin_password).'&lang=en&submit=Login');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true );
curl_setopt($ch, CURLOPT_AUTOREFERER, true );
curl_setopt($ch, CURLOPT_COOKIEJAR, 'tmp/cookies.txt');
curl_exec($ch);
curl_close($ch);
// }
// { create domain
$ch=curl_init($postfixmailadmin_url.'create-domain.php');
curl_setopt($ch, CURLOPT_COOKIEFILE, 'tmp/cookies.txt');
curl_setopt($ch, CURLOPT_POST, true );
curl_setopt($ch, CURLOPT_POSTFIELDS, 'fDomain='.$domain.'&fAliases=25&fMailboxes=25&submit=Add Domain');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true );
curl_exec($ch);
curl_close($ch);
// }
// { create email accounts
foreach($emails as $email){
$ch=curl_init($postfixmailadmin_url.'create-mailbox.php');
curl_setopt($ch, CURLOPT_COOKIEFILE, 'tmp/cookies.txt');
curl_setopt($ch, CURLOPT_POST, true );
curl_setopt($ch, CURLOPT_POSTFIELDS, 'fDomain='.$domain.'&fUsername='.$email['username'].'&fPassword='.$email['password'].'&fPassword2='.$email['password'].'&fActive=on&fMail=on&submit=Add Mailbox');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true );
curl_exec($ch);
$numemails++;
echo $numemails.':'.$email['username'].'@'.$domain.' ';
curl_close($ch);
}
// }
flush();
ob_flush();
}
Make sure to up the timeout length of your PHP scripts – it took me about 10 minutes to transfer 1607 emails.
Also, be aware that this does not transfer the /contents/ of the email accounts – just recreates them on the other server – I don’t even want to touch that particular problem…