Category Archives: web development

unwatermarking images

I’ve started a website where I intend to sell thousands of products from a number of distributors through drop-shipping (the products go directly to the customer).

For reasons that I don’t understand, the distributors have watermarked their images, and don’t provide unwatermarked versions unless you’re an already well-established customer of theirs.

For the purpose of this demo, a watermark is a constant-colour “stamp” which is given opacity and pasted into the original image.

As I intend to be a good customer, I figured it would be okay for me to simply “unwatermark” the images.

There are a number of instructions online which show how to /fake/ an unwatermaking – by basically smudging the area where the watermark is.

However, as most watermarking appears to follow a single method, it is actually possible to simply reverse the process and remove the watermark, after a little trial and error.

Let’s consider an example. Here is an image, a stamp, and the merge of the two:

(original is here)

  • demo1
  • demo2
  • demo3

To reverse this, you need to know what algorithm was used to create the watermark, and what the original watermark was.

Most people use a fairly simple method to watermark their images:

The stamp is one single colour, usually gray (#808080 in RGB) which will be visible on images which are both light and dark.

The stamp is then given an opacity (30% in my case above), and pasted directly over the original image.

The formula for any particular colour channel (R, G and B) on any pixel is: C3=(1-p)C1+pC2, where p is opacity (0 to 1), C1 is the colour value for the original image, C2 is the stamp’s colour value, and C3 is the resulting image’s colour value.

To reverse the watermarking, you need to convert the formula to see what it is in respect to C1: C1=(C3-pC2)/(1-p).

As most stamps will be using a middle gray (#808080), you just have to guess at the opacity. .3 is a good start.

For some reason I’m not yet sure of, the code I came up with did unwatermark the image, but too much… the points where the watermark were, ended up being too bright. So I needed to add a darkening aspect, reducing the brightness of the result of the above calculation.

I’m not going to hold your hand if you can’t make this work, but here’s the code I ended up with (assumes the images are exactly 400×400 in size). The original should be ‘original.jpg’, and the stamp should be ‘stamp.png’ (with white where transparent pixels should be).

$p=.3; // opacity

$f1=imagecreatefrompng('stamp.png');
imagepalettetotruecolor($f1);
$f2=imagecreatetruecolor(400, 400);
$f3=imagecreatefromjpeg('original.jpg');
imagepalettetotruecolor($f3);

for ($x=0;$x<400; ++$x) {
  for ($y=0; $y<400; ++$y) {
    $rgb1=imagecolorat($f1, $x, $y);
    $rgb3=imagecolorat($f3, $x, $y);
    $r3 = ($rgb3 >> 16) & 0xFF;
    $g3 = ($rgb3 >> 8) & 0xFF;
    $b3 = $rgb3 & 0xFF;
    if ($rgb1==16777215) { // white. just copy
      $c=imagecolorallocate($f2, $r3, $g3, $b3);
      imagesetpixel($f2, $x, $y, $c);
      continue;
    }
    $r1 = ($rgb1 >> 16) & 0xFF;
    $g1 = ($rgb1 >> 8) & 0xFF;
    $b1 = $rgb1 & 0xFF;
    $r2=c($r1, $r3, $p);
    $g2=c($g1, $g3, $p);
    $b2=c($b1, $b3, $p);
    $c=imagecolorallocate($f2, $r2, $g2, $b2);
    imagesetpixel($f2, $x, $y, $c);
  }
}
imagejpeg($f2, 'unwatermarked.jpg');

function c($c1, $c2, $p) {
  $c=c1($c1, $c2, $p);
  $c3=$c-(255-$c)*.2;
  return $c3<0?0:(int)$c3; 
} 
function c1($c2, $c3, $p) {
  $c=($c3-$c2*$p)/(1-$p);
  return $c>255?255:(int)$c;
}

installing kv-WebME in CentOS 6

I’m setting up a new server and need to install kv-WebME in it. The previous instructions (for Fedora 16) were fine for an older version of WebME, but the most recent requires more uptodate packages.

So, first, we need to tell CentOS 6 to use more uptodate packages than are provided by default.

su -
rpm --import  http://apt.sw.be/RPM-GPG-KEY.dag.txt
rpm -ivh http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm

Now we need to install PHP (at /least/ 5.3), Apache, MySQL, etc.

yum install httpd mysql-server php php-gd php-mysql php-xml svn zip unzip

Add a user account for the website, and download the kv-webme repository

adduser webme
su - webme
svn co https://kv-webme.googlecode.com/svn/trunk/ kv-webme
chmod 755 /home/webme
exit

Now add the web configuration to /etc/httpd/conf/httpd.conf

NameVirtualHost *:80
<Directory /home/webme/kv-webme>
  AllowOverride All
</Directory>
<VirtualHost *:80>
  DocumentRoot /home/webme/kv-webme
</VirtualHost>

And finally, turn it all on.

service httpd start
service mysqld start
chkconfig --level 35 mysqld on
chkconfig --level 35 httpd on

That’s it!

straightening an image of horizontal lines

I’m working on a mobile app for photographing sheet music and then playing it.

When I first approached this, I considered using a Hough transform, which is a mathematical tool for finding lines in an image. It produces a matrix based on MC space (the tangent and y offsets of the lines).

I could then use the matrix to figure out what was being shown on the sheet.

That method is very computationally expensive.

After considering it, trying it, then abandoning it, a better solution came to me while I was thinking about something totally different.

Sheet music is composed of mostly horizontal lines, while everything that is not a horizontal line is part of the notation itself.

So, all I need to do is first locate the horizontal lines, and everything else will be easy to find.

The first problem, then, is how to make sure that the sheet itself is level.

How I ended up doing this was to measure mean difference of the average colours of each ‘y’ coordinate of the image, and try offsetting one side of the sheet up and down until I reached the maximum mean difference.

This is easier to understand visually.

Let’s consider this image:

As a human, we find it easy to spot the skew and fix it, but a computer is not so intuitive.

Here is the same image with the “x” coordinates of each “y” coordinate averaged out (motion-blurred, basically)

That’s a simple average of the “x” coordinates, and there already appears to be a pattern.

Next we shift/skew one side of the image up or down a few pixels and test it again. In my tests, I use a naive “brute-force” test of all offsets from -15 to +15. Here are blurs of a -11 offset and a +11 offset:


-11

+11

Obviously, the right one is the -11 one. But how do we tell a computer what the “obvious” solution is?

Well, the right answer is probably to come up with a way to measure which one is more “noisy”, but I couldn’t think of a simple way to do that.

Instead, what I did was to measure the average colour in the each image, use that average to find the mean difference in each image (how far from the average “gray” each line is), and the one we are looking for is the one with the highest mean difference.

Having found the right offset (-11), we then simply shift the pixels in the image by that much (in Y and X space), and end up with these images:


original image

straightened

The next task is to fix skewing, but it will use basically the same technique.

demo

split testing

I wrote a split-testing plugin (also known as “A/B testing”) for my CMS early last year. It wasn’t very good, so I wrote an external split-testing tool instead, which you can you for your own websites if you want.

Split-testing application

It works similarly to Google’s “web optimiser”, but I think it’s easier to use.

As a test, I chose to optimise the front page of KV Sites, to see if I could encourage people to contact me or read other pages on the website.

In particular, I wanted to see if it was better to explain what I do, or to assume that the reader knows and just wants to get on with.

As an example, consider the following two extracts:

Indirect speach – explaining what I do

Web Development

Reduce costs by automating and networking your business.

Proven record, with a number of large projects completed. Example projects include Duffy Transport's management application, the crop prediction software used by Cropworks, ongoing development of the WebME CMS, and technical support for a number of projects by other web development companies.

vs.:

Direct speech – asking what the reader wants

I am looking for Web Development

If you are looking for an online solution which to reduce your office expenses, we have experience writing solutions which can cut hours of work per day off your load. Read more about our web application development here.

It seems obvious that the direct speech version would work better, but when your job is involved, it’s better to be certain than to work based on assumptions.

So, I set up a split test. On the homepage of KV Sites, I sent the HTML for both versions to the browser, and set the different versions to display randomly using CSS.

Each version includes a “more info” link (of sorts), so I added the “conversion” code (which records when people go to the page you want them to) to the pages they were linked to, and started recording.

It took a few weeks for the information to come in, as my site does not have heavy traffic, and I didn’t want to artificially boost it in any way, but here’s how it turned out:

Name Visitors Conversions
front page direct marketing 237 v2: 12
v1: 24

It’s obvious from the above that “v1″ (the direct speech one) is twice as effective as “v2″ (indirect speech).

I’m very encouraged by these results. They mean that it is possible to both a) improve “conversions” based on text changes, and b) provide numbers proving those results.

code golf

I came across a new (to me) game yesterday – Code Golf.

The game involves coming up with an algorithm to solve a programming problem, and trying to condense the code for the algorithm into the smallest number of bytes possible.

The first one I tried was this. It was a fascinating problem, and took me a day of musing on it (in the back of my head as I did other things) before I had a solid solution.

After writing the solution, which took 1380 bytes, it was time to start “golfing” it.

At first, I thought I’d try my compressor on it. This shrank it to 825 bytes, but after compression, it couldn’t be worked with anymore, so I thought I’d try compressing it manually.

This resource was fascinating, and gave me a load of pointers.

There were a few small points I came up with myself while working on it:

I prefer to use “\n” instead of “;” for command endings, as it makes code more readable. (the game is about shrinking the code, not obfuscating it)

A saving can be had by combining nested loops:

// before
for(i=5;i--;)for(j=5;j--;)M[i][j]=0
// after
for(i=25;i--;)M[0|i/5][j%5]=0

If you need to push into an array on multiple lines, make a shortcut for the push method:

// before (example)
a=[]
cond1()&&a.push(1)
cond2()&&a.push(2)
cond3()&&a.push(3)
cond4()&&a.push(4)
// after
a=[]
P=a.push
cond1()&&P(1)
cond2()&&P(2)
cond3()&&P(3)
cond4()&&P(4)

If possible, find a maths way of identifying interesting points, instead of comparisons.

// before
if((x==4&&y==4&&z==3)||(x==4&&y==3&&z==4)||(x==3&&y==4&&z==4))dosomething()
// after
if(x*y*z==48)dosomething()

I think this game is really interesting, and it will sharpen my own skills as a programmer, as it taxes the mind not only to find the solution to a problem, but also to express that solution as concisely as possible.

In the end, I was able to solve the problem in 668 characters – that’s 142 characters less than my compressor was able to manage.

musical intervals trainer, web version

last weekend, I wrote an intervals trainer app for practicing recognising intervals.

I want other people to use it, but haven’t got a Google development account yet so can’t upload an app.

So, today, I improved the app and made a web-accessible version.

try it out!

it’s designed to move up from very simple intervals (major/minor 2nd intervals, with only natural notes) to more difficult intervals (diminished/augmented, with double sharps and flats), but it’s also designed to only get more difficult at a rate that /you/ can manage.

to do this, the app uses a “levels” system, where each level has one more extra type of interval or note type, and you are tested on them. over 50% of the time, the question will be from the level you’re on, and the rest of the time, the question will be randomly chosen from every other level that you have already passed.

get 10 in a row correct, and you go to the next level.

but, get 5 wrong in a row, and you go down a level.

at the moment, there are 24 levels – all the way up to augmented 8ths – can you get through all the levels?

give it a try!

Adding a trigger-based plugin to WebME

One of my clients has his own fork of WebME that he keeps current with the SVN version.

He mentioned a collision recently, where a hack he had written into the online store plugin was overwritten by updates.

So, in this post, I’ll demonstrate how to add a plugin that uses a trigger to run some code.

First, we define what needs to be done.

The client wants that when an order in an online store has been processed, an email is sent out to that client.

This immediately points out where the trigger point goes. The processing of orders is done in the file ww.plugins/online-store/verify/process-order.php, so we need to put the trigger in there as well, at the end of the OnlineStore_processOrder() function:

  Core_trigger('after-order-processed', array($order));

What happens at that point is that the CMS will check all plugins to see if there are any that have a trigger of the name “after-order-processed”, and if there are, then that trigger will be fired.

So, next we need to create the plugin. In fact, it’s so easy I’ll just write it straight out. I’m calling this one “DemoPlugin”, so we start by creating the directory /ww.plugins/demo-plugin”, and placing the file plugin.php in it:

<?php
$plugin=array(
  'name' => 'DemoPlugin',
  'triggers'      => array(
    'after-order-processed'=>'DemoPlugin_afterOrderProcessed',
  ),
  'description' => 'Sends an email after an order is processed',
);

function DemoPlugin_afterOrderProcessed($PAGEDATA, $order) {
  mail('kae.verens@gmail.com', 'subject line', print_r($order, true));
}

Simple!

Now, in order to do this, I edited one file which is part of the “official” WebME package, to add the trigger. If you find you need to do this, please contact me and tell me what you’re trying to do, and what you edited, so I can add it to the SVN version and not break your code in future releases.

FB and G+, the new OS-wars

Back when Linux was a gangly youth, there was a great excitement every time you did an update. Unlike Windows and Mac with their expensive three-year-nothing-then-a-service-pack-with-cool-stuff routines, in Linux, there were so many cool free packages around, and /every one/ of them would have something new almost every week.

So, I spent quite a lot of time compiling and recompiling, everything from wget up to the KDE behemoth. And I’d be reading the weekly release notes as well to see if there was any new trick out that week.

I can’t tell you how excited I was when I first installed a copy of Linux where I didn’t have to configure X11 using a text-mode Xconfigureateur!

Or when I recompiled my kernel the first time, or when I recovered from a bad update straight from the GRUB command-line. Or the first time I ejected a CD from across the room (logged in remotely), or freaking out the wife by playing some music on her laptop from a different room.

In the last few years, Linux development has matured, though, so there’s not the same edge-of-the-seat excitement that there used to be, but I think there’s still hope for the techadrenaline junkies out there, because Facebook and Google+ are the new cool, and there’s a /ton/ of stuff that can be done with that!

My eyes are on how FB and G+ evolve. now that FB has competition, I expect some /really/ cool stuff is going to come out of the woodwork.

first and last saturday of a month

I’m working on an expenses application. In order to do it, I need to display the entire month, with full weeks in each (from Saturday to Friday).

To do this, I need to know what date the Saturday in the first week falls, and the date of the first Saturday in the following month.

Here’s the code (assuming year is a 4-digit year, and month is a 1-12 number):

      // { start date
      var start=new Date(year, month-1, 1);
      var day=start.getDay();
      day=(day+1)%7; 

      start=new Date(start-3600000*24*day);

      // }
      // { end date
      month++;
      if (month==13) {
        month=1;
        year++;
      }
      var end=new Date(year, month-1, 1);
      var day=end.getDay();
      day=7-((day+1)%7);

      end=new Date((+end)+3600000*24*day);

      // }

can’t be bothered explaining it – just trust that it’s right ;-)

separating buttons in jquery-ui dialog

By default, the jQuery-UI dialog will place buttons on the right side of the popup:

This causes a problem because if you have “OK” right next to “Delete”, and you click the wrong one, well …

The obvious solution is to move the “Delete” to the opposite side.

To do that, add the following two lines after creating the dialog:

$('.ui-dialog-buttonset').css('float','none');
$('.ui-dialog-buttonset>button:last-child').css('float','right');

Now the buttons are on opposite sides: