jsquash, a javascipt compressor
Last week, I wrote the first part of a JavaScript compressor – it basically removed any unneeded spaces and semi-colons.
This week, I decided to write the compression part. I believe I’ve done pretty well, even if it’s quite slow to run.
try it out! (tested in Firefox. if this breaks your browser, …sorry!)
The two engines I was influenced by are the Dean Edwards’ “Packer”, and dithered.com Chris’s compressor (I don’t know his surname). Dean’s engine is incredibly fast, but I felt that the method used by Chris’s was a bit easier to understand and extend.
There were some design considerations, other than “compress the javascript”. My main goals were:
- don’t hog the CPU. I carefully broke loops apart into
setTimeoutblocks, to simulate threading. this makes the script very slow to finish (hours in the case of large scripts!), but it means that the browser is usable while it’s working away. - don’t restrict the users’ code. The script figures out for itself what characters it can use in its substitutions, so you don’t need to avoid using any characters.
Surprisingly, the script appears to create stronger compression than both Dean’s and Chris’s scripts. I am surprised by this because Dean’s algorithms are usually beyond my touch.
A handy thing about this script is that you do not have to wait for it to finish – at any time, you can click the “save” button to export a copy of the current compression – the script will continue in the background.
The dithered algorithm runs in the opposite direction to mine. In that script, you tell it the maximum size string to search for, and it searches for replacements with that length and then gradually lower. Mine starts with a default of 2, and searches upwards, depending on whether there is a reasonable chance that there are strings of a larger length to be found.
Now, to work on speeding it up!
Well done. With everything I tried your algorithm produced better compression results than all the others. The decompression seems to be fast enough to be not very noticable.
Will you release the source in unsquashed form as well? I’d like to port the algorithm to some other language so that I can use it as part of the compiling an packaging process of Applications.
the source is in this directory: here.
I have noticed a few bugs when the compression gets into the high-bit characters (128+) – some characters seem to cause breakages, but I haven’t yet figured out which. I think the way to find out is to run an
evalof the compressed code after each compression, within atry{}catch(e){}, and report on those characters that cause errors.I’m currently working on converting the code to a QT application (using kdevelop in Linux). I haven’t written C (or C++) in about 6 years, so it’s hit-or-miss whether I’ll release it this side of the end of the month.
If you want, I can undo the
setTimeout-based simulated threading that I did, so you can view the algorithm with itsforloops intact.For-Loops would be great, though your code is still understandable.
Actualy C++ was my choice as well, but I’m going to do a CLI-Programm that can easily be included in make-Files. If you like we could join forces and isolate the Algorithm itsself in a simple Library which can be used by a CLI tool as well as by a GUI tool.
Is the high-bit-character Problem a problem while compressing or while decompressing? In the first case the C++ code could be a solution.
The problem happens during the decompression. I’ll be working on it later on, and will try to isolate the actual characters which cause the problems. I believe the problem is probably a bug in regexps in Firefox (haven’t deeply tested this in IE yet). If I can isolate it, I’ll come up with a workaround, and will also report the bug.
I didn’t even think of CLI – basically, I’m building an exact clone of the application, using KDE GUI widgets.
That would probably be very useful, though – I’ve also thought of adding a “compress this” to context menus for Konqueror’s file manager, allowing some-one to simple right click on a .js and compress it in an off-hand way.
I’ll build up the loops again today.
“That would probably be very useful, though – I’ve also thought of adding a “compress this†to context menus for Konqueror’s file manager, allowing some-one to simple right click on a .js and compress it in an off-hand way.”
Actually that would be a good place to use a CLI tool or a library 🙂
I think you should not overwrite the original file with that context menu entry. Maybe replace “.js” by “.z.js” or somethin similar in the filename. Just to prevent users from making their js-files unreadable.
true. I was thinking that also. What I would do is similar to the habit I’m trying to cultivate – automatically create a _source directory, please the original in that (checking first with the user if there’s already a copy in there!), then do the squishy thing.
I’m replacing the loops here – massive speed increase!
Didn’t get to work on my C++ version over the weekend – a wedding happened… maybe tonight 🙂
I have some C++ code now. Actually it is just the first step (removing spaces and comments). I grabed your code and made C++ code from it.
Maybe the architecture can be improved as well, but first I suggest, we should try and get working code. This evening I hope to find the time to transfer the rest of your code to C++. Do you have access to a CVS- or SVN-server that we could use or should we start a sourceforge-project to share code?
Another important thing: I suggest, we use the GPL or the LGPL as the Licence. What do you think?
OK, I transfered everything to C++. It compiles, but I haven’t testet it yet. Maybe I get around to do that tomorrow.
I have to admitt, that I haven’t realy unterstood the algorithm by now. I need to read the code again in a quiet hour 🙂
My own effort compiles, but then crashes when you hit the “squash this” button. here it is. Not sure what I’m doing wrong…
I’ll write up a post explaining the algorithm.
As for the sourceforge thing – yeah, why not! I think this could turn into a few applications. It would be handy to have somewhere central to host it. I’m just learning to use SVN myself, and haven’t tried setting up a remote server yet.
Maybe it crashes because you are using cmp in kjsquashwidget.cpp without having created it. With cmp created it didn’t work for me, but it stopped crashing.
Pingback: » How the jsquash compressor works « klog
I have made a compression tool too. Have a look at it at http://www.bananascript.com/