Scaling FieldMotion

I wrote a quick overview of how I approach the scaling-up of FieldMotion, over at the FieldMotion blog.

Basically, we started off with a huge monolithic block of code and data, then looked carefully at it to see how we could tear it apart into logical chunks.

The simplest to start with was data, so we separated out the database into a MySQL master/slave cluster, and a MongoDB replicated shard cluster.

Afterwards, we looked at the actual services that were performed by the server and started separating those out so they were completely independent from the main block.

At the moment, FieldMotion runs across more than 60 servers, and can tolerate sudden catastrophes on pretty much any part of it. We actually had an incident recently where an entire datacentre went offline for about 8 hours, but no-one noticed because we make sure each of our replicas is in a different datacentre.

yet another update

Time flies. I keep on planning to do things, and then failing to do them because there isn’t enough time, in between working 12 hours a day and trying not to fall asleep as soon as I get home.

I finished the basics of my next book, Live Forever, which I put up in website form so I can figure out through statistics which pages (a lot of them!) need work. Tonight, I’m working on the Cancer chapter so haven’t put that in there yet.

Over the weekend, I hope to get a start on a new project, which will help to design 100% nutrition diets based on common supermarket produce. There are known recommended daily allowances (RDAs) for all nutrients, but when you make your dinner, you don’t calculate an optimal meal because it’s just not practical or easy. The new project is designed to get around that by offering meal plans that are affordable and personalisable (you will be able to put your preferences into it). We’ll see if that gets off the ground!

In CoderDojo, some of my students (I really mentor them, more than teach, but what do you call someone you mentor? Mentoree?) are working on some interesting projects for this year’s Coder Dojo conference and next year’s Young Scientist. Two examples: programmable magnetic levitation, and a laser harp.

In work, we’ve moved beyond the frantic development stage that all companies go through, and are now in stabilisation mode, making sure the system is bulletproof and can scale well beyond current needs. I still find it interesting, even though the work I’m doing at the moment is not flashy and user-visible. Today, for example, I was writing a logging system to make sure that even though users access our mobile servers in a “round robin” method at the moment and the logs of their visits are therefore scattered among the servers, I can still aggregate them on the other end into something that can be searched easily. Not flashy, but quietly satisfying.

what I’m up to these days

I’m going to start blogging again soon. I’m getting more and more into Arduino stuff, and want to be able to explain stuff properly to my students at the Monaghan Coder Dojo, so will be writing articles explaining what I’m teaching.

In work (FieldMotion, where we do field service management software), we’re doing some really interesting things with Zapier, which lets us easily integrate with more than 750 applications.

We’re also doing a new thing with the reports that we generate, where we can apply “skip logic” to the actual reports, resulting in personalised reports depending on the questions answered in the job forms.

I’ve had to take a break from the 3D Printer stuff, because the printer I had (a Makibox) broke down so often that I was spending more time fixing it than using it. I’m not quitting with 3D printing – just need to wait for a bit of cash to come in so I can afford a better printer. Once I have that new printer, the first thing I’m doing is making a second printer with it!

a more precise measurement of the number of cows murdered for the new UK fivers

image taken from https://www.theguardian.com/business/2016/dec/02/5-animal-fat-bank-note-british-vegetarians-being-stupid-says-inventor

There is a Vice article that claims to have an accurate measurement of how many cows were killed to make the new UK five-pound notes.

The say it’s half a cow.

Ignoring that you can’t really kill just half of a cow, let’s look at their maths.

How much do cows weigh? Between 1,100kg for a male (bull) and 720kg for a female. So, on average, a cow weighs 910kg.

Not true. If you want the average weight of a cow, you need to remember that 50% of male cows are murdered before they become adults, so the average needs to take that into account.

It looks like Vice got their weights from Google, which says 1100kg for a male, and 720kg for a female.

Given a 2:1 female:male ratio, the average is more like (720*2+1100)/3=850kg per cow.

The body fat content of an average cow is 25 percent. Therefore, the amount of fat in an average cow’s body is 227.5kg.

Vice appears to take the word of this question‘s answerer when it states 25%.

However, if you trust the word of the Canadian government, then it’s more like 15%. The Oklahoma state government says it’s less than 18% (including bone, skin), making the 15% sound about right.

So, 15% of 720Kg. Therefore, the amount of fat in an average cow’s body is 108kg, not 227.5kg. Less than half what Vice stated.

How many kilograms of this fat is contained in offcuts you could use to make tallow? About 40kg, according to a man at the James Elliott butcher in Islington.

Tallow is rendered from suet, which is made from the fat found around the loins and kidneys. So we can’t just use all the fat from the cow. According to the Oklahoma Department of Agriculture, about 4% of a cow’s weight is suet. That’s 28.8kg (720*.04), not 40kg.

How much tallow is used in one note, according to the Bank of England? “A trace”, which chemically means less than 100 parts per million, or 0.01 percent. A polymer consultant I called confirmed that the tallow present in a given polymer would be a fraction of a single percentage.

Again, where do they come up with these numbers? No references given.

First off, when some PR guy says “there’s just a trace”, they are not speaking as an analytical chemist. They are saying “stop asking. not enough per note to make a difference on a scale”.

They say that trace means less than 100ppm. But atmospheric CO2 is a trace gas, and that is 335ppm.

Here’s an actual definition of what “trace” means – 0.1%. That’s still a “fraction of a single percentage” as Vice said, but it’s ten times larger than the 0.01% they pulled out of nowhere.

New £5 notes weigh 0.7g, therefore there is roughly 0.00007 g of tallow present in one £5 note.

0.1% of 0.7g is 0.0007. Ten times larger than the figure Vice comes up with.

How many fivers are in circulation now, and therefore will be around by May of 2017, when all the old paper ones have been phased out? 329 million notes.

To work out how much tallow will be used in total in all of these fivers, we need to multiply 0.00007g by 329 million, which gives us 23,030g, or 23kg.

Again, multiply by ten. 329,000,000*0.0007g=230,300g, or 230kg.

And if you get about 40kg of tallow-worthy fat from the average cow, how many cows would you need to make every single £5 note in circulation?

Well, since it’s actually 28.8kg per cow… take the 230kg required, divide it by 28.8kg, and you get:

8 cows.

Not half a cow.

16 times larger.

You might say “yeah – but who cares? it’s a fucking cow!”.

Well yeah – people that say things like that are not going to be budged anyway.

quick method to clone a MySQL database

let’s say you have a MySQL database on db1.db and you want to clone it to db2.db

the “official” way to do this is to run a “mysqldump” on db1.db and then import the resulting .sql file into the db2.db server.

There are problems with this approach:

  • mysqldump locks the source database, making it inaccessible while the dump is happening.
  • mysqldump creates files which may be many times the size of the source database’s binary files, potentially exhausting the space on your source server before it’s even done.
  • the resulting file then needs to be imported into the target server, which could take hours depending on the size.

I needed to clone some databases in a hurry that are about 20G in size. The method I used ended up taking less than half an hour to complete, and the source database (db1.db) only had to be down for less than a minute, instead of the potential /hours/ in the mysqldump method.

  1. use rsync on db2.db to copy the data directories from db1.db to db2.db:
    cd /var/lib/ && rsync root@db1.db:/var/lib/mysql ./ -rva –progress –delete
  2. use rsync on db2.db to copy binary logs from db1.db to db2.db:
    cd /var/log/ && rsync root@db1.db:/var/log/mysql ./ -rva –progress –delete
  3. repeat 1&2 (the first time around would take some time. the second time around will be quick)
  4. on db1.db, stop the database
    service mysqld stop
  5. on db2.db, repeat 1&2 one last time
  6. on db1.db, start the database again, and start the slave service if you need to
    service mysqld start
  7. on db2.db, remove auto.cnf and any innodb log files
    cd /var/lib/mysql/ && rm -f auto.cnf ib_logfile*
  8. start the database, and start the slave if needed
    service mysqld start

With the above method, your source database will be down for only a minute or so (steps 4-6).

The reason that 1&2 are repeated 3 times:

  1. clone the db1.db database from scratch. this will take a while
  2. because it took so long to run #1, there are probably a lot of changes. repeat to get those changes
  3. when you stop db1.db, some files will get final changes as they are changed. grab those after db1.db has been stopped

You need to delete any existing innodb logs (step 7) which might cause the system to attempt to “fix” some tables it might think are broken. but, because we did a clean shutdown in step 4, this is not necessary. so delete the log files (they will be recreated automatically).

If you are doing the clone because you want to create a new slave database, then the database needs a new internal ID that it will send to the master. By deleting auto.cnf, you force the MySQL server to create a new unique ID.

front right top corner

I’ve done the back corners of the printer. Now, I can tackle the front.

The front corners are where I will put the motors that control the X/Y coordinates of the hot-end.

So far, everything I’ve printed is symmetrical, but the two belts are at different heights, so in this case, one motor will be higher than the other.

I’ve decided that the right motor (right when facing out from the printer. left when facing the printer) will be the top motor.

I’ve designed the model for this so that it can wrap over the end of the case edge (and you can screw into it) and bolt the motor into the model.

front right top corner, with model Nema-17 motor in place

front right top corner, with model Nema-17 motor in place

front right top corner model

front right top corner model

It’s best to print this one on its side, so there is no support needed, and less cleanup in that space between the walls. After printing this out for the first time, I found that the wall space in my print was too tight, so I adjusted the STL file to add 1mm more space. this should not matter much.

motor and printed model added to box

motor and printed model added to box

back top corners

CoreXY printers have two timing belts overlaid on each other around the box. To allow the belts to move, bearings are placed in various corners. Today, I’ll tackle the back top corners of the printer box.

In the image below (taken from a scene of this video), you can see how it’s handled usually:

image showing back corners of CoreXY belt system

image showing back corners of CoreXY belt system

Because I’m trying to avoid using any rods are other forms of complex structure, I decided to come up with a printed solution that I could attach to the wooden corners of the box.

The design with bearings and a washer in place will look like this:

back top corners of print, with two bearings and a washer in place

back top corners of print, with two bearings and a washer in place

This slots neatly over the wood at the back top corners of the box.

The design is not yet perfect. I anticipate there will be pressure towards the center of the box on the bottom bearing, so I should have screw holes at the bottom of those walls as well. But, I think this will do for the “bootstrap” printer.

An improvement I will be making as soon as the prototype is complete, is to replace the metal bearings with 3d-printed bearings, like in this video. That will get me closer to having a purely 3d-printed 3d printer. Also, 3d-printed bearings will be cheaper than metal bearings, reducing the cost for future printers.

So to create the corners, we will need to print out two each of the outer back top corners, and the inner back top corners. Don’t slot them together until you have your bearings. Otherwise you will find it difficult (or impossible) to separate them without breaking them.

inner back top corner. bearings and washer go on the pole

inner back top corner. bearings and washer go on the pole

Screenshot from 2016-03-23 20-13-32

outer back top corner. the hole on the top slots onto the inner corner’s pole to keep it still

Once your pieces are printed, place an LM8UU bearing on each pole, then a washer, and then another LM8UU bearing. Slot the bottom piece with the pole into the top piece so that the pole goes into its corresponding circular hole in the top piece. You might need to shave the top of the pole slightly to make this fit. Don’t shave too much.

Finally, place the corner pieces over the back top corners of the box and bolt them in place. For the other edge and corner pieces so far, you could use screws, but this one will need bolts because there will be inward pulling force on the pieces from the belts going through them.

where to put the inner back top corner pieces

where to put the inner back top corner pieces

I don’t yet have the bearings for the corners, so the photo below is of installation on one side without the bearings. When the bearings arrive, I’ll update this post.

back top corner. the belts loop around the pole on this (after bearings are added)

back top corner. the belts loop around the pole on this (after bearings are added)

putting the box together

KV Printer 1 will be basically a 50cm^3 cube, giving quite a large printable area.

Obtain a 5mm plywood sheet and cut 4 50cm^2 squares in it. These form the base and walls.

Next, we need to stick this together at the corners. To do that, print out 2 corner pieces and 6 edge pieces. Using these as templates, drill 2mm diameter holes in all corners of the wooden squares (they’ll be 20.5mm in from X and Y), then screw the squares together like in the third image below.

outer corner for 3d printer

outer corner for 3d printer

outer edge piece for 3d printer

outer edge piece for 3d printer

placement of outer corner and edge pieces

placement of outer corner and edge pieces

Notice that we have not yet fastened the back top edges together. That will be done in the next post.

The finished product at this stage looks like this:

printer box after installation of back bottom corner and side edge pieces

printer box after installation of back bottom corner and side edge pieces

building a new 3D printer

after working with the MakiBox 3D printer for 8 months, I think I’ve learned enough about its failings to start building my own.

I’ve started building a 3D printer of my own, based on the SmartCore idea, but with enough changes that this will be my own design.

Makibox (on the right) printing out pieces for the new KVPrinter version 1. The wood on the left is for the walls and base of KV Printer 1

Makibox (on the right) printing out pieces for the new KVPrinter version 1. The wood on the left is for the walls and base of KV Printer 1

The MakiBox printer’s major failing (as far as I’m concerned) is in how it controls the X/Y position of the hot-end.

To do this, it has two long horizontal threaded rods, against the back wall and the left wall. These rods have long arms positioned on the threads, extending out above the print bed. Where the arms cross each other, the hot-end hangs down. Thus, the position of the hot-end can be adjusted by turning the rods.

The problem with this method is easy to see when you consider an analogy. Hold a pencil normally, and draw a 1mm line. Now, hold the pencil by the eraser end and try draw a 1mm line. The precision is just not there. The further away from the fingers the pencil lead gets, the harder it is to control it precisely.

One solution to this which I thought of, is to use a Bowden cable (bicycle brake cables, for example) to fix the position of the arms at the screw side to the position of the arms at their opposite sides. This would work, and would increase the precision of prints drastically, but it’s a lot of work and would look ugly.

After seeing the SmartCore printer, I decided that instead of fixing what I have, I would use what I have to make a new printer. In a way, I am printing a new printer. At least, parts of one.

The SmartCore printer is based on the CoreXY positioning technology, which is similar to the Bowden solution I came up with. Here is a video showing CoreXY in motion

In CoreXY, the hot-end (or drawing thing in the video) is positioned on a moving platform. It can move in X along the platform, and the platform itself moves in Y along rods in the sides of the frame.

To reduce cost in my own printer, I will replace the Y and Z rods with ledges that the platform will slide along.

My calculations suggest that the material cost of my printer will end up being below €150. If this ends up being correct, and the printer is as good as I hope it to be, then I will sell kit packages of the printer for €200.

Bill of Materials:

item amt cost per piece total
nema 17 motors 2 €18.61 €37.22
nema 17 motors 2 €12.675 €25.35
rods, 8x500mm 2 €4.58 €9.16
lm8uu bearings 12 €0.5075 €6.09
608 bearings 10 €0.237 €2.37
timing belts (meters) 5 €1.004 €5.02
controller board 1 €25.71 €25.71
pfte bowden tube 1 €7.97 7.97
hot end 1 €8.53 €8.53
psu, 12v 20a 1 €21.29 €21.29
wood
Total €148.71

I’m working on construction at the moment. I’ll write more articles as I go.

Makibox 3D Printer

I had the option to get my birthday present about two months early. Jumped at the chance.

Makibox, a 3D printer company, is selling off its entire stock of printers (makiboxclearance.co.uk), so it was a chance to get something cheap that I can hack on.

The package only took a week or so deliver, which is much better service than I expected, based on some of the messages I’d seen online.

I bought the unheated version (here) in kit form.

It took a few hours to put the machine together. I didn’t try printing anything until the next day.

The printer works by raising and lowering a print bed (the Z axis), and moving a “hot end” around on top of that in X and Y. The hot end hangs from the centre of two crossbeams, one of which moves in X and the other in Y.

The first problem I encountered, was that when I went to print for the first time, the hot end immediately started carving a pretty pattern into the bed. The printer didn’t know where the bed was, so was lowering the hotend too far down.

This kept happening even after I used the “bed leveling wizard” in Cura, the first step of which is /supposed/ to define where the bed is. But, no matter how accurately I did the first step, it totally ignored that and reset automatically to a level where it thought the bed was a few millimetres lower than it actually was, making the hot end drive straight into the bed.

It took me a while to figure out the problem – that the bed depth was “hard-coded” into the printer’s hardware – before every print, it would raise the bed right up until the platform-raising piece on the X axis screw touched against the “end-stop” switch at the top.

The solution to that was to glue something to the top of the platform-raising piece so it would hit the switch sooner. In the end, I glued a scrabble piece and a sim card (I had them at hand) on. This artificially lowered the expected bed depth by about 2cm, which is much more than is needed for the hot-end that comes with the printer, but is perfect for the replacement hot-end I ordered next.

The original hot-end sucks. They even say it themselves – in their words, “the standard hotend in the makibox kit is not the greatest piece of engineering ever made by man, it does have a tendency to burn out”.

The first problem I encountered with that hot-end was that it has no way of cooling off. There is an aluminium wall on one side of the base-plate, which could hold a heat-sink, but the heat-sink would be a case of “too little, too late”, as the hot-end should really be cooled right above the heating element, not 3cm above it. The problem is that when the hot-end’s heat spreads upwards, the plastic being pushed into it melts too soon, and it ends up like trying to push goo through a small hole at the bottom of a can, using a piece of spaghetti.

I /was/ going to try solve this by wrapping some tubing around the hollow bolt above the heating element, and run water through it, but the hot-end just stopped working on me completely, so I decided to pay for a better solution.

This solution was the E3D V6 (Lite), which has a proper heatsink, and a fan.

The E3D V6 took a few days to arrive, and when it did, there was a few hours assembly needed. The hardest part was figuring out how to connect the Bowden tube to the Makibox’s extruder. I managed this in the end by taking an M6 (I think. maybe it was M5?) nut and screwing it directly onto the end of the Bowden tube, then the new tube would dock into the extruder just like the original.

The next problem is one I’m still working on solving. The hot-end is positioned by moving two beams. The hot-end hangs from where the beams cross each other. The problem is that the beams are moved by long screws on /one end/ of the beam. The part that connects to the screw tries to keep the beams perpendicular to the screw, but it’s like trying to lift a plank of wood by lifting just one end – difficult.

The solution for this, I think, is to run some strings around a series of wheels that guide the strings such that when one end of the Y axis moves (for example), the other end is pulled by the string to keep the beams perpendicular to each other.

So, the first prints I’m doing are holders for the wheels. The prints are really terrible, as the printer is obviously not yet in perfect working order, but after I finish fixing this problem, I can print them again in better quality 🙂