14 Feb

rewriting the trading algorithm

I have the script now running at a realistic 1.14% per day return, taking into account the average number of times that I’ve had to pay a trading fee, the average offset of what price a trade happens vs what price the algorithm wanted it to happen, and other unpredictable things.

The script as it is, is very hard to train in any way other than almost a brute force fashion. Because most of the variables (length of EMA long tail, number of bars to measure for ATR, etc) are integers, I can’t hone into the right values using a momentum-based trainer, so I need to do it using long, laborious loops.

There is one improvement that I could do, which would probably double or more the return, without drastically changing the script. At the moment, there is a lot of “down time”, where the script has some cash sitting in the wallet, and is waiting for a good opportunity to jump on a deal. If the script were to consider other currencies at the same time, then it would be able to use that down time to buy in those currencies.

On second thought, I think the returns would be much higher, because when the script currently makes a BUY trade, its usually sold again within about 30 minutes. That means that it could potentially make 48 BUY trades in a day, each with an average of maybe a .5% return. That’s about a 27% return ((1+(.5/100))48 = 1.27).

That would be nice, but it’s impossible with the GDAX market, because the GDAX market only caters to four cryptocurrencies at the moment. Also, while the money aspect is nice, I’m actually doing this more for the puzzle-solving aspect. I get a real thrill out of adding in the real-life toils and troubles (fees, unpredictable trades, etc) and coming up with solutions that return interest despite those. So, I’m not going to refine the hell out of the script. Instead, I’m going to start working on another version.

Before I started on the current version, I had made a naive attempt at market prediction using neural networks. While some of the results looked very realistic, none of them stood up to scrutiny. I failed. But, I also learned a lot.

I’m going to make another neural-net-based attempt, using a different approach.

The idea I have is that instead of trying to predict the upcoming market values, I’ll simply try to predict whether I should buy or sell. This reduces the complexity a lot, because instead of trying to predict an arbitrary number, I’m only outputting a 1 (buy), 0 (nothing), or -1 (sell). This can then be checked by running the market from day 1 of the test data using each iteration of the network, and then adjusting the weights of the network based on whether the end result was higher or lower than the last time it was run.

I noticed that the best tunings from my current script only made a very few trades, on the very few days where the market basically dropped like a stone and then rebounded. But I can’t assume those will happen very often, so I like to see a few trades happen every day. With the neural network, I can increase the number of trades by simply adjusting the output function that decides whether a result is -1, 0, or 1 (this is usually done by converting the inputs into a value between -1 and 1, then rounding to an integer based on whether the value is above/below 0.7/-0.7. That value can be adjusted easily to provide more 1/-1 hits).

The current approach also involves a lot of math to measure ATR (average true range), running EMA (exponential moving average), etc. With the neural network approach, I will just need to squish the numbers down to form a pattern between -1 and 1 (where 0 is the current market price) and run directly against those numbers.

Because neural networks are a lot more “touchy-feely” than the current EMA/stop-gain/stop-loss approach I’m using, I will be able to “hone in” on good values – something I cannot do at the moment because of the step-like nature of integers.

I won’t be using any off-the-shell networks, as I can’t imagine how to write a good trainer for them. Instead, I’ll write my own, using a cascade-correlation approach.

Cascade-correlation is an approach to neural networks which allows a network to “grow” and gradually learn features of the input data. You train a network layer against every single input, until the output stops improving. You then “freeze” that network layer so is not adjusted anymore. You then create a second layer which trains against every single input and the previously trained network. You continue adding layers until there is no more noticeable improvement.

The point of this is that the first training layer will notice a feature in the data that produces a good result, and will train itself to recognise that feature very well. The second layer will then be able to ignore that feature (because it’s already being checked for), and find another one that improves the results. It’s like how you decide what animal is in a picture – does it have 6 legs (level one), is it red (level two), does it have a stinger (level three) – it’s a scorpion! Instead of trying to learn all the features of a successful sale at once, the algorithm picks up a new one at each level.

Around Christmas, I was playing with the FANN version of cascade correlation, and I think it’s very limited. It appears to create its new levels based on all inputs, but only the last feature-detection layer. Using the above example, this would make it difficult to recognise a black scorpion, as it would not be red. I believe that ideally, each feature layer should be treated as a separate new input, letting the end output make decisions based on multiple parallel features, not a single linear feature decision.

29 Jan

using LIMIT trades to reduce fees

I made two enhancements to my Litecoin market script over the weekend.

The first one was based on something I did last week, where it occured to me that if you buy when the ATR is low, you will probably lose money through fees because the market value will not shift enough for there to be a profit.

For example, if the value is 200 when you buy and 201 when you sell, then you will lose money. Let’s say you you €10 of coins. It will increase in value to €10.05 by the time you sell them, but you will pay just over €0.06 in fees.

An ATR limit large enough to discourage the script from buying when the margin is so small would stop this issue.

However, I realised that a limit that worked at around a value of 200 would not be effective at 400.

The solution was to make a new limit which is inversely proportional to the market value. Let’s say the number is 50. Then it would look for an ATR of 0.25 if the market value was 200, and an ATR of 0.125 if the value was 400.

This made a remarkable difference in my simulation. It now estimates a 3.15% return per day based on the configuration figures I found.

Last week’s version ended up with about 14 buys in a 50 day period, which meant that there was only about one buy every 4 days, making it look like it wasn’t doing anything.

Now, it has what looks like about 32 events per day. A lot of them are repeats, where a sell signal might pop up from a chandelier exit followed by another from exponential moving average or simple moving average, but it’s still a lot more, making the script feel a lot more alive.

This is helped by me changing the trade method.

I had it on MARKET trades, which are virtually guaranteed sales/buys, but also are guaranteed 0.3% fees.

I’ve changed that to LIMIT trades that work in a way that might not trade at exactly what was requested, but should not trigger a fee at all (at least, I haven’t had a single fee yet in my tests!).

How it works: let’s say the market value is 200 right now. The script will check the order books, which might currently have an “ask” of 200.01 and a “bid” of 199.99 (for example). If we are trying to sell, then we add an sell/ask order of 200.02 (current+0.01). If the market value goes down, then we cancel that order and create a new one based on whatever the new value is.

And vice versa with buys/bids.

This means we probably won’t get exactly what we want, because we are relying on market jitter to make the trade happen. But at least we won’t have a fee to worry about!

21 Jan

automated script for the Litecoin market

Over Christmas, I started looking into how stock markets work and decided to give it a shot. The simplest way I found to start off on it was actually the cryptocoin market. I chose Coinbase’s GDAX market as the one I’d work on.

At first, I had a naive idea that I just needed to watch the numbers on the market for times when the latest figure is the lowest in a long time. Then you buy. And then the opposite happens -if the latest bar is the highest in a long time, sell.

It turns out that doesn’t work. I wrote a testing application that downloaded 6 months of per-minute data about the LTC-EUR market and ran simulations against it to figure out what would happen if I was to trade based on my algorithms. The first one (above) sucked.

So I started looking a bit further into how traders actually do it themselves.

It turns out it’s pretty simple, if you’re willing to put the testing time in and come up with some good configuration numbers.

The first thing I checked out was called “MACD” (Moving Average Convergence Divergence). That uses a simple moving average (SMA) of the market value to generate two lines – a “long” average based on 26 figures, and a “short” average based on 12 figures. When the short average crosses over the long, it signals an action. For example, if the current short value is higher than the current long, and the last calculations were the opposite (short under long), then that indicates you should Buy, because it looks like there is an upwards trend. The opposite happens when the crossover shows the short going under the long. Sell.

The 12 and 26 figures are traditional. You could work based on them, but my tests showed that there are different figures that can give you better results. I guess it depends on the market. My current settings here are 25/43 instead of 12/26.

The next thing I worked on was a “Chandelier Exit”. This is a strategy for cutting your losses when the market suddenly drops more than usual. To do this, you measure the “ATR” (average true range) for the last n periods (traditionally 22). You then multiply the ATR by a volatility value (traditionally 3), subtract that from the current High value, and if the current market value is below that, Sell. My current values for this are a volatility of 5.59 based on an ATR of 18 bars.

I then looked at exponential moving average based MACD. The standard moving average is a straightforward average of n numbers, but the EMA puts more weight on the more recent numbers, so it reacts quicker to changes in the market.

After trying to tune the EMA for a while, I found that if I use EMA instead of SMA, then I get worse results, because the script would buy quickly when it saw an upward trend, but that might turn out to be just jitter and you’ll lose it all immediately afterwards. It’s safe to sell when the market drops, it’s not safe to buy when the market looks like it’s just starting to rise. it’s better to take your time.

So, I added a switch to my code so that I could decide whether to use SMA or EMA for buys and sells, etc.

I found that the combination that gives the best results uses only SMA for buys, but then uses all of SMA, EMA and Chandelier exits to signal a sell. Oh – EMA of 40 and 80.

Doing this, I’ve been able to come up with a configuration of my script that gives an average return of about 1.1%. This means that if you were to invest €5000, then there would be “interest” of about €55 per day. Or if you can keep your money in the game, it starts to grow. €50 invested for 365 days at 1.1% interest per day is €2711.

If you’re interested in going the script a shot, you can download it from here.

I keep on having more ideas on how to improve this.

12 May

split testing

I wrote a split-testing plugin (also known as “A/B testing”) for my CMS early last year. It wasn’t very good, so I wrote an external split-testing tool instead, which you can you for your own websites if you want.

Split-testing application

It works similarly to Google’s “web optimiser”, but I think it’s easier to use.

As a test, I chose to optimise the front page of KV Sites, to see if I could encourage people to contact me or read other pages on the website.

In particular, I wanted to see if it was better to explain what I do, or to assume that the reader knows and just wants to get on with.

As an example, consider the following two extracts:

Indirect speach – explaining what I do
Web Development
Reduce costs by automating and networking your business.
Proven record, with a number of large projects completed. Example projects include Duffy Transport's management application, the crop prediction software used by Cropworks, ongoing development of the WebME CMS, and technical support for a number of projects by other web development companies.

vs.:

Direct speech – asking what the reader wants
I am looking for Web Development
If you are looking for an online solution which to reduce your office expenses, we have experience writing solutions which can cut hours of work per day off your load. Read more about our web application development here.

It seems obvious that the direct speech version would work better, but when your job is involved, it’s better to be certain than to work based on assumptions.

So, I set up a split test. On the homepage of KV Sites, I sent the HTML for both versions to the browser, and set the different versions to display randomly using CSS.

Each version includes a “more info” link (of sorts), so I added the “conversion” code (which records when people go to the page you want them to) to the pages they were linked to, and started recording.

It took a few weeks for the information to come in, as my site does not have heavy traffic, and I didn’t want to artificially boost it in any way, but here’s how it turned out:

NameVisitorsConversions
front page direct marketing237v2: 12
v1: 24

It’s obvious from the above that “v1” (the direct speech one) is twice as effective as “v2” (indirect speech).

I’m very encouraged by these results. They mean that it is possible to both a) improve “conversions” based on text changes, and b) provide numbers proving those results.

23 Jun

growing up

For a long time, whenever I did something that I thought was interesting, I would write it in an article.

I don’t write as much as I used to. Not because what I’m doing is not interesting, but because it’s taking a lot longer now to complete the interesting jobs, now that they’re basically full projects and not just little snippets here and there.

As an example, we’re doing some interesting work over in the KV WebME project. The most recent is an upgrade to image galleries allowing the gallery layouts to be defined using templates, instead of saying “you want layout 1, or layout 2?”. This work is then also used in the products plugin to let people sell images in their online stores.

There are a number of bits in that project that deserve full posts themselves, but as I basically commissioned the piece and got others to do the work, it’s no longer mine to describe. For me, the cool little tricks are now just a smaller part of a bigger picture.

The bigger picture right now is 20eu.com, where you can create your own online store within literally minutes for only €20 (compare that to the “free” getting business online project, which doesn’t have an e-commerce aspect).

It’s now harder for me to write about, because there are no longer single cool aspects that I can point to that can be re-used by other people.

I was walking to work today with Conor (an employee), discussing stuff along these lines, and these points stood out:

  • In the beginning, I was a programmer, and every task had something new to me but nothing I could write about that would interest experienced programmers.
  • After a few years, I was a good programmer, and I did less tasks, but they were larger, and there were often aspects to them that were brand new, so I ended up being one of the first to build them (it helps that my field is Ajax, which is basically new-born).
  • And now, I’m tired of being just a programmer and have started branching into managing other programmers. I’m more interested now in getting full projects done than in the nitty-gritty.

Unfortunately, this means there is less to talk about that is even vaguely techie. I feel like I’m shifting focus into marketing and project management.

Ick!

On the plus side (for me), it means that eventually, I’ll have enough resources that I can get the projects done that I’ve always wanted to do.

So, I plan on starting to write about the business end of my work.

Don’t worry – I’ll categorise it correctly, so if you’re only interested in my PHP or Linux posts, then just change your reader settings to only read from those RSS feeds.