Gekko CLI users – it’s the TOML to JSON translator you’ve all be waiting for

As the Gekko community has grown more and more strategies are appearing which is super awesome. What can be a little frustrating is that most of these strategies provide TOML files (for using the Gekko UI). This seems like a small thing to translate right?

Well yes, if you’re downloading 1 strategy file, but what if you’re downloading one of the big strategy strategy repos such as  Gekko Strategies. There are at least 40 strategies in there and an installer (made by the owner of the original repo xFFFFF), and who wants to trawl through 40+ TOML files and rewrite them into JSON and add them to their config. Enter the TOML to JSON translator you’ve all been waiting for!

So what does this TOML to JSON config converter do?

It takes all the TOML files, converts them all into JSON that Gekko needs, then appends them to your Gekko config file meaning all you then need to do is change the config.tradingAdvisor.method to the new Strategy of choice and away you go, it will run in immediately without any hassle.

Added to the Bruteforce repo, the Gekko TOML to JSON converter script is pretty straight forward to use. Grab the Bruteforce repo and install it.

Before doing anything open the the file “toml-config-converter.js” and go to line 22 which looks like:

var strategiesFolder = ‘../gekko/config/strategies/’;
var configFile = ‘../gekko/config-new.js’;
var outputConfigFile = ‘../gekko/config-ready-to-use.js’;

This is the important part.

strategiesFolder is where all your TOML files are, this doesn’t have to be your Gekko/config/strategies file, it could be someones installed repo.

configFile is your default config file, typically it is from the vanilla Gekko install with your database and other base settings updated.

outputConfigFile is the new config that is generated by the TOML to JSON converter.

Note: do not make configFile and outputConfigFile the same file. I have no idea what will happen but I imagine nobody will hear your screams.

I should add that any TOML references to strategies with hyphens in (“-“) will be discarded and not copied across.

 

 

 

Gekko Nodejs Bruteforce backtester app

Do you want to run backtests against multiple candle sizes, coins, history sizes and store the data for every single run?

If the answer is yet then read on…

After getting a little bored of running backtests manually I wrote a new bruteforce backtester application for Gekko in Nodejs.  It’s pretty straight forward to setup, and writes everything you backtest into a nice and neat csv file that allows you to sort to find the most profitable settings. This is what it can do:

  1. Set multiple history periods
  2. Set multiple candle sizes
  3. Set multiple strategies
  4. Set multiple exchange and trading pairs
  5. Set random ranges for each strategy config
  6. Writes all outputs and strategy configs out into a csv so you can review what strat and related settings are going to make you the most cold hard crypto

I’ve put it on the Gekkowarez github and you can find all the install instructions there.

I guess there is an obvious question to answer; why use this instead of the Gekko Genetic Algorithm ?

Well it’s a fair question. I use this after having just built or come across a new trading strategy and I don’t know basics like;

What’s the right candle size?

What’s the right history setting?

What type of trading pair is this best for – high volume, usd or btc currency?

I can do a certain amount of optimization with the brute force approach as well. Ultimately, in about 10 minutes I’ve got a pretty clear picture of whether a trading strategy will work – and that’s a big win for me!

Of course, I will then turn to the GA when it’s time to get really optimal!

Example csv output can be found here:

Gekko Bruteforce backtester example output

 

How to manage thresholds when backtesting against many coin pairs

The problem – it’s just so annoying….

Something that always bugs me is when I have to track different thresholds (the values that determine when indicators / events should trigger a buy or sell) when testing against multiple coins. I would end up with endless commented lines such as:

//if(rsi>80){ //usdtbtc

if(rsi>70){ //btcsia

Now typically you should be putting these thresholds into your config rather than hard coding them. For example:

config.Pumprider = {
parameters:{
historySize: 10,
optInFastPeriod:10,
optInSlowPeriod:21,
optInSignalPeriod:9,
optInPenetration:2,
StopLoss:{
percentage:5,
},
},
thresholds:{
accumulationpersistence:3,
distributionpersistence:3,
uptrendpersistence:3,
downtrendpersistence:3,
}
}

So great, now I have to comment out lines in the config now, or have multiple configs. Whichever way you work, this is a really annoying and messy problem.

What’s the solution?

I think it makes a lot of sense to stick to using the thresholds in the config so I am using this:

config.zuki_linreg_slope = {

thresholds:{

btcethlong: 0.0002,
btcethshort: 0.0002,
btcdgblong: 0.00000001,
btcdgbshort: 0.00000001

}

}

Then in my strategy I include this in the header:

var long = 0;
var short = 0;
var exchange = config.watch.exchange;
var currency = config.watch.currency;
var asset = config.watch.asset;

and then this in the init:

let pair = currency.toLowerCase()+asset.toLowerCase();
let thresholdkeylong = pair+”long”;
let thresholdkeyshort = pair+”short”;
long = this.settings.thresholds[thresholdkeylong];
short = this.settings.thresholds[thresholdkeyshort];

Then we use long and short in our buy/sell logic:

if (hasbought===false && ema < long ){ }else if (hasbought=== true && ema> short){ //DGB

}

What’s actually happening?

Essentially we set a currency/asset based set up thresholds, identify the currency/asset pair being used in a backtest using the config.watch object, then used this to grab the right threshold values. In this case we assign the values to Long and Short, this is good for a simple example, but of course you could extend this to anything that is coin pair specific.

As a super cool side benefit this made testing with this gekko multi batch backtesting tool a dream.

How to prevent a gekko strategy from opening with a sell

Typically you’re going to start a strategy against a trading account that holds one side of the trading pair you are trading against. This means that if you are going to trade ETH/BTC you will start with ETH (the asset) in your trading account and zero Bitcoin (the currency).

Because Gekko doesn’t link your trading account to the trading strategy you can run into problems in 3 scenarios:

  1. When you first start your strategy Gekko may try and sell the currency first, which is not going to work given you have started with zero Bitcoin (the currency).
  2. When Gekko crashes (yes this happens!) or something else stops your Gekko instance while your strategy holds ETH (the asset).
  3. When Gekko crashes (yes this happens!) or something else stops your Gekko instance while your strategy holds BTC (the currency).

What is the solution?

Ensure your strategy knows which trade type to make first. This is actually really straight forward.

  1. Set a “firsttimetradetype” variable as “buy” (if you hold asset) in the Strategy config.
  2. Set a “firsttimetrade” variable as boolean false.
  3. Then inside the Strategy buy / sell logic check which is the type to trade first, and the “firsttimetrade” variable being false. If so then perform the trade and set the “firsttimetrade” to true.

In the event that Gekko crashes and you are left with currency and zero asset then you can switch the “firsttimetradetype” variable to sell.

In conclusion

The bottom line here is this initially seems like a simple problem to solve, but because Gekko separates the Trading (and trading account) from the Strategy the reality is there not a hands free solution.

The separation of trading, trading account and strategy is a fundamental weakness of the trading bot and it will be interesting to see how people try and solve it moving forward.

Live trading – segmentation fault – fixed use Node 7.1.0

I’ve been using Node 7.6.0 for running Gekko since, well, for as long as I can remember. I did try 7.8.0 once and that did not end well for me. Anyway, to cut a long story short I have recently moved to Postgresql from SQLite (largely thanks to Gen and his crazy linux skills). Tonight I tried to push a gekko live for the first time with a Postgresql backend and bang: Segmentation Fault.

Now, it’s not clear to me why, but after 3 hours I thought why not try another Node version. So using NVM (nvm install 7.1.0) I quickly tested and was stunned to see it worked. I’ll be honest, I don’t know the root cause. But it works and to my knowledge there are no issues running Gekko running on 7.1.0 but maybe I’ll bump into something shortly.

Long Gekko backtest causing econnect timeout solution

When I run backtests over 3-5 months that use multiple talib (yuck!) indicators I often get an econnect error. This was a major pain to fix, but seems to be to do with the socket timeout. Below is the fix:

Edit server.js and add the line in bold below the websocket server reference:

const wss = new WebSocketServer({ server: server });
wss._server.timeout = 50000000;

18 month speed back test

Not using Talib was a revelation to me, I was running back tests for hours and it was driving me mad. After working with the GA and the time back tests were taking I just gave up and instead started trying to work on native Gekko indicators.

The following back test took over 30 minutes to run with talib.

How to put a Stop Loss into your Gekko Strategy

This gets asked for a lot and is actually really easy to do, even for a noob coder like me.

In plain english the idea is to calculate a stop loss at the point of a buy and then trigger is alongside your sell logic.

In code terms it looks something like this:


//In the strategy init create the stoploss variable:
this.stop = "";


if(logic that determines a sell || this.stop != "" && price<this.stop){
       
//if you want to show a stoploss being triggered in the console:
        if(this.stop != "" && price<this.stop){
            console.log("stoplosss triggered - "+ this.stop);
        }

        this.direction="short";
        if(this.trend == "long"){
            this.stop = "";
            this.trend = this.direction;
            this.advice(this.direction);
        }
   
    }else if(logic that determines a buy){
        if(this.stop==""){
//sets up the stoploss, you should make the .2 a variable in the strategy config really
            this.stop = price-(price*.2);
            this.direction="long";
            this.trend = this.direction;
            this.advice(this.direction);
        }
    }

Because at the time of writing Gekko is an all or nothing bot, ie it trades 100% of current/asset per instruction, there is likely no need to expand upon this stop loss. However, once you can control the amount per trade you can start building in more complex stop loss rules based upon confidence levels in your strategies.

Avoid memory issues when running long backtests

Node has a memory limit of 1.5gig. In version 0.5.0 something caused long backtests (e.g. 16 month 1m candles) to fail with memory exhaustion.

Initially not using talib indicators seemed to be the fix. However, I’ve recently encountered the issue again when only using native indicators which is very annoying when running a GA over night…

Anyway, I am a noob, all I needed was to expand the memory. You do this by adding the flag listed below to the node command. Now, it seems all is good.

node –max-old-space-size=8192 server.js

 

Talib and windows

Talib is a nodejs library that Gekko can use to run more complex technical indicators that are not natively included in Gekko. Things like Bollinger Bands etc. But there are a couple of things to consider:

  1. Talib is just horrifically slow, so much so it basically renders using the Genetic Algorithm backtester pointless. I should note that the speed does not impact live trading, just backtesting.
  2. Talib is horrible to get working with Windows. Some people have achieved it and there is a useful thread here, but after the 3rd day I gave up. I recommend that you use Bash on Windows and install Nodejs and Gekko on that. It’s absolutely brilliant, and one of the best things Microsoft have ever created.

Here is a guide to installing Nodejs on Windows Bash.