Skip to the page
EXAMPLES/VET A TRADING BOT

Vet a trading bot

For developers with a trading bot. Your bot buys a stock when it rises above its 20-day high and sells when it falls below its 10-day low or 8% under the price it paid. It looks good on paper, and you're about to put real money behind it. First you want to know if it beats the simplest alternative, which is to buy every stock and hold them. If it doesn't, you'd be paying its trading costs and taking its risk for nothing. This script runs the bot and buy-and-hold on sixteen markets and counts which did better on each.

Don't take it live yet. The bot beat buying and holding on 6 of 16 markets, about what a coin toss would give, so this run finds no edge. On one market its trading cost 2,044, about an eighth of its profit there.

6 of 16markets where the bot did better
10 of 16markets where buy-and-hold did better
2,044what the bot's trading cost on one market

Sixteen markets instead of one

Flip a coin once and get heads, and you've learned nothing about the coin. A backtest on one price history is one flip. Whether the bot looks clever on it or not, you can't tell skill from luck. This script gives the bot sixteen flips, one per market, against buy-and-hold on the same market.

The question

Does my breakout bot do better than buying everything and holding it, before I put real money behind it?

The fair test

Sixteen different markets, with the bot and buy-and-hold on the same one each time. The test counts who won each market, so one lucky market cannot carry the result.

The result

The bot won 6 and lost the rest. The test puts the chance of a split at least that lopsided by luck alone at 0.45, so this run finds no edge. Its trading also cost about an eighth of its profit on one market.

tradefloor against a backtest

A backtest gives the bot one price history and has to guess what its trading cost. tradefloor builds as many markets as you ask for and runs the bot and buy-and-hold in each. It fills the bot's orders against an order book and measures the cost.

Needed for this decisionBacktest on price historytradefloor
More than one market to test onOne historySixteen markets here, and more by changing one line
A fair comparison with buy-and-holdSame history, but one sampleSame market each time, counted with a paired sign test
What the bot's own orders costA fixed slippage guessMeasured against the same market without the bot's orders
A result someone else can checkDepends on their dataThe same seed gives the same market on any machine

The script

Install with pip install tradefloor. The script needs no API key and runs in under a minute. history_days=20 gives each market twenty days of price history before day 0, so the bot's first decision already has a 20-day high to compare with.

import tradefloor as tf

market = tf.Universe.random(10, seed=111)
SEEDS = range(16)
DAYS = 60
STOP = 0.08  # sell if a holding falls 8% below its entry price


class Breakout:
    # buy above the 20-day high, sell below the 10-day low or at the stop
    def __init__(self):
        self.entry = {}

    def act(self, obs):
        orders = {}
        for ticker in obs.tickers:
            # checked at every step, since there are no stop orders to rest
            price, held = obs.price(ticker), obs.position(ticker)
            bars = obs.history.bars(ticker, last=20)
            if held > 0 and (price < self.entry[ticker] * (1 - STOP)
                             or price < min(bar["low"] for bar in bars[-10:])):
                orders[ticker] = -held
            elif held == 0 and price > max(bar["high"] for bar in bars):
                # one equal slot per name, the size buy-and-hold holds
                slot = obs.portfolio.net_worth() / len(obs.tickers)
                orders[ticker] = int(slot / price)
                self.entry[ticker] = price
        return orders


def entrants():
    return {"breakout": Breakout(), "buy_and_hold": tf.baselines.BuyAndHold()}


# 20 days of warm-up, so the first decision already has a 20-day high
ranking = tf.rank(entrants, seeds=SEEDS, universe=market, days=DAYS,
                  history_days=20, workers=4)
bot, bench = ranking.records["breakout"], ranking.records["buy_and_hold"]
test = ranking.separation("breakout", "buy_and_hold")

# per seed, in currency: the bot's P&L less buy-and-hold's
excess = {f"seed {s}": round(x) for s, x in zip(ranking.seeds, bot.excess_pnls)}

# seed 0 again, priced against the same market with the bot's orders left out
cost = tf.tca.analyse(Breakout(), seed=0, universe=market, days=DAYS,
                      history_days=20)
paid = cost.shortfall()
seed0 = {"breakout before costs": round(bot.pnls[0] + paid),
         "breakout after costs": round(bot.pnls[0]),
         "buy_and_hold": round(bench.pnls[0])}

n = len(ranking.seeds)
print(f"{n} markets, {DAYS} days each, {len(market)} companies, 1,000,000 to start")
print(f"{'':14}{'median P&L':>12}{'mean vs B&H':>13}{'ahead':>8}")
print(f"{'breakout':14}{bot.median_pnl:>+12,.0f}{bot.mean_excess_pnl:>+13,.0f}"
      f"{bot.seeds_ahead:>5}/{n}")
print(f"{'buy_and_hold':14}{bench.median_pnl:>+12,.0f}")
print(f"seed 0: traded {cost.as_dict()['notional']:,.0f} in {len(cost.fills)} fills, "
      f"costs {paid:,.0f} ({cost.shortfall_bps():.1f} bps), P&L "
      f"{seed0['breakout before costs']:+,} before costs, "
      f"{seed0['breakout after costs']:+,} after")

if test["p_value"] < 0.05:
    better = "beats" if test["wins_a"] > test["wins_b"] else "trails"
    verdict = f"the bot {better} buy-and-hold on these markets"
else:
    verdict = "no clear edge over buy-and-hold, so this is no case to go live"
print(f"Verdict: ahead on {test['wins_a']} of {test['paired_seeds']} markets, "
      f"p = {test['p_value']:.2f}: {verdict}.")
16 markets, 60 days each, 10 companies, 1,000,000 to start
                median P&L  mean vs B&H   ahead
breakout            +2,341       -1,181    6/16
buy_and_hold        +8,580
seed 0: traded 2,197,199 in 22 fills, costs 2,044 (9.3 bps), P&L +16,981 before costs, +14,937 after
Verdict: ahead on 6 of 16 markets, p = 0.45: no clear edge over buy-and-hold, so this is no case to go live.

What costs took on one market

Market 1 fell, so the bot's lead there comes from sitting in cash, not from its timing. The bot spends much of its time in cash and buy-and-hold never does, so the gap on each market mostly tracks whether that market rose or fell.

Adapting it

Replace the body of Breakout.act with your own rule. Raise SEEDS to detect a smaller edge, or DAYS for longer markets. The strategy comparison guide explains the sign test and what tf.rank reports.

Limits

  • Ten made-up companies on the default preset, and 60 trading days per market. Longer runs or another roster may differ.
  • Profit only, with no adjustment for risk. The bot carries far less exposure than buy-and-hold.
  • The cost line is for one market, not an average over all sixteen.
  • Equities and cash only, and no stop orders resting in the book. The stop is checked at each decision step, so a price can fall past it between checks.
  • Sixteen markets can't detect a small edge. A real but small difference needs many more markets.

Reference

tf.rank, Ranking.separation and tf.tca.analyse are in Agents and evaluation.