Vet a trading bot
For developers with a trading bot. Your bot buys a stock when it rises above its 20-day high and sells when it falls below its 10-day low or 8% under the price it paid. It looks good on paper, and you're about to put real money behind it. First you want to know if it beats the simplest alternative, which is to buy every stock and hold them. If it doesn't, you'd be paying its trading costs and taking its risk for nothing. This script runs the bot and buy-and-hold on sixteen markets and counts which did better on each.
Don't take it live yet. The bot beat buying and holding on 6 of 16 markets, about what a coin toss would give, so this run finds no edge. On one market its trading cost 2,044, about an eighth of its profit there.
The bot against buy-and-hold, market by market
Profit of the bot less profit of buy-and-hold, each market starting from 1,000,000. Bars to the right are markets the bot won.
Bot less buy-and-hold
Sixteen markets instead of one
Flip a coin once and get heads, and you've learned nothing about the coin. A backtest on one price history is one flip. Whether the bot looks clever on it or not, you can't tell skill from luck. This script gives the bot sixteen flips, one per market, against buy-and-hold on the same market.
The question
Does my breakout bot do better than buying everything and holding it, before I put real money behind it?
The fair test
Sixteen different markets, with the bot and buy-and-hold on the same one each time. The test counts who won each market, so one lucky market cannot carry the result.
The result
The bot won 6 and lost the rest. The test puts the chance of a split at least that lopsided by luck alone at 0.45, so this run finds no edge. Its trading also cost about an eighth of its profit on one market.
tradefloor against a backtest
A backtest gives the bot one price history and has to guess what its trading cost. tradefloor builds as many markets as you ask for and runs the bot and buy-and-hold in each. It fills the bot's orders against an order book and measures the cost.
| Needed for this decision | Backtest on price history | tradefloor |
|---|---|---|
| More than one market to test on | One history | Sixteen markets here, and more by changing one line |
| A fair comparison with buy-and-hold | Same history, but one sample | Same market each time, counted with a paired sign test |
| What the bot's own orders cost | A fixed slippage guess | Measured against the same market without the bot's orders |
| A result someone else can check | Depends on their data | The same seed gives the same market on any machine |
The script
Install with pip install tradefloor. The script needs no API key and runs in under a minute. history_days=20 gives each market twenty days of price history before day 0, so the bot's first decision already has a 20-day high to compare with.
import tradefloor as tf
market = tf.Universe.random(10, seed=111)
SEEDS = range(16)
DAYS = 60
STOP = 0.08 # sell if a holding falls 8% below its entry price
class Breakout:
# buy above the 20-day high, sell below the 10-day low or at the stop
def __init__(self):
self.entry = {}
def act(self, obs):
orders = {}
for ticker in obs.tickers:
# checked at every step, since there are no stop orders to rest
price, held = obs.price(ticker), obs.position(ticker)
bars = obs.history.bars(ticker, last=20)
if held > 0 and (price < self.entry[ticker] * (1 - STOP)
or price < min(bar["low"] for bar in bars[-10:])):
orders[ticker] = -held
elif held == 0 and price > max(bar["high"] for bar in bars):
# one equal slot per name, the size buy-and-hold holds
slot = obs.portfolio.net_worth() / len(obs.tickers)
orders[ticker] = int(slot / price)
self.entry[ticker] = price
return orders
def entrants():
return {"breakout": Breakout(), "buy_and_hold": tf.baselines.BuyAndHold()}
# 20 days of warm-up, so the first decision already has a 20-day high
ranking = tf.rank(entrants, seeds=SEEDS, universe=market, days=DAYS,
history_days=20, workers=4)
bot, bench = ranking.records["breakout"], ranking.records["buy_and_hold"]
test = ranking.separation("breakout", "buy_and_hold")
# per seed, in currency: the bot's P&L less buy-and-hold's
excess = {f"seed {s}": round(x) for s, x in zip(ranking.seeds, bot.excess_pnls)}
# seed 0 again, priced against the same market with the bot's orders left out
cost = tf.tca.analyse(Breakout(), seed=0, universe=market, days=DAYS,
history_days=20)
paid = cost.shortfall()
seed0 = {"breakout before costs": round(bot.pnls[0] + paid),
"breakout after costs": round(bot.pnls[0]),
"buy_and_hold": round(bench.pnls[0])}
n = len(ranking.seeds)
print(f"{n} markets, {DAYS} days each, {len(market)} companies, 1,000,000 to start")
print(f"{'':14}{'median P&L':>12}{'mean vs B&H':>13}{'ahead':>8}")
print(f"{'breakout':14}{bot.median_pnl:>+12,.0f}{bot.mean_excess_pnl:>+13,.0f}"
f"{bot.seeds_ahead:>5}/{n}")
print(f"{'buy_and_hold':14}{bench.median_pnl:>+12,.0f}")
print(f"seed 0: traded {cost.as_dict()['notional']:,.0f} in {len(cost.fills)} fills, "
f"costs {paid:,.0f} ({cost.shortfall_bps():.1f} bps), P&L "
f"{seed0['breakout before costs']:+,} before costs, "
f"{seed0['breakout after costs']:+,} after")
if test["p_value"] < 0.05:
better = "beats" if test["wins_a"] > test["wins_b"] else "trails"
verdict = f"the bot {better} buy-and-hold on these markets"
else:
verdict = "no clear edge over buy-and-hold, so this is no case to go live"
print(f"Verdict: ahead on {test['wins_a']} of {test['paired_seeds']} markets, "
f"p = {test['p_value']:.2f}: {verdict}.")16 markets, 60 days each, 10 companies, 1,000,000 to start
median P&L mean vs B&H ahead
breakout +2,341 -1,181 6/16
buy_and_hold +8,580
seed 0: traded 2,197,199 in 22 fills, costs 2,044 (9.3 bps), P&L +16,981 before costs, +14,937 after
Verdict: ahead on 6 of 16 markets, p = 0.45: no clear edge over buy-and-hold, so this is no case to go live.What costs took on one market
Profit on market 1, before and after the bot's trading costs
Currency, 60 days, with buy-and-hold on the same market
Market 1 fell, so the bot's lead there comes from sitting in cash, not from its timing. The bot spends much of its time in cash and buy-and-hold never does, so the gap on each market mostly tracks whether that market rose or fell.
Adapting it
Replace the body of Breakout.act with your own rule. Raise SEEDS to detect a smaller edge, or DAYS for longer markets. The strategy comparison guide explains the sign test and what tf.rank reports.
Limits
- Ten made-up companies on the default preset, and 60 trading days per market. Longer runs or another roster may differ.
- Profit only, with no adjustment for risk. The bot carries far less exposure than buy-and-hold.
- The cost line is for one market, not an average over all sixteen.
- Equities and cash only, and no stop orders resting in the book. The stop is checked at each decision step, so a price can fall past it between checks.
- Sixteen markets can't detect a small edge. A real but small difference needs many more markets.
Reference
tf.rank, Ranking.separation and tf.tca.analyse are in Agents and evaluation.