Compare strategies
To compare agents fairly, run them on many markets. The signatures and the fields of each result are in the reference, Agents and evaluation.
Each code block on this page runs by itself. The agent they compare is the Quoter from Write an agent, copied into each block, on a universe of 40 generated companies.
One market and the baselines
tf.reference_agents() builds five baselines: buy-and-hold, a random trader, momentum, mean-reversion and the Oracle, which reads the model's fair value. tf.evaluate gives each agent a separate copy of the market, so no agent uses up liquidity another expected, and tf.versus_buy_and_hold gives each agent's P&L minus buy-and-hold's.
import tradefloor as tf
market = tf.Universe.random(40, seed=111)
class Quoter:
# bid a cent under the last price at each open, cancel what is left at the close
def act(self, obs):
if obs.is_first_step_of_day:
bid = round(obs.price("AAA") - 0.01, 2)
return {"AAA": tf.Limit(500, bid)}
if obs.is_last_step_of_day:
return {"AAA": tf.Cancel()}
return None
agents = tf.reference_agents(seed=3)
agents["quoter"] = Quoter()
scores = tf.evaluate(agents, seed=7, universe=market, days=10)
for name, gap in tf.versus_buy_and_hold(scores).items():
print(f"{name:15} {gap:+12,.0f}")random -42,092 momentum -53,386 mean_reversion -48,137 oracle -6,341 quoter +930
On pt-v22, compare with buy-and-hold. A shock there moves fair value for good. The Oracle's P&L then follows the market's month and isn't a ceiling, and tf.capture_ratio returns nothing and says why. To put several agents in one market, where one agent's trades move the prices another meets, use a World.
Across seeds with rank
tf.rank runs the agents on many seeds, each a different market on one universe. It calls the factory once per seed, which gives each seed new agents with no state carried over.
import tradefloor as tf
market = tf.Universe.random(40, seed=111)
class Quoter:
# bid a cent under the last price at each open, cancel what is left at the close
def act(self, obs):
if obs.is_first_step_of_day:
bid = round(obs.price("AAA") - 0.01, 2)
return {"AAA": tf.Limit(500, bid)}
if obs.is_last_step_of_day:
return {"AAA": tf.Cancel()}
return None
def entrants():
# new agents for every seed, so no state carries from one market to the next
agents = tf.reference_agents(seed=3)
agents["quoter"] = Quoter()
return agents
ranking = tf.rank(entrants, seeds=range(8), universe=market, days=10)
print(ranking.report())
print(ranking.separation("quoter", "buy_and_hold"))8 seeds on universe 9be68b9bc37e... under model pt-v22
quoter vs buy_and_hold +10,326 a seed ahead 6/8 wins 5/8
buy_and_hold the benchmark median pnl +5,998 wins 1/8
mean_reversion vs buy_and_hold -12,428 a seed ahead 2/8 wins 2/8
random vs buy_and_hold -19,128 a seed ahead 1/8 wins 0/8
momentum vs buy_and_hold -38,482 a seed ahead 1/8 wins 0/8
The Oracle 'oracle' ran and has no row, because it is not a ceiling on this model (below). Its P&L over buy_and_hold's was +5,052 a seed, ahead on 5 of 8 seeds.
No capture ratio on pt-v22. As on pt-v20 and pt-v21, market moves there mostly stick: each shock moves fair value for good, so even perfect knowledge of the model's fair value leaves little edge. The Oracle made money in 6 of 8 test markets and took buy-and-hold's sign in all 8, so a fraction of its P&L would measure the month, not the agent. Compare against buy-and-hold instead.
{'a': 'quoter', 'b': 'buy_and_hold', 'wins_a': 6, 'wins_b': 2, 'ties': 0, 'paired_seeds': 8, 'decisive': False, 'p_value': 0.2890625}Reading the result
Each row of the report is one agent against the benchmark, buy-and-hold by default, measured seed by seed on the same market. The figure before "a seed" is the mean of the agent's P&L minus buy-and-hold's, in currency. On pt-v22 it's the figure to quote. "ahead" counts the seeds on which the agent earned more than buy-and-hold, and "wins" the seeds on which it had the highest P&L of every ranked agent.
separation(a, b) is a paired sign test between two agents. The Quoter was ahead on 6 of 8 seeds and behind on 2. At 6 wins in 8 and p = 0.29, this run can't tell them apart, because a fair coin lands 6 to 2 or further from even that often. decisive is true only when one agent won on every seed, and it counts wins without testing significance. Report the paired count with any pooled P&L, and add seeds before you call a winner. ranking.records["quoter"] holds the per-seed figures behind the row, described in AgentRecord.
A row can also carry REFUSED, UNUSABLE or RAISED lines, which count the steps an agent's orders were refused, its act returned something other than a dict, or its code raised. Read those before the P&L. Ranking lists every member of the result.
Execution cost
tf.tca.analyse runs the same seed twice, with and without the agent's orders, and prices each fill against the market where it never traded. Real data can't supply that untraded market, and arrival price and VWAP stand in for it there.
import tradefloor as tf
market = tf.Universe.random(40, seed=111)
class RoundTrip:
# buy 2,000 AAA, sell them back three steps later
def act(self, obs):
if obs.step == 0:
return {"AAA": 2000}
if obs.step == 3:
return {"AAA": -2000}
return None
ex = tf.tca.analyse(RoundTrip(), seed=42, universe=market, days=1)
print(round(ex.shortfall_bps(), 2))11.25
A positive shortfall is a cost. The engine has no slippage formula. A large order walks the book and pays for the depth it takes, and cost grows with size along the square-root law that real metaorders follow (How it is measured, row C9). tca.analyse takes market orders only and refuses a tf.Limit, because the part of a limit order that waits fills inside a session, where the untraded run has no price to compare it with. tca.analyse and Execution have the arguments and the result.
Next steps
- Fork a market changes one input in a copy of a market and compares the two futures.
- Record and replay makes a comparison that involves a model reproducible.
- Read Skill or luck in a trading strategy, a guide that runs two strategies on 40 markets and shows how one market can crown either.
- Watch Skill or luck?, a 3:02 video recorded on the hosted app.