A/B test an AI agent
For teams building AI trading agents. Your agent is Claude Haiku 4.5 with a short prompt telling it to grow the portfolio and answer each day in JSON. Version 2 adds one sentence: hold less of volatile stocks, and halve every position when the VIX is above 30. That caution might cost your users money in normal times, and it might protect them in the next crash. This script asks Claude for every decision under both prompts on the same ordinary and crisis markets, so you can decide whether to ship the change.
Don't ship version 2 yet. In one market that was copied so both versions met the same crisis, version 2 lost more than version 1: 274,759 against 169,739. Across twelve crisis markets, version 2 did better on 8 of 12, a split a coin toss often gives, and in ordinary markets on 6 of 12. Run it on more markets before you decide.
The same market, calm and in a crisis
Net worth from 1,000,000. On day 5 the market is copied and one copy goes into a crisis. Each version trades both copies.
Version 2 in the crisis675,040Version 1 in the crisis756,380Version 2 calm949,799Version 1 calm926,119
Both versions on the same markets
Car makers crash-test two designs into the same wall at the same speed. If each car hit a different wall, a better result could just mean a softer wall. Testing an agent on one stretch of history has the same weakness, and that stretch may not include a crisis at all. Here both versions trade the same twenty-four markets, and half of them have a crisis.
The question
Does the extra sentence in version 2 cost money in normal times, and does it save money in a crisis?
The fair test
Claude trades the same twelve ordinary markets and the same twelve crisis markets under each prompt. Then one market is copied on day 5, and the crisis lands on the same day in the copy each version trades.
The result
In the one shared crisis version 2 lost more. Over twelve crisis markets it did better on 8. Both could easily be luck, so the change isn't ready to ship.
tradefloor against a backtest
A backtest gives both versions one sample, the one history it replays, and the crisis you care about may not be in it. tradefloor starts the crisis on the day you choose and runs both versions through it on many markets. It can also copy one market so both versions meet the crisis at the same moment.
| Needed for this decision | Backtest on price history | tradefloor |
|---|---|---|
| A crisis to test against | Only if the dates happen to include one | Any shipped scenario, started on the day you choose |
| Both versions under the same conditions | Same history, but one sample | Twenty-four paired markets and a sign test |
| The crisis landing at the same moment for both | No | Yes. A fork copies the market, and the crisis lands on both copies. |
| Model calls that cost nothing to repeat | Every rerun calls the model again | Record the answers once and replay them for free |
Claude's answers, recorded once
The first run asked Claude Haiku 4.5 for all 790 decisions and saved the answers in two recordings, one per version. That took about fifteen minutes. Later runs find the recordings beside the script and replay them with no API key and no cost, and the numbers come out identical each time. Each answer is stored with the exact market data Claude was shown, and if you change the experiment the run stops with ReplayMiss and doesn't reuse old answers.
The recordings show version 2 following its rule. On days the VIX was above 30 it answered with lines such as "VIX at 50 exceeds 30 threshold, triggering mandatory position halving across all holdings". Whether the rule made money is a separate question, and the twelve crisis markets answer it.
To replay, download example-agent-team.py, example-agent-team-v1.json and example-agent-team-v2.json into one folder, pip install tradefloor and run python example-agent-team.py. To record your own, delete the two recordings, pip install anthropic, set ANTHROPIC_API_KEY and run it again. Claude answers a little differently each time and a fresh recording gives different numbers, so check that the decision still holds.
The script
import json
import pathlib
import tradefloor as tf
from tradefloor.integrations.callable import callable_agent
from tradefloor.integrations.common import AdapterInfo, Transcript, digest
MODEL = "claude-haiku-4-5-20251001"
ASK = ("You manage a stock portfolio in a simulated market. Each trading day you get "
"the market as JSON. Grow the portfolio. Reply with only a JSON object: "
'{"actions": [{"symbol": "AAA", "side": "BUY", "quantity": 100}], '
'"rationale": "one sentence"}. Quantities are shares, and an empty actions '
"list keeps what you hold.")
PROMPTS = {"v1": ASK,
"v2": ASK + " Risk rule: hold less of volatile stocks, and halve every "
"position when the VIX is above 30."}
market = tf.Universe.random(8, seed=11)
crisis = tf.Scenario.load("liquidity_crisis")
# Claude is called once and every answer recorded. With the recordings beside
# this file, every later run replays them: no API key, the same numbers.
here = pathlib.Path(__file__).parent
recordings = {v: here / f"example-agent-team-{v}.json" for v in PROMPTS}
live = not all(path.exists() for path in recordings.values())
tapes = {v: Transcript() if live else Transcript.load(path) for v, path in recordings.items()}
def ask_claude(prompt):
import anthropic # needed only to record
client = anthropic.Anthropic()
def ask(payload):
reply = client.messages.create(
model=MODEL, max_tokens=600, system=prompt,
messages=[{"role": "user", "content": json.dumps(payload)}])
return reply.content[0].text
return ask
def to_decision(raw, payload):
# runs on every replay too; a reply that is not JSON holds
text = raw.strip().removeprefix("```json").removeprefix("```").removesuffix("```")
try:
return json.loads(text)
except ValueError:
return {"actions": [], "rationale": "unreadable reply"}
def agent(version):
info = AdapterInfo(framework="callable", provider="anthropic", model=MODEL,
instructions_digest=digest(PROMPTS[version]))
if live:
return callable_agent(ask_claude(PROMPTS[version]), postprocess=to_decision,
info=info, mode="live", recorder=tapes[version])
return callable_agent(postprocess=to_decision, info=info, mode="replay",
transcript=tapes[version])
def both():
return {"v1": agent("v1"), "v2": agent("v2")}
# Paired A/B tests: both versions meet the same markets, ordinary and in a crisis
tests = {
"ordinary, 15 days": tf.rank(both, seeds=range(12), universe=market, days=15,
benchmark="v1", workers=8),
# other seeds, so no question asked in the ordinary test is asked again here
"crisis, 15 days": tf.rank(both, seeds=range(12, 24), universe=market, days=15,
benchmark="v1", scenario=crisis.starting_at(5)),
}
pnl_gap = {kind: [round(g) for g in r.records["v2"].excess_pnls] for kind, r in tests.items()}
# One shared past, then the same crisis for both versions from day 5
net_worth = {}
for name in ("v1", "v2"):
world = tf.World(seed=24, universe=market, agent=agent(name)) # a market of its own
path = []
for day in range(5):
world.run(1)
path.append(round(world.net_worth()))
calm, stressed = world.fork("calm", "crisis")
stressed.apply(crisis, at=0)
net_worth[f"{name} calm"], net_worth[f"{name} crisis"] = list(path), list(path)
for day in range(15):
for arm, label in ((calm, "calm"), (stressed, "crisis")):
arm.run(1)
net_worth[f"{name} {label}"].append(round(arm.net_worth()))
if live:
for version, path in recordings.items():
tapes[version].save(path)
answers = sum(len(tape) for tape in tapes.values())
print(f"{MODEL}, {answers} answers {'recorded' if live else 'replayed'}")
print()
print("Paired test, v2 against v1 on the same markets")
print(f"{'markets':18}{'count':>6}{'v2 ahead':>10}{'p':>7}{'mean v2 - v1':>14}")
for kind, r in tests.items():
s = r.separation("v2", "v1")
print(f"{kind:18}{s['paired_seeds']:>6}{s['wins_a']:>10}{s['p_value']:>7.2f}"
f"{r.records['v2'].mean_excess_pnl:>+14,.0f}")
print()
print("One market forked on day 5, then 15 days calm or in crisis")
print(f"{'version':8}{'calm end':>12}{'crisis end':>12}{'crisis cost':>13}")
crisis_loss, ends = {}, {}
for name in ("v1", "v2"):
calm_end, crisis_end = net_worth[f"{name} calm"][-1], net_worth[f"{name} crisis"][-1]
crisis_loss[name] = calm_end - crisis_end
ends[name] = {"calm": calm_end, "crisis": crisis_end}
print(f"{name:8}{calm_end:>12,}{crisis_end:>12,}{-crisis_loss[name]:>+13,}")
ordinary, stress = (r.separation("v2", "v1") for r in tests.values())
worse_day_to_day = ordinary["p_value"] < 0.05 and ordinary["wins_b"] > ordinary["wins_a"]
better_in_crisis = stress["p_value"] < 0.05 and stress["wins_a"] > stress["wins_b"]
decision = "ship v2" if better_in_crisis and not worse_day_to_day else "not yet, add markets"
print(f"Verdict: {decision}. v2 was ahead on {ordinary['wins_a']} of {ordinary['paired_seeds']} "
f"ordinary markets (p = {ordinary['p_value']:.2f}) and {stress['wins_a']} of "
f"{stress['paired_seeds']} crisis markets (p = {stress['p_value']:.2f}).")claude-haiku-4-5-20251001, 790 answers replayed Paired test, v2 against v1 on the same markets markets count v2 ahead p mean v2 - v1 ordinary, 15 days 12 6 1.00 -2,857 crisis, 15 days 12 8 0.39 +148,230 One market forked on day 5, then 15 days calm or in crisis version calm end crisis end crisis cost v1 926,119 756,380 -169,739 v2 949,799 675,040 -274,759 Verdict: not yet, add markets. v2 was ahead on 6 of 12 ordinary markets (p = 1.00) and 8 of 12 crisis markets (p = 0.39).
The p column is the chance of a split at least that lopsided if the two versions were equally good. At 1.00 the ordinary markets show no difference, and at 0.39 the gap in the crisis markets could easily be luck. The copied market is a single draw, and a large gap on one market proves little by itself.
Adapting it
Put the two prompts you want to compare in PROMPTS, or replace ask_claude with a call to another model. The framework adapters on the LLM adapters page take the same mode="live" and mode="replay". To test another scenario, change liquidity_crisis.
Limits
- One model and one recording. Claude answers differently from run to run, so record again, or on more markets, before quoting the size of the gap.
- Twelve short markets can't detect a small cost, so finding no difference in ordinary markets doesn't show there is none.
- One scenario, started early so the short runs reach it. Its shocks are stated assumptions, not a forecast, and other scenarios may rank the versions differently.
- Each version trades a separate copy of the market, and the two never meet in one market.
- Eight simulated companies on the default preset.
Reference
tf.rank and Ranking.separation are in Agents and evaluation. The callable adapter is in LLM adapters, and forks are in Fork a market.