Skip to the page
EXAMPLES/A/B TEST AN AI AGENT

A/B test an AI agent

For teams building AI trading agents. Your agent is Claude Haiku 4.5 with a short prompt telling it to grow the portfolio and answer each day in JSON. Version 2 adds one sentence: hold less of volatile stocks, and halve every position when the VIX is above 30. That caution might cost your users money in normal times, and it might protect them in the next crash. This script asks Claude for every decision under both prompts on the same ordinary and crisis markets, so you can decide whether to ship the change.

Don't ship version 2 yet. In one market that was copied so both versions met the same crisis, version 2 lost more than version 1: 274,759 against 169,739. Across twelve crisis markets, version 2 did better on 8 of 12, a split a coin toss often gives, and in ordinary markets on 6 of 12. Run it on more markets before you decide.

274,759what one crisis cost version 2, against 169,739 for version 1
8 of 12crisis markets where version 2 did better
6 of 12ordinary markets, an even split
790decisions recorded from Claude

Both versions on the same markets

Car makers crash-test two designs into the same wall at the same speed. If each car hit a different wall, a better result could just mean a softer wall. Testing an agent on one stretch of history has the same weakness, and that stretch may not include a crisis at all. Here both versions trade the same twenty-four markets, and half of them have a crisis.

The question

Does the extra sentence in version 2 cost money in normal times, and does it save money in a crisis?

The fair test

Claude trades the same twelve ordinary markets and the same twelve crisis markets under each prompt. Then one market is copied on day 5, and the crisis lands on the same day in the copy each version trades.

The result

In the one shared crisis version 2 lost more. Over twelve crisis markets it did better on 8. Both could easily be luck, so the change isn't ready to ship.

tradefloor against a backtest

A backtest gives both versions one sample, the one history it replays, and the crisis you care about may not be in it. tradefloor starts the crisis on the day you choose and runs both versions through it on many markets. It can also copy one market so both versions meet the crisis at the same moment.

Needed for this decisionBacktest on price historytradefloor
A crisis to test againstOnly if the dates happen to include oneAny shipped scenario, started on the day you choose
Both versions under the same conditionsSame history, but one sampleTwenty-four paired markets and a sign test
The crisis landing at the same moment for bothNoYes. A fork copies the market, and the crisis lands on both copies.
Model calls that cost nothing to repeatEvery rerun calls the model againRecord the answers once and replay them for free

Claude's answers, recorded once

The first run asked Claude Haiku 4.5 for all 790 decisions and saved the answers in two recordings, one per version. That took about fifteen minutes. Later runs find the recordings beside the script and replay them with no API key and no cost, and the numbers come out identical each time. Each answer is stored with the exact market data Claude was shown, and if you change the experiment the run stops with ReplayMiss and doesn't reuse old answers.

The recordings show version 2 following its rule. On days the VIX was above 30 it answered with lines such as "VIX at 50 exceeds 30 threshold, triggering mandatory position halving across all holdings". Whether the rule made money is a separate question, and the twelve crisis markets answer it.

To replay, download example-agent-team.py, example-agent-team-v1.json and example-agent-team-v2.json into one folder, pip install tradefloor and run python example-agent-team.py. To record your own, delete the two recordings, pip install anthropic, set ANTHROPIC_API_KEY and run it again. Claude answers a little differently each time and a fresh recording gives different numbers, so check that the decision still holds.

The script

import json
import pathlib

import tradefloor as tf
from tradefloor.integrations.callable import callable_agent
from tradefloor.integrations.common import AdapterInfo, Transcript, digest

MODEL = "claude-haiku-4-5-20251001"
ASK = ("You manage a stock portfolio in a simulated market. Each trading day you get "
       "the market as JSON. Grow the portfolio. Reply with only a JSON object: "
       '{"actions": [{"symbol": "AAA", "side": "BUY", "quantity": 100}], '
       '"rationale": "one sentence"}. Quantities are shares, and an empty actions '
       "list keeps what you hold.")
PROMPTS = {"v1": ASK,
           "v2": ASK + " Risk rule: hold less of volatile stocks, and halve every "
                       "position when the VIX is above 30."}

market = tf.Universe.random(8, seed=11)
crisis = tf.Scenario.load("liquidity_crisis")

# Claude is called once and every answer recorded. With the recordings beside
# this file, every later run replays them: no API key, the same numbers.
here = pathlib.Path(__file__).parent
recordings = {v: here / f"example-agent-team-{v}.json" for v in PROMPTS}
live = not all(path.exists() for path in recordings.values())
tapes = {v: Transcript() if live else Transcript.load(path) for v, path in recordings.items()}


def ask_claude(prompt):
    import anthropic  # needed only to record
    client = anthropic.Anthropic()

    def ask(payload):
        reply = client.messages.create(
            model=MODEL, max_tokens=600, system=prompt,
            messages=[{"role": "user", "content": json.dumps(payload)}])
        return reply.content[0].text
    return ask


def to_decision(raw, payload):
    # runs on every replay too; a reply that is not JSON holds
    text = raw.strip().removeprefix("```json").removeprefix("```").removesuffix("```")
    try:
        return json.loads(text)
    except ValueError:
        return {"actions": [], "rationale": "unreadable reply"}


def agent(version):
    info = AdapterInfo(framework="callable", provider="anthropic", model=MODEL,
                       instructions_digest=digest(PROMPTS[version]))
    if live:
        return callable_agent(ask_claude(PROMPTS[version]), postprocess=to_decision,
                              info=info, mode="live", recorder=tapes[version])
    return callable_agent(postprocess=to_decision, info=info, mode="replay",
                          transcript=tapes[version])


def both():
    return {"v1": agent("v1"), "v2": agent("v2")}


# Paired A/B tests: both versions meet the same markets, ordinary and in a crisis
tests = {
    "ordinary, 15 days": tf.rank(both, seeds=range(12), universe=market, days=15,
                                 benchmark="v1", workers=8),
    # other seeds, so no question asked in the ordinary test is asked again here
    "crisis, 15 days": tf.rank(both, seeds=range(12, 24), universe=market, days=15,
                               benchmark="v1", scenario=crisis.starting_at(5)),
}
pnl_gap = {kind: [round(g) for g in r.records["v2"].excess_pnls] for kind, r in tests.items()}

# One shared past, then the same crisis for both versions from day 5
net_worth = {}
for name in ("v1", "v2"):
    world = tf.World(seed=24, universe=market, agent=agent(name))  # a market of its own
    path = []
    for day in range(5):
        world.run(1)
        path.append(round(world.net_worth()))
    calm, stressed = world.fork("calm", "crisis")
    stressed.apply(crisis, at=0)
    net_worth[f"{name} calm"], net_worth[f"{name} crisis"] = list(path), list(path)
    for day in range(15):
        for arm, label in ((calm, "calm"), (stressed, "crisis")):
            arm.run(1)
            net_worth[f"{name} {label}"].append(round(arm.net_worth()))

if live:
    for version, path in recordings.items():
        tapes[version].save(path)
answers = sum(len(tape) for tape in tapes.values())
print(f"{MODEL}, {answers} answers {'recorded' if live else 'replayed'}")
print()
print("Paired test, v2 against v1 on the same markets")
print(f"{'markets':18}{'count':>6}{'v2 ahead':>10}{'p':>7}{'mean v2 - v1':>14}")
for kind, r in tests.items():
    s = r.separation("v2", "v1")
    print(f"{kind:18}{s['paired_seeds']:>6}{s['wins_a']:>10}{s['p_value']:>7.2f}"
          f"{r.records['v2'].mean_excess_pnl:>+14,.0f}")
print()
print("One market forked on day 5, then 15 days calm or in crisis")
print(f"{'version':8}{'calm end':>12}{'crisis end':>12}{'crisis cost':>13}")
crisis_loss, ends = {}, {}
for name in ("v1", "v2"):
    calm_end, crisis_end = net_worth[f"{name} calm"][-1], net_worth[f"{name} crisis"][-1]
    crisis_loss[name] = calm_end - crisis_end
    ends[name] = {"calm": calm_end, "crisis": crisis_end}
    print(f"{name:8}{calm_end:>12,}{crisis_end:>12,}{-crisis_loss[name]:>+13,}")

ordinary, stress = (r.separation("v2", "v1") for r in tests.values())
worse_day_to_day = ordinary["p_value"] < 0.05 and ordinary["wins_b"] > ordinary["wins_a"]
better_in_crisis = stress["p_value"] < 0.05 and stress["wins_a"] > stress["wins_b"]
decision = "ship v2" if better_in_crisis and not worse_day_to_day else "not yet, add markets"
print(f"Verdict: {decision}. v2 was ahead on {ordinary['wins_a']} of {ordinary['paired_seeds']} "
      f"ordinary markets (p = {ordinary['p_value']:.2f}) and {stress['wins_a']} of "
      f"{stress['paired_seeds']} crisis markets (p = {stress['p_value']:.2f}).")
claude-haiku-4-5-20251001, 790 answers replayed

Paired test, v2 against v1 on the same markets
markets            count  v2 ahead      p  mean v2 - v1
ordinary, 15 days     12         6   1.00        -2,857
crisis, 15 days       12         8   0.39      +148,230

One market forked on day 5, then 15 days calm or in crisis
version     calm end  crisis end  crisis cost
v1           926,119     756,380     -169,739
v2           949,799     675,040     -274,759
Verdict: not yet, add markets. v2 was ahead on 6 of 12 ordinary markets (p = 1.00) and 8 of 12 crisis markets (p = 0.39).

The p column is the chance of a split at least that lopsided if the two versions were equally good. At 1.00 the ordinary markets show no difference, and at 0.39 the gap in the crisis markets could easily be luck. The copied market is a single draw, and a large gap on one market proves little by itself.

Adapting it

Put the two prompts you want to compare in PROMPTS, or replace ask_claude with a call to another model. The framework adapters on the LLM adapters page take the same mode="live" and mode="replay". To test another scenario, change liquidity_crisis.

Limits

  • One model and one recording. Claude answers differently from run to run, so record again, or on more markets, before quoting the size of the gap.
  • Twelve short markets can't detect a small cost, so finding no difference in ordinary markets doesn't show there is none.
  • One scenario, started early so the short runs reach it. Its shocks are stated assumptions, not a forecast, and other scenarios may rank the versions differently.
  • Each version trades a separate copy of the market, and the two never meet in one market.
  • Eight simulated companies on the default preset.

Reference

tf.rank and Ranking.separation are in Agents and evaluation. The callable adapter is in LLM adapters, and forks are in Fork a market.