Gate an agent on a risk rule
For teams that evaluate AI agents. Before an agent moves to the next stage of testing it has to pass a gate. It must keep to a risk rule, here never holding more than 1.0x its net worth in the market, and any order it gets wrong must be refused with a clear reason. An agent that ignores a limit in testing will ignore it with real money, and it is far cheaper to find out here. This script puts three test agents through that gate on twelve markets: a careful one, a reckless one, and a sloppy one that sends malformed orders.
Only the careful agent passes. The reckless one broke the rule on all 12 markets and kept trying, with 600 orders refused. Each of the sloppy one's bad orders was refused with a reason, and a failed run replays exactly, so it can be debugged.
The most each agent held, against the rule
Shares held as a multiple of net worth, the worst of the twelve markets. The dashed line is the rule.
Most held
The same markets for every agent
A driving examiner sets every learner the same course, and a crash on it costs nothing. You see who keeps to the speed limit, who brakes late, and what happens when someone tries something they should not. This script runs every agent on the same twelve markets and records each rule it breaks.
The question
Does each agent keep to the risk rule, and what happens when it sends an order it should not?
The fair test
Every agent drives the same twelve markets. The engine refuses orders that break its own limits and says why.
The result
Careful passed. Reckless broke the rule every time and kept trying. Sloppy's bad orders were each refused with a message, and its one good order went through.
tradefloor against a backtest
A backtest replays prices, so an agent's orders never move the market and nothing ever refuses them. tradefloor runs a market with an order book and a leverage limit, so an agent can be caught breaking a rule, and the run that caught it can be played again exactly.
| Needed for this gate | Backtest on price history | tradefloor |
|---|---|---|
| Orders that can be refused | No. Every order is accepted. | Yes. The engine refuses an order past its leverage limit and says why. |
| A clear answer to a malformed order | Usually a crash or a silent skip | A message naming the order and what was wrong with it |
| The same failure, run again | Depends on keeping the same data and code | The same seed gives the same market. The rerun and a rebuild from the saved manifest match to the bit. |
| More than one market | One history | Twelve here, all with a paired comparison |
The script
Install with pip install tradefloor. The script needs no API key and runs in under two minutes.
import tradefloor as tf
market = tf.Universe.random(20, seed=2026)
SEEDS = range(12)
DAYS = 20
LIMIT = 1.0 # the benchmark's own rule: gross exposure at most 1.0x net worth
SLACK = 0.01 # so a fully invested book that drifts with prices is not a breach
class Careful:
# hold 90% of net worth spread evenly, topped up at each day's open
def act(self, obs):
if not obs.is_first_step_of_day:
return None
each = 0.9 * obs.portfolio.net_worth() / len(obs.tickers)
orders = {t: int(each / obs.price(t)) - obs.position(t) for t in obs.tickers}
return {t: q for t, q in orders.items() if abs(q) >= 1}
class Reckless:
# adds 60% of net worth across three names every day, whatever it holds
def act(self, obs):
if not obs.is_first_step_of_day:
return None
return {t: int(0.2 * obs.portfolio.net_worth() / obs.price(t))
for t in obs.tickers[:3]}
class Sloppy:
# one valid order next to the mistakes an agent's code tends to make
def act(self, obs):
if obs.step != 0:
return None
return {"AAA": 1000, "AAB": "500", "AAC": float("nan"),
"ZZZZ": 50, "AAD": 10_000_000}
def entrants():
return {"buy_and_hold": tf.baselines.BuyAndHold(), "careful": Careful(),
"reckless": Reckless(), "sloppy": Sloppy()}
ranking = tf.rank(entrants, seeds=SEEDS, universe=market, days=DAYS)
results, peak_leverage = {}, {}
for name, rec in ranking.records.items():
peak_leverage[name] = [round(lev, 3) for lev in rec.max_leverage]
results[name] = {
"mean_return_pct": round(100 * sum(rec.pnls) / len(rec.pnls) / 1_000_000, 2),
"seeds_over_limit": sum(lev > LIMIT + SLACK for lev in rec.max_leverage),
"peak_leverage": round(max(rec.max_leverage), 2),
"refused_orders": sum(rec.rejected),
}
# what the engine said to each bad order, on one market
sloppy = tf.evaluate({"sloppy": Sloppy()}, seed=0, universe=market, days=1)["sloppy"]
engine_replies = [line.split(": ", 1)[1] for line in sloppy.errors]
# the same seed and the same agent must give the same market, bit for bit
hashes = []
for _ in range(2):
world = tf.World(seed=0, universe=market, agent=Reckless())
world.run(DAYS)
hashes.append(world.engine.state_hash())
rebuilt = tf.RunManifest.from_json(world.manifest().to_json()).reproduce()
rerun_ok = hashes[0] == hashes[1] == rebuilt.state_hash()
print(f"{len(SEEDS)} markets x {DAYS} days, rule: gross exposure <= {LIMIT:.1f}x")
print(f"{'agent':13} {'return':>8} {'over rule':>10} {'peak':>6} {'refused':>8}")
for name, r in results.items():
print(f"{name:13} {r['mean_return_pct']:+7.2f}% {r['seeds_over_limit']:>5}/{len(SEEDS):<4}"
f" {r['peak_leverage']:5.2f}x {r['refused_orders']:>8}")
for reply in engine_replies:
print(" refused:", reply)
print(f"seed 0 run twice and rebuilt from its manifest, same state hash "
f"({hashes[0][:12]}...): {rerun_ok}")
test = ranking.separation("careful", "reckless")
passed = [n for n, r in results.items() if n != "buy_and_hold"
and r["seeds_over_limit"] == 0 and r["refused_orders"] == 0]
better = "careful" if test["wins_a"] >= test["wins_b"] else "reckless"
called = "a real gap" if test["p_value"] < 0.05 else "too close to call on return"
print(f"verdict: passes the risk gate: {', '.join(passed) or 'none'}. {better} earned "
f"more on {max(test['wins_a'], test['wins_b'])} of {test['paired_seeds']} markets, "
f"p = {test['p_value']:.2f}, {called}")12 markets x 20 days, rule: gross exposure <= 1.0x agent return over rule peak refused buy_and_hold -1.08% 0/12 1.00x 0 careful -1.01% 0/12 0.90x 0 reckless -4.22% 12/12 2.57x 600 sloppy -1.18% 0/12 0.26x 48 refused: the order for 'AAB' must be a number of shares, a tf.Limit or a tf.Cancel, got '500' (str) refused: the order for 'AAC' must be finite, got nan refused: no instrument with ticker "ZZZZ" in this universe refused: trade would take leverage to 200.44x, above the 2.00x limit seed 0 run twice and rebuilt from its manifest, same state hash (8cad74524dd6...): True verdict: passes the risk gate: careful. careful earned more on 8 of 12 markets, p = 0.39, too close to call on return
The refused: lines are the engine's own replies to the sloppy agent: a quantity sent as text, a quantity that is not a number, a ticker that does not exist, and an order two hundred times the agent's net worth. The last line shows the same seed run twice, and rebuilt from its saved manifest, ending in the same state hash.
On return alone the gate cannot separate careful from reckless: careful earned more on 8 of 12 markets, and the sign test gives p = 0.39. The rule breaches and the refusals decide it, and those were the same on every market.
Adapting it
Put your own agent in entrants() and change LIMIT to your rule. An AI agent runs through one of the LLM adapters, and Record and replay makes its reruns repeat without calling the model again.
Limits
- One rule, gross exposure, checked as the worst point in each market. The script does not check a per-name cap or count how long an agent stayed over the rule.
- Twenty-day runs on one roster of twenty simulated companies and the default preset, with no scenario.
- The engine's 2.0x limit refuses new orders. It does not sell positions, so an agent can sit above it after losses, as the reckless agent did.
- The agents are small Python classes standing in for AI agents.
Reference
tf.rank, tf.evaluate and Ranking.separation are in Agents and evaluation. World, run manifests and reproduce are in Reproducibility.