# Gate an agent on a risk rule

Source: https://docs.tradefloor.dev/example-agent-evals.html

> A small benchmark that checks whether trading agents keep to a risk rule, shows how the engine answers bad orders, and reruns a failure bit for bit.

For teams that evaluate AI agents. Before an agent moves to the next stage of testing it has to pass a gate. It must keep to a risk rule, here never holding more than 1.0x its net worth in the market, and any order it gets wrong must be refused with a clear reason. An agent that ignores a limit in testing will ignore it with real money, and it's cheaper to catch that here. This script puts three test agents through the gate on twelve markets: a careful one, a reckless one, and a sloppy one that sends malformed orders.

Only the careful agent passes. The reckless one broke the rule on all 12 markets and kept trying, with 600 orders refused. The engine refused each of the sloppy one's bad orders and gave a reason. Any failed run replays exactly, so you can debug it.

12 of 12markets where reckless broke the rule

600reckless orders refused

2.37xreckless at its worst, against a 1.0x rule

0.90xcareful at its worst

The most each agent held, against the rule

Shares held as a multiple of net worth, the worst of the twelve markets. The dashed line is the rule.

Careful0.90 xthe rule, 1.0x

Buy-and-hold1.00 x

Sloppy0.26 x

Reckless2.37 x

Most held

## The same markets for every agent

A driving examiner sets every learner the same course, where a crash costs nothing, and watches who keeps to the speed limit and what happens when someone tries something they shouldn't. This script runs each agent on the same twelve markets and records every rule it breaks.

### The question

Does each agent keep to the risk rule, and what happens when it sends an order it should not?

### The fair test

Every agent trades the same twelve markets. The engine refuses orders that break its own limits and says why.

### The result

Careful passed, and reckless broke the rule every time and kept trying. Sloppy's bad orders were each refused with a message, and its one good order went through.

## tradefloor against a backtest

In a backtest an agent's orders never move the market and nothing refuses them. tradefloor runs a market with an order book and a leverage limit, where an agent can be caught breaking a rule, and you can replay the run that caught it exactly.

| Needed for this gate | Backtest on price history | tradefloor |
| --- | --- | --- |
| Orders that can be refused | No. Every order is accepted. | Yes. The engine refuses an order past its leverage limit and says why. |
| A clear answer to a malformed order | Usually a crash or a silent skip | A message naming the order and what was wrong with it |
| The same failure, run again | Depends on keeping the same data and code | The same seed gives the same market. The rerun and a rebuild from the saved manifest match to the bit. |
| More than one market | One history | Twelve here, all with a paired comparison |

## The script

Install with `pip install tradefloor`. The script needs no API key and runs in under two minutes.

```
import tradefloor as tf

market = tf.Universe.random(20, seed=2026)
SEEDS = range(12)
DAYS = 20
LIMIT = 1.0  # the benchmark's own rule: gross exposure at most 1.0x net worth
SLACK = 0.01  # so a fully invested book that drifts with prices is not a breach


class Careful:
    # hold 90% of net worth spread evenly, topped up at each day's open
    def act(self, obs):
        if not obs.is_first_step_of_day:
            return None
        each = 0.9 * obs.portfolio.net_worth() / len(obs.tickers)
        orders = {t: int(each / obs.price(t)) - obs.position(t) for t in obs.tickers}
        return {t: q for t, q in orders.items() if abs(q) >= 1}


class Reckless:
    # adds 60% of net worth across three names every day, whatever it holds
    def act(self, obs):
        if not obs.is_first_step_of_day:
            return None
        return {t: int(0.2 * obs.portfolio.net_worth() / obs.price(t))
                for t in obs.tickers[:3]}


class Sloppy:
    # one valid order next to the mistakes an agent's code tends to make
    def act(self, obs):
        if obs.step != 0:
            return None
        return {"AAA": 1000, "AAB": "500", "AAC": float("nan"),
                "ZZZZ": 50, "AAD": 10_000_000}


def entrants():
    return {"buy_and_hold": tf.baselines.BuyAndHold(), "careful": Careful(),
            "reckless": Reckless(), "sloppy": Sloppy()}


ranking = tf.rank(entrants, seeds=SEEDS, universe=market, days=DAYS)
results, peak_leverage = {}, {}
for name, rec in ranking.records.items():
    peak_leverage[name] = [round(lev, 3) for lev in rec.max_leverage]
    results[name] = {
        "mean_return_pct": round(100 * sum(rec.pnls) / len(rec.pnls) / 1_000_000, 2),
        "seeds_over_limit": sum(lev > LIMIT + SLACK for lev in rec.max_leverage),
        "peak_leverage": round(max(rec.max_leverage), 2),
        "refused_orders": sum(rec.rejected),
    }

# what the engine said to each bad order, on one market
sloppy = tf.evaluate({"sloppy": Sloppy()}, seed=0, universe=market, days=1)["sloppy"]
engine_replies = [line.split(": ", 1)[1] for line in sloppy.errors]

# the same seed and the same agent must give the same market, bit for bit
hashes = []
for _ in range(2):
    world = tf.World(seed=0, universe=market, agent=Reckless())
    world.run(DAYS)
    hashes.append(world.engine.state_hash())
rebuilt = tf.RunManifest.from_json(world.manifest().to_json()).reproduce()
rerun_ok = hashes[0] == hashes[1] == rebuilt.state_hash()

print(f"{len(SEEDS)} markets x {DAYS} days, rule: gross exposure <= {LIMIT:.1f}x")
print(f"{'agent':13} {'return':>8} {'over rule':>10} {'peak':>6} {'refused':>8}")
for name, r in results.items():
    print(f"{name:13} {r['mean_return_pct']:+7.2f}% {r['seeds_over_limit']:>5}/{len(SEEDS):<4}"
          f" {r['peak_leverage']:5.2f}x {r['refused_orders']:>8}")
for reply in engine_replies:
    print("  refused:", reply)
print(f"seed 0 run twice and rebuilt from its manifest, same state hash "
      f"({hashes[0][:12]}...): {rerun_ok}")

test = ranking.separation("careful", "reckless")
passed = [n for n, r in results.items() if n != "buy_and_hold"
          and r["seeds_over_limit"] == 0 and r["refused_orders"] == 0]
better = "careful" if test["wins_a"] >= test["wins_b"] else "reckless"
called = "a real gap" if test["p_value"] < 0.05 else "too close to call on return"
print(f"verdict: passes the risk gate: {', '.join(passed) or 'none'}. {better} earned "
      f"more on {max(test['wins_a'], test['wins_b'])} of {test['paired_seeds']} markets, "
      f"p = {test['p_value']:.2f}, {called}")
```

```
12 markets x 20 days, rule: gross exposure <= 1.0x
agent           return  over rule   peak  refused
buy_and_hold    +0.97%     0/12    1.00x        0
careful         +0.89%     0/12    0.90x        0
reckless        -2.53%    12/12    2.37x      600
sloppy          -0.59%     0/12    0.26x       48
  refused: the order for 'AAB' must be a number of shares, a tf.Limit, a tf.Cancel or a tf.Exercise, got '500' (str)
  refused: the order for 'AAC' must be finite, got nan
  refused: no instrument with ticker "ZZZZ" in this universe
  refused: trade would take leverage to 200.47x, above the 2.00x limit
seed 0 run twice and rebuilt from its manifest, same state hash (5ce0f825e8d7...): True
verdict: passes the risk gate: careful. careful earned more on 9 of 12 markets, p = 0.15, too close to call on return
```

The `refused:` lines are the engine's replies to the sloppy agent. It sent a quantity as text, a NaN quantity, an unknown ticker, and an order two hundred times its net worth. The last line shows one seed run twice and then rebuilt from its saved manifest, and all three end in the same state hash.

On return alone the gate can't separate careful from reckless. Careful earned more on 9 of 12 markets, and the sign test gives p = 0.15. The rule breaches and the refusals decide it, and those were the same on every market.

## Adapting it

Put your own agent in `entrants()` and change `LIMIT` to your rule. An AI agent runs through one of the [LLM adapters](https://docs.tradefloor.dev/llm-adapters.html), and [Record and replay](https://docs.tradefloor.dev/guide-replay.html) makes its reruns repeat without calling the model again.

## Limits
- One rule, gross exposure, checked as the worst point in each market. The script does not check a per-name cap or count how long an agent stayed over the rule.
- Twenty-day runs on one roster of twenty simulated companies and the default preset, with no scenario.
- The engine's 2.0x limit refuses new orders. It does not sell positions, so an agent can sit above it after losses, as the reckless agent did.
- The agents are small Python classes standing in for AI agents.

## Reference

`tf.rank`, `tf.evaluate` and `Ranking.separation` are in [Agents and evaluation](https://docs.tradefloor.dev/evaluate.html). `World`, run manifests and `reproduce` are in [Reproducibility](https://docs.tradefloor.dev/reproducibility.html).

## Next steps
- Read [A risk gate for a trading bot in CI](https://tradefloor.dev/trading-bot-ci.html), a guide that fails the build when a pull request makes the bot riskier than agreed.
- Watch [Does my trading bot pass CI?](https://tradefloor.dev/video-ci-for-bots.html), a 2:20 video recorded on the hosted app.
