Agents and evaluation

Writing an agent

An agent is any Python object with an act(obs) method. At each decision step the harness passes it an observation and reads back orders. By default a run has 6 decision steps a day, 65 simulated minutes apart, and each agent starts with 1,000,000 in cash and may hold positions worth up to twice its net worth. obs.step counts steps across the whole run, and obs.step_of_day, obs.is_first_step_of_day and obs.is_last_step_of_day count within the day. An agent may also declare privileged = True and keep an explain(day) method, both described below. A first agent builds and scores one on a seeded market.

What act returns

A dict of ticker to order, where each value is one of these:

Return None or {} to trade nothing. Values are shares, never weights, and any finite number of shares is accepted, fractions included. To hold a target weight, send the target holding less the shares already held and the shares in orders still waiting on the ticker, with the target rounded to whole shares if you want them:

target = int(weight * obs.portfolio.net_worth() / obs.price(ticker))
order  = target - obs.position(ticker) - waiting

The target alone, such as 0.2 * net_worth / price, is the holding to reach, and sent as an order while a position exists it buys the position again. Sizing to a target weight works through an example with a waiting order.

There are no stop or bracket orders, so check a stop yourself at each step. A refused entry, such as a string or NaN, gets a line in the scorecard's errors and the rest of the dict still trades, so read errors before the P&L.

What the agent sees

The observation holds the roster and last prices, each name's order book and average volume, the agent's own positions, cash and waiting orders, daily bars and the published economic figures in obs.history, and a view of the market in obs.engine. What obs.engine and obs.portfolio serve depends on the agent's access: ordinary by default, privileged with privileged = True on the agent, or trusted with trusted_agents=True on the call. The levels are the same under evaluate, rank, World and tca.analyse, and The agent's view sets out what each one serves.

Ordinary and privileged agents get the read-only MarketView and PortfolioView, which raise tf.SandboxError for anything they do not serve, recorded in errors. obs.hidden is None unless the agent is privileged, when it is a read-only HiddenState that also carries the size of each day's news. obs.engine.bars() refuses under evaluate, which records no days, so daily bars come from obs.history. An LLM framework run through an adapter sees only the payload the adapter builds, the same at every level: What the model sees.

Warm-up history

history_days=N on evaluate, rank, World or tca.analyse runs the market for N days before day 0 with nobody trading, and obs.history holds those days at the first decision, labeled -N to -1. Each scored day is added after its last step. N is at most 2,520, ten years of 252 days, and no scenario applies during the warm-up. Warm-up history runs a breakout rule on it.

Against buy-and-hold, across seeds

evaluate scores one market, which is one sample. Compare each agent with buy-and-hold on the same market with versus_buy_and_hold, and run rank over many seeds before calling a winner. Compare strategies does both and reads the paired sign test. To put several agents in one market, use a World.

Explaining moves

An agent may also have explain(day). The harness calls it after each close and expects the name of the factor that moved prices most that day. On pt-v20 random_noise wins almost every day, so the scorecard reports explanation_baseline, what naming the most common winner every day scored, and explanation_edge, the accuracy less that baseline, which is the figure to quote.

Execution cost

tf.tca.analyse prices every fill against the same seed run without the agent's orders. Execution cost in Compare strategies prices a round trip and says what the figure measures, and tca.analyse below has the arguments.

evaluate

Score each agent on its own copy of the same market.

tf.evaluate(
    agents: dict[str, Agent | StrategySpec],
    *, seed: int,
    universe: Sequence[Instrument],
    macro: Macro | None = None,
    days: int = 5,
    steps_per_day: int = 6,
    ticks_per_step: int = 65,
    cash: float = 1000000.0,
    max_leverage: float | None = 2.0,
    start: tuple[int, int, int] = (9, 30, 3),
    scenario: Scenario | None = None,
    model: str | ModelParams | None = None,
    cash_interest: bool = False,
    trusted_agents: bool = False,
    history_days: int = 0,
    margin_interest: bool = True,
) -> dict[str, Scorecard]

Returns a dict of Scorecard, keyed by the names you passed.

evaluate builds one engine and forks it for each agent and for an untraded baseline, so every agent meets the same market and none uses liquidity another expected. Agent objects are not copied, so state an agent keeps persists across its own steps. An agent's fills in a step reach its market once, on the first tick of the next step. A bad entry in an agent's dict is refused with a line in errors and the rest trades. A return other than a dict, or an exception from act, trades nothing that step.

evaluate warns, without changing anything it runs, when an agent failed on every step, when every order in a step was a fraction of a share (weights sent as shares), and when a spec's top_k is more than half the roster.

reference_agents

The five baseline agents.

tf.reference_agents(*, seed: int = 0) -> dict[str, Agent]

seed seeds the random agent only. The keys are buy_and_hold, random, momentum, mean_reversion and oracle. The Oracle declares privileged = True and reads the model's fair value, holding the same gross exposure as the others. On pt-v20 it is a reference agent and not a ceiling, so compare agents with buy-and-hold there.

baselines.Balanced

A fixed-weight portfolio of the equities and the simulated rate indices, 60/40 by default, with a drift band. It is one of the baselines reference_agents leaves out.

tf.baselines.Balanced(
    *, equity: float = 0.6,
    bonds: dict[str, float] | None = None,
    band: float | None = 0.05,
    max_participation: float = 0.05,
)

It needs a roster with the rate indices, from Universe.random(..., bonds=True) or Universe.with_bonds(). Without them it raises ValueError at its first step, which evaluate records in errors.

u = tf.Universe.random(20, seed=101, bonds=True)
scores = tf.evaluate(
    {"sixty_forty": tf.baselines.Balanced()},
    seed=3, universe=u, days=120,
    scenario=tf.Scenario.load("curve_shock"),
    cash_interest=True)

capture_ratio

tf.versus_buy_and_hold(scores, *, reference: str = "buy_and_hold") -> dict[str, float]
tf.capture_ratio(scores, *, oracle: str = "oracle") -> dict[str, float]
tf.capture_withheld(scores, *, oracle: str = "oracle") -> str | None
tf.oracle_is_ceiling(model=None) -> bool

versus_buy_and_hold gives each agent's P&L less buy-and-hold's in the same market, in currency. It is the comparison to quote on pt-v20. It leaves out tampered agents and warns, returning {}, when buy-and-hold did not run.

capture_ratio gives each agent's P&L as a fraction of the Oracle's: 1.0 is the Oracle's P&L. On pt-v20 it returns {} and warns, because a shock there moves fair value for good and the Oracle's P&L follows the market's month. capture_withheld returns that reason, or None where a ratio is reported. oracle_is_ceiling answers for a preset name, a ModelParams or the default.

leaderboard

tf.leaderboard(scores, by: str = "pnl") -> list[Scorecard]

Sorts scorecards by one field, best first, with tampered cards last.

rank

Run the same agents on many seeds and compare them seed by seed.

tf.rank(
    make_agents: Callable[[], dict[str, Agent | StrategySpec]],
    *, seeds: Iterable[int],
    universe: Sequence[Instrument],
    macro: Macro | None = None,
    days: int = 5,
    steps_per_day: int = 6,
    ticks_per_step: int = 65,
    cash: float = 1000000.0,
    max_leverage: float | None = 2.0,
    start: tuple[int, int, int] = (9, 30, 3),
    scenario: Scenario | None = None,
    oracle: str = "oracle",
    workers: int = 1,
    model: str | ModelParams | None = None,
    trusted_agents: bool = False,
    benchmark: str = "buy_and_hold",
    history_days: int = 0,
    margin_interest: bool = True,
) -> Ranking

The arguments shared with evaluate mean the same, and the others are these:

Ranking

report() names the benchmark by its label, as in vs buy_and_hold or vs flat. On pt-v20 it also says that the Oracle ran and has no row, with its P&L over the benchmark's. An agent whose adapter refused actions gets a REFUSED line with the count and the first refusal, an agent whose act() returned something other than a mapping gets an UNUSABLE line with the count and the first seed and step, and an agent that raised gets a RAISED line.

separation(a, b) returns a dict with a, b, wins_a, wins_b, ties, paired_seeds, decisive and p_value. Both agents meet the same market on each seed, so each pair differs only in the agent. decisive is true only when one agent won on every seed, and it counts wins without testing significance.

AgentRecord

One agent's results across every seed.

tca.analyse

Run the same seed with and without the agent's orders, so every fill is priced against the market where it never traded.

tf.tca.analyse(
    agent,
    *, seed: int,
    universe: Sequence[Instrument],
    macro: Macro | None = None,
    days: int = 1,
    steps_per_day: int = 6,
    ticks_per_step: int = 65,
    cash: float = 1000000.0,
    max_leverage: float | None = 2.0,
    start: tuple[int, int, int] = (9, 30, 3),
    scenario: Scenario | None = None,
    model: str | ModelParams | None = None,
    trusted_agents: bool = False,
    history_days: int = 0,
) -> Execution

The agent is any object with act(obs) returning share quantities, as on Writing an agent. tca.analyse takes market orders only and refuses a tf.Limit, because the part of a limit order that waits fills inside a session, where the untraded run has no price to compare it with. days is 1 by default, unlike evaluate. history_days runs the market that many days before day 0 with nobody trading, as in evaluate, and both runs start day 0 from the warmed market. No scenario applies during the warm-up. Execution cost explains what the comparison measures.

Execution

What the trading cost, against the run without the agent's orders.

Scorecard

One agent's result on one seed, as evaluate returns it, with errors to read before the P&L.

sharpe, volatility_pct, time_in_market and avg_gross_exposure are read-only properties. They and exposure_curve are left out of as_dict(), so the traded digest hashes what it did before they were added.

The repr prints the four figures after the impact, then the counts and flags that are set. Over fewer than Scorecard.SHARPE_MIN_DAYS (20) scored days it prints sharpe=n/a (short run), because the standard error of an annualized Sharpe ratio is 3.5 or more there. The property still returns the figure: buy-and-hold in One market and the baselines reads sharpe=n/a (short run), vol=9.6%, in_market=100%, exposure=1.00x after ten days on a 1.05% gain, and its sharpe is +2.78. The counts and flags include errors=1, free-borrowing, history_days=20 and explanation=0.967 vs baseline 0.983 (edge -0.016).

RunManifest

A finished run as one document: the roster, the day-0 economy, the scenario, the order log, the strategy when it is a StrategySpec, the package version, the model's coefficients and the market digest a replay has to reproduce. Recording the market shows the round trip.

tf.RunManifest.of(
    engine: Engine,
    *, seed: int,
    universe: Sequence[Instrument],
    macro: Macro | None = None,
    scenario: Scenario | None = None,
    strategy: StrategySpec | str | None = None,
    universe_source: Any = None,
    label: str = "",
    derived_from: Any = None,
    ledger: DayLedger | None = None,
    agent_access: dict[str, Any] | None = None,
) -> RunManifest

reproduce() raises ValidationError on a mismatch and names the component that disagreed. It first compares the era digest, a small fixed calculation, with the one in the manifest, and refuses to replay on a build that does different arithmetic. It checks the market and carries no score: result holds the market's digest, the number of days and draws_consumed, and nothing else. evaluate and rank write no manifest, so to let a reader check a score, publish the agent, the call and the seeds.