# How it is measured

Source: https://docs.tradefloor.dev/how-its-measured.html

> How the default model is graded: checks registered before the run, exam seeds the tuning never saw, long-run rows over 21 years, the one-year table, the grade.

Each validation criterion and its acceptance range is committed before the grade. The grade runs on holdout seeds the tuning never used, and the scripts and their output are published. pt-v22 passed 190 of the 190 registered criteria that count toward its grade. On the one-year table, all 19 statistics fall inside acceptance ranges read from real market data.

## What validation establishes
- pt-v22 passes 190 of the 190 pre-registered validation criteria that count toward its grade, on holdout seeds the tuning never used. The criteria cover crash sizes in replays of 2008 and 2020, the frequency of large index falls, the duration of high-VIX spells, rate and bond moves, earnings in a contraction and the cost of trading size. Each has an acceptance range fixed before the grade.
- The leak-finding strategies earn about nothing. The market doesn't pay you for trading on a late headline, past prices alone, value or momentum screens, published economic data or a rate decision.
- Over one year, all 19 statistics of the one-year table fall inside acceptance ranges read from real market data. The rows for large index falls and for the index's drift are read over 360 seeds, as the grade defines them. `tf.envelope.check()` answers for runs of up to 252 trading days.

## What validation does not establish
- That a strategy that makes money here will make money in a real market. Matching some statistics of real prices doesn't put real opportunities into simulated ones, so a profit here is evidence about the simulator.
- That anything outside the registered rows and the one-year table is realistic. Some behaviors aren't checked at all. The named gaps lists measured failures and the uses each one rules out.
- That a result carries to another preset, or past 252 days on a statistic the long-run rows don't cover.

The scripts that grade every row, the one-year table and rows D1 and V1 are in the library repository, in [validation/pt-v22/](https://github.com/simoncoombes/tradefloor/tree/main/validation/pt-v22), with the outputs they wrote (The published grade).

## Registration, then one grade

1Measure real marketsThe criteria are built on statistics from the S&P 500 since 1928, the VIX since 1990, 40 large US stocks, Treasury and corporate yields, Federal Reserve decisions and Shiller's long-run earnings. One example is how many days a decade the index falls more than 5%. Each statistic becomes a criterion with an acceptance range wide enough to cover how much the real figure moves from one decade to the next.

2Write the rulesMost behaviors come from a published source, such as the square-root cost of size. A new rule goes in switched off, and a test runs one fixed market on all the shipped presets to check that no other preset's output changed by a single bit.

3Screen on practice seedsCandidate settings run in grids over many simulated histories. Each market comes from a seed, and the seeds are split in two. Practice seeds are used while choosing settings, and exam seeds are kept back for the grade.

4Try to cheat itSome checks are strategies built to find leaks. They trade on a headline 5 minutes late, use rules that read only past prices, run value and momentum screens, time the market on published economic data and trade on a rate decision. A strategy passes only if it earns about nothing, as it would in a real market. An independent audit found two leaks, and both were fixed in the model.

5RegisterThe checks, their pass marks and the exact settings to be graded are committed. Nothing on the list can change after the results come in.

6Grade onceThe model runs once on the exam seeds and has to pass every row. If it fails one it doesn't ship. Any change then needs a new registration and fresh seeds before another grade.

7FreezeA model that passes ships as a preset whose values never change after release. It carries a fingerprint of its coefficients and known-answer digests, and each release checks those on five platforms.

## The seeds

The final settings for pt-v22 were chosen on practice seeds. The grade ran on fresh exam seeds that no earlier run had used. [grade-seeds.json](https://github.com/simoncoombes/tradefloor/blob/main/validation/pt-v22/programme/r13reg/grade-seeds.json) lists them with the proof that they were unused. They cover 270 histories of 21 years with nothing imposed, replays of 2008 and 2020 with the real VIX, the real 2020-21 and 2022 economies driven through the model, the packaged 2008 recession (`recession_2008`) and the leak-finding strategies.

## The registered rows

Each row is a validation criterion with an acceptance range. A crash size has to be within 30% of the real value, a frequency has to be half to twice the real rate, and any edge a trading strategy could learn has a hard limit. pt-v22 passes 190 of the 190 that count toward its grade; the table shows each reading against its pass mark.

A rows replay 2008 and 2020, B rows count events and long-run figures over the free histories, C rows are leak and consistency checks and the cost of size, R rows cover rates, bonds and the driven 2022 market, S rows the 2008 recession, D2, F1 and L1 the driven 2020 path, V1 variance over two and five years, and D1 the one-year table below. Where a row has both replays, 2008 comes before 2020. The rows for the contracts, the leak checks and the rest of pt-v22's registration are summed up below the table, one line for each group.

| Row | What it checks | pt-v22 | Real | Pass mark | Within it |
| --- | --- | --- | --- | --- | --- |
| A1 | Worst month's volatility in the 2008 and 2020 replays within 30% of real | 100 / 98.3 | 84.3 / 94.5 | Within 30% | yes |
| A2 | Maximum drawdown in the 2008 and 2020 replays within 30% of real | 0.444 / 0.343 | 0.568 / 0.339 | Within 30% | yes |
| A3 | Peak stock correlation in the 2008 and 2020 replays within 0.15 of real | 0.813 / 0.864 | 0.748 / 0.872 | Within 0.15 | yes |
| B1 | Share of sessions with the VIX above 30 | 0.068 | 0.082 | 1/2x to 2x | yes |
| B2 | Mean length of a fear spell above VIX 30, sessions | 24 | 22 | 1/2x to 2x | yes |
| B3 | 20% bear markets per decade | 1.69 | 1.16 | 1/2x to 2x | yes |
| B4 | 10% corrections per decade | 4.62 | 2.84 | 1/2x to 2x | yes |
| B5 | Sessions down more than 5% per decade | 7.2 | 6.2 | 1/2x to 2x | yes |
| B6 | Share of sessions with the VIX under 15 | 0.308 | 0.326 | 1/2x to 2x | yes |
| B7 | Index annual volatility, % | 18.4 | 18.1 | Within 20% | yes |
| B8 | Long-run index return, % a year | 7 | 6.25 | Within 2 points of the target | yes |
| C1 | Crash rate in years 3-21 against years 1-2 | 1.12 | 1 | 2/3x to 1.5x | yes |
| C2 | Histories touching the VIX ceiling, of 270 | 5 | 0 | At most 9 of 270 (1 per 30) | yes |
| C3 | Edge from reading a headline 5 ticks late, bp | 17.9 | 0 | Under 20 bp | yes |
| C4a | 65-minute lag-1 autocorrelation of print returns, median name; Roll spread over quoted | −0.044 / 1.5 | 0 / 1 | At or above −0.05, or Roll at most 2x quoted | yes |
| C4b | The price-only rule furthest over its line on tf-suite-2026.1 (named in worst): median points over buy-and-hold, markets beaten of 20 | −3 / 8 | 0 / 10 | Every rule at most +5 points and 14 of 20 | yes |
| C5 | Stock-level (idiosyncratic) variance ratio at 60 sessions, median name, 90 histories of 2,660 sessions (three seed sets of 30; the median over the histories) | 0.952 | 0.924 | 0.8 to 1.05 | yes |
| C6 | Rank IC of the value signal on published fundamentals against the next 20 sessions: whole history, first 60 sessions | 0.002 / 0.0024 | 0.0093 / 0.0093 | −0.03 to 0.05; −0.03 to 0.05 | yes |
| C7 | Rank IC of 12-1 and 6-1 month momentum against the next 20 sessions | 0.0053 / 0.0011 | 0.0267 / 0.0413 | −0.04 to 0.095; −0.02 to 0.1 | yes |
| C8 | Lo-MacKinlay one-day loser-minus-winner book, bp a day | −1.06 | −1.74 | −6.4 to 2.9 | yes |
| D2 | The driven 2020-21 market: the index's maximum drawdown and the sessions from the pre-crash high back to it, medians over histories | 0.382 / 168 | 0.339 / 126 | 0.237 to 0.441; 63 to 252 | yes |
| F1 | The driven 2020 path's fast crash: the high to the lowest close within 60 sessions, and the sessions it took | 0.382 / 31.5 | 0.339 / 23 | 0.237 to 0.441; 12 to 46 | yes |
| L1 | Look-through: sessions from the index's trough to the aggregate earnings level's trough on the driven 2020 path | 71 | 68 | 1 to 136 | yes |
| C10 | Timing rules on published macro data against buy-and-hold, and the drift after a published turn | 0.0059 / 0.5 / −1.25 / 0.988 | −2 / 0.324 / −5.9 / 4.3 | Every rule at most +1.0 point a year and ahead in at most 2/3; drifts no larger than the S&P's after NBER turns | yes |
| R5 | The driven 2022 market: the index's maximum drawdown | 0.221 | 0.254 | 0.178 to 0.33 | yes |
| R7a | The equal-weight index's mean log move from the first price readable after a changed policy rate to the end of the first 65-minute bar, less the mean first bar of all days, bp: after a hike, after a cut | 1.78 / −0.727 | 0 / 0 | Each within 5 bp, or 2 se where wider | yes |
| R7b | The audit's rate-news agent through tf.evaluate: mean annual excess over holding in points, histories ahead | −1.22 / 2 | 0 / 15 | At most 0 points a year, ahead in at most 20 of 30 | yes |
| S1a | The packaged recession: share of the index's paired log fall, at its lowest, won back 252 sessions later, mean over seeds | 0.88 | 0.62 | 0.45 to 1 | yes |
| S2 | The packaged recession: the index's own rise in the 252 sessions after its low, percent, mean over seeds | 60.6 | 69 | 25 to 80 | yes |
| S1b | The packaged recession: seeds whose cycle has left contraction and trough within 24 months of the onset | 180 | 180 | Every seed | yes |
| V1 | The index's variance ratio at two and five years relative to one, years 2-21 of the pooled free histories | 0.909 / 0.825 | 0.93 / 0.87 | 0.75 to 1.15; 0.55 to 1.2 | yes |
| C9 | The cost of size in the agent-facing book: exponent and coefficient of the average cost against Q/V, in sigma units | 0.459 / 0.474 | 0.5 / 0.5 | Exponent in 0.4 to 0.7 and coefficient in 0.33 to 0.67 | yes |
| R1 | Sd of the 2-year Treasury yield's daily change, bp (FRED DGS2 2015-2025) | 5.3 | 5.23 | 3.65 to 6.8 | yes |
| R2 | Sd of the 10-year Treasury yield's daily change, bp (FRED DGS10) | 5.89 | 5.41 | 4.54 to 6.27 | yes |
| R4 | Daily correlation of the roster index with an IG bond's return (minus the corporate yield's change; SPY against LQD) | 0.293 | 0.272 | 0.15 to 0.39 | yes |
| B9 | Sd of annual index log returns, years 2-21, percent; the start-up drift: sd of the first 60 sessions' index return over the steady state's, across 50 markets | 16.8 / 1.14 | 17.4 / 1 | 13.9 to 20.9; 0.6667 to 1.5 | yes |
| D1 | Ruled bands of the one-year realism table in, on all four cells | all | all | Every band in | yes |
| R6 | The driven 2022 market: the market P/E's log change per 100 bp of the corporate yield, monthly averages | −2.08 | −5.43 | −10.86 to −2.72 percent | no |

The other rows that count toward the grade, each listed with its reading in [validation/pt-v22/](https://github.com/simoncoombes/tradefloor/tree/main/validation/pt-v22):
- 32 rows for the futures, options and margin: all within their marks.
- 59 further rows registered for pt-v22: all within their marks.
- 44 leak checks: all within their marks.
- 11 rows carried over from pt-v21's grade: all within their marks.
- 6 rows restated at registration 19: all within their marks.
- The exploit search: 0 exploits found over 509 trading rules.

Shown for comparison and not counted: `R6`, `OF4`, `SV1`, `SV2`, `SV3`, `SV4`, `SV5_9D`, `SV5_6M`, `SV5_1Y`, `SV6_median`, `SV6_sd`, `SV7`, `SV8`, `SV9`, `OB1r`, `OB1v`, `OB1d`, `OB2r`, `OB2v`, `OB2d`, `OB3r`, `OB3v`, `OB3d`, `OB4r`, `OB4v`, `OB4d`, `OB5`, `OB6`, `XD4`, `VF1`, `VF2`, `VF3`, `VF4_F1`, `VF4_F2`, `VF4_F4`, `VF5`, `VF6_h21`, `VF6_h63`, `VF6_h126`, `VF7`, `VF8`.

## The one-year table

The second test is a panel of statistics measured over one year, taking the median across 30 seeds. Each is compared with a band read from the longest real record the statistic allows. The 14 shape statistics and crisis dispersion are measured on a fixed roster of 40 companies. The four index rows use a roster that changes with each seed, since a crash rate measured on one roster describes only that roster.

| Statistic | What it checks | Real band | pt-v22 | In band |
| --- | --- | --- | --- | --- |
| annualised_vol_pct | How much prices move in a year | 12 to 41 | 19.9 | yes |
| excess_kurtosis | How fat the tails are | −13 to 24 | 6.72 | yes |
| return_acf1 | Does yesterday predict today | −0.07 to 0.06 | −0.0049 | yes |
| abs_return_acf1 | Does a wild day follow a wild day | 0.02 to 0.17 | 0.105 | yes |
| abs_return_acf5 | The same, one week apart | −0.03 to 0.1 | 0.0281 | yes |
| abs_return_acf20 | The same, one month apart | −0.05 to 0.06 | 0.0044 | yes |
| cross_sectional_corr | How much names move together | 0.09 to 0.49 | 0.311 | yes |
| volume_abs_return_corr | Do big moves come with volume | 0.35 to 0.64 | 0.593 | yes |
| leverage_effect | Do falls raise volatility | −0.11 to 0 | −0.0434 | yes |
| volume_change_acf1 | Does volume mean-revert | −0.3 to −0.2 | −0.241 | yes |
| corr_asymmetry | Do names couple more when falling | −0.15 to 0.23 | 0.0646 | yes |
| corr_asymmetry_lagged | The same, one day later | −0.15 to 0.33 | 0.0734 | yes |
| sector_excess_corr | Do industries move together | 0.04 to 0.23 | 0.126 | yes |
| corr_persistence_acf1 | Does correlation stay high after a panic | −0.48 to 0.69 | 0.0505 | yes |
| crisis_sector_dispersion | Does one industry lead a crisis | 0.79 to 1.74 | 1.32 | yes |
| index_drift_pct | Which way the index goes | 1.1 to 10.3 | 7.33 | yes |
| fear_gauge_dn1 | How much fear a bad day buys | 0.39 to 3.03 | 1.98 | yes |
| fear_gauge_dn3 | The same, on a much worse day | 2.6 to 9.58 | 5.75 | yes |
| index_tail_dn3_pct | How often the index falls hard | 0.64 to 2.34 | 1.25 | yes |

The table certifies runs of up to 252 trading days. At 504 days pt-v22 holds all 15 statistics of the two-year panel inside bands read from real data, but `tf.envelope.check()` still certifies one year only. For longer runs the evidence is the long-run rows above. Some statistics pass on the low side, such as volatility clustering, which is close to real one day apart and fades faster than real after that. The named gaps below say what that rules out.

Ask the package before you lean on a statistic:

```
import tradefloor as tf

verdict = tf.envelope.check(horizon_days=504, statistics=["abs_return_acf20"])
print(str(verdict).splitlines()[0])
```

```
OUTSIDE the envelope
```

The verdict goes on to name each reason. Here the horizon is past the certified 252 days, and lag-20 volatility clustering is one of the named gaps.

## The named gaps

Each gap below is something the model gets wrong, with the uses it rules out. `tf.envelope.check()` refuses a question that asks for one of them.

| Gap | What falls short | Do not use it for |
| --- | --- | --- |
| horizon | The certified horizon is 252 days | Multi-year backtests, and anything keyed on volatility dynamics beyond one year |
| decay-shape | Volatility memory decays faster than real past the first day | Strategies whose edge depends on volatility clustering past lag 8: volatility forecasts over more than a week, and vol targeting and risk parity on a one-month or longer estimate |
| scenario-magnitude | A driven scenario moves prices at about half to four-fifths of the real size | Sizing a scenario's impact rather than detecting it |
| macro-range | The endogenous macro state cannot reach its own crisis regimes | Studying inflation regimes or policy crises from the endogenous economy alone |
| roster-concentration | A concentrated roster is granted on pt-v19 only, for four sector mixes and the shape rows | Citing the certification for a concentrated roster on a level or crisis row, past 504 days, on any preset but pt-v19 (pt-v20 and the default, pt-v21, included), or for a sector mix other than the four measured |
| external-flow | Every statistic is measured under the preset's own shock flow, and the news, economic shocks, earnings revisions, VIX levels and ticks outside the regular session a host adds are outside it | Citing the certification, the two-year panel or the long-run criteria for a market a host drives with its own news or economic shocks beyond the flow in CALIBRATED_FLOW |

The envelope doesn't name these limits, though they're measured too:
- By default nothing in the market learns your pattern and trades against you. [Populated mode](https://docs.tradefloor.dev/populated.html) adds traders that do. When they trade ahead of a predictable series of orders, the series costs about 2.4% more than it does in isolated mode, far less than published studies find in real markets.
- The impact of order flow injected into a market grows more slowly with its size than published studies find, with Y of 0.06 to 0.63 against 0.5 to 1.
- With the VIX held at 65 the market is 5.2 times as volatile as at a VIX of 5, against a real 6.2.
- Some rows pass close to a limit. The daily correlation of the index with a Treasury bond's return reads -0.14 against a real -0.16, near the floor of -0.36.

## The published grade

The registration, the scripts that graded pt-v22, their inputs and the outputs of the grading run are in [validation/pt-v22/](https://github.com/simoncoombes/tradefloor/tree/main/validation/pt-v22) in the library repository. pt-v21's grade stays in [validation/pt-v21/](https://github.com/simoncoombes/tradefloor/tree/main/validation/pt-v21). [validation/README.md](https://github.com/simoncoombes/tradefloor/blob/main/validation/README.md) says how to check the grade on a laptop or run it again. The one-year table is also published as data in [envelope.json](https://docs.tradefloor.dev/envelope.json), and [docs/STATISTICS.md](https://github.com/simoncoombes/tradefloor/blob/main/docs/STATISTICS.md) defines each statistic and gives the source of its band.

## What the grade does not say

A pass means the model matches real markets on these figures and wasn't tuned to the exam. It doesn't predict real returns, and some behaviors aren't checked at all. The coefficients pt-v22 carries over from earlier presets were chosen with a scoring rule over statistics that overlap the one-year table. The table helped choose those coefficients, so it's weaker evidence than the rows graded on exam seeds.
