Skip to the page
MODEL AND VALIDATION/HOW IT IS MEASURED

How it is measured

Each validation criterion and its acceptance range is committed before the grade. The grade runs on holdout seeds the tuning never used, and the scripts and their output are published. pt-v22 passed 190 of the 190 registered criteria that count toward its grade. On the one-year table, all 19 statistics fall inside acceptance ranges read from real market data.

What validation establishes

  • pt-v22 passes 190 of the 190 pre-registered validation criteria that count toward its grade, on holdout seeds the tuning never used. The criteria cover crash sizes in replays of 2008 and 2020, the frequency of large index falls, the duration of high-VIX spells, rate and bond moves, earnings in a contraction and the cost of trading size. Each has an acceptance range fixed before the grade.
  • The leak-finding strategies earn about nothing. The market doesn't pay you for trading on a late headline, past prices alone, value or momentum screens, published economic data or a rate decision.
  • Over one year, all 19 statistics of the one-year table fall inside acceptance ranges read from real market data. The rows for large index falls and for the index's drift are read over 360 seeds, as the grade defines them. tf.envelope.check() answers for runs of up to 252 trading days.

What validation does not establish

  • That a strategy that makes money here will make money in a real market. Matching some statistics of real prices doesn't put real opportunities into simulated ones, so a profit here is evidence about the simulator.
  • That anything outside the registered rows and the one-year table is realistic. Some behaviors aren't checked at all. The named gaps lists measured failures and the uses each one rules out.
  • That a result carries to another preset, or past 252 days on a statistic the long-run rows don't cover.

The scripts that grade every row, the one-year table and rows D1 and V1 are in the library repository, in validation/pt-v22/, with the outputs they wrote (The published grade).

Registration, then one grade

1Measure real marketsThe criteria are built on statistics from the S&P 500 since 1928, the VIX since 1990, 40 large US stocks, Treasury and corporate yields, Federal Reserve decisions and Shiller's long-run earnings. One example is how many days a decade the index falls more than 5%. Each statistic becomes a criterion with an acceptance range wide enough to cover how much the real figure moves from one decade to the next.
2Write the rulesMost behaviors come from a published source, such as the square-root cost of size. A new rule goes in switched off, and a test runs one fixed market on all the shipped presets to check that no other preset's output changed by a single bit.
3Screen on practice seedsCandidate settings run in grids over many simulated histories. Each market comes from a seed, and the seeds are split in two. Practice seeds are used while choosing settings, and exam seeds are kept back for the grade.
4Try to cheat itSome checks are strategies built to find leaks. They trade on a headline 5 minutes late, use rules that read only past prices, run value and momentum screens, time the market on published economic data and trade on a rate decision. A strategy passes only if it earns about nothing, as it would in a real market. An independent audit found two leaks, and both were fixed in the model.
5RegisterThe checks, their pass marks and the exact settings to be graded are committed. Nothing on the list can change after the results come in.
6Grade onceThe model runs once on the exam seeds and has to pass every row. If it fails one it doesn't ship. Any change then needs a new registration and fresh seeds before another grade.
7FreezeA model that passes ships as a preset whose values never change after release. It carries a fingerprint of its coefficients and known-answer digests, and each release checks those on five platforms.

The seeds

The final settings for pt-v22 were chosen on practice seeds. The grade ran on fresh exam seeds that no earlier run had used. grade-seeds.json lists them with the proof that they were unused. They cover 270 histories of 21 years with nothing imposed, replays of 2008 and 2020 with the real VIX, the real 2020-21 and 2022 economies driven through the model, the packaged 2008 recession (recession_2008) and the leak-finding strategies.

The registered rows

Each row is a validation criterion with an acceptance range. A crash size has to be within 30% of the real value, a frequency has to be half to twice the real rate, and any edge a trading strategy could learn has a hard limit. pt-v22 passes 190 of the 190 that count toward its grade; the table shows each reading against its pass mark.

A rows replay 2008 and 2020, B rows count events and long-run figures over the free histories, C rows are leak and consistency checks and the cost of size, R rows cover rates, bonds and the driven 2022 market, S rows the 2008 recession, D2, F1 and L1 the driven 2020 path, V1 variance over two and five years, and D1 the one-year table below. Where a row has both replays, 2008 comes before 2020. The rows for the contracts, the leak checks and the rest of pt-v22's registration are summed up below the table, one line for each group.

RowWhat it checkspt-v22RealPass markWithin it
A1Worst month's volatility in the 2008 and 2020 replays within 30% of real100 / 98.384.3 / 94.5Within 30%yes
A2Maximum drawdown in the 2008 and 2020 replays within 30% of real0.444 / 0.3430.568 / 0.339Within 30%yes
A3Peak stock correlation in the 2008 and 2020 replays within 0.15 of real0.813 / 0.8640.748 / 0.872Within 0.15yes
B1Share of sessions with the VIX above 300.0680.0821/2x to 2xyes
B2Mean length of a fear spell above VIX 30, sessions24221/2x to 2xyes
B320% bear markets per decade1.691.161/2x to 2xyes
B410% corrections per decade4.622.841/2x to 2xyes
B5Sessions down more than 5% per decade7.26.21/2x to 2xyes
B6Share of sessions with the VIX under 150.3080.3261/2x to 2xyes
B7Index annual volatility, %18.418.1Within 20%yes
B8Long-run index return, % a year76.25Within 2 points of the targetyes
C1Crash rate in years 3-21 against years 1-21.1212/3x to 1.5xyes
C2Histories touching the VIX ceiling, of 27050At most 9 of 270 (1 per 30)yes
C3Edge from reading a headline 5 ticks late, bp17.90Under 20 bpyes
C4a65-minute lag-1 autocorrelation of print returns, median name; Roll spread over quoted−0.044 / 1.50 / 1At or above −0.05, or Roll at most 2x quotedyes
C4bThe price-only rule furthest over its line on tf-suite-2026.1 (named in worst): median points over buy-and-hold, markets beaten of 20−3 / 80 / 10Every rule at most +5 points and 14 of 20yes
C5Stock-level (idiosyncratic) variance ratio at 60 sessions, median name, 90 histories of 2,660 sessions (three seed sets of 30; the median over the histories)0.9520.9240.8 to 1.05yes
C6Rank IC of the value signal on published fundamentals against the next 20 sessions: whole history, first 60 sessions0.002 / 0.00240.0093 / 0.0093−0.03 to 0.05; −0.03 to 0.05yes
C7Rank IC of 12-1 and 6-1 month momentum against the next 20 sessions0.0053 / 0.00110.0267 / 0.0413−0.04 to 0.095; −0.02 to 0.1yes
C8Lo-MacKinlay one-day loser-minus-winner book, bp a day−1.06−1.74−6.4 to 2.9yes
D2The driven 2020-21 market: the index's maximum drawdown and the sessions from the pre-crash high back to it, medians over histories0.382 / 1680.339 / 1260.237 to 0.441; 63 to 252yes
F1The driven 2020 path's fast crash: the high to the lowest close within 60 sessions, and the sessions it took0.382 / 31.50.339 / 230.237 to 0.441; 12 to 46yes
L1Look-through: sessions from the index's trough to the aggregate earnings level's trough on the driven 2020 path71681 to 136yes
C10Timing rules on published macro data against buy-and-hold, and the drift after a published turn0.0059 / 0.5 / −1.25 / 0.988−2 / 0.324 / −5.9 / 4.3Every rule at most +1.0 point a year and ahead in at most 2/3; drifts no larger than the S&P's after NBER turnsyes
R5The driven 2022 market: the index's maximum drawdown0.2210.2540.178 to 0.33yes
R7aThe equal-weight index's mean log move from the first price readable after a changed policy rate to the end of the first 65-minute bar, less the mean first bar of all days, bp: after a hike, after a cut1.78 / −0.7270 / 0Each within 5 bp, or 2 se where wideryes
R7bThe audit's rate-news agent through tf.evaluate: mean annual excess over holding in points, histories ahead−1.22 / 20 / 15At most 0 points a year, ahead in at most 20 of 30yes
S1aThe packaged recession: share of the index's paired log fall, at its lowest, won back 252 sessions later, mean over seeds0.880.620.45 to 1yes
S2The packaged recession: the index's own rise in the 252 sessions after its low, percent, mean over seeds60.66925 to 80yes
S1bThe packaged recession: seeds whose cycle has left contraction and trough within 24 months of the onset180180Every seedyes
V1The index's variance ratio at two and five years relative to one, years 2-21 of the pooled free histories0.909 / 0.8250.93 / 0.870.75 to 1.15; 0.55 to 1.2yes
C9The cost of size in the agent-facing book: exponent and coefficient of the average cost against Q/V, in sigma units0.459 / 0.4740.5 / 0.5Exponent in 0.4 to 0.7 and coefficient in 0.33 to 0.67yes
R1Sd of the 2-year Treasury yield's daily change, bp (FRED DGS2 2015-2025)5.35.233.65 to 6.8yes
R2Sd of the 10-year Treasury yield's daily change, bp (FRED DGS10)5.895.414.54 to 6.27yes
R4Daily correlation of the roster index with an IG bond's return (minus the corporate yield's change; SPY against LQD)0.2930.2720.15 to 0.39yes
B9Sd of annual index log returns, years 2-21, percent; the start-up drift: sd of the first 60 sessions' index return over the steady state's, across 50 markets16.8 / 1.1417.4 / 113.9 to 20.9; 0.6667 to 1.5yes
D1Ruled bands of the one-year realism table in, on all four cellsallallEvery band inyes
R6The driven 2022 market: the market P/E's log change per 100 bp of the corporate yield, monthly averages−2.08−5.43−10.86 to −2.72 percentno

The other rows that count toward the grade, each listed with its reading in validation/pt-v22/:

  • 32 rows for the futures, options and margin: all within their marks.
  • 59 further rows registered for pt-v22: all within their marks.
  • 44 leak checks: all within their marks.
  • 11 rows carried over from pt-v21's grade: all within their marks.
  • 6 rows restated at registration 19: all within their marks.
  • The exploit search: 0 exploits found over 509 trading rules.

Shown for comparison and not counted: R6, OF4, SV1, SV2, SV3, SV4, SV5_9D, SV5_6M, SV5_1Y, SV6_median, SV6_sd, SV7, SV8, SV9, OB1r, OB1v, OB1d, OB2r, OB2v, OB2d, OB3r, OB3v, OB3d, OB4r, OB4v, OB4d, OB5, OB6, XD4, VF1, VF2, VF3, VF4_F1, VF4_F2, VF4_F4, VF5, VF6_h21, VF6_h63, VF6_h126, VF7, VF8.

The one-year table

The second test is a panel of statistics measured over one year, taking the median across 30 seeds. Each is compared with a band read from the longest real record the statistic allows. The 14 shape statistics and crisis dispersion are measured on a fixed roster of 40 companies. The four index rows use a roster that changes with each seed, since a crash rate measured on one roster describes only that roster.

StatisticWhat it checksReal bandpt-v22In band
annualised_vol_pctHow much prices move in a year12 to 4119.9yes
excess_kurtosisHow fat the tails are−13 to 246.72yes
return_acf1Does yesterday predict today−0.07 to 0.06−0.0049yes
abs_return_acf1Does a wild day follow a wild day0.02 to 0.170.105yes
abs_return_acf5The same, one week apart−0.03 to 0.10.0281yes
abs_return_acf20The same, one month apart−0.05 to 0.060.0044yes
cross_sectional_corrHow much names move together0.09 to 0.490.311yes
volume_abs_return_corrDo big moves come with volume0.35 to 0.640.593yes
leverage_effectDo falls raise volatility−0.11 to 0−0.0434yes
volume_change_acf1Does volume mean-revert−0.3 to −0.2−0.241yes
corr_asymmetryDo names couple more when falling−0.15 to 0.230.0646yes
corr_asymmetry_laggedThe same, one day later−0.15 to 0.330.0734yes
sector_excess_corrDo industries move together0.04 to 0.230.126yes
corr_persistence_acf1Does correlation stay high after a panic−0.48 to 0.690.0505yes
crisis_sector_dispersionDoes one industry lead a crisis0.79 to 1.741.32yes
index_drift_pctWhich way the index goes1.1 to 10.37.33yes
fear_gauge_dn1How much fear a bad day buys0.39 to 3.031.98yes
fear_gauge_dn3The same, on a much worse day2.6 to 9.585.75yes
index_tail_dn3_pctHow often the index falls hard0.64 to 2.341.25yes

The table certifies runs of up to 252 trading days. At 504 days pt-v22 holds all 15 statistics of the two-year panel inside bands read from real data, but tf.envelope.check() still certifies one year only. For longer runs the evidence is the long-run rows above. Some statistics pass on the low side, such as volatility clustering, which is close to real one day apart and fades faster than real after that. The named gaps below say what that rules out.

Ask the package before you lean on a statistic:

import tradefloor as tf

verdict = tf.envelope.check(horizon_days=504, statistics=["abs_return_acf20"])
print(str(verdict).splitlines()[0])
OUTSIDE the envelope

The verdict goes on to name each reason. Here the horizon is past the certified 252 days, and lag-20 volatility clustering is one of the named gaps.

The named gaps

Each gap below is something the model gets wrong, with the uses it rules out. tf.envelope.check() refuses a question that asks for one of them.

GapWhat falls shortDo not use it for
horizonThe certified horizon is 252 daysMulti-year backtests, and anything keyed on volatility dynamics beyond one year
decay-shapeVolatility memory decays faster than real past the first dayStrategies whose edge depends on volatility clustering past lag 8: volatility forecasts over more than a week, and vol targeting and risk parity on a one-month or longer estimate
scenario-magnitudeA driven scenario moves prices at about half to four-fifths of the real sizeSizing a scenario's impact rather than detecting it
macro-rangeThe endogenous macro state cannot reach its own crisis regimesStudying inflation regimes or policy crises from the endogenous economy alone
roster-concentrationA concentrated roster is granted on pt-v19 only, for four sector mixes and the shape rowsCiting the certification for a concentrated roster on a level or crisis row, past 504 days, on any preset but pt-v19 (pt-v20 and the default, pt-v21, included), or for a sector mix other than the four measured
external-flowEvery statistic is measured under the preset's own shock flow, and the news, economic shocks, earnings revisions, VIX levels and ticks outside the regular session a host adds are outside itCiting the certification, the two-year panel or the long-run criteria for a market a host drives with its own news or economic shocks beyond the flow in CALIBRATED_FLOW

The envelope doesn't name these limits, though they're measured too:

  • By default nothing in the market learns your pattern and trades against you. Populated mode adds traders that do. When they trade ahead of a predictable series of orders, the series costs about 2.4% more than it does in isolated mode, far less than published studies find in real markets.
  • The impact of order flow injected into a market grows more slowly with its size than published studies find, with Y of 0.06 to 0.63 against 0.5 to 1.
  • With the VIX held at 65 the market is 5.2 times as volatile as at a VIX of 5, against a real 6.2.
  • Some rows pass close to a limit. The daily correlation of the index with a Treasury bond's return reads -0.14 against a real -0.16, near the floor of -0.36.

The published grade

The registration, the scripts that graded pt-v22, their inputs and the outputs of the grading run are in validation/pt-v22/ in the library repository. pt-v21's grade stays in validation/pt-v21/. validation/README.md says how to check the grade on a laptop or run it again. The one-year table is also published as data in envelope.json, and docs/STATISTICS.md defines each statistic and gives the source of its band.

What the grade does not say

A pass means the model matches real markets on these figures and wasn't tuned to the exam. It doesn't predict real returns, and some behaviors aren't checked at all. The coefficients pt-v22 carries over from earlier presets were chosen with a scoring rule over statistics that overlap the one-year table. The table helped choose those coefficients, so it's weaker evidence than the rows graded on exam seeds.