The 82% wall — what 97 experiments taught the loop

The first post described the setup: a Karpathy-style autoresearch loop pointed at a swing-trading backtest. One editable file (strategy.py), a frozen harness, hard risk gates, and AI agents running experiments in git worktrees around the clock. That post was written when the program was 83 experiments old and a day young.

The program has now run 97 experiments across 19 waves, under three different mandates, with ten holdout checkpoints and three reverts — and it has hit a wall that I believe is the honest frontier of this universe. This post is the retrospective: what was tried, what compounded, what blew up out of sample, and what actually mattered. (Same disclaimer as always: this is a research program that measures backtests. Nothing here is investment advice.)

The scoreboard first

Everything is measured against the strategy a reasonable person would write first: hold every 2x name above its own 200-day moving average, equal weight. That baseline is genuinely hard to beat on growth — because the twelve tradable names are the biggest winners of two decades, and the baseline rides them with maximum hindsight.

The champion keeps 95% of the baseline's out-of-sample growth at 29 points less drawdown, and — the constraint that shaped the entire program — it still earns a Calmar of 0.44 on a control cohort of ten large caps that never got 2x ETFs because they stagnated or fell (INTC, GE, BA, T, …). The baseline scores 0.03 on that cohort. That number is the difference between an edge and a survivorship artifact.

Act one: buy the risk budget

The first mandate scored validation Calmar with a hard −30% drawdown gate. The opening move was a thirteen-experiment parallel wave — one mechanism each, every quant idea you would reach for first:

All thirteen were rejected — and the wave was still the most valuable thing the program ever ran, because it produced a map. Market-level signals cannot fix a 2x single-stock tail (the deepest drawdowns are earnings gaps the index never sees). Momentum tilts concentrate into exactly the names that gap. And two mechanisms were orthogonal solutions to different problems: the vol-expansion switch solved the tail, and Kelly sizing was the only thing that made money on the losers cohort.

exp/015 — the hybrid of those two — became the first strategy ever to pass every gate. From there the loop laddered small, boring, orthogonal accepts: a recovery channel for names 20–45% under their highs (+0.01), a no-trade band that cut turnover 18% (+0.05), and then the single largest gain of the entire program: averaging the Kelly estimate across three horizons (21/63/126 days), worth +0.36 Calmar at lower drawdown. Not a new signal — variance reduction on the noisiest estimate. Estimation-error reduction beat signal addition, decisively, and nothing else ever came close.

The first trap

Waves 5 and 6 found free money: crank gross exposure in "calm bull" and "fresh bull" regimes (SPY above its 200-day, VIX low, or just after an up-cross). Validation Calmar marched 1.87 → 1.93 → 2.05 → 2.18 → 2.28. Every accept passed every gate.

Then the holdout milestone — a split the agents never optimize against, only checked after accepts — told a different story:

Validation climbing while holdout falls is the signature of overfitting, and the program's own charter (§9) forced the response: five accepted experiments were declared an overfit lineage and reverted. The 2020–21 fresh-bull pattern the boosts exploited simply didn't repeat benignly in the 2022 bear. A validation-only loop would have shipped that strategy.

Act two: spend it on growth

At this point the risk problem was solved but growth sat at ~54% CAGR against the baseline's 88%. Mandate v2 (my call, as owner) flipped the score to raw validation CAGR and moved the drawdown floor to −45%: the program had proven it could buy risk control cheaply, now spend some of it.

The unlock was structural: deploy the full book proportional to signal strength (water-filling under a 25% per-name cap) — but with a thin-signal guard that throttles gross when few names have conviction. Flat gross increases were tested three ways and always collapsed the control cohort; conviction-conditional gross never did. The ladder ran 65.8% → 70.1% → 71.9% in three waves, and holdout milestone #6 recorded the program's headline result: 53.2% holdout CAGR vs the baseline's 53.4% — out-of-sample growth parity — at −45% vs −67% drawdown.

Then sector indices went in as features, and the single biggest growth lever of the program appeared: sector relative strength, worth +10.6pp CAGR as a symmetric tilt — and unusable, because boosting sector leaders concentrates into high-beta winners and breaks both the drawdown and the control gate. The fix took two waves to find and became my favorite result of the run: use relative strength only as a brake. Cut the laggards, never boost the leaders. exp/077 (+1.9pp, control intact) was the only variant in the family that improved growth and control simultaneously. Composed with a breadth throttle (exp/082, after I lowered the control floor from 0.45 to 0.42 and locked it), the champion reached 75.4% @ −30.9%.

The wall

After exp/082, four consecutive waves rejected everything. The pattern was remarkably consistent: an 81–82% validation CAGR class exists in this universe — the loop reached it through eight independent configurations (hot vol switches, leader tilts, guard retunes, gross ladders) — and every single one lands the control cohort at 0.34–0.41 against the locked 0.42 floor. The extra nine points of growth are winners-only edge. That is the 82% wall: not a failure to find growth, but a refusal to accept growth that doesn't survive on the losers.

The stretch also produced my favorite bug: a cleanup pass "accepted" an experiment on a CAGR improvement of 4×10⁻¹⁶ — float-summation noise from a code path that changed nothing. The judge now requires wins to exceed 0.01pp. Agents will ladder on numerical dust if you let them.

Act three: buy back the drawdown

Mandate v3 inverted the problem once more: hold the growth, score the drawdown. The research-driven wave (CPPI governors, emergency vol caps, defensive sleeves) mostly falsified the literature at 2x: a "crisis-only" volatility cap is impossible when the book runs 55–65% vol normally, and a drawdown governor keyed to a book proxy de-grosses in recoveries.

One idea survived the validation gauntlet: park idle cash in treasuries (IEF). "Bonds as default cash" posted 77.4% CAGR with the largest control margin ever recorded (0.53) and was accepted under a dominance amendment. Holdout milestone #10 killed it in one look: 44.9% CAGR @ −46.5%. The 2022 joint bond-equity selloff — precisely the caveat in the literature — makes an always-on bond sleeve toxic out of sample. Reverted, with a standing rule for any future attempt: the sleeve must be trend-gated on the bond's own price. The holdout earned its keep twice.

What actually made a difference

Ranked by measured impact across 97 experiments:

1. Estimation-error reduction over signal addition. Tri-horizon Kelly averaging: +0.36 Calmar at lower drawdown. The largest single accept, and it added zero new information.
2. Composing orthogonal mechanisms. Every champion was a composition: tail control (vol switch) × cross-sectional edge (Kelly) × cost control (band) × conditional gross (guard). Mechanisms that shared an axis interfered; orthogonal ones stacked cleanly.
3. Brakes, not boosts. Sector RS as a de-size-only brake passed; as a symmetric tilt it failed everywhere. Fast cuts with slow re-adds hurt (patience on the cut side is load-bearing), regime gross boosts overfit, and every winner-boosting mechanism died on the control gate.
4. The control cohort was the whole game. In 97 experiments, exactly two mechanism families ever lifted loser-cohort performance: Kelly-style risk-adjusted sizing and a sector-vol dimmer. Every other point of CAGR the loop found was survivorship beta in disguise, and the gate caught it.
5. A held-out split with teeth. Two lineages — five accepts in one, a dominant-looking champion in the other — were reverted on holdout divergence. Both were validation darlings. The gap between "passed every gate" and "real" is exactly the width of the data you refuse to optimize against.

And one meta-lesson about running agent loops: the harness is part of the strategy. Agents merged unjudged hybrids until accepts pinned exact files, laddered on float noise until the judge got an epsilon, and cheerfully optimized whatever the score actually measured rather than what I meant. Each patch made the next hundred experiments trustworthy.

Where it stands

The division of labor ended up cleaner than I expected. The agents were relentless at mechanism search — wave 1 alone covered the standard quant playbook in an afternoon, and rejection lessons compounded into genuinely sharp wave plans. What they could not do is move the goalposts: every phase transition (Calmar → CAGR → drawdown, gate calibrations, the locked floor) was an owner decision, made by reading the frontier the loop had mapped and deciding which constraint to price. The loop finds the frontier; you choose where on it to stand.

Within this 22-name universe and this mandate, ~75–76% validation CAGR (holdout ~51% @ −37%) appears to be the honest frontier. The one structural door left is universe expansion — more names with real 2x ETFs means a higher diversification ceiling, which lifts gross and control together. That redefines the baseline and the target itself, so it's a decision for the owner, not the loop.

The live dashboard has every curve and scorecard, and the experiment journal has all 97 entries — every hypothesis, every verdict, every lesson, exactly as the loop logged them.