Autoresearch for trading

Earlier this year Andrej Karpathy released autoresearch: a single-GPU LLM training script, a fixed evaluation, and an AI agent that edits the training file, runs for five minutes, keeps the change if val_bpb improved, and discards it otherwise. The human writes PROGRAM.md in plain English; the agent runs hundreds of experiments overnight. Pointed at nanochat, it found real improvements in a codebase that was already well optimized.

The design is the interesting part, not the LLM. It works because the loop has three properties: one editable file, a fixed harness the agent cannot touch, and a single number that decides keep-or-discard. Nothing about that is specific to training language models. So I built the same loop for a problem I actually care about: swing-trading 2x-leveraged single-stock ETFs (NVDL, TSLL, AAPU, and friends) — instruments that compound spectacularly when timed and lose 90%+ when held naively.

This post is about what the loop looks like translated to trading, and what 83 experiments taught it in its first day. (Standard disclaimer, and an honest one: this is a research program that measures backtests. It never touches a brokerage, and nothing here is investment advice.)

The translation

The agent may edit exactly one file: strategy.py, a single function that maps features (prices, volume, VIX, sector indices — anything computable at that day's close) to portfolio weights for the next day's open. Everything else — execution model, transaction costs, the metrics battery, the train/validation/holdout splits, the judge — is frozen. Experiments run in isolated git worktrees; a candidate either earns a merge to main or is deleted, and either way the hypothesis, scorecard, and a one-sentence lesson go into an append-only journal.

Markets punish optimism much harder than language modeling does, so the keep-or-discard rule can't be a single number. A change is kept only if it improves validation CAGR and clears hard gates: drawdown and Sharpe floors, turnover limits so it stays a swing program, and the one I care most about — the control cohort.

The survivorship trap, and the gate that fixes it

The twelve names with real 2x ETFs have them because they were the biggest winners of the last two decades. Any strategy tuned on NVDA-went-up-300x inherits a colossal hindsight edge. So the harness backtests every candidate twice: once on the winners, and once on ten deliberately mediocre large caps (INTC, GE, BA, T, ...) as if 2x products existed for them. Strategy logic must be symmetric across all 22 names, and the edge has to survive on the losers, or the change is rejected no matter how good the headline number looks.

This single gate did more to keep the research honest than everything else combined. Wave after wave, mechanisms that looked brilliant on the tradable cohort — VIX throttles, SPY regime filters, breakout gates — turned out to carry zero information on the control names. They weren't finding edge; they were re-deriving "buy the stocks that went up."

What the loop found

The baseline to beat: equal weight across the names, each gated by its 200-day moving average — published folk wisdom. On validation (2015–2020) it compounds at 88.4%/yr but rides a −61.7% drawdown, and on the control cohort it has essentially no edge (Calmar 0.03). Classic survivorship beta.

Thirteen waves of parallel experiments later, the standing champion combines six accepted mechanisms, each found in a different wave:

• Rolling Kelly sizing — the only wave-1 mechanism with real edge on the losers cohort;
• volatility-expansion de-sizing — cut exposure when short-term vol runs ahead of long-term vol; the tail solution;
• conviction-conditional full deployment — spend gross exposure only when breadth of signal supports it;
• rebalance bands — trade only when targets drift meaningfully;
• sector relative-strength as a brake — cut laggards-vs-their-sector, but never boost leaders (the symmetric version broke both risk gates);
• a market-breadth throttle on top of the brake — the final composition.

Champion scorecard as of milestone #8: validation 75.4% CAGR at −30.9% max drawdown (Calmar 2.44 vs the baseline's 1.43), with genuine control-cohort edge (~15x the baseline). On the untouched holdout (2021→now, including the 2022 bear): 50.8% CAGR at −37.5% vs the baseline's 53.4% at −66.5% — 95% of the baseline's growth at 29 points less drawdown, and roughly double its Calmar. The loop is still running, so the dashboard's numbers will have moved by the time you read this.

The agents will overfit if you let them

The most instructive moment wasn't an accept. Around experiment 60, a lineage of regime-conditional tweaks pushed validation Calmar from 1.87 to 2.28 over two waves — while the holdout, checked only at scheduled milestones, fell for the second consecutive time and breached the drawdown mandate. The protocol called it: the entire lineage was declared overfit and reverted, five accepts rolled back at once. Validation-holdout divergence is the trading equivalent of a train/val gap, and no amount of agent enthusiasm is allowed to argue with it.

That's the real lesson of the autoresearch pattern, and it transfers to any domain: the value isn't that the agent is smart, it's that the harness is incorruptible. The agent cannot edit the judge, cannot see the holdout, cannot merge its own work, and cannot delete the record of a failure. Under those constraints, even overnight-scale trial and error compounds into something honest. Remove any one of them and you get a very fast overfitting machine.

The program's current state is itself a finding: every constraint — control floor, drawdown envelope, dev-consistency — is now binding simultaneously. That's the definition of an efficient frontier. Pushing growth further is no longer an engineering problem; it's an owner decision about which constraint to relax, priced in the journal to the second decimal. (Exactly one such decision has been taken: the control floor moved 0.45 → 0.42, which admitted a candidate that was better on both growth and drawdown, and the journal records the price — a thinner survivorship margin — next to the ruling.)

The whole thing — harness, strategy, journal, this dashboard — is on GitHub. Fork it, change PROGRAM.md, and point your own agents at it.