The chart of an autonomous research loop always goes up. That is the problem.
If a loop proposes experiments, runs them against a fixed validation set, and keeps whatever scores best, the best-so-far line can only rise. It rises whether the loop is finding real signal or harvesting noise. On a problem where the differences between models are smaller than the noise, it will mostly be harvesting noise, and the chart will look wonderful.
I built an autonomous research loop over the Numerai tournament, adapted from the autoresearch pattern: an agent edits a training file, a frozen harness scores it, and a keep rule decides. Over 73 experiments it kept very few. This post is about why that was mostly the right outcome, and the handful of ways it was failing without telling me.
Why Numerai is the hard version of this problem#
In LLM pretraining, running a hundred experiments against a pinned validation shard is nearly harmless. Architecture differences are large relative to noise.
Numerai is the opposite on every axis. Per-era correlations are tiny. Candidate models often differ by less than seed noise. Eras overlap because the target looks weeks ahead, so neighbouring eras are not independent samples. And the data is non-stationary, so what worked last year is weak evidence about next quarter.
Run fifty experiments against a fixed search set, pick the maximum, and you will reliably produce a model that looks excellent and scores close to zero live. The project's instructions to the agent say it plainly: most of your experiments should be discarded, and that is the loop working correctly, not failing.
The machinery that makes "discard" the default#
A frozen harness. One file owns the data splits, the scorer, the number of seeds and the time budget. It sits under a byte-identity lock; the agent may read it and never edit it. The agent changes only the training file: model family, feature set, training target, era weighting, post-processing.
A purged three-way split. Eras are split newest-first into holdout, search and train, with purge gaps between them so overlapping targets do not leak across the boundaries. The holdout eras live on a separate storage volume that the research functions do not mount. They are physically unreachable from research, not just off-limits by convention.
A keep rule set by measured noise. Before any experiment, the loop runs the unmodified baseline across three seeds. The spread of the primary metric across those seeds is the noise floor, and it is the bar. A candidate is kept only if it beats the best result in its lineage by at least one seed standard deviation. Hard rejects come first: too much exposure to any single feature, a non-positive mean correlation, a blown time budget, or a drawdown much worse than the champion's.
A counted holdout gate. A kept candidate still does not become the champion. The gate reads holdout once, counts that read against a fixed budget, and crowns a new champion only if the holdout result is consistent with the search claim. Search scores are selected, so they are biased upward. Consistency is what catches a search win that was really selection noise.
What the 73 experiments found#
The experiments ran on two harness versions, and the results are not comparable across them. I changed the scoring target between them on purpose, and the ledger stamps every row with its harness version so nobody, including the agent, compares the two by accident.
On the first harness (48 experiments), there was one real keep. Weighting training eras by recency with a long half-life, combined with a modest increase in model complexity, improved the primary metric by about 9% over baseline and lowered seed variance, drawdown and feature exposure at the same time. Everything else either failed to clear the bar or broke a constraint:
- Recency weighting alone cut seed variance by about 40% without raising exposure. Shorter half-lives made things worse. Focusing only on the most recent eras degraded performance badly.
- Most ensembling attempts (across seeds, feature subsets or era blocks) pushed feature exposure over the cap or blew the time budget.
- Small feature sets halved correlation. Importance-based feature selection still breached the exposure cap.
- Alternative model families mostly timed out or crashed inside the budget. Gradient-boosted trees were the only family that reliably finished within the constraints.
- One objective change produced a mean correlation of zero with a nonsensically good Sharpe ratio, which is exactly the kind of metric gaming the hard rejects exist for.
On the second harness (25 experiments so far), there were no keeps. The rejections were spread across causes: six over the time budget or crashed, eight on drawdown, four short of the bar, one on exposure, and six that were not experiments at all.
That last category is the interesting one.
Six experiments that never reached the model#
Six of the 25 second-harness experiments were flagged NO-OP: the metrics were identical to an existing row. The proposed change never reached the model.
It got worse before it got better. Earlier, a run of five experiments silently reverted the champion and "tested" an old baseline instead. The step that summarises findings for the next round of proposals then learned false negatives from them: it concluded those ideas did not work, when they had never been tried.
Two fixes came out of that. Each proposal is now a complete training file, not a patch, stamped with the hash of the champion it was written against. If the champion has moved, the proposal is retired without running. LLMs produce malformed patches; complete files fail legibly. And any result identical to an earlier row is flagged and excluded from evidence, so a no-op can never be mistaken for a negative result.
Claimed versus realised#
The number I watch most is not the champion's score. It is the gap between what a candidate claimed on search and what it realised on holdout.
One candidate claimed about 1.78 on the search set and realised about 0.14 on holdout. Earlier candidates showed the same pattern at smaller scale. If search gains stop showing up on holdout, the loop is manufacturing false progress, and the rising chart is a picture of that.
This is also why the publishing step is guarded mechanically. The component that writes public summaries cannot see a raw score at all. It accepts a claim object, and a claim can only assert improvement when an out-of-sample confirmation step verified it. There is no field to put an unconfirmed number into. A flat week still gets published, as a negative result.
The loop was green and idle#
Here is the failure I did not see coming.
For weeks, both of my research loops (this one, and a sibling that enters Kaggle competitions) ran on schedule and reported green. The Numerai loop accepted 0 of 35 candidates over 18 nightly cycles. The Kaggle loop accepted 0 of 205 trials in one stretch. Neither could tell.
Every guardrail I had was aimed at false progress. One fired if the keep rate was too high. One fired if half the runs crashed. One needed more champion models than existed before it could say anything. None of them was aimed at the loop not moving at all.
So I built a small shared control plane whose first job is futility detection. Its rule is that green must mean "the improvement mechanism ran", not "nothing threw". It distinguishes three ways a loop can be stuck, because the fixes are unrelated:
- No supply: the generator produced nothing to test.
- No clearance: trials run and are measured, and nothing passes the rule.
- Repetition: proposals exist but duplicate what has already been tried.
A monitor that only counts trials sees "no supply" and "repetition" as the same thing. And a state it cannot read is an alarm, never a pass, because "I could not check" must never mean "proceed".
The monitor is verified the way I now verify every monitor. Each mode has a committed input that makes it fire and a healthy input that must pass. It writes a timestamped heartbeat, and something dumber outside it fails when that heartbeat goes stale. A futility monitor that fails silently repeats the exact mistake it exists to prevent.
A cron on the wrong branch#
The most embarrassing finding was not statistical.
GitHub fires scheduled workflows only from the default branch. A cron defined anywhere else is stored, displayed in the UI, and never runs, with no error. Four of the Kaggle loop's agent workflows were sitting on a feature branch. Its structural-search plane produced 0 of 799 expected runs.
The fix is about twenty lines: a CI check that every workflow with a schedule exists, identically, on the default branch. Prose in a README did not prevent this. A check in CI does.
Spend is a measurement too#
The experiments run on CPU containers on Modal, and the research budget is authorised against Modal's own billing report rather than an internal estimate. That decision came from the estimate being wrong: it ran well below the bill until I found a default rate baked into the container and repriced each experiment on the client instead.
The same benchmark discipline paid for itself in another way. Running the champion on the same seeds at 16 and at 8 vCPUs produced identical metrics (per-era differences around 1e-16) at about half the billed cost per experiment, and the 8-vCPU run was faster end to end. More threads bought nothing at that feature sampling rate.
There was also a shared-budget failure. Both loops drew on one workspace credit, and when the research budget was computed from the whole workspace invoice, the other loop's spend made each Numerai experiment look several times more expensive than it was. Research stopped for a week. Each app now meters only itself.
What changed in the third harness#
The one-seed-standard-deviation bar turned out to be too strict in a specific way: on the second harness it sat about 15% above the champion, and no candidate cleared it. The holdout gate had a related problem: it rejected a candidate that genuinely beat the live model on holdout, because the candidate's search claim had been even higher.
The third harness version replaced the fixed margin with a paired per-era test against the champion, and judges holdout against a published benchmark on the same eras. Changing the measurement is a deliberate, versioned act, never something the loop can do to itself. Optimising the referee is the one thing the agent is explicitly forbidden to try.
What I took from it#
- A rising best-so-far line is not evidence of anything. Track claimed versus realised.
- Set the keep bar from measured noise, and expect most experiments to be discarded.
- Make the holdout physically unreachable from research, and count every read.
- Treat a no-op as a distinct outcome. Never let it count as a negative result.
- Write a monitor for the loop not moving, not only for the loop moving wrongly.
- Prove every monitor can go red.
Seventy-three experiments and almost no keeps sounds like a failed project. Most of it was the methodology refusing to be fooled. The parts that were real failures were the ones that looked like success.