Research loops: autonomous ML experiments, and a C++ feed handler

The problem#

Autonomous research loops are easy to build and hard to trust. A loop that proposes experiments, runs them, and keeps the winners will always produce a rising chart. The question is whether anything real is rising, or whether it is selecting noise with great efficiency.

This project is three loops and the machinery around them:

• numerai-autoresearch, an autonomous research loop over the Numerai tournament, adapted from the autoresearch pattern (an agent edits a training file, a frozen harness scores it, a keep rule decides).
• auto-kaggle, a loop that enters tabular Kaggle competitions, trains, submits, and climbs without supervision.
• mlops-auto, a shared control plane whose main job is to tell me when either loop has stopped improving while still reporting green.

Plus Titan, a low-latency market-data ingester in C++20, which is a different kind of research: how fast can one machine turn a websocket frame into an order book update.

Architecture#

flowchart TB
    subgraph Clock["Schedulers"]
        GHA["GitHub Actions<br/>3-day Numerai cycle"]
        MC["Modal cron<br/>Kaggle sweeps + dispatch"]
    end
    subgraph Ideas["Idea sources"]
        LLM["LLM proposals<br/>complete train files"]
        TPE["Optuna tuner"]
        RND["random control arm"]
    end
    BANDIT["bandit allocates slots"]
    HARNESS["frozen harness<br/>splits, scorer, time budget"]
    KEEP{"keep rule<br/>beat champion by<br/>one seed std"}
    GATE{"holdout gate<br/>counted reads"}
    STATE[("ledgers in git<br/>runs in W&B")]
    EFF["mlops-auto efficacy check<br/>NO_SUPPLY / NO_CLEARANCE / REPETITION"]

    GHA --> BANDIT
    MC --> BANDIT
    LLM --> BANDIT
    TPE --> BANDIT
    RND --> BANDIT
    BANDIT --> HARNESS --> KEEP
    KEEP -->|keep| GATE
    KEEP -->|discard| STATE
    GATE --> STATE
    STATE --> EFF

Numerai. A daily GitHub Actions wake-up runs a full cycle every three days: ingest live scores, run up to six experiments on Modal CPU containers, apply the keep rule, gate new candidates on holdout, retrain when due, and publish a W&B report. The harness that owns the data splits, the scorer, the seed count and the time budget is under a byte-identity lock. The holdout eras live on a separate storage volume that research functions do not mount, so they are physically unreachable, not merely off-limits.

Kaggle. Four planes, none always-on: code in git, Modal cron as the clock, GitHub Actions for agent work, and compute on my PC first, Kaggle's free notebooks second, and Modal as paid failover. The hosts share heartbeats and trial claims, so Modal returns without spending anything while the PC is alive. Search is two-tier: Optuna tunes hyperparameters continuously and cheaply; a Claude Code agent makes structural changes (features, model family, CV scheme, ensembling) a few times a week and opens a PR for each.

mlops-auto. Stdlib-only, read-only, and network-free when it renders a verdict. It names three futility modes separately because their fixes are unrelated: nothing was proposed, things ran but nothing cleared the bar, or proposals duplicated what was already tried.

Stack choices and why#

• A keep rule set by measured noise. Numerai's per-era correlations are tiny and noisy, eras overlap because the target looks weeks ahead, and the data is non-stationary. The loop runs the unmodified baseline across three seeds first, and the spread across seeds becomes the bar: a candidate is kept only if it beats the best result in its lineage by at least one seed standard deviation. Most experiments are supposed to be discarded.
• A purged three-way split and a counted holdout. Search scores are selected, so they are biased upward. The gate promotes only if holdout agrees with the search claim, and every holdout read is counted against a fixed budget.
• Complete files, not patches. Each proposal is a full training file stamped with the hash of the champion it was written against. Stale proposals are retired without running. LLMs produce malformed patches; complete files fail legibly.
• A bandit over idea sources, with a random arm. Discounted Thompson sampling allocates slots between the LLM, the tuner, a templated ensemble arm and a random control. The random arm is how I find out whether the "smart" arms beat chance.
• CPU on Modal, priced from the bill. A local GPU never finished a run. An 8-vCPU vs 16-vCPU benchmark on the same seeds produced identical metrics at about half the billed cost, so the loop moved to 8. Spend is measured against Modal's own billing report, because the internal estimate ran well below the invoice until it was fixed.
• Kaggle safety in code. Nothing fabricated is ever submitted: stale prediction files are deleted before training, every submission is validated against the sample file, unreadable quota fails closed, and a trial truncated by its time budget can never become champion because its score is systematically optimistic.

What broke / what I learned#

Green and idle. Both loops failed silently for weeks while every guard worked as designed. Numerai accepted 0 of 35 candidates over 18 nightly cycles; auto-kaggle accepted 0 of 205 trials in one stretch. Both reported green, because every existing guard was aimed at false progress (too many keeps, too many crashes) and none at the loop not moving at all. mlops-auto exists to make green mean "the improvement mechanism ran", not "nothing threw".

A cron on the wrong branch never runs. GitHub fires scheduled workflows only from the default branch. Four agent workflows sat on a feature branch, and the Kaggle structural-search plane produced 0 of 799 expected runs with no error anywhere. The fix is a 20-line CI check that every scheduled workflow exists on the default branch.

NO-OPs are poison. Several experiments silently reverted the champion and "tested" an old baseline, and the summariser then learned false negatives from them. Six of the 25 experiments on the second harness version were flagged NO-OP: the change never reached the model. These rows are now excluded from evidence.

Claimed is not realised. Candidates that looked strong on search collapsed on holdout. Publishing is guarded mechanically: the write-up generator cannot see a raw score, only a claim that was confirmed out of sample.

The full story of the first 73 Numerai experiments is in What 73 autonomous experiments taught me.

Titan: a C++20 market-data ingester#

Titan reads a public level-2 crypto market-data websocket feed and maintains in-memory order books, with tick-to-book latency measured on every update. It also feeds a Python trade service in the same repo, where strategies propose trades in paper mode. Nothing reaches the exchange until I approve a proposal, and live trading needs two separate switches: live mode in the config, and a second one, off by default, before the bot may approve its own orders.

flowchart LR
    WS["websocket feed<br/>TLS"] -->|"read, stamp T1"| A
    subgraph A["Thread A (pinned)"]
        P["simdjson On-Demand parse<br/>into 64-byte records"]
    end
    A -->|"try_push"| R[("lock-free SPSC ring<br/>65,536 slots")]
    R -->|"try_pop"| B
    subgraph B["Thread B (pinned)"]
        BOOK["flat order book per product"]
        H["latency histogram, stamp T2"]
    end

The hot path, from frame read to book update, may not use mutexes, heap allocation, node containers like std::map, syscalls other than the socket read, or floating point for prices and sizes. A counting global operator new in the tests and a ThreadSanitizer build enforce part of that. Records are exactly one cache line. The ring is hand-written: each side caches the other's index and only re-reads it when the ring looks full or empty, so in steady state neither side touches the other's cache line.

Two things I learned the hard way. First, the order book keeps the best price at the back of each sorted array, because updates cluster near the top of book, and with the best at index 0 every top-of-book insert would memmove the whole side. Second, snapshots arrive best-first, the worst possible order for that layout, so a snapshot is loaded unsorted and sorted once at the end, which took the slowest snapshot apply from 108 ms to 11 ms.

Measured on WSL2 with pinned threads and a TSC clock: replay of a 15-minute capture runs at 1.3 µs p50 and 13.6 µs p99 tick-to-book, and live at 7.7 µs p50 and 38.8 µs p99. Replay with the recorded spacing matches live, so the gap comes from idle time between frames rather than the network. A one-hour live run had zero sequence gaps and zero parse errors.

Status#

The Numerai loop runs every three days and is on its third harness version, which replaced the one-seed-std keep bar with a paired per-era test after the old bar proved unreachable. Auto-kaggle runs continuously on tabular Playground competitions. mlops-auto's efficacy checks and publishing guard are built and tested; its shared ledger and dispatcher are deliberately unbuilt until a score moves, because a framework layered over two flat loops produces flat results with better logging. Titan has its first four milestones done; a known limit is that fixed-point sizes overflow on a few very low-priced coins.