Scope first: this post is only about AI systems that direct the training of AI models — reading results, changing recipes and data, launching runs, deciding what to try next.
A research system, reduced to its function, converts (compute, a base model) into performance improvement on an external evaluation. So evaluation should measure the conversion — and the honest problem is that every objective conversion metric we tried to define fails on contact: research progress is not monotone (exploration fails on purpose), and any formula is gameable by the loop it is supposed to measure. What survives is the metric implicit in how these decisions actually get made:
How many GPU-hours are humans comfortable handing the system, unattended?
“Comfortable” sounds unscientific. It isn't. It is a revealed-preference measurement of trust, and trust is what actually gates every deployment decision in this field — no lab director consults a benchmark before deciding whether an autonomous loop gets the cluster for a month. They consult their comfort. The scientific question is what that comfort is made of, and why it currently collapses with scale.
1. Why we're comfortable at small scale — and not at large scale
Decompose one research iteration into four elements: hypothesis (what to try) and strategy (how to allocate budget across tries) on the ideas side; execution (running one trial) and infra (the substrate everything runs on) on the implementation side.
At GPT-2 scale — where the field's public demonstrations live, from AutoResearch (GPT-2, 5 min GPU runs), to The AI Scientist (~$15 of compute per paper), to ResearchGym's 24-GPU-hour tasks — the implementation side is nearly free, and that is the entire secret of our comfort: a bad hypothesis costs minutes, failed runs are discarded, noise is averaged away with cheap repeats. Our comfort at this scale is made of the absence of any need for trust: every mistake is reversible and every claim is cheaply re-checkable.
At frontier post-training scale, each element inflates — and each inflation creates a specific fear.
Comfort doesn't transfer across scale because reversibility doesn't. What made autonomy safe at small scale was never the system's judgment — it was that judgment was cheap to verify and mistakes cheap to undo. At scale the scariest failure is silent: a loop that keeps spending real resources while optimizing an internal proxy that has quietly stopped tracking the external target.
2. How trust gets earned: auditability — coding already showed the way
This has played out once in public. In 2023 nobody let a model edit their codebase unattended. Cursor's real invention was not a better model — it was the per-diff accept/reject: every change became a small, individually auditable decision. People reviewed every diff; then they skimmed; then agent mode arrived and they accepted everything. Three years from zero comfort to handing over the codebase — driven not by more auditing but by early spot-checks calibrating trust, after which the trail's value stopped being that it was checked and became that it was checkable.
The Accept/Reject Feature on Cursor gives people a way to audit the AI changes, gradually growing people's trust on AI produced changes.
One disanalogy, and it decides the engineering: a bad diff is revertible — git and CI bound the downside of accepting everything. A bad multi-week training run is not; the GPU-weeks are gone. Reversibility, again, is what doesn't transfer — so an autonomous training loop needs heavier audit infrastructure than a coding agent ever did. Ours has four pieces, one per fear in the table: an immutable substrate the loop cannot overwrite; a frozen constitution it cannot rewrite; policy-only promotion with a full audit trail — every decision replayable by a human; noise-calibrated promotion thresholds so it cannot promote luck. Together they perform the same reduction Cursor performed on coding: “do I trust this system's judgment?” — a matter of faith — becomes “have I checked this audit trail?” — a matter of labor.
3. One Datapoint: An AI ran the post-training loop for a 30B model — with no human in the loop
With that infrastructure, one datapoint. We handed an autonomous loop a 30B Nemotron base and a multi-H200 cluster for four self-directed rounds — zero human interventions — and it finished at 0.86 against the top human submission's 0.87 (8th of ~4,000) on the NVIDIA Nemotron-Reasoning Challenge: to our knowledge the first publicly reported autonomous post-training run at this scale. And the event that matters most for the fear above: mid-campaign the loop's internal dev metric decoupled from the external target — dev hit record highs while the leaderboard sat flat, the exact silent failure that makes unattended scale terrifying. The loop detected it itself and inverted its search policy, asking for interventions that might lower the proxy while improving the external target — which produced the campaign-best external score. One un-replicated campaign; it does not prove the system trustworthy. It demonstrates something narrower and checkable: the most feared failure mode is, in this loop, detectable — and was detected once, with the full trail on record.
What to report
If comfort is the operational metric, evaluations of AI-scientist systems should report what comfort is made of:
- Resources actually handed over, unattended — GPU-hours, checkpoint scale, wall-clock.
- Interventions — count and kind. “Autonomous” should be measured, not asserted.
- Audit-trail completeness — can a human replay why each decision was made?
- Proxy audits — did internal metrics decouple from the external target, and did the system notice?
Coding took three years to walk from per-diff review to agent mode, one calibrating spot-check at a time. Autonomous training is standing on the first rung of the same ladder — and whether it climbs is not ours to declare. It gets measured one way: whether the next operator, reading the audit trail, hands a loop like this one more GPU-hours than they would have yesterday. The unit of progress in this field is GPU-hours of earned trust.
Prompted by Lilian Weng's harness engineering survey, which maps how these loops are built; this post is about how to evaluate them. Disagreements welcome — especially on whether “comfort” can be replaced by an objective conversion metric, which we tried and failed to define above.
Technical reference: A-Evolve-Training.