Index of everything on this site: https://ai.hiina.space/index.md
The hypergradient answer is yes, with a ceiling. Maclaurin, Duvenaud and Adams (2015) computed exact gradients of validation loss with respect to learning-rate schedules, momentum, initialization and regularization by reversing SGD-with-momentum through an entire run (abstract-only) (arXiv 1502.03492). Baydin et al. adapt the learning rate online using its own gradient (abstract-only) (arXiv 1703.04782). Lorraine, Vicol and Duvenaud used the implicit function theorem to tune millions of hyperparameters, such as every weight of a data-augmentation network, at "a few times" the cost of normal training (abstract-only) (arXiv 1911.02590). The ceiling is short-horizon bias. Differentiable windows are far shorter than real training, which biases choices toward tiny step sizes; even a 100-step horizon picked learning rates several orders of magnitude too small (abstract-only) (Wu, Ren, Liao, Grosse).
At LLM scale, practice routes around the outer gradient with an invariance argument instead. Under muP, optimal hyperparameters stay stable across width. Settings found on a 40M-parameter proxy beat published GPT-3 6.7B hyperparameters at about 7% of that model's pretraining cost (abstract-only) (Yang et al.). CompleteP extends transfer to depth and reports 12 to 34% compute-efficiency gains (abstract-only) (Dey et al.). Careful baselines matter more than any of this. With proper tuning, matrix optimizers (Muon, Soap, Kron) gave about 1.4x over AdamW at 0.1B but only about 1.1x at 1.2B. No optimizer reached the 2x claimed earlier, and tuning only the learning rate in the GPT-3 recipe gave AdamW itself up to 2x (Wen et al., ICLR 2026). Moonshot's roughly 2x claim for Muon at scale comes from a different setup and is in tension with that result (Liu et al.).
So can evolution be made differentiable? Inference: there are three defensible readings. First, ES is already a stochastic gradient of a Gaussian-smoothed objective; that is how Metz et al. and Jackson et al. use it in outer loops, and why it survives chaotic landscapes. Second, evolutionary operators can be parameterized as soft, attention-based functions and meta-trained, as LES and LGA do, though they train with ES rather than backprop. Third, discrete selection could be relaxed with Gumbel or perturbed-optimizer tricks; this review found no canonical "differentiable GA" paper, so this third reading is unsourced. The first two exist and work.
Learning-to-learn began with LSTM optimizers trained by gradient descent (abstract-only) (Andrychowicz et al.). Metz et al. (2019) diagnosed why meta-gradients fail. Long unrolls make the outer loss nearly discontinuous and explode gradients, while short unrolls bias them, sometimes anti-correlating with the true gradient. Their fix combines smoothing with reparameterization and ES gradient estimators weighted by inverse variance; the two estimators' variances differed by up to about 20 orders of magnitude (arXiv 1810.10180). VeLO then spent about 4,000 TPU-months meta-training a hypernetwork optimizer. It beat learning-rate-tuned baselines on all 83 tasks in its suite, but it lags beyond about 500M parameters. On an 8B LM it underperformed an untuned Adafactor run at a quarter of the batch size, and it degraded beyond about 200K steps (Metz et al. 2022).
The recent turn is toward cheapness. Celo meta-trained in about 24 GPU-hours on four tiny image tasks and beat 15 hand-designed optimizers on 17 unseen tasks, though it was unstable on AlgoPerf (Moudgil et al.). Celo2 (ICLR 2026) meta-trained in 4.5 GPU-hours on four 8x8 MLP tasks and was evaluated up to GPT-3 XL at 1.3B. On GPT-2 124M it reached 3.35588 validation loss against Muon's 3.35636, using a tuned RMS multiplier; the default Celo2 scored 3.36785. At 1.3B, the figure caption says it outperforms tuned AdamW and VeLO, while the results text says only "competitive". No loss numbers are given in the text, and that run used 10 tokens per parameter because of compute limits (Moudgil, Knyazev, Belilovsky). Lion, found by program search, claims up to 5x less pretraining compute on JFT but admits gains are small or not significant in some settings (abstract-only) (Chen et al.). Wen et al. note that Lion's optimal weight decay differs sharply from AdamW's, so comparisons are sensitive to baseline tuning.
Inference: the same pathology recurs at every level. A long-horizon outer objective is chaotic, and success comes from smoothing (ES) and from shrinking what is learned so extrapolation is safer. Celo2 hands step size to a tunable scalar and borrows Muon's Newton-Schulz orthogonalization, so its result is weak evidence that a learned rule beats hand design. "Learned optimizers beat AdamW or Muon at scale" is not established independently of the authors.
LES (ICLR 2023) learns an evolution strategy's update rule as self-attention over candidates and fitnesses. Meta-trained on low-dimensional analytic functions, it generalizes to unseen problems, population sizes and horizons, can be distilled into an explicit heuristic, and can train itself self-referentially from scratch (Lange et al.). LGA does the same for genetic algorithms. Its selection and mutation-rate operators use fewer than 1,500 meta-parameters, meta-evolved by OpenAI-ES on 10 BBOB functions over 50 generations, and the result works for 4,000 generations. It overfits the meta-training distribution without regularization (Lange et al.). The evosax library packages about 30 ES and GA algorithms, including the discovered ones, for vectorized JAX execution (Lange).
In RL, Jackson et al. meta-learned objectives conditioned on training progress. Meta-gradients through truncated backprop were myopic, while ES, which evaluates the whole agent lifetime, learned a schedule that moves from optimistic and entropy-seeking early to pessimistic late. Meta-trained on one MinAtar game, it beat PPO on Brax and MinAtar (arXiv 2402.05828). DiscoPOP moves the same loop up a level. An LLM proposes and implements preference-optimization losses guided by prior results, and finds one that blends logistic and exponential losses (abstract-only) (Lu et al.). On the biology side, Kashtan and Alon showed that goals varying over shared subgoals make simulated networks evolve modularity and adapt faster: evolution of evolvability (metadata only) (PNAS 2005). Inference: Celo's task augmentation is plausibly the same mechanism, varied meta-training tasks producing transferable modular rules. That link is speculative.
The benchmarks paint a consistent picture. On RE-Bench's seven ML-engineering environments, the best agents scored about 4x human experts at a 2-hour budget. Humans narrowly led at 8 hours and scored about 2x the best agent at 32 total hours, a best-of-k style budget rather than one 32-hour run (abstract-only) (METR). On MLE-bench, o1-preview with AIDE medalled in 16.9% of 75 Kaggle competitions (abstract-only) (Chan et al.). Meta's Automated LLM Speedrunning Benchmark (NeurIPS 2025 D&B) asked agents to reproduce 19 NanoGPT speedrun records from the previous record's script. Without hints, all agents recovered under 20% of the human speedup on average. The best hinted configuration reached 0.46 +/- 0.04, and agents building on their own code degraded quickly over successive records (Zhao et al.). Sakana's AI Scientist produced papers for under $15 each, and v2 had one fully autonomous workshop submission score above the average human acceptance threshold (both abstract-only) (v1, v2).
A year later the picture shifts on narrow tasks. Prime Intellect's "auto-nanogpt" (May 2026) ran Codex (GPT-5.5 xhigh) and Claude Code (Opus 4.7 xhigh) on the speedrun's optimizer track for two weeks: about 10,000 runs and about 14,000 H200-hours. Both beat the 2,990-step human baseline; Claude reached 2,930 steps and Codex 2,950. The mutations were mostly Muon-family variants and schedule changes. The authors report that agents "struggle to come up with new ideas", a novelty-gated track failed to beat baseline, Claude often stopped to wait for user input, and the monitoring agent sometimes misreported results. They call the result a lower bound and "pretty yolo". It is single-source (Prime Intellect).
Labs now publish figures on AI building AI. Anthropic states that "as of May 2026, more than 80% of the code we merge into Anthropic's codebase was authored by Claude", up from low single digits before Claude Code's February 2025 preview, and calls the figure conservative. Its "best next step" agreement rose from 51% (Opus 4.5, November 2025) to 64% (Mythos Preview, April 2026), but those numbers come from 129 sessions selected because the human's choice had room to improve, so they are not like-for-like (Anthropic). A separate page reports that Claude "leads" 26% of Anthropic's AI R&D work, weighted over a 542-node task tree of about 15,000 tasks sampled in July 2026 and rated on Epoch AI's AL0 to AL5 scale. Claude collaborates or better on over 90%, and it is "not operating fully autonomously for any measured subset" (Anthropic). GovAI notes it found no lab disclosure of what share of next-model development is done by AI (Chan et al.).
Inference: agents excel when the objective is a fast scalar, the search is local mutation of a strong existing script, and evaluations are cheap. They are weak at recombining distant ideas and at novelty. In genome terms they are good mutation operators and poor crossover operators. The disclosed AI-builds-AI evidence is code share and task-level automation; the most concrete quantified contribution to a successor model is still AlphaEvolve's 1% of Gemini training time.
The intuition that speedrun contests are already outer-loop search
holds up, and the record log is unusually good data. Track 1 trains a
GPT-2-small-class model to validation loss 3.28 on 8xH100. Record #1
(llm.c baseline, 28 May 2024) took 45 minutes. Record
#3 (4 October 2024) introduced Muon at 24.9 minutes, credited to
@kellerjordan0 and @jxbz. By #12 (19 November
2024) it was 5.03 minutes, and #20 (16 January 2025, six contributors)
reached 2.992. After that came a long tail: Polar Express replacing
Newton-Schulz (#38), NorMuon (#41), multi-token prediction (#53, 1.988
min), bigram hash embeddings (#62, 1.655), MUDD skip connections (#81,
1.363), a prefix-token auxiliary loss (#88, 1.243) and #91 at 1.126.
Then #92 (30 August 2026) hit 0.665 minutes, a 40.9%
single-step drop that bundled sampled softmax, full FP8 MLP, an
84.6M-row hashed n-gram table, the "ANVIL" optimizer stack, depth
reduction and CUDA-graph capture. Five records are co-credited to
entries the README labels "AI System": hiverge.ai (#32), Locus (#60),
Aster (#69), Station (#72) and Recursive (#87, 1.256 min, 11 June 2026).
Each is paired with a named human, the labels are self-reported, and how
autonomous each run was is not documented (modded-nanogpt
README).
Horizontal gene transfer is visible in the log too. Track 2 (GPT-2 Medium, target 2.92) went from 5.8 hours to 22.98 minutes by November 2025. Record #18 then reached 17.35 minutes on 31 December 2025 with the description "Bulk transfer short track features" and no credited contributor: features moved wholesale from one lineage to another. The repo also has a separate optimization track, which Prime Intellect used. The CIFAR-10 analogue, airbench, reaches 94% in 3.29 seconds on one A100, but its paper does not say how the baseline hyperparameters were found (Jordan). This review did not locate an MNIST speedrun or a "smallest ImageNet" contest.
Inference: the speedrun is an open, asynchronous, multi-lineage evolutionary process. It has a fixed fitness function (time to target loss), a mutable genome (the training script), a selection rule (beat the record under fixed rules), recombination by pull request, cross-track transfer, and a non-stationary environment: PyTorch upgrades and rule changes produce some "improvements" that are environmental rather than mutational. The trajectory looks like a rugged landscape. It fell from 45 to about 3 minutes in eight months, crept from 3 to about 1.1 over 19 months through many 2 to 6% steps, then took one large multi-mutation jump. Nobody has published an analysis of the log as a search process. Experiment 1 does it.
Inference: putting the section together gives a nested stack, with the method switching as the outer objective gets longer, noisier and less differentiable.
| Level | Object optimized | Methods that work today | Selection signal |
|---|---|---|---|
| 0 | Weights | SGD, AdamW, Muon | Exact gradient |
| 1 | Hyperparameters, schedules | Proxy sweeps plus muP or CompleteP; PBT; Hyperband; hypergradients only at short horizons | Held-out loss on proxies |
| 2 | Optimizer rule | Hand design; learned (VeLO, Celo2); program search (Lion) | Meta-validation tasks |
| 3 | Algorithm, objective, kernels | LLM code evolution (AlphaEvolve, DiscoPOP); speedrun humans and agents; ES-learned RL objectives | Executable benchmarks |
| 4 | Agents and scaffolds | DGM and HGM archives; ADAS | Agent benchmarks |
| 5 | Researchers and labs | Humans, with growing AI share | Papers, leaderboards, products |
| 6 | Market | Not modelled | Revenue, funding, talent |
The switch from gradients to evolution tracks horizon length and smoothness. ES beat truncated meta-gradients exactly when the objective was lifetime-long (Metz 2019, Jackson 2024). Higher levels also have fewer, costlier fitness evaluations, so they lean on proxies, and Wen et al.'s 1.4x-to-1.1x shrinkage shows proxy conclusions can mislead.
Text licensed CC BY 4.0 (attribution: Hiina, ai.hiina.space). Source: https://github.com/hiinaspace/ai-hiina-space