--- id: llm-evolution-and-meta-optimization/08-five-experiments-cheapest-first title: 'LLM evolution and meta-optimization: 8. Five experiments, cheapest first' report: https://ai.hiina.space/reports/llm-evolution-and-meta-optimization.md section: 8 version: 1 updated: '2026-10-27' author: Hiina --- Index of everything on this site: https://ai.hiina.space/index.md # LLM evolution and meta-optimization: 8. Five experiments, cheapest first - Section 8 of 10 of the report "LLM evolution and meta-optimization" by Hiina (v1, updated 2026-10-27) - Report overview and contents: https://ai.hiina.space/reports/llm-evolution-and-meta-optimization.md - Previous: https://ai.hiina.space/reports/llm-evolution-and-meta-optimization/07-what-nobody-has-done-yet.md - Next: https://ai.hiina.space/reports/llm-evolution-and-meta-optimization/09-conclusion.md ## 8. Five experiments, cheapest first The setups, costs and timings below are my estimates, built from the unit costs in the cited papers. None comes from a source. ### Experiment 1: read the modded-nanogpt log as an evolutionary record - **Hypothesis.** Speedrun progress follows rugged-landscape dynamics. Gains per record decay roughly geometrically within a "paradigm", large jumps come from bundled multi-mutation records or cross-lineage transfer, and optimizer-family mutations have a higher survival rate than architecture mutations. - **Setup.** Fetch the raw README from both tracks (92 Track 1 records plus Track 2). For each record, extract date, time, delta from previous, contributor handles and the AI System label. Annotate mutation category (optimizer, architecture, attention, precision, kernel or systems, environment upgrade) with a cheap LLM pass and hand-check the result. Then trace which features later records remove or replace by diffing the record scripts in `records/`. Compute gain by category, feature half-life, contributor co-authorship network, and the share of total improvement from AI-System-labelled records. - **Cost.** No GPU. Under $10 of API spend for annotation, and a weekend. - **Interesting result.** A clear signature that distinguishes environmental "improvements" (software upgrades) from mutational ones. Evidence that AI-System entries cluster in particular mutation categories, which would test the "agents are mutators, not recombiners" reading. A measured cross-track transfer coefficient (how much of a Track 1 feature's gain survives on Track 2). - **Closest prior work.** [Meta's Automated LLM Speedrunning Benchmark](https://arxiv.org/pdf/2506.22419) uses record transitions as tasks, and [Prime Intellect's auto-nanogpt](https://primeintellect.ai/auto-nanogpt) uses the optimizer track as a fitness function. Neither analyses the log itself. ### Experiment 2: a Price-equation decomposition of the Hugging Face model graph - **Hypothesis.** Most trait change in the open-weight ecosystem is transmission bias (fine-tuners changing traits as they copy), not selection (some lineages out-reproducing others). Selection, where present, acts mainly between families (base models), not within them. Inference: this would quantify Boudry and Friederich's domestication reading. - **Setup.** Start from the ModelAtlasData CSV and a fresh Hub crawl of `base_model` and `base_model_relation`. Unlike Laufer et al., keep adapters, merges and quantizations as typed edges. Define fitness as descendant count, with downloads as a robustness check. Define traits as license permissiveness, card length, declared language, quantization, and merge versus fine-tune. Compute the Price covariance and transmission terms per generation, then a two-level decomposition with base family as the group. Add tokenizer-hash and config features from small file downloads as more heritable traits. - **Cost.** CPU only, a few days, no API. - **Interesting result.** Merge edges showing a different selection signature from fine-tune edges, for example merges reproducing more but transmitting traits less faithfully. Or between-family selection dominating, which would say base-model releases, not fine-tuner choices, drive ecosystem composition. Either would be the first quantitative levels-of-selection result for AI models. - **Closest prior work.** [Laufer, Oderinwale, Kleinberg](https://arxiv.org/pdf/2508.06811) on inheritance and drift; [Cebrian](https://arxiv.org/abs/2603.22287) on founder effects; [Frank](https://arxiv.org/pdf/2507.18549) for the formalism. ### Experiment 3: the merge-speciation curve and a compatibility gene - **Hypothesis.** Merge quality between two fine-tunes is predictable from cheap weight statistics: task-vector cosine, sign agreement, delta magnitude. Merge quality falls off sharply rather than gradually as fine-tuning distance grows, a speciation threshold. An M2N2-style archive with fitness sharing beats plain CMA-ES at fixed evaluation budget, and its advantage grows when the pool includes nearly incompatible parents. - **Setup.** Pick a 1.5B to 3B base (Qwen2.5 or Llama-3.2) and 8 to 12 public same-base fine-tunes from the Hub, plus deliberately "drifted" fine-tunes made with QLoRA and continued pretraining at increasing step counts. For every pair, compute the weight statistics and measure the merged model on a few hundred items from GSM8K and one other skill. Then run CMA-ES and an M2N2 reimplementation over a genome of about 30 to 60 per-layer coefficients plus a DARE drop-rate gene, population 16 to 32, fitness on 200 held-out items. Merge on CPU with mergekit and evaluate in vLLM on a single 24 GB GPU. - **Cost.** Electricity. One to three days on one 24 GB consumer GPU; no API. - **Interesting result.** A single scalar, such as sign agreement in middle layers, that predicts merge success with high rank correlation. That would be the compatibility metric M2N2's authors asked for, and a measured species boundary. Also interesting: evolved recipes that converge on the same layer-wise pattern across different parent pairs, an inherited "body plan". - **Closest prior work.** [M2N2](https://arxiv.org/pdf/2508.16204), [Sakana Evolutionary Model Merge](https://arxiv.org/pdf/2403.13187), [DARE](https://arxiv.org/pdf/2311.03099). ### Experiment 4: does subliminal transfer decay with lineage distance? - **Hypothesis.** Subliminal trait transfer is graded, not binary. Its strength falls with weight-space distance from the teacher (CKA or sign agreement) and is predicted better by weight distance than by output-phylogeny distance. Inference: if true, that separates genetic inheritance from cultural resemblance experimentally. - **Setup.** Teacher: Qwen2.5-0.5B or 1.5B with a LoRA that instills a measurable preference (a favourite animal or number, as in the paper). Generate filtered number-sequence data from it. Students, at increasing distance: the same base, same-base fine-tunes that have drifted by 1k, 10k and 100k steps of continued pretraining (from Experiment 3), a sibling with the same tokenizer but a different initialization (another Qwen2.5 size), and an unrelated family. Fine-tune each student with LoRA on the teacher's data, and measure the preference shift over at least 5 seeds per student, matching the paper's N of 5 or more. Compute REEF-style CKA and a PhyloLM-style output distance for each teacher-student pair. - **Cost.** Electricity. Two to four days on one 24 GB consumer GPU; no API. - **Interesting result.** A smooth transfer-versus-distance curve with a measurable half-distance would make "species" quantitative and give a forensic test: transfer strength becomes evidence of shared initialization. A sharp cliff would confirm the binary story with a located boundary. Any transfer to the different-initialization sibling would contradict the published boundary. - **Closest prior work.** [Cloud et al.](https://arxiv.org/pdf/2507.14805); [REEF](https://arxiv.org/abs/2410.14273) for the distance measure; [PhyloLM](https://arxiv.org/html/2404.04671v5). ### Experiment 5: a mini Darwin Godel Machine on a speedrun fitness, comparing selection levels - **Hypothesis.** On a fast, executable training objective, clade-level parent selection (HGM's clade-metaproductivity) beats both greedy and DGM-style parent selection at equal evaluations. Letting the genome include the agent's own prompt and tool scaffold, not just the training script, raises the ceiling. This tests both levels of selection and a two-part genome (harness plus artifact). - **Setup.** Fitness: time to target accuracy on airbench (CIFAR-10, seconds per run on a consumer GPU) or a 10 to 30M-parameter NanoGPT to a fixed loss (a few minutes per run), with fixed seeds and repeated evaluations to control noise. The mutator is a cheap API model or a local Qwen-Coder in 4-bit, editing the training script in a sandbox. Keep the evaluator read-only and outside the genome, per DGM's hacking result. Run three archives of about 80 iterations each: greedy, DGM sampling, and HGM clade estimates. Log the full parent-child tree, then compute stepping-stone statistics and the correlation between parent score and best-descendant score. - **Cost.** About $50 to $300 in API, depending on model and context length, plus several days of single-GPU time for fitness evaluations. - **Interesting result.** Replicating HGM's metaproductivity mismatch in a non-SWE domain would make it a general property of LLM-driven evolution. Agents rediscovering known speedrun mutations (Muon-family changes, zero-init projections) in the same order humans did would mean the search follows a shared landscape. A measurable gain from also evolving the scaffold would be a first data point on full-tuple genomes. - **Closest prior work.** [DGM](https://arxiv.org/pdf/2505.22954), [HGM](https://arxiv.org/pdf/2510.21614v2), [Prime Intellect auto-nanogpt](https://primeintellect.ai/auto-nanogpt), [ShinkaEvolve](https://sakana.ai/shinka-evolve/). --- Text licensed CC BY 4.0 (attribution: Hiina, ai.hiina.space). Source: https://github.com/hiinaspace/ai-hiina-space