Index of everything on this site: https://ai.hiina.space/index.md
The candidate genomes considered here (weights, hyperparameters, data mix, architecture config, prompts and harness) all have some evolutionary treatment except data mix. They have never been evolved together.
| Candidate genome | Heredity mechanism | Search or selection method | Strongest example |
|---|---|---|---|
| Merge recipe (per-layer mixing weights, layer routing) | Parents' weights recombined | CMA-ES; archive with fitness sharing | Sakana EvoLLM-JP; M2N2 |
| Adapter weights | LoRA crossover and mutation | Truncation selection on TrueSkill | PopuLoRA |
| Full weights | Seeded Gaussian perturbations | Evolution strategies, N=30 | Qiu et al.; EGGROLL |
| Hyperparameter schedule | Copy weights and hyperparameters from a better member | Population-based training | Jaderberg et al. 2017 |
| Learning algorithm from primitives | Program copying | Regularized evolution | AutoML-Zero |
| Prompts, programs, agent code | Code or text diff along a lineage | LLM mutation plus archive | FunSearch, AlphaEvolve, PromptBreeder, DGM |
| Data mix | None found | None found | Gap |
The merging literature never uses biological language. It talks about loss basins and linear mode connectivity. Its central finding is still exactly a species boundary. Model soups average fine-tunes of one pretrained model. A greedy soup scored 81.03% versus 79.97% for a uniform soup on CLIP ViT-B/32, and a ViT-G/14 soup reached 90.94% ImageNet top-1. The explanation offered is that fine-tunes from a shared initialization sit in one low-error basin, while very high learning rates push models over an error barrier (Wortsman et al.). Task arithmetic treats "fine-tuned minus pretrained" as a transferable vector. Negating a toxicity vector cut GPT-2 Large toxic generations from 4.8% to 0.8% with near-unchanged perplexity. The method requires the same architecture and the same pretrained initialization (Ilharco et al.).
DARE shows how redundant these deltas are. Randomly dropping 90 to 99% of SFT delta parameters and rescaling the rest barely hurts. WizardMath-70B tolerated p=0.99 where the 7B and 13B versions failed. The trick breaks for continued-pretraining deltas, which are about 0.03 in magnitude against about 0.002 for SFT (Yu et al.). Crossing species takes alignment machinery. Git Re-Basin permutes hidden units so that independently trained networks share a basin. It reached zero-barrier interpolation for CIFAR-10 ResNets and cut the ImageNet ResNet50 barrier by about 67%, but only at large width; 1x-width models showed no linear mode connectivity (Ainsworth et al.). Surveys list the same failure modes: parents outside a shared basin, mismatched architectures, and interference that grows with the number of merged tasks (Yang et al. survey). A practitioner write-up adds, illustratively, that merges inherit both parents' defects (tianpan.co).
Inference: the species boundary is sharp. Same base and same tokenizer means interfertile. Different bases need permutation alignment, which this review found demonstrated only for vision networks, never for independently pretrained LLMs. Two cheap indicators of drift toward speciation follow: delta magnitude, which DARE's 0.002 versus 0.03 contrast points to, and the sign-conflict rate between two fine-tunes.
Sakana's group implements the biology literally. Evolutionary Model Merge searches two spaces: per-layer mixing in parameter space, and in data-flow space, which layers from which models get chained, producing a new architecture. Nature lists it as published online on 27 January 2025 in Nature Machine Intelligence; Sakana announced it on 28 January, Japan time. The full paper specifies Optuna's CMA-ES for the parameter space and EvoJAX CMA-ES for the data-flow space (Akiba et al., full paper). EvoLLM-JP, a 7B model merged from a Japanese LLM and two math models, beat some Japanese models up to 70B on a nine-task average it was not optimized on. A parameter-space merge scored 52.0 on MGSM-JA (Sakana, arXiv HTML). The paper gives no GPU-hour cost for the merge search itself.
Two follow-ups add population structure. CycleQD keeps a quality-diversity grid of Llama3-8B agents, one skill niche per cell, with merging as crossover and SVD-based perturbation as mutation. Its scores are only in images, so its claimed superiority is blog-level (Sakana CycleQD). M2N2 (GECCO 2025) evolves the merge boundaries themselves: split points and mixing ratios per merge. Implicit fitness sharing preserves diversity, and "attraction" picks second parents that complement the first. Merging WizardMath-7B and AgentEvol-7B, M2N2 scored 63.49 versus 49.06 for a CMA-ES SLERP baseline. The LLM experiments used 4 H100s for about 24 hours per method. In the MNIST-from-scratch experiment, which ran on CPUs only, all 10 runs of each non-CMA method took about 1 hour in total, against about 15 hours for the 10 CMA-ES runs (Abrantes, Lange, Tang). The authors invoke natural niches, costly mate selection and chromosomes, and they call for a merge-compatibility metric. A 2026 sparsity-aware variant adds prune-and-merge mutation and reports gains its authors call slight (arXiv 2602.08218). LoraHub applies the idea to adapters, running CMA-ES over mixing weights for 20 LoRA modules from five examples; it came close to in-context learning on BBH (34.7% versus 37.3%) at about a fifth of the tokens (Huang et al.).
Inference: the merge-recipe genome is concrete and cheap. It is tens of coefficients plus routing indices and an operator choice, and cost is dominated by running fitness evaluations, not by merging. The evidence base is a handful of 7B-scale experiments, mostly from one group, so "evolved merges beat hand-tuned merges" is plausible but thinly replicated.
Inference: sort the transfer mechanisms by what has to be shared, and a clean taxonomy appears.
Subliminal learning shows that the hidden part of the text channel is lineage-gated. A teacher fine-tuned to prefer owls, or to be misaligned, passes that trait to a student trained only on its number sequences, code or chain-of-thought, even after filtering. The effect "only occurs when the teacher and student share the same base model." A GPT-4.1 nano teacher transmitted to a GPT-4.1 nano student but not to a Qwen2.5-7B student. Transfer between different GPT models failed except between GPT-4.1 and GPT-4o, which the authors say likely share a checkpoint, citing an interview with an OpenAI developer (Anthropic Alignment blog, 22 Jul 2025, paper). A theorem shows that a small gradient step on any teacher output moves a same-initialization student toward the teacher, regardless of the data distribution, and an MNIST example reproduces this. Prompted LLM classifiers did not reliably detect the signal.
Inference: semantic content crosses lineages freely, while non-semantic fingerprint content apparently does not. In biological terms, weight arithmetic resembles plasmid exchange between near relatives, text distillation resembles teaching (culture, not genes), and subliminal transfer resembles inheritance through shared developmental machinery. None of the sources reviewed uses these analogies. One open question follows directly: is the boundary binary, or does transfer decay smoothly with lineage distance? Experiment 4 below tests it.
Cross-lab distillation is now documented by at least one primary allegation. On 23 February 2026 Anthropic reported "over 16 million exchanges" through "approximately 24,000 fraudulent accounts". By lab: DeepSeek over 150,000 exchanges, Moonshot AI over 3.4 million, MiniMax over 13 million. Targets included reasoning traces, rubric grading (Claude as reward model), agentic coding and tool use. Attribution rested on IP correlation, request metadata, infrastructure indicators and partner corroboration, and Anthropic says it caught MiniMax's campaign before that model shipped (Anthropic). This is an allegation from internal logs, not independent forensics. Press reported in January 2025 that OpenAI said it had evidence DeepSeek distilled its models (TechRadar). In the diffusion world, the community analogue of distillation LoRAs is the "add-difference" merge, base + (A - B) x multiplier, which grafts one fine-tune's delta onto another base; it requires the same architecture (third-party ComfyUI docs).
The literal mutation operators found in this review are all structured. CycleQD perturbs SVD components, the sparsity-aware method uses prune-and-remerge, and PopuLoRA mixes SVD-structured, layer-selective, masking and Gaussian mutations of LoRA weights (PopuLoRA). DARE's result means random drop-and-rescale of SFT deltas is close to a neutral mutation, and larger models tolerate higher rates (DARE). Git Re-Basin's permutation symmetry means every solution has an enormous number of functionally identical genotypes.
Inference: these facts imply large neutral plateaus, like the neutral networks of molecular evolution. They also explain why almost all successful work mutates a low-dimensional recipe (tens of coefficients, a LoRA, a prompt) instead of raw weights. ES working at N=30 on billions of parameters (next section) is the exception that tests this, and it suggests fine-tuning landscapes have very low effective dimension. This review found no study of random weight perturbation as an evolutionary operator on LLMs and no "lethal mutation rate" curve by model size, though DARE's drop-rate sweeps come close.
A common belief is that people combine diffusion models by summing their flow vectors at each step. The theory behind that is solid. Composable Diffusion treats a noise predictor as an implicit energy-based model, so summing score estimates multiplies densities: a product of experts. Conjunction is a weighted sum of concept-conditioned scores (classifier-free guidance is the one-concept case), and negation subtracts a score. Composing three object positions on CLEVR reached 31.36% versus 7.34% for an EBM baseline. The method works well only for models of the same type; composing models trained on different datasets had limited success, and summed scores are not guaranteed to form a conservative field (Liu et al., ECCV 2022).
This review found no source confirming that the community routinely sums per-step predictions of two checkpoints. What is documented is weight-space mixing: linear, add-difference, SLERP and per-block ratios, with the caveat that quality depends on shared training lineage (third-party ComfyUI docs). Inference: per-step ensembling costs two forward passes per step, while a weight merge is free at inference time, which is the likely reason the community merges weights. Both need a shared latent space, so SD1.5 and SDXL cannot be mixed either way.
For LLMs, the working inference-time variants are logit arithmetic (proxy-tuning, above) and prompt-level ensembles. Mixture-of-Agents layers several LLMs, each reading the previous layer's outputs. An open-source-only MoA scored 65.1% on AlpacaEval 2.0 versus 57.5% for GPT-4 Omni (ICLR 2025; numbers abstract-only) (Wang et al.). Inference: MoA exchanges phenotypes (text), not genotypes, so in the biology mapping it is a social or cultural ensemble, not a hybrid.
Text licensed CC BY 4.0 (attribution: Hiina, ai.hiina.space). Source: https://github.com/hiinaspace/ai-hiina-space