Research
PUBLISHED
UYIR
Decomposing Evolutionary Mixture-of-LoRA Architectures
TMLR 2026 · arXiv 2605.11153
Read on arXiv →What It Is
A factorial decomposition of an evolutionary mixture-of-LoRA system — isolating whether the win comes from the router rewrite, the per-domain leave-one-out evaluation scope, or the lifecycle (death + α-blend inheritance + SVD mutation + slot reallocation).
The Problem
Evolutionary LoRA systems bundle multiple design choices together — new routers, new fitness signals, lifecycle dynamics — and report aggregate gains. Nobody knows which factor actually carries the win. UYIR runs the controlled factorial.
Approach
- Built a from-scratch ~150M widened-D substrate (D=1536, V=32K) for clean attribution
- 5-of-8 partial 2³ factorial across (router rewrite × per-domain LOO eval × lifecycle), n=3 seeds, 25K adaptation steps per cell
- Controllable synthetic sandbox to locate where evolutionary search is load-bearing vs degrades the gradient solution
- Base-perturbation probe directionally refutes a "genomic-context" reframe of the lifecycle role
Results
- Router rewrite carries the entire +0.0426 nat balanced log-PPL improvement (t=12.86, p=0.006)
- Full evolutionary system vs static B3 baseline: +0.015 nats (t=1.94, p=0.19) — does not clear α=0.05 at n=3
- Lifecycle is a net drag of ≈ −0.028 nats (t=−4.46, p=0.047) — death + inheritance + mutation is costing performance, not adding it
- Synthetic sandbox: evolutionary search on the routing channel is load-bearing only when adapters are pre-aligned to the task — in every other regime tested it underperforms, ties, or actively degrades the gradient solution
Stack
PyTorchLoRA / PEFTFrom-scratch GPT (~150M)PythonStatistical attribution