Research
PUBLISHED
KALAVAI
Predicting When Independent Specialist Fusion Works
NeurIPS 2026 · arXiv 2603.22755
Read on arXiv →What It Is
A quantitative model for post-hoc cooperative LLM training — predicts fusion gain before any compute is spent. Contributors fine-tune checkpoint copies independently; KALAVAI fuses them via lightweight MoE routing in 500 steps.
The Problem
Fine-tuning a model for one domain degrades performance in others. Teams maintain N separate models with no principled way to combine them. KALAVAI proves when fusion works — and quantifies the gain before you run it.
Approach
- Derived linear gain law: gain ≈ 0.83 × divergence − 2.80 (R² = 0.872, n=8, range 3–26% divergence)
- Shared initialization is mandatory — fusion fails without it
- Lightweight MoE routing (500 steps) with learned dispatch — weight averaging degrades by 3.4% vs best specialist
- Validated across 4 model families (Pythia 410M / 1B / 6.9B, Qwen 1.5B) with full audit trail
Results
- +7.72% at 410M (±0.02%), +7.49% at 1B (±0.01%), +6.53% at 6.9B (±0.024%)
- +21.76% cross-lingual gain (Tamil, Yoruba, Welsh, Code) — Yoruba perplexity 41.9 → 7.7
- +16.79% from a simulated 20-contributor federation (10 languages + 10 domains)
- Fusion gain approaches zero below ~3.3% divergence — predictable no-go zone
Stack
PyTorchHuggingFace TransformersMoE routingPythonStatistical evaluation