A few people have asked me, in person and in DMs, what the KALAVAI paper is actually about — without the abstract, without the math, without the figures. This is my honest attempt to explain it.
For most of the last two years I kept running into the same wall, and the wall always had the same shape. Training a useful language model in any language that wasn't English meant one of two things. Either you were OpenAI, Anthropic, Google, or Meta — institutions with the data, the compute, the people, and the funding to do everything at once and end up with a single dense model that knows everything well enough — or you were a researcher with one GPU, one domain, one corpus, and an honest paper about a single narrow capability that the wider field treated as marginal. There was no third option. You had everything, or you had a corner. The shape of the wall was the shape of the economics.
That bothered me. It bothered me because the actual world doesn't live in those two states. The world has a Yoruba speaker in Lagos sitting on a folder of digitised novels, a Welsh-language teacher in Cardiff with a curriculum corpus that has never seen a transformer, a Tamil literature professor in Chennai whose archive has hundreds of years of poetry, a clinician in Bangalore with a private corpus of de-identified discharge summaries, a contracts partner in São Paulo with a decade of annotated case law. Each of these people has something — a domain, a dataset, often a GPU, almost always a question they desperately want a language model to be able to answer. None of them has everything. And, for reasons that turned out to be mostly historical accident rather than mathematical necessity, no one had given them a protocol that let them combine what they had into something none of them could build alone.
That's the question KALAVAI tries to answer — quite literally. It is the answer to one sentence: what is the simplest, cheapest, most institutionally realistic protocol that lets twenty independent contributors, each with a GPU and a domain, train together and end up with one model none of them could have built alone?
The protocol fits in a paragraph
I am going to try to describe the entire system in one paragraph, because if I can't do that, I haven't actually understood it.
A coordinator publishes a single starting checkpoint — say, Pythia-410M at training step 10,000 — to a shared registry. Anyone who wants to contribute downloads that exact revision. Each contributor then fine-tunes their own copy of that checkpoint on their own domain — on their own hardware, on their own schedule, with their own data — and at no point do they share gradients, share activations, share batches, or communicate with any other contributor in any way. Zero communication during training. When a contributor's specialist is done, they upload the resulting checkpoint. Once a set of checkpoints exists, a lightweight router — a single linear layer, no bigger than a single matrix multiplication — is trained for 500 steps on a small mixed-domain dataset. The router learns which of the N specialists is best suited for which kind of token. At inference, all N specialists are run in parallel for every input, and the router decides — on a token-by-token basis — which specialist's output to weight, and how much. The fused model behaves like one model with N specialisations.
That's it. The full protocol. Standard PyTorch, standard HuggingFace, standard training loop. No custom CUDA kernels. No distributed-training framework. No federated-learning library. No LoRA, no adapters, no parameter-efficient anything. The mechanism is the protocol, not the infrastructure — which is the part I think is most often missed when people first read it.
# Step 1: Everyone starts from the same checkpoint.
base = load("pythia-410m", revision="step10000")
# Step 2: Each contributor trains on their own domain, independently.
# No communication. No gradient sharing. No data sharing.
specialist_code = train(copy(base), code_data, steps=2000)
specialist_science = train(copy(base), science_data, steps=2000)
specialist_fiction = train(copy(base), fiction_data, steps=2000)
# Step 3: A single linear layer learns to dispatch tokens to specialists.
# 500 steps. One matrix. Tiny.
router = nn.Linear(hidden_size, 3, bias=False)
fused = MoE(specialists=[code, science, fiction], router=router)
train_router(fused, mixed_data, steps=500)
# Step 4: The fused model outperforms every individual specialist
# on its own domain. +7.72% at 410M, corrected per-domain eval.
The fused model beats every individual specialist on every domain we tested. +7.72% at 410M parameters (three seeds, ±0.02%, corrected per-domain equal-weight evaluation), +7.49% at 1B, +6.53% at 6.9B. The mechanism holds at every scale. There is no curve that bends back down. There is no scale at which the trick stops working. That was already enough to write a paper.
The result I didn't expect
When I started this, I assumed the headline finding would be the fusion gain itself. The fusion gain is real, it's reproducible in thirty minutes on a consumer GPU, and it sits comfortably in the literature alongside Branch-Train-Mix, PHATGOOSE, and the rest of the post-hoc-fusion family. But it turned out the more interesting finding — the one I keep getting questions about — was that you can predict how well the cooperative will work before you train a single specialist.
Across every condition we tested — eight setups in total, spanning English code-and-science, four under-resourced languages, three private professional domains, and a 20-contributor federation — fusion gain scales linearly with how much the specialists diverge from the base model:
gain ≈ −2.80 + 0.83 × divergence (R² = 0.872, n = 8, slope 95% CI [0.52, 1.15])
Divergence here is the mean per-token KL divergence of each specialist from the base model, averaged across all specialists. It's a single number that captures how much each specialist has moved from where it started. Crucially, it is computable in minutes on a small validation set as soon as each contributor finishes training — long before anyone has touched the router, long before any inference cost has been paid. The formula gives a prediction. The cooperative gives an answer. The residuals between them are small.
If your specialists diverge by 15% from base — roughly the level you get from a normal English-domain fine-tune — the formula expects about +10% fusion gain. If your specialists diverge by 25%, which is what cross-lingual settings reach, expect about +18%. And below approximately 3.3% mean divergence, the formula predicts the gain goes to zero. We tested this directly: Qwen-1.5B at full pre-training shows 3.16% mean divergence and produces +1.06% gain. The mechanism still works — the model is improved — but the magnitude is small. The formula said it would be small. The formula was right.
That is the part of the paper I am most proud of, and it is also the part that took the longest to find. The first version of the regression had n=4 and an R² in the high 0.7s and I was unsure whether the apparent linearity was real or a small-sample artifact. Adding cross-lingual conditions, the private-domain experiment, and the 20-contributor federation tightened the line, and the residuals stayed bounded. The formula is the most useful thing in the paper precisely because it tells you whether a cooperative is worth running before you run it. That single property changes the economics of trying.
The first surprise: all specialists must run at inference
This was the result I had to repeat to myself a few times before I believed it.
When I designed the original architecture, I assumed there would be a fast path. The reasoning is intuitive: if the routing decision is "this token belongs to the code specialist," then surely you should run only the code specialist on that token, save the compute of running the other two, and pocket the inference-cost savings. So we built that. A domain classifier with 99.3% accuracy — almost perfect routing — dispatching each token to a single specialist.
The fast path produced −21.1% degradation relative to the base model. Same routing decisions, single-expert dispatch, near-perfect accuracy, and the result was catastrophically worse than doing nothing.
The slow path — running all three specialists in parallel for every token, combining their outputs through the trained router — produced +14.1% improvement under the original mixed-batch evaluation. The same routing decisions, the same accuracy, the same model weights, but evaluated by a joint inference path instead of a single-expert dispatch. The difference between the two is 35 percentage points. That is the difference between a useful system and a system that is worse than no fine-tuning at all.
The explanation, once we found it, was simple and almost embarrassing: each specialist forgets what it wasn't trained on. The code specialist's loss on science data is worse than the base model's loss on science data. The science specialist is similarly worse than base on code. Fine-tuning is a specialisation procedure, but it is also, equivalently, a forgetting procedure — and the forgetting is rapid and severe in the absence of replay. When you dispatch a token to a single specialist, that specialist is the only one with a vote, and on any token even slightly outside its training distribution it votes catastrophically wrong. Joint inference works because the other specialists, the ones who would have been "wrong" for that token, contribute a small residual that the router can use to suppress the dominant specialist's hallucination. The router doesn't only select — it also suppresses. The 35-point gap is the suppression value.
This is the finding that changes what kind of system KALAVAI is. The point of the protocol is not "split the work across specialists at inference." The point is "combine all the specialists' votes at inference, and let the router learn how to weight them." This is also why the protocol is not an inference-efficiency story. It pays the inference cost in exchange for the training accessibility. That trade is deliberate.
The second surprise: shared initialisation is non-negotiable
The other constraint that turned out to be load-bearing was the one I almost relaxed first.
The protocol says: every contributor starts from the same checkpoint. Same architecture, same weights, same tokenizer, same revision. I initially treated this as a convenience — surely the system should be flexible enough to handle contributors who fine-tune from slightly different starting points? In a real federation, people have different cached models, different revisions, different convenience reasons for grabbing the version they grabbed. Surely the protocol could accommodate that?
It cannot. When we tested it directly — contributors starting from genuinely different checkpoints of the same model family, let alone different model families — the router fails to learn coherent dispatch. The fusion gain drops to zero or goes negative. The reason, when you stare at it for long enough, is also simple: the geometry of the representation space is set by the starting checkpoint. Fine-tuning moves each specialist a small distance through that space, but the directions of movement are coherent — every specialist's hidden states are still pointing into the same coordinate system, and a small linear router can read those coordinates and dispatch sensibly. If specialists start from different checkpoints, their hidden-state coordinate systems are not aligned. The router would need to learn a per-specialist projection to even begin combining them, and the 500-step training budget is nowhere near enough to learn that projection from scratch.
Shared initialisation is the one constraint of the protocol that does not flex. Everything else — domain choice, training duration, freeze depth, contributor count, hardware — has wiggle room. Shared init does not.
The third surprise: the router learns where to switch mid-sentence
If you'd asked me before I ran the experiments whether a router trained on 500 steps of mixed data could learn to switch specialists inside a single sentence — switching mid-clause, without ever having been told where the clause boundaries are — I would have said "no, that's far too much to ask of a 500-step linear layer." I was wrong.
I gave the fused model the prompt "Derive the equation for protein folding using Python pandas." It is a sentence designed to be hostile to a document-level classifier: it contains science vocabulary ("derive," "equation," "protein," "folding") and code vocabulary ("Python," "pandas") in a single eleven-word span, with no obvious structural break. A document-level router that classified the entire prompt would have to pick one and accept the consequences. A token-level router would have to discover, without supervision, that the science tokens belong to the science specialist and the code tokens belong to the code specialist — and switch between them within the sentence.
The router did exactly that. Without any supervision signal, without any domain labels, without any cue about where the boundary should fall, the router put high science weight on the scientific vocabulary and high code weight on the programming terms, mid-sentence, cleanly. The same pattern held across narrative/science hybrids, technical/narrative hybrids, and multi-domain sentences with three switches. The router behaves as a near-hard switch in practice: the top-1 expert receives more than 95% of the routing weight in 99.7% of tokens. We confirmed that hard routing (argmax dispatch with all specialists still computed) and soft routing (learned softmax weights) produce identical perplexity — the value is in the selection and suppression, not in the specific weighting scheme.
The reason this is surprising is that the router was never told there were domain boundaries inside sentences. It learned to look for them anyway, because the training signal — minimise loss on mixed-domain data — rewards exactly that behaviour and nothing else gets you the gain.
The training duration crossover, or: when frozen layers stop being optional
There's one more finding from the paper that I keep wanting to mention because it changes how you think about the protocol as a practitioner, not as a researcher.
For short training horizons — up to about 2,000 steps per specialist — freezing the first K layers of the model is either optional or marginally hurtful. Specialists with no frozen layers reach the highest single-experiment peaks (+8.12% at 2,000 steps with freeze=0). For a sprint cooperative — one weekend, one round of training — freeze=0 is the right answer.
But around 5,000 steps the picture flips. Unfrozen specialists begin to over-specialise; their representations drift far enough from the shared initialisation that the router can no longer combine them cleanly. By 10,000 steps the unfrozen configuration has degraded to +5.83%, while a configuration with the first 4 layers frozen holds at +7.33%. By 20,000 steps the gap is 3 percentage points, and unfrozen is clearly losing.
The practical guideline that falls out: if you are training under 5,000 steps, don't freeze. If you are training over 5,000 steps, freeze the first 4–8 layers. That's the kind of rule that only feels obvious in hindsight, and it has direct consequences for how a real cooperative is structured. A long-running cooperative — months of contributor effort, deep specialisation — needs frozen anchors. A short-burst cooperative — a weekend hackathon, a sprint — does not. The mechanism is structurally different in the two regimes, and the data tells you exactly where the crossover is.
What Yoruba taught me
The cross-lingual experiment was the one I was most nervous about, because I had a hypothesis I didn't want to test and find broken: I wanted to know whether KALAVAI could do anything for languages that the base model had essentially never seen.
Pythia-410M is an English model. Its perplexity on Yoruba is 41.9 — meaning, roughly, it has no idea what's coming next. Its perplexity on Welsh is 102.7, which is worse than random. Both languages are functionally invisible to the base model. The question was: if a Yoruba speaker fine-tunes a copy of Pythia on whatever Yoruba text they can find — the digitised novels of D. O. Fagunwa, public-domain newspaper archives, Wikipedia in Yoruba — and a Welsh speaker does the same with whatever Welsh text they have, and a Tamil literature student does the same with literary Tamil, and a code contributor adds Python on top of all of that, and then the four checkpoints get fused — does anything happen?
Something happens.
Yoruba perplexity drops from 41.9 to 7.7. That is a 5.4× reduction. Welsh drops from 102.7 to 22.1, a 4.6× reduction. The Tamil specialist learns from a literary corpus no frontier lab will ever prioritise, because there is no commercial reason to prioritise it. The fused model handles all four languages simultaneously; the router learns to dispatch tokens correctly based on language alone, with no script-level supervision. The cooperative produces a multilingual model that none of the four contributors could have built individually, and that no English-only lab would ever choose to build.
Cost per contributor, end to end, including failed runs and the few hours spent on data cleaning: roughly $5 to $10 in electricity. That is not a typo. Each language specialist was a single-GPU fine-tune of a 410M model for 2,000 steps on whatever public-domain text the contributor could find. The router added a few cents of compute on top. The total compute cost of building a four-language model that meaningfully reduces perplexity on three under-resourced languages is well under the cost of a single dinner.
That is the number that I keep coming back to, because it is the number that changes the economics of training a useful language model in a language that frontier labs ignore. The standard answer — "we'll get to Yoruba eventually, but we need the data and the compute" — assumes the data and the compute have to come from one place. KALAVAI does not require that assumption. The data and the compute can come from the people who already have them, and the cooperative can stitch the result together at the end.
I keep being asked whether this is "really useful" or just "a research curiosity." My honest answer is that we have not yet run a real-world cooperative end-to-end. The 20-contributor experiment in the paper simulates the federation on single machines because we don't yet have the infrastructure to actually run twenty unrelated contributors against a shared checkpoint registry with asynchronous submission. But the mechanism works in simulation, the cost numbers are measured rather than projected, and the bottleneck has moved from "is the math possible" to "can we build the infrastructure." That second question is engineering. The first question was the harder one.
What this makes possible
I want to spell out, briefly and concretely, what the protocol unlocks — not because I think every one of these is going to happen in the next year, but because the shape of what becomes possible is what matters here.
A hospital network with five hospitals, each with thousands of records in different specialties, where privacy laws prevent pooling, can now train five specialists on private data that never leaves their respective servers and share only the trained checkpoints. The fused model understands all five specialties. No data was shared. No privacy regulation was violated. This was structurally impossible with synchronous federated learning, which leaks information through gradient sharing. Zero communication during training is a stronger privacy property than zero data sharing.
A country — a small one, with one language, one ministry of justice, one ministry of education — can train a sovereign national model without negotiating with a hyperscaler. The university trains the language specialist. The ministry of justice trains the legal specialist on national law. The education department trains on school textbooks. Each institution uses the hardware it already has. The fused model is a national AI that no foreign company built and no foreign company controls. Digital sovereignty stops being a question of negotiation and starts being a question of coordination.
A consortium of law firms across jurisdictions — Indian, UK, EU, Brazilian, US — can each train on their proprietary corpora and contribute a checkpoint without sharing the underlying case databases that are their competitive advantage. The fused model can reason about cross-border contracts in ways no single firm's model can. The data stays in the firm. The capability is shared.
Endangered languages, the use case I personally care about most, become economically buildable. Yoruba is not a typo or a special case — it is the easy case. Once the protocol exists and the formula tells you the expected gain, a small group of contributors can build a multilingual model with a long tail of under-resourced languages at a cost that does not require a grant. The cooperative changes the economics from "one organisation needs everything" to "each community contributes what it has."
These are not five different stories. They are one story, told five times: when training does not require communication, contribution does not require capitulation.
What I am explicit about not claiming
I want to be honest about the boundary conditions of this work, because reviewers will be honest about them whether I am or not.
No inference efficiency. N specialists run in parallel at inference. For N=3, the inference cost is approximately 2.5× a single dense model. KALAVAI is a training-time democratisation; it does not lower the cost of serving the resulting model.
No frontier-scale evidence. The largest model we tested is 6.9 billion parameters. The mechanism is consistent through that scale, and the predictive formula is satisfied at every tested scale, but we have no direct evidence above 7B. The paper is explicit about this and I want to be explicit about it here.
No real cooperative demonstrated. All experiments in the paper simulate the cooperative on single machines. Heterogeneous hardware, asynchronous contributor submission, contributor reliability, checkpoint authentication — these are open engineering problems. The mechanism works; the infrastructure to run it at scale across the open internet does not yet exist.
No guaranteed downstream benchmark gains. Perplexity improvements are clear and consistent. Standard benchmark accuracy improvements (ARC, HellaSwag, WinoGrande, PIQA) at 1B are modest — less than one percentage point on most. This is consistent with the wider literature finding that perplexity and benchmark accuracy do not reliably track at sub-7B scales. A cooperative aligned to benchmark-relevant domains, rather than the code/science/fiction split we trained on, would present a stronger evaluation. We have not yet run that experiment.
Limited architecture generality so far. All primary results use Pythia. Qwen-1.5B shows +1.06% — small but positive, and consistent with the predictive formula's floor. The mechanism's behaviour across diverse architecture families remains under-tested.
I would rather state the boundary conditions clearly than be surprised by them in a reviewer's footnote.
What KALAVAI is really about
The name is the Tamil word கலவை — mixing, fusion, blending. The protocol mixes independently trained specialists into something none of them could be alone. The Murai Labs thesis I keep coming back to is the same one this paper sits inside: disciplined orchestration beats bigger models. Not because bigger models are wrong — they aren't — but because they are not the only path forward, and they are the path that requires the most concentrated capital and the most concentrated data, and those concentrations are not the only resource the world has to offer.
KALAVAI is an attempt to make cooperative training a protocol rather than a framework or a service. The distinction matters. A framework is a piece of code that someone has to maintain. A service is a vendor that someone has to trust. A protocol is a small set of constraints — shared initialisation, joint inference, learnable suppression — that, once satisfied, let twenty independent parties produce a single useful model without any one of them needing to trust, coordinate with, or share data with any of the others. The protocol is small enough that I can describe it in a paragraph. The empirical formula is small enough to fit in one line. The training run reproduces in thirty minutes on a consumer GPU.
That is the part that I think is interesting and that I am going to keep building on. The paper is the first step. The infrastructure is the next step. The cooperative — a real one, with real contributors, real hardware diversity, real institutional friction — is the step after that. None of those steps are easy. All of them are now possible in a way they weren't before.
The arXiv paper has the full method, the full ablations, the full audit chain (322 automated checks pass before any reported number is published), and the corrected per-domain evaluation protocol that took us three iterations to get right. The code reproduces the 410M result in thirty minutes on any 24GB consumer GPU. The models — three specialist checkpoints plus the fused MoE — are public on HuggingFace, MIT licensed. If you have a GPU and a domain you care about, the protocol is ready for you to try.
Resources
- Paper: arxiv.org/abs/2603.22755
- GitHub: Murai-Labs/Kalavai
- Live walkthrough: murailabs.com/kalavai
- Models on HuggingFace
- Under review at NeurIPS 2026.
Thanks to everyone who reviewed early drafts of the paper and pushed back on the predictive model — that formula went through five revisions before it stuck. And thanks to the engineering review chain that caught the original mixed-batch evaluation inconsistency; the corrected per-domain protocol is what every number in this post and the paper uses.