The Bug Was Never in the Weights: Quantizing GLM-5.3-Flash to 4 Bits for Two DGX Sparks

Here is the problem in one number. GLM-5.3-Flash, the mixture-of-experts model Z.ai released on August 26, has about 320 billion parameters. Stored the way it was trained, in 16-bit floats, the weights come to 598.5 GiB. The two DGX Spark boxes in my office have 128 GB of memory each. The model does not fit, and it does not fit by a lot.

That gap is what quantization is for. It is the reason almost every large model anyone actually runs outside a hyperscaler has been through it, and the idea is almost embarrassingly simple. A weight is a number like 0.0137. You do not need all sixteen bits of it. Take a small group of weights, pick one scale for the group, divide each weight by that scale, round to the nearest point on a coarse grid, and store the grid index instead of the number. At 4 bits the grid has sixteen points. When it is time to do the math you multiply the index back by the scale and get 0.0134 or 0.0141 instead of 0.0137, and the model mostly does not notice. Four bits per weight instead of sixteen turns 598 GiB into 181 GiB, and 181 GiB fits on two Sparks with room left for a long context window.

The catch is the word mostly. Quantization is lossy in a way that is unusually hard to see. A model that has been squeezed too far does not crash, and it does not stutter. It keeps producing fluent, grammatical, confident text while the meaning quietly comes unstuck from reality. The grammar survives longer than the sense. So the real question was never "can I make it fit." It was "what did I lose, and how would I know?"

I wanted a proper answer to that, not a recipe copied from a model card. This post is the version of that week with the reasoning left in. The repo has every table and every log reference.

The short version: I quantized a 320B model to 4 bits, measured the damage on every one of its 37,152 expert tensors, found a recipe that does 23% less damage than the stock one, and then discovered that the worst thing happening to the model's output was not in the weights at all. It was one wrong index in the serving code.

Why two Sparks, and why 4 bits

The target is tensor-parallel serving across both nodes, over the 200G ConnectX-7 link between them. The format is NVFP4, weight-only. Each weight is a 4-bit float; weights are grouped in blocks of sixteen, each block carries an 8-bit scale, and each tensor carries one 32-bit global scale on top of that. The activations flowing through the model stay in 16-bit, which is what the "W4A16" you will see on model cards means. That two-level scale is the bookkeeping I mentioned above, and I promise it comes back later in this story.

Not everything gets squeezed. Only the routed-expert feed-forward weights do, which is 37,152 tensors and about 97% of the parameters. The remaining 1,618 tensors stay in 16-bit: attention, the shared experts, the routers, the embeddings, the output head, the norms, the multi-token-prediction head, and the first few dense layers. I did not invent that partition. LibertAI published it, and my starting point, which I call P1, is an exact replica of their card: NVIDIA's ModelOpt 0.45.0, the same 43-entry ignore list, no calibration data, weights streamed shard by shard through CPU memory. The only thing I added was a tap. Every tensor, as it was quantized, wrote down how far the round trip moved it: the cosine between the original and the reconstruction, the relative error, and what happened to its scales.

That file turned out to be the most useful thing I made all week.

37,152 instruments, one flat landscape

I had a plan for that file. Everyone who quantizes a mixture-of-experts model has the same instinct, and I had it too: some experts must be fragile, some layers must be sensitive, and the clever recipe protects those by keeping them in higher precision. Build the exemption list. Ship the checkpoint. I expected the tap to tell me which tensors to protect.

It told me not to bother.

The error landscape is flat. Not roughly flat. Flat in a way that made me re-run the analysis to check for a bug. Across the whole model the per-tensor cosine similarity sits between 0.99544 and 0.99604, so the worst tensor in the model is 1.07 times the average. The worst 1% of tensors carry 1.06% of the total error, which is what you would get if the error were spread perfectly evenly. I ran an ANOVA across the 288 expert indices looking for hot experts and got p = 0.52; the spread between expert indices is 0.993 times what pure sampling noise would produce. A "bad" expert in one layer tells you nothing about the same index in the next layer. The correlation between adjacent layers' expert rankings is 0.0001, which is the kind of number you only see when there is genuinely nothing there. Eighty-six percent of the variation in error is tensor-by-tensor noise that no structural label explains.

I priced the best exemption list I could build anyway, using the oracle answer: take the 372 worst tensors and keep them in higher precision. That buys a 1.06% reduction in mean error for a 2.6% larger checkpoint. Exempting the entire multi-token-prediction layer buys 1.23% for 5.9% more bytes. Neither is worth doing, and the reason is one curve:

Exemption lists work when the error has a tail. This error does not have one.

One real structural signal did survive. Layer 45, the multi-token-prediction layer that drafts speculative tokens, is a mild outlier: about 6% worse than the rest of the model, uniformly, and every one of the 245 scale clamps in the whole checkpoint lives there. But whether that layer deserves 16-bit experts is a question about whether the drafts get accepted, not about weight error. On my stack, with layer 45's experts quantized and only its head kept in 16-bit, speculative decoding accepted 51.6% of draft tokens, an average of 2.07 accepted out of every 4 proposed. That held up fine. (Draft acceptance, if you want the mechanism.)

A footnote that cost me an afternoon. LibertAI's card reports a round-trip error of 0.0925 and a cosine of 0.99665. My replica matched their error to three decimal places. But those two numbers cannot both be per-tensor averages, because for this kind of statistic the squared error is very nearly twice one-minus-cosine, and 0.0925 implies a cosine near 0.99575, not 0.99665. I spent that afternoon assuming my quantizer was wrong. It was not. Their cosine was aggregated some other way. I mention it because the next person will hit the same wall, and because my own pair, 0.0921 and 0.99576, does satisfy the identity.

Fix the quantizer, not the exemption list

If there is no tail to protect, the only place left to find error is in the quantizer itself. So I rewrote ModelOpt's stock quantizer in plain code, checked it produced bit-identical output to the library before changing anything, and then tried levers on a sample of 872 tensors: the worst 372, 400 chosen at random, and 100 from layer 45.

The winner is boring and old. Instead of setting each block's scale from its largest value, try nine multipliers on that scale, from 0.70 to 1.75, and keep whichever one reconstructs the block best. It won on 872 out of 872 tensors, including the ugly ones: the single worst tensor went from 0.0956 error to 0.0849. It costs nearly nine times the quantize time, which on CPU streaming means hours instead of minutes, and it still needs no calibration data.

I built the full checkpoint with it, called it P2, and the numbers were lovely. Mean squared error dropped from 0.8487% to 0.6335%, a quarter less, and every single one of the 37,152 tensors improved. Not most of them. All of them.

Then I ran the behavioral gates and the model was worse.

The gates are a small battery of things I actually use this model for. One of them hands the model a codebase and asks it to find every call site of a function across files, at 32k and again at 100k tokens of context. The stock checkpoint went 6 for 6 on both. P2 went 0 for 4. A tool-calling probe went 0 for 2. A quarter less weight error, and the model got worse at its job.

I had written down before building P2 that this was the outcome I was most afraid of, that improving the weights would not improve the behavior. Writing it down first is the discipline I try to hold myself to on everything: decide what would kill the exciting story before you have the data, so you cannot talk yourself out of it afterward. But predicting a failure is not the same as understanding it, and the explanation came from looking at what the scale search was actually doing to individual blocks.

Multipliers below 1.0 shrink a block's scale. That makes the grid finer for the bulk of the block's values, which is why the error goes down, but it also means the block's largest weights no longer fit on the grid and get clipped. On this model the clipping was diffuse, around 4% of a block's energy clipped at meaningful depth, but it was showing up on somewhere between 9% and 20% of the blocks in every single tensor. A little deep clipping, everywhere. The objective was delighted to trade a few large weights for many precise small ones. The model was not.

So P3 forbids multipliers below 1.0. No clipping is possible by construction, and it costs about 2.5 points of the gain. This was the experiment that decided things, and I had written that down too: if the gates came back, clipping was the cause.

They came back. The long-code probe went from 0 for 4 to 2 for 3 at both context lengths, and tool-calling went from 0 for 2 to 1 for 3. P3 keeps almost all of P2's improvement, 22.8% below the stock recipe, with no clipping at all. That is the checkpoint on Hugging Face.

I want to be careful about what this shows, because someone else got close to it from the other side. voska's card for their own 4-bit build says they tried an MSE-optimal clip search and threw it out: "~9% lower weight L2, no better in practice; optimising weight-space error is the wrong objective at 4 bits." I think that is almost right, and the gates let me say where the almost is. Optimizing weight error was fine. One specific lever for doing it had a side effect the objective could not see. Take away the side effect and the gain stops hurting. Weight error is a proxy, and a proxy always tells you about itself first.

The bug that was never in the weights

If you only read one section, read this one.

Serve this family of checkpoints on an unpatched vLLM and the model will, now and then, emit U+FFFD, the Unicode replacement character, the little diamond with a question mark in it. Tool calls come back malformed. It happens mostly in Korean, Tamil, Japanese, anything that takes more than one byte per character, and almost never in English, which is exactly why an English smoke test passes. The server logs nothing. The probe that catches it belongs to tonyd2wild, who first measured the pattern on this model family: 4, 9, and 8 corruption events across three runs on checkpoints packed by ModelOpt, and 0, 0, 0 on a checkpoint packed by a different tool. I ran the same probe against my own P3 on stock vLLM and got 8 events in 6 runs. Korean tool-calling produced 1, 3, and 4. English code and JSON produced none.

The cause is one line. Remember the two-level scales from the top of this post, one per block and one per tensor? In a mixture-of-experts model, each expert has a "gate" projection and an "up" projection, and vLLM fuses those two weights into one matrix for speed. When it repacks that fused matrix for the marlin kernel, it grabs the global scale like this:

w13_weight_scale_2 = layer.w13_weight_scale_2[:, 0]

Index zero. The gate projection's scale, applied to both halves. vLLM even knows this might be wrong, because it logs a warning when the two scales differ ("w1_weight_scale_2 must match w3_weight_scale_2. Accuracy may be affected.") and then carries on. The up half gets reconstructed with the wrong scale, off by the ratio between the two, separately for every expert. Whether the two scales happen to match is a property of how the checkpoint was made, not of the code, and you can count it without touching a GPU by reading the safetensors headers:

On my checkpoint, seven out of ten expert pairs have mismatched scales, and one of them is off by a factor of ten. RedHatAI's checkpoint, made with a different tool that forces the two scales to be equal, is immune. That is the split tonyd2wild saw. Any checkpoint made the ModelOpt way, with separate gate and up projections, is affected, including LibertAI's.

I wanted the proof to not depend on a running server, so I did it on a laptop: the original 16-bit weights, ModelOpt's own reconstruction, layer 10, expert 0, where the two scales differ by a factor of 1.23. The reconstruction error on the up projection:

The wrong scale adds nearly three times as much error as the quantization itself did. An expert whose ratio is 0.93 goes 0.1124 to 0.0924 once compensated. An expert whose ratio happens to be exactly 1.0 is untouched, as it should be.

The fix is twenty lines that run before the repack: multiply the up half's block scales by the ratio of the two global scales, clamped to the largest value an 8-bit float can hold. It ships as a patched copy of one vLLM file that you mount over the original inside the container, so there is no image to rebuild. On my checkpoint the compensation averages 1.02 per layer, maxes at 2.0, and hits the clamp on about three in ten thousand scales. I got it wrong once on the way: the fused scale tensor is three-dimensional and split across the two nodes, so the half-split has to know which rank it is on. My first version did not, and failed in a way that took a while to read. I also did not try the other obvious fix, requantizing both halves to the larger scale. It might be better. I have not measured it.

Same checkpoint, same probe, only the repack changed:

Zero is easy to fake with an empty string, so I read the outputs. I ran the original Korean and English probes ten times each, then Japanese, Chinese, Tamil, Arabic, Russian, Hindi, an emoji-heavy prompt, and a code-switching prompt three times each, then two 8,000-token Korean essays. All clean, and the Korean is real, structured, coherent Korean, not a lucky absence of bad bytes.

Changing the numerics could have changed the behavior, so I re-ran the gates on the patched build. Long-code at 32k went 2 of 3 to 3 of 3. At 100k it went 2 of 3 to 1 of 3, which at temperature 1.0 with three samples is noise. Tool-calling went 1 of 3 to 2 of 3. Code execution stayed at 0 of 2. Neutral to slightly better, with the corruption gone. It is on by default.

I wrote all of this up on vllm-project/vllm#54150, where the original reporter had asked the right question and left it open: are the weights damaged, or is vLLM's ModelOpt path wrong? It is the second one. Their report came from four RTX PRO 6000s, a different Blackwell chip on the same vLLM commit: 86 corruption events in 6 runs on LibertAI's checkpoint, 94 on another ModelOpt build, 0 on RedHatAI's. Mine adds the GB10. The bug does not care which Blackwell it is on. It happens when the weights are repacked, before any kernel runs.

Where this sits among everyone else's work

None of this happened alone, and the people whose work I leaned on deserve a straight account of how mine compares.

LibertAI published the recipe I replicated as P1: the partition, the format, the data-free CPU streaming. Their checkpoint has the same repack problem, because the problem is in how ModelOpt packs scales, not in anything they chose. I did not re-run my scale census on their weights before they left my cluster, so that particular claim is reasoning, not measurement. The corruption on their checkpoint is measured, in the vLLM issue.

RedHatAI's checkpoint is immune, and I confirmed it clean on my own stack, 0 for 6, matching the issue reporter's control. It quantizes activations too, which mine does not, and on my stack it has a chat-parser quirk on long code prompts that I did not get to the bottom of before release. Their card lists GPQA 90.57, AIME25 86.67, MATH-500 94.87. Those are their numbers, with no baseline column to compare against.

inco.ai shipped the day-0 NVFP4 checkpoint and the DFlash 2 drafter, and their launch post reports FP8-to-NVFP4 parity on their suite and up to 4.4 times the throughput at concurrency 1 on their engine. DFlash 2 is a block-diffusion drafter, about 2.3 GiB, lossless by rejection sampling. Because it drafts from token IDs rather than from the target's internals, its acceptance rate survives quantizing the target, which is why the community reaches for it when a quantized model's own MTP head starts getting rejected. My MTP held at 51.6%, so I shipped without it. I have not measured it and am not claiming anything about it.

EXL3, Mia-AiLab's 4-bit-per-weight recipe, was the sparring partner. On malaiwah's third-party panel it posts a much lower KL divergence from the original model than NVFP4 does, 0.0246 against 0.0605, though that panel mixes KV-cache formats and I have no KL harness of my own. That advantage did not show up on my gates:

† P3's code-execution cell used to read 1 of 2 until I hardened the grader to check for empty output at the token cap. The old pass was an empty string. EXL3's diorama failures were the model thinking for more than 200,000 characters and then producing nothing, at two different temperatures. EXL3's genuine advantages are operational: faster decoding with DFlash 2, a million tokens of context, a KV pool nearly twice mine. On this battery they do not translate into better output. That makes three quantizer families now where weight-space distance and behavior have come apart.

KVarN, a calibration-free KV-cache quantizer from Huawei's CSL group that stores keys at 4 bits and values at 2 and reports 2.77 times the cache capacity at accuracy parity on GLM-4.7-Flash, looked like the obvious next lever, and nobody had tried composing it with NVFP4 weights. It does not run on a GB10. Its sparse-MLA path bottoms out in a compiled kernel built on tensor-memory instructions the GB10 does not have, and there is no Triton fallback; adding the architecture to the build produces a silent empty stub. The spike took an hour to prove that. When FlashInfer or FlashMLA ships a sparse-MLA kernel for sm_12x, it deserves another look.

The stack also carries two community fixes for the GB10 itself, mounted the same way as my patch: tonyd2wild's fix for a hard crash on top-k above 24K, and vcruz305's fix for an out-of-bounds read at the tail of the KV pool. I would not serve this model at long context on this vLLM without them.

What I am explicit about not claiming

This is not a benchmark suite. Each gate cell is two or three samples at temperature 1.0. A one-cell difference is noise, and I have tried to only read the cells where the direction was large and I had predicted it.

Code execution is at risk on this checkpoint. P3 is 0 for 2 on that probe under the strict grader. P2 was 2 for 2. If that is your workload, do not assume P3 is fine.

Very long single-shot generation is unproven on P3. The only checkpoint that ever rendered the 64k-token diorama scene was P2, once.

The older gate numbers predate the runtime fixes and cannot be reattributed to the current stack.

I have no KL-divergence measurement of my own. Every KL number above is someone else's, taken in their setup.

This checkpoint carries no activation scales. It is validated only on the marlin MoE backend and will produce garbage on vLLM's default FlashInfer NVFP4 path. Do not drop --moe-backend marlin.

I would rather write the boundary conditions myself than have someone else find them in a footnote.

What I am taking away from the week

The map was worth building because it killed the plan. I instrumented 37,152 tensors expecting to find the ones to protect, and the answer was that there are none. That is a cheap thing to learn from a JSONL file and an expensive thing to learn by shipping a checkpoint with an exemption list that buys nothing. The measurement that ruins the exciting hypothesis is the one to run first.

P2 improved every tensor in the model and got worse at the job. I keep coming back to that. The quantizer optimizes weight error because weight error is what it can compute, and that works right up until the lever you pull has a side effect the number cannot see. The behavioral gates are not a check you run after the quantization. They are the experiment.

And I spent most of the week inside the quantizer while the biggest quality problem on the whole path was [:, 0] in a function I had never read. The single most useful thing I did was run someone else's probe against my own checkpoint and take the answer seriously when it came back dirty.

If you serve any ModelOpt-packed NVFP4 mixture-of-experts on vLLM, mine, LibertAI's, anyone's, run the probe. It is in the repo and takes a few minutes. If it comes back dirty, the patch is twenty lines, and the mechanism, the proofs, and the upstream thread are all there to argue with.

Resources

• Weights: huggingface.co/murai-labs/GLM-5.3-Flash-NVFP4-Murai — 181.3 GiB, 120 shards
• Patch, probe, launcher, recipe: github.com/Murai-Labs/GLM-5.3-Flash-NVFP4-Murai
• Upstream report: vllm-project/vllm#54150
• Hardware: 2× NVIDIA DGX Spark (GB10, sm_121, 128 GB unified each), 200G ConnectX-7. Serving image radixark/vllm-glm53-flash:sm121-v8, TP=2, marlin MoE backend, fp8 KV cache, MTP speculative decode with 4 draft tokens.
• If the scale-and-grid idea at the top of this post was new to you, or you want to see the "fluent gibberish" failure with your own eyes, Chapter 27 of Under The Hood builds a quantizer from scratch and pushes it down to 2 bits until the meaning falls out. It is free to read. Project 14 does the same for speculative decoding, MTP and DFlash included. The whole book is on Leanpub with a reader discount, Kindle, and paperback.

tonyd2wild's probe found this class of corruption first, and chriswritescode-dev's report on the vLLM tracker is what sent me looking. If your own stack contradicts any number here, the probe and the scale-census script are public precisely so they can be argued with.