DEV Community 18m ago

Cutting 64 of 256 Experts from a GGUF by Byte Copy: What the Headers Say About GLM-5.3-SLIM on One B200

TL;DR: Scantist's GLM-5.3-SLIM-GGUF is a 192-expert version of Unsloth's 256-expert UD-IQ2_M file, and it is built by copying bytes, not by requantizing. We read both sets of GGUF headers on November 11, 2026. The file drops from 238.58 GB to 179.86 GB, a 24.6% cut, and the header math says 93.4% of that cut is the expert slice, 6.4% is a removed next-token-prediction (MTP) layer and 0.2% is router rows.

Every one of the 225 routed-expert tensors in the new file is exactly 0.75 times the size of its source tensor, in the same quant type, and all 1,407 other non-router tensors keep their names, shapes and types.

The practical point for anyone sizing hardware: the cut buys fit, not speed. Both files read about 21.5 GB of weights per decoded token, because top-8 routing is unchanged. On one B200 the smaller file leaves roughly 11.6 GiB for everything else, which is the difference between "does not load" and "about 100K tokens of shared KV cache".

What did Scantist actually ship?

The repository Scantist/GLM-5.3-SLIM-GGUF was created on November 2, 2026. When we read it on November 11 it showed 1,303 likes and 11,066 downloads. It holds six shards of one model plus an importance-matrix file and three evaluation scripts.

The base is unsloth/GLM-5.3-GGUF, folder UD-IQ2_M, which also has six shards. We parsed the tensor headers of all twelve shards with HTTP range requests, so no weights were downloaded. Here is what the headers say.

Unsloth UD-IQ2_M / Scantist E192
Routed experts per MoE layer (expert_count): 256 / 192
Experts used per token: 8 / 8
Transformer blocks: 79 / 78
MTP layer (nextn_predict_layers): 1 / 0
Tensors: 1,809 / 1,782
Parameters (from tensor shapes): 753.9B / 562.7B
File bytes, all shards: 238,577,585,701 / 179,861,261,600
Average bits per weight: 2.532 / 2.557

The average bits per weight goes up slightly even though nothing was requantized. That is not a contradiction. The experts are stored at about 2.42 bits and most other tensors at 4 to 8 bits, so when you remove a quarter of the experts, the high-precision tensors weigh a little more in the average.

Why is the saving 24.6% and not 25%?

Cutting 64 of 256 experts sounds like a clean 25%. The headers show why it lands a bit lower on the file and higher on the experts. We split the 58,716,322,560 bytes of tensor savings into three pieces.

Expert slice, layers 3 to 77 (75 layers): 54,842,621,952 bytes saved, 93.40% share of the saving, 22.99 points of the 238.6 GB file
MTP layer (block 78, attention, experts, indexer, everything): 3,755,716,608 bytes saved, 6.40% share, 1.57 points
Router rows (ffn_gate_inp, exp_probs_b): 117,984,000 bytes saved, 0.20% share, 0.05 points
Sum: 58,716,322,560 bytes, 100%, 24.61 points

The three rows add up to the byte. The expert slice is exactly 25% of the routed-expert bytes in those 75 layers (54.84 of 219.37 GB), but those experts are only 92% of the original file, because attention, shared experts, embeddings and the output matrix do not shrink. A quarter of 92% is 23.0 points. The MTP layer then adds 1.57 points by disappearing altogether, which is how the total reaches 24.6%.

How do you remove experts from a GGUF without requantizing?

In GGUF the expert axis is the last, slowest-varying dimension of each routed-expert tensor. The header for blk.20.ffn_down_exps.weight lists dimensions [2048, 6144, 256] in the source and [2048, 6144, 192] in the new file. One expert is therefore one contiguous byte range of the same length in both files, and keeping experts means copying ranges.

For the IQ3_XXS type used by that tensor, a block of 256 weights takes 98 bytes, so one expert is 2048 x 6144 / 256 x 98 = 4,816,896 bytes.

Two more checks make the claim hard to argue with:

All 225 routed-expert tensors in the new file (75 layers times gate, up and down) have the same quant type as their source and exactly 0.75 times its bytes. We found zero exceptions.
The 1,407 tensors that are neither routed experts nor router rows have identical names, shapes and types in both files.

That matches what the model card says: kept experts are copied byte for byte, in their original order, as experts 0 to 191. The card also reports that router rows are bit-identical to those of an independent NVFP4 checkpoint built with the same expert selection. We did not verify that part, since it needs the weights.

The removed MTP layer has a quirk worth knowing. Block 78 in the Unsloth file stores its experts as Q3_K (down) and Q2_K (gate and up), while every other layer uses IQ types. It is the only place those two types appear, so the "0.75 times" rule holds for the 75 kept layers and the MTP layer has to be accounted for separately.

Does a smaller file decode faster?

No, and the header math shows why. A token only reads the experts the router picks. With 8 of 256 or 8 of 192, the bytes per expert are the same, so the weight bytes read per token barely move. Counting every tensor except the token embedding (a row lookup) and taking 8 experts per MoE layer:

Unsloth, without the MTP layer: Non-expert bytes per token 14.79 GB, 8 routed experts, 75 layers 6.86 GB, Total weight bytes per token 21.64 GB
Scantist E192: Non-expert bytes per token 14.67 GB, 8 routed experts, 75 layers 6.86 GB, Total weight bytes per token 21.53 GB

Scantist reports 34.5 tokens per second for a single sequence on one B200 with an f16 KV cache. Multiplied by 21.5 GB per token, that is an implied 743 GB/s of weight traffic. NVIDIA lists B200 memory bandwidth at about 8 TB/s, so the single-sequence run uses roughly 9% of it.

The model card says as much: the speeds are limited by the llama.cpp kernels for these low-bit types, not by memory bandwidth. Our figure ignores KV reads and activations, so treat it as a rough ceiling check, not a profile.

The consequence for edge and on-prem teams is simple. Dropping experts is a way to make a model fit, and then batching is what buys throughput: throughput climbs from 34.5 tok/s with one sequence to 203.3 tok/s with 64, most likely because the weights read in one step are shared across all sequences in the batch. These numbers come from Scantist's llama-batched-bench runs, not ours.

What does fitting one B200 leave for context?

The model card reports 183,359 MiB on one B200 and 170,896 MiB of weights on the GPU (the 624 MiB token embedding stays in host memory). Our header total for the E192 tensors is 171,520 MiB, which agrees once you subtract that embedding. The 256-expert file is 227,516 MiB, which overshoots the card's memory figure by 44,157 MiB.

KV cache per token is where we can check the card against the config. The metadata gives a latent width of 512 plus 64 rotary dimensions, so one cached vector is 576 values, or 1,152 bytes at f16. Across 78 layers that is 89,856 bytes, or 87.75 KiB per token, which matches the card's MLA figure exactly. The card adds 5.25 KiB per token for the DSA indexer for a total of 93 KiB. We could not derive the indexer share from the headers, so that part is the card's number. At q4_0 the same arithmetic gives 24.68 KiB for MLA, and adding the card's indexer figure scaled by the 18/64 q4_0 ratio gives 26.16 KiB, close to the 26.2 KiB the card lists.

Setting (card's measurements):

f16 KV, recommended: 102,400 tokens, KV pool GPU used 181,678 MiB, Free 1,681 MiB
f16 KV, too tight: 110,592 tokens, KV pool GPU used 182,454 MiB, Free 905 MiB
q4_0 KV, five long requests: 343,040 tokens, KV pool GPU used 182,226 MiB, Free 1,133 MiB

By our arithmetic the f16 pool costs 9,300 MiB at 102,400 tokens, so about 970 MiB of the reported usage is something other than weights, the 512 MiB compute buffer and KV. At 110,592 tokens that remainder is about 1,000 MiB. The q4_0 run carries about 2,041 MiB, so do not extrapolate the overhead linearly.

One long reasoning request of 68,608 tokens is 6.08 GiB at f16, which is 67% of the 102,400-token pool. That is why the card suggests q4_0 for long-reasoning workloads: the same request costs 1.71 GiB and five fit. The price, per the card, is 11% lower single-sequence speed at 32K tokens of context and 20% lower at 63K.

What does the quality evidence say?

Scantist reports these results from its own scripts:

GPQA-Diamond (198), T=1.0, high reasoning effort, 65,536 max tokens, one sample: 84.8% (168/198)
HumanEval pass@1 (164), greedy, thinking off, serial: 92.7% (152/164)
HumanEval pass@1 (164), same, 8 concurrent requests: 89.6% (147/164)
MMLU, 570-question subset, zero-shot, greedy, thinking off: 71.4% (407/570)

The card is candid about the limit of this evidence: the 256-expert UD-IQ2_M and the full-precision model were not run on the same harness, so none of these numbers measures what dropping 64 experts costs. The serial and concurrent HumanEval runs differ by 9 discordant problems (exact McNemar p = 0.18), which is a useful reminder of how much batch numerics alone can move a score.

The card also warns that the earlier FP8 tests of the same expert set showed a clear loss on Chinese knowledge QA and a smaller loss on science QA. The cosine similarities in the card's integrity section (0.94 for IQ2_XXS, 0.98 for IQ3_XXS, 0.994 for IQ4_XS) show that the kept experts map to the right source experts, against under 0.003 for a neighbouring expert. They are not a quality score for the pruned model.

Can you check this yourself?

The script below reads one shard header from each repository with a 4 MB range request and prints the shape and byte count of the same expert tensor. It needs only the Python standard library.

import struct , urllib . request
BASE = "https://huggingface.co/%s/resolve/main/%s"
BLOCK = { "IQ2_XXS" : ( 256 , 66 ) , "IQ3_XXS" : ( 256 , 98 ) , "IQ4_XS" : ( 256 , 136 ) , "IQ2_S" : ( 256 , 82 ) }
FMT = { 0 : "B" , 1 : "b" , 2 : "H" , 3 : "h" , 4 : "I" , 5 : "i" , 6 : "f" , 7 : "?" , 10 : "Q" , 11 : "q" , 12 : "d" }
TYPE = { 16 : "IQ2_XXS" , 18 : "IQ3_XXS" , 22 : "IQ2_S" , 23 : "IQ4_XS" }
def head ( url , n = 4_000_000 ) :
  req = urllib . request . Request ( url , headers = { "Range" : f "bytes=0-{n-1}" } )
  return urllib . request . urlopen ( req ) . read ( )
def tensors ( buf ) :
  p = 4
  def u ( f ) :
    nonlocal p
    v = struct . unpack_from ( "<" + f , buf , p ) [ 0 ] ; p += struct . calcsize ( f ) ; return v
  def s ( ) :
    nonlocal p
    n = u ( "Q" ) ; v = buf [ p : p + n ] . decode ( ) ; p += n ; return v
  u ( "I" ) ; nt , nk = u ( "Q" ) , u ( "Q" )
  for _ in range ( nk ) : # shards 2..N carry only a few metadata keys
    s ( ) ; t = u ( "I" ) ; s ( ) if t == 8 else u ( FMT [ t ] )
  for _ in range ( nt ) :
    name = s ( ) ; dims = [ u ( "Q" ) for _ in range ( u ( "I" ) ) ] ; ty = u ( "I" ) ; u ( "Q" )
    yield name , dims , ty
for repo , path in [ ( "unsloth/GLM-5.3-GGUF" , "UD-IQ2_M/GLM-5.3-UD-IQ2_M-00003-of-00006.gguf" ) , ( "Scantist/GLM-5.3-SLIM-GGUF" , "GLM-5.3-UD-IQ2_M-E192-00003-of-00006.gguf" ) ] :
  for name , dims , ty in tensors ( head ( BASE % ( repo , path ) ) ) :
    if name == "blk.20.ffn_down_exps.weight" :
      blk , size = BLOCK [ TYPE [ ty ] ]
      per_expert = dims [ 0 ] * dims [ 1 ] // blk * size
      print ( f "{repo.split('/')[0]:9s} dims={dims} {TYPE[ty]} " f "experts={dims[2]} bytes/expert={per_expert:,} tensor={per_expert*dims[2]:,}" )

Output from our run on November 11, 2026:

unsloth dims=[2048, 6144, 256] IQ3_XXS experts=256 bytes/expert=4,816,896 tensor=1,233,125,376
Scantistdims=[2048, 6144, 192] IQ3_XXS experts=192 bytes/expert=4,816,896 tensor=924,844,032

192 / 256 = 0.75, and 924,844,032 / 1,233,125,376 = 0.75 exactly. To reproduce the full tables, loop the same parser over all six shards of each repository, sum bytes by tensor name, and compare each expert tensor against its source.

What did we not measure?

We did not run the model, so every speed, memory-use and benchmark figure above is Scantist's. Our own numbers are the file sizes, tensor bytes, parameter counts, per-token weight bytes and KV arithmetic, all derived from the Hugging Face API and the GGUF headers.

We did not reproduce the claim that a 192 GB Mac can run the file. The weights leave 24.5 GiB of headroom on a 192 GiB machine, but macOS caps how much unified memory the GPU may use by default, so check that limit before you plan around it. The per-token read figure also leaves out KV reads and activations.

What should an edge team take from this?

If you are choosing between a smaller expert count and a smaller bit width, do the header math first. A byte-slice like this one is cheap, reversible and lossless with respect to the source file, but it is a lossy change to the model, and the evidence for its cost is thin here. Separate three questions that are easy to blur: does the file fit, how much context is left, and how fast does a token come out. Pruning experts moves the first two. Batching and kernels move the third.

FAQ

Does removing experts make GLM-5.3 faster? Not by itself. Top-8 routing reads the same number of expert bytes per token, about 6.86 GB in both files, and the whole weight read is about 21.5 GB per token. The gain is that the model fits on one 192 GB-class GPU.

Why is the saving 24.6% and not 25%? The expert slice removes exactly 25% of the routed-expert bytes in 75 layers, which is 23.0 points of the file because attention, embeddings and shared tensors stay. The removed MTP layer adds 1.57 points. The three pieces sum to 58,716,322,560 bytes.

Is the 192-expert file requantized? The header evidence says no. All 225 routed-expert tensors keep their quant type and are exactly 0.75 times the size of their source, and the other tensors keep their names, shapes and types.

How much KV cache does one token cost? The MLA part is 87.75 KiB at f16 by our arithmetic from the config (576 values, 2 bytes, 78 layers). The card lists 93 KiB with the DSA indexer, and 26.2 KiB at q4_0.

Does the pruned model score well? Scantist reports 84.8% on GPQA-Diamond and 92.7% on HumanEval, but did not run the 256-expert baseline on the same harness, so the cost of the pruning is not measured.

Which numbers here are our own? File sizes, tensor bytes, parameters, per-token weight bytes and KV arithmetic. Speeds, memory use and benchmark scores are Scantist's.