[Feature]: GLM 5.3 Flash running in 192GB

[nomonkeynodeal]

There was a small patch involved in getting this model to work: https://huggingface.co/JJ48-24/GLM-5.3-Flash-AWQ-W4A16-aggr-w8-mtpq8c-int4g32-mixA

In 192GB VRAM (4 x 48gb 4090), it runs pretty well without MTP at 256k context. vLLM says ~320k should be possible. I only got MTP running at 192k so far.

no MTP:

run	cache	pp tok/s	tg tok/s	e2e tok/s	TTFT s	total s	p_tok	c_tok
w	warmup	171.5	97.0	92.5	0.128	2.768	22	256
2	bust	4606.5	96.0	43.8	3.175	5.843	14626	256
3	bust	4602.9	95.9	43.8	3.178	5.848	14630	256
4	bust	4598.1	95.9	43.8	3.181	5.849	14627	256
5	hit	62734.2	96.0	88.3	0.233	2.900	14627	256
avg bust		4602.5	95.9	43.8	3.178	5.847	14628	256
avg cached		62734.2	96.0	88.3	0.233	2.900	14627	256

  --env NCCL_P2P_DISABLE=1
  --env NCCL_ALGO=Ring
  --env NCCL_PROTO=Simple
  lazymio/vllm-backport:latest-sm89
  models/JJ48-24/GLM-5.3-Flash-AWQ-W4A16-aggr-w8-mtpq8c-int4g32-mixA
  --served-model-name default_assistant_model --tensor-parallel-size 4
  --gpu-memory-utilization 0.98 --max-model-len 262144 --max-num-seqs 1 --max-num-batched-tokens 2048
  --reasoning-parser glm45 --enable-auto-tool-choice --tool-call-parser glm47
  --enable-prompt-tokens-details --enable-prefix-caching --trust-remote-code
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[1]}'

MTP:

run	cache	pp tok/s	tg tok/s	e2e tok/s	TTFT s	total s	p_tok	c_tok
w	warmup	36.2	138.3	104.1	0.609	2.460	22	256
2	bust	3475.5	166.5	44.6	4.209	5.746	14627	256
3	bust	4356.9	173.3	53.0	3.357	4.835	14627	256
4	bust	4384.3	184.2	54.2	3.336	4.726	14628	256
5	hit	29403.6	188.7	138.1	0.497	1.854	14628	256
avg bust		4072.2	174.7	50.6	3.634	5.102	14627	256
avg cached		29403.6	188.7	138.1	0.497	1.854	14628	256

  --env NCCL_P2P_DISABLE=1
  --env NCCL_ALGO=Ring
  --env NCCL_PROTO=Simple
  lazymio/vllm-backport:latest-sm89
  ${models}/JJ48-24/GLM-5.3-Flash-AWQ-W4A16-aggr-w8-mtpq8c-int4g32-mixA
  --served-model-name default_assistant_model --tensor-parallel-size 4
  --gpu-memory-utilization 0.986 --max-model-len 196608 --max-num-seqs 1 --max-num-batched-tokens 1792
  --reasoning-parser glm45 --enable-auto-tool-choice --tool-call-parser glm47
  --enable-prompt-tokens-details --enable-prefix-caching --trust-remote-code --port ${PORT}
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[1]}'
  --speculative-config '{"method":"mtp","num_speculative_tokens":3,"draft_sample_method":"greedy"}'

DeepSeek created the patch. I was going to submit a PR, but I don't what I'm doing and I think that might actually make more of a mess.

The vllm-backport fails running this model with:
KeyError: 'model.language_model.layers.42.self_attn.in_proj_qkvbfg_a.weight_packed'

The cause is not the quantization format or any kernel — both are already supported. It is two hard-coded assumptions in the GLM-5.3-Flash model code that self-attention weights are always BF16.

The patch:

Two files, ~25 lines each.

1. New helper (added to both files):

def _self_attn_quant_config(quant_config):

if quant_config is None:
    return None
try:
    from vllm.model_executor.layers.quantization.compressed_tensors.compressed_tensors import (  # noqa: E501
        CompressedTensorsConfig,
    )
except Exception:
    return None
if not isinstance(quant_config, CompressedTensorsConfig):
    return None
for scheme in getattr(quant_config, "target_scheme_map", {}).values():
    if scheme.get("format") != "pack-quantized":
        continue
    if scheme.get("input_activations") is not None:
        continue
    weight_quant = scheme.get("weights")
    qtype = getattr(weight_quant, "type", None)
    if getattr(qtype, "value", qtype) == "int":
        return quant_config
return None

2. kda.py — after the existing try/finally:

    if _self_attn_quant_config(saved_quant_config) is not None:
        self.quant_config = saved_quant_config
   

3. model.py — replace the quant_config=None at line 331 with:

            quant_config=_self_attn_quant_config(quant_config),
   

Should be safe for existing checkpoints.

[wtdcode]

I'm not sure why it needs a new model? What's the original issue?

[nomonkeynodeal]

The original issue is that your awq is 191GB. As far as I know, this is the first time GLM 5.3 Flash can be run on 4 x 48GB 4090s fast in vLLM.

[vikcious]

I'm not sure why it needs a new model? What's the original issue?

For running the model in 8x 24gb 3090 or 4x 48gb 4090, the model îs a bit to big at 192Gb.
Also it would be nice if you could mention in the recipe at least the minimum VRAM size for a specific model.

And many thank for dealing with it so far, for all your efforts! 🙏