[Bug]: Host RAM OOM in process_weights_after_loading (Marlin repack) when most of a MoE's experts are CPU-offloaded

[noonghunna]

Your current environment

• 2× RTX 3090 (sm_86, 24 GiB each, PCIe, no NVLink), 196 GiB host RAM, 32-core host
• lazymio/vllm-backport:latest-sm86, vLLM v0.12.0
• Model: wtdcode/GLM-5.3-Flash-AWQ-W4A16 (compressed-tensors pack-quantized, W4A16)
• Experts CPU-offloaded: --cpu-offload-params experts --offload-backend uva

What happens

With only 48 GiB of VRAM the routed experts (~146.7 GiB) have to live on the host, so this is
offload as the primary weight store rather than a few-GiB top-up. Loading gets all the way
through reading the checkpoint, then dies in process_weights_after_loading — immediately after
int_wna16.py logs Using MarlinExperts — with a host OOM kill, not a Python exception and
not a CUDA OOM:

INFO  [uva.py:65] Total CPU offloaded parameters: 73.55      <- per rank, TP=2
INFO  [default_loader.py:430] Loading weights took 271.22 seconds
INFO  [int_wna16.py:409] Using MarlinExperts
...~75 s later...
Out of memory: Killed process (VLLM::Worker_TP)
  total-vm:231892748kB  anon-rss:1117884kB  shmem-rss:90272184kB
RuntimeError: Engine core initialization failed. Failed core proc(s): {'EngineCore': 1}

Reproduce

docker run --gpus all --ipc=host --shm-size 200g \
  -v <weights>:/model:ro lazymio/vllm-backport:latest-sm86 \
  /model --tensor-parallel-size 2 \
  --cpu-offload-params experts --cpu-offload-gb 90 --offload-backend uva \
  --max-model-len 4096 --gpu-memory-utilization 0.85 \
  --disable-custom-all-reduce --enforce-eager

It is not a configuration problem

Two runs varying everything plausible landed in the same place, ~71 s and ~75 s after the
MarlinExperts line:

	run 1	run 2
MTP head	present	removed (num_nextn_predict_layers: 0)
max-model-len	4096	204800
gpu-memory-utilization	0.85	0.90
Total CPU offloaded parameters	73.55 / rank	73.55 / rank (identical)
shmem-rss at kill	90.59 GiB / rank	86.09 GiB / rank
outcome	host OOM in repack	host OOM in repack

Note the offload total is byte-identical across both — dropping the MTP layer changes what is
resident but not what is offloaded, since the offloaded set is the 42 MoE decoder layers.

Two observations

1. shmem-rss at the kill exceeds the reported offload total by 12.5–17 GiB per rank
   (86–91 GiB resident vs 73.55 GiB offloaded). Two ranks put 172–181 GiB of page-locked memory
   resident on a 196 GiB host before any repack transient is counted. I have not attributed
   that gap and am not claiming a cause — but if some of it is reclaimable, it is the difference
   between loading and not.
2. Because the memory is page-locked it cannot be swapped, so adding swap does not help.

Relationship to #67

#67 fixes a separate, earlier ceiling: Tensor.pin_memory() rounds to the next power of two
(~1.78×), so before it the experts needed ~260 GiB and died before the offload total was ever
logged. With it they occupy their true 146.7 GiB and loading reaches the repack — which is where
this issue begins. #67 is necessary to see this at all; it does not address it.

Note on where this belongs

The offloader (vllm/model_executor/offloader/*) and int_wna16.py both carry upstream vLLM
copyright, and prefetch.py is adapted from SGLang — so this may be an upstream issue rather
than a fork one. Filing here because this fork is what makes the model runnable on Ampere at all,
and its users are the ones who will hit it.

Question

Is offloading the majority of a model's weights (rather than a few GiB) a regime you intend to
support? If so, is the Marlin repack expected to hold both pre- and post-repack expert layouts
resident, and is there a streaming or per-tensor variant that would avoid the peak?

[wtdcode]

CPU offload is never tested honestly. I will have a look when I have enough GPUs.