[Bug]: Qwen3-VL E/PD handoff drops per-frame timestamp tokens and frame markers for video inputs; encoder-side mRoPE ids are sliced against the shorter PD prompt #20138

System Info

• TensorRT-LLM 1.3.0rc25 (785c948), PyTorch backend, LLM API only (no serving framework involved); transformers 5.5.4; PyTorch 2.12.0a0 (NGC 26.05 build).
• Re-checked on main at 760a584 (2026-10-12): the code involved is unchanged, see "Status on main" below.
• Model: Qwen/Qwen3-VL-8B-Instruct (bf16). Qwen/Qwen3-VL-2B-Instruct goes through the same code path and is expected to behave the same (not run).
• One GPU per run: the encoder (MultimodalEncoder) and the PD (LLM) share the GPU and the process. Reproduced on two different GPU types.
• Input: synthetic 10 s / 24 fps / 1280x720 H.264 clips; 40 uniformly spaced frames passed as VideoData with metadata {fps, total_num_frames, duration, frames_indices}; mm_processor_kwargs={"do_sample_frames": False} (-> video_grid_thw [20, 36, 66], 11,880 visual tokens, E->PD handle [11880, 16384] bf16).
• Greedy decoding: SamplingParams(temperature=0, top_k=1, max_tokens=64).

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

For video inputs, the E/PD handoff (MultimodalEncoder -> DisaggregatedParams -> LLM.generate_async(disaggregated_params=...)) rebuilds a prompt on the PD side whose layout differs from the aggregated path: the per-frame-group timestamp tokens and the per-group <|vision_start|> / <|vision_end|> markers are missing. The mRoPE position ids/deltas shipped from the encoder, however, follow the aggregated layout, and the PD side slices them to its own (shorter) prompt length without any check. Visual-token positions and the decode-time mRoPE delta end up misaligned. No error is raised; the output departs from the aggregated path within the first few tokens and the model loses its sense of time, while frame content is still recognized.

Minimal reproduction (public APIs only; LLM API, because the trtllm-serve E/PD protocol does not carry multimodal / mRoPE handles yet, see the TODO at tensorrt_llm/serve/openai_protocol.py L320):

import os, cv2, numpy as np, torch
from transformers import AutoTokenizer
from tensorrt_llm import LLM, MultimodalEncoder, SamplingParams
from tensorrt_llm.llmapi import KvCacheConfig
from tensorrt_llm.inputs.multimodal_data import VideoData

MODEL = "Qwen/Qwen3-VL-8B-Instruct"

def load_video(path, n=40):
    cap, raw = cv2.VideoCapture(path), []
    fps = cap.get(cv2.CAP_PROP_FPS)
    while (r := cap.read())[0]:
        raw.append(r[1])
    idx = np.linspace(0, len(raw) - 1, n).astype(int)
    frames = [torch.from_numpy(cv2.cvtColor(raw[i], cv2.COLOR_BGR2RGB)).permute(2, 0, 1).float() / 255 for i in idx]
    return VideoData(frames=frames, metadata={"fps": fps, "total_num_frames": len(raw),
                                              "duration": len(raw) / fps, "frames_indices": idx.tolist()})

if __name__ == "__main__":  # required: LLM() starts its executor via MpiPoolSession, which re-imports this file
    tok = AutoTokenizer.from_pretrained(MODEL)
    prompt = tok.apply_chat_template([{"role": "user", "content": [{"type": "video"}, {"type": "text",
              "text": 'List the events as JSON {"events": [{"start_s", "end_s", "description"}]}.'}]}],
              tokenize=False, add_generation_prompt=True)
    mm = {"prompt": prompt, "multi_modal_data": {"video": [load_video("clip_10s_720p_24fps.mp4")]},
          "mm_processor_kwargs": {"do_sample_frames": False}}
    sp = SamplingParams(temperature=0, top_k=1, max_tokens=64)
    kw = dict(model=MODEL, max_num_tokens=16384, max_seq_len=32768, disable_overlap_scheduler=True,
              kv_cache_config=KvCacheConfig(free_gpu_memory_fraction=0.5))

    # Path A: aggregated (raw video into the LLM)
    llm = LLM(**kw); out_a = llm.generate(mm, sp); llm.shutdown()

    # Path B: E/PD handoff on the same GPU
    os.environ["TLLM_MULTIMODAL_DISAGGREGATED"] = "1"          # PD model skips building the ViT
    llm = LLM(**kw)
    enc = MultimodalEncoder(model=MODEL, max_num_tokens=65536)  # grid 20x36x66 = 47,520 pre-merge patches > default 16384
    enc_out = enc.generate_async(mm).result()
    dp = enc_out.disaggregated_params
    dp.request_type = "context_and_generation"
    out_b = llm.generate_async({"prompt": prompt}, sampling_params=sp, disaggregated_params=dp).result()

    print(out_a.outputs[0].token_ids[:8], out_b.outputs[0].token_ids[:8])  # differ within the first few tokens
    print(len(out_a.prompt_token_ids), len(enc_out.prompt_token_ids), len(out_b.prompt_token_ids))

Expected behavior

Path B produces the same greedy tokens as path A (same weights, same frames, same processor kwargs, same sampling), or at least sees the same prompt layout, with mRoPE positions consistent with it.

actual behavior

8 clips x 40 frames, Qwen3-VL-8B. Our harness runs the same flow as the script, with a longer structured-output prompt (JSON events with start/end times):

• Exact match A vs B: 0/8; all 8 clips diverge at output token 0.
• A: all 8 answers start with {"events": [{"start_s": 0.0, "end_s": 10.0, ... (correct duration).
• B: all 8 answers start with ```json; 3/8 answer "start_time": 0, "end_time": 100 for the 10 s clip. The description of what is shown in the frames is correct, so the embeddings themselves are fine; timing is lost.
• Layout (measured with a diagnostic run on 2 of the clips; identical for both): L_agg = L_encoder = 12,231 and L_pd = 12,073, i.e. delta = 158 = 20 timestamps x 6 tokens + 38 frame markers (19 extra <|vision_start|> + 19 extra <|vision_end|>). With the mask-based layout helpers, the aggregated prompt has 20 runs x 594 <|video_pad|>; the PD rebuild has a single run of 11,881 (<|vision_start|> + 11,880 pads).
• The encoder's mrope_position_ids have shape [3, 1, 12231] and deltas -11,220, identical to the aggregated path; the PD side slices them to [0:12073]. First differing prompt index: 3 (where the aggregated prompt has its first timestamp token, <0.1 seconds>).
• Path A is deterministic: a rerun several days later matched the first run on 8/8 clips.
• Not a transport problem: handle shape/dtype are as expected, and E and PD run on the same device.
• The minimal script above, run as-is on 9 runs (8 different synthetic clips plus one rerun): len(A) = len(encoder) = 12,069 and len(B) = 11,911 (delta 158) on 9/9 runs. With this short prompt both paths emit the same first 3 tokens (```json and a newline), then A continues with an object {"events": ... and B with a list [{"start_s" ...; deterministic across runs. With the longer prompt above, the outputs differ from token 0.

additional notes

Root cause (line numbers at 785c948)

1. Aggregated path: the HF Qwen3-VL processor replaces the video placeholder with one group per temporal patch, <t seconds><|vision_start|> + 594 x <|video_pad|> + <|vision_end|>, i.e. 20 groups for 40 frames (temporal_patch_size 2, 36x66/4 = 594 tokens per group). Each timestamp such as <0.1 seconds> is 6 tokens here.
2. PD path: Qwen3VLInputProcessorBase.build_disagg_prefill_multimodal_inputs (tensorrt_llm/_torch/models/modeling_qwen3vl.py L473-515) tokenizes the raw text prompt, which contains one <|vision_start|><|video_pad|><|vision_end|>, and calls _expand_prompt_token_ids_for_mm_handoff (L74-166), which expands the single <|video_pad|> into tensor_size[0] = 11,880 contiguous pads (a single run). The 20 timestamps (20 x 6 = 120 tokens) and the 19 extra <|vision_start|> / <|vision_end|> pairs (38 tokens) are never inserted -> expected delta = 158 tokens.
3. mRoPE: the encoder (mm_encoder_only) runs the model's input processor on the raw video, so its mrope_config (position ids [3, 1, L_agg] and deltas) is computed on the aggregated layout. _forward_step_mm_encoder_only (tensorrt_llm/_torch/pyexecutor/model_engine.py L7561-7655, mRoPE extraction L7622-7653) returns it, and it travels as DisaggregatedParams.mrope_position_ids_handle / mrope_position_deltas_handle. On the PD side, BaseLLM._preprocess (tensorrt_llm/llmapi/llm.py, is_mm_disagg branch L924-991) puts these handles into multimodal_data["mrope_config"] (L974-982) next to the PD-rebuilt prompt, and model_engine.py L5354-5360 takes mrope_pos_ids[:, :, begin_compute : begin_compute + len(prompt_tokens)] with no check that mrope_pos_ids.shape[-1] == len(prompt). The deltas (computed for L_agg) are then used for decode.
4. The TODO at llm.py L937-939, "TODO(TRTLLM-12869): Pass encoder-side MM layout through DisaggregatedParams so prefill does not rebuild prompt tokens, positions, lengths, runs, special offsets, and cumsum here", describes the missing piece. For Qwen3-VL video the rebuild is not only redundant but wrong.

Images are not affected: a single image has no timestamp tokens and exactly one <|vision_start|> / pad run / <|vision_end|>, so the rebuilt layout matches the aggregated one.

Status on main (760a584, 2026-10-12)

• modeling_qwen3vl.py: _expand_prompt_token_ids_for_mm_handoff (L76-166) and build_disagg_prefill_multimodal_inputs (L475-517) are identical to 785c948: the text prompt is tokenized and each placeholder is expanded into one contiguous run.
• llm.py: the TODO(TRTLLM-12869) is at L976-978; the rebuilt prompt is used at L979-986, and the encoder mRoPE handles are attached unchanged at L1013-1021. DisaggregatedParams still has no field for the encoder-side prompt token ids; the multimodal_positions / multimodal_lengths fields added in [TRTLLM-16621][feat] Support DeepSeek-V4.1-Flash #19727 sit with the P->D parameters and are not used by the E/PD branch in llm.py.
• The encoder step is in tensorrt_llm/_torch/pyexecutor/engine/runners/mm_encoder.py; the aggregated-layout mRoPE ids are forwarded at L92-101 / L126-127. The context-phase mRoPE slicing moved to tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/runner.py with [TRTLLM-15762][refactor] Migrate decoder execution and private resources into the runner family #19896: L3399-3404 slices mrope_pos_ids[:, :, begin_compute : begin_compute + len(prompt_tokens)], and the copy at L4066-4085 checks only the tensor rank and the leading (3, 1) dimensions, not the length.
• Other input processors built on Qwen3VLInputProcessorBase share this path and are expected to behave the same (not run): Cosmos3InputProcessor (modeling_cosmos3.py L59, @support_multimodal_disaggregated at L68) and the Qwen3.5 VLM wrappers in modeling_qwen3_5.py, which register Qwen3VLInputProcessorBase directly (L823 and L840).
• tests/unittest/_torch/multimodal/test_qwen3vl_disagg_prompt.py (L33) asserts the current single-run layout without timestamps for video, so a fix has to update that expectation.

Why the existing tests don't catch it

TestVideoMMEEPD (tests/integration/defs/accuracy/test_epd_disagg_multimodal.py L241) scores multiple-choice accuracy on VideoMME. Many questions can be answered from frame content alone, so losing timestamps and shifting positions mostly hurts temporal questions and can stay within the accuracy threshold; #13864 reports identical VideoMME scores for the aggregated and E/PD paths (Qwen3-VL-2B, 54.667 both). There is no token-level parity test between the aggregated path and E/PD for video.

Proposed fix

(a) Per TRTLLM-12869: carry the encoder-side prompt_token_ids (already expanded by the HF processor) and the MM layout (positions, lengths, runs, special token offsets, embedding lengths) in DisaggregatedParams, and have the PD side use them instead of rebuilding. We have a PoC that monkeypatches build_disagg_prefill_multimodal_inputs to take the encoder RequestOutput.prompt_token_ids and derive the layout with the existing mask helpers in tensorrt_llm/inputs/multimodal.py (_compute_mm_masks, _find_mm_token_start_pos_from_masks, _find_mm_token_runs_from_mask, _find_mm_embedding_lengths_from_masks; the same calls inputs/registry.py makes for the aggregated path, L1448-1463 on main). With this PoC the E/PD path matches the aggregated path exactly on 8/8 clips (greedy, 64 tokens); the PD prefill is 158 tokens longer than with the stock rebuild (PD TTFT +1.4%).

(b) Alternatively, ship video_grid_thw plus the per-group timestamps (or the video metadata) with the handle and let the PD side rebuild the identical token sequence, reusing the HF processor's timestamp formatting. #19702 recently added per-item encoder-to-prefill metadata (mm_embedding_metadata) for Nemotron-H EVS; the same channel could carry the Qwen3-VL layout.

In both cases, add a defensive check where the context-phase mRoPE ids are sliced (main: runners/decoder/runner.py L3399-3404) that mrope_position_ids.shape[-1] equals the prompt length, so a layout mismatch fails loudly instead of silently misaligning (a similar length check was raised in #11170 for a different case), and update test_qwen3vl_disagg_prompt.py to expect the aggregated layout.

Impact

Silent quality degradation (temporal grounding, timestamps, event ordering) for every video request served through the Qwen3-VL E/PD handoff, in any serving stack built on it (e.g. a video-enabled NVIDIA Dynamo E/PD deployment with a TensorRT-LLM encoder). Throughput/latency measurements are barely affected (the PD prompt is ~1.3% shorter), which makes the problem easy to miss in performance-only validation.

Related: TRTLLM-12869 (the TODO in llm.py), #13864, #19702, #11170.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.