[None][test] Migrate disaggregated E2E tests to C++ bounce buffers by chuangz0 · Pull Request #19588 · NVIDIA/TensorRT-LLM

chuangz0 · 2026-09-23T08:09:36Z

Description

Disaggregated E2E configurations that set kv_cache_bounce_size_mb without selecting an implementation still exercise the Python bounce buffer. Select the C++ NIXL agent buffer on both context and generation workers in the TinyLlama serving test and all 20 affected perf/perf-sanity configurations (42 worker configurations total), retaining the Python transceiver runtime.

Raise the 14 performance-worker capacities configured as 384 MiB to 512 MiB. The C++ buddy allocator rounds usable capacity down to a power of two, so 384 MiB provides only 256 MiB of usable space. Other capacities remain unchanged. Performance configurations use the C++ default admission gates; transfers that do not qualify use standard NIXL.

The TinyLlama correctness E2E uses dedicated descriptor admission settings so its short workload exercises C++ bounce. It verifies normal generated output and the context worker's bounce path engaged for write marker, with executor debug logging enabled on that worker. The shared runner now supports context-worker log assertions and retains worker logs when an assertion fails.

Test Coverage

Existing test_disaggregated_overlap_transceiver_runtime_python_bounce[TinyLlama-1.1B-Chat-v1.0] remains registered in the GB200 pre-merge and QA test lists.
Passed all applicable pre-commit and commit-message hooks, including YAML validation, legacy lint, test-list validation, and DCO.
Passed configuration audits: both peers select C++ bounce with matching settings; all 40 performance workers use power-of-two capacities without admission overrides; no 384 MiB bounce capacity remains in perf/perf-sanity configurations.
Passed 8 CPU-only isolated runner checks covering context/generation log assertions, rejection of fallback or wrong-role markers, environment propagation, process termination, and failure-log retention.
GPU E2E and performance runs were not executed locally: this machine has one RTX A6000 and no configured model weights; the serving E2E targets GB200/GB300.

PR Checklist

Reviewed the applicable coding, test coverage, API, dependency, ownership, documentation, and architecture checklist items.

Dev Engineer Review

The diff only reformats imports, docstrings, and a logger call in test_disaggregated.py. It does not change runtime behavior or configuration. No current review findings were supplied, so severity counts are unavailable.

QA Engineer Review

No test behavior or test-list entries changed. The TinyLlama test remains listed in the GB200 CI list and the manual-QA list. The supplied results report eight CPU-only runner checks passed and no local GPU E2E or performance runs. Three earlier CI runs failed without test-level details; a later run reported PR_Github success and the L0 pipeline unstable. Coverage verdict: needs follow-up, because the reported CI failures remain unexplained.

Per-File QA Perspective

tests/integration/defs/disaggregated/test_disaggregated.py: The diff contains formatting-only changes. The TinyLlama test remains listed in tests/integration/test_lists/test-db/l0_gb200_multi_gpus.yml and tests/integration/test_lists/qa/llm_function_core.txt.

coderabbitai · 2026-09-23T08:28:03Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: NVIDIA/TensorRT-LLM/.coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 485e96fe-ce55-445f-93ac-5fcbebb31aa3

📥 Commits

Reviewing files that changed from the base of the PR and between 9d2fcc6 and 432c777.

📒 Files selected for processing (2)

tests/integration/defs/disaggregated/test_configs/disagg_config_overlap_transceiver_runtime_python_bounce.yaml
tests/integration/defs/disaggregated/test_disaggregated.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

The integration configuration enables NIXL agent bounce buffers on context and generation workers. The test helper can capture and check log markers for either worker role. Performance profiles also enable agent bounce buffers, with selected profiles increasing KV-cache bounce size from 384 MB to 512 MB.

Changes

NIXL bounce-buffer coverage

Layer / File(s)	Summary
Configure the Python transceiver test `tests/integration/defs/disaggregated/test_configs/disagg_config_overlap_transceiver_runtime_python_bounce.yaml`	Both workers enable the agent bounce buffer and set `min_descriptor_count` to `"1"` and `max_average_descriptor_size` to `"4MB"`.
Check worker log markers `tests/integration/defs/disaggregated/test_disaggregated.py`	`run_disaggregated_test` captures logs and checks requested markers for either worker role. The bounce test disables `TRTLLM_USE_PY_NIXL_KVCACHE` and checks the context-worker marker `bounce path engaged for write`.
Enable bounce buffers in performance profiles `tests/scripts/perf-sanity/disaggregated/-NIXL.yaml`, `tests/scripts/perf/disaggregated/-NIXL.yaml`	The profiles enable agent bounce buffers for context and generation NIXL transceivers. Selected profiles increase `kv_cache_bounce_size_mb` from 384 to 512. The files also add license headers.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Other

Sequence Diagram(s)

sequenceDiagram
  participant Test as Disaggregated test
  participant Workers as Context and generation workers
  participant Helper as run_disaggregated_test
  Test->>Workers: Run with Python NIXL KV-cache handling disabled
  Workers-->>Helper: Context-worker logs
  Helper->>Helper: Check for bounce path engaged for write

Suggested reviewers: brnguyen2

Merge Risk: ⚪ Minimal · up to 432c7

The available evidence establishes no concrete regression in the bounce-buffer configurations or their test. The GPU-specific run was not performed locally, but the reported CI failures are not tied to a specific test or current-head defect.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 4 functions across 1 files. (1 skipped: 1…	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title follows the required [None][type] format and clearly states that disaggregated E2E tests are migrating to C++ bounce buffers.
Description check	✅ Passed	The description includes the required Description, Test Coverage, and PR Checklist sections. It explains the motivation, implementation, affected configurations, validation performed, and limitations …
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 4 functions across 1 files. (1 skipped: 1 unsupported.)

Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create a new PR

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai · 2026-09-23T09:47:39Z

⚠️ Fork-based autofix is unavailable. Re-run autofix from a branch in the upstream repository.

chuangz0 · 2026-09-23T09:47:45Z

/bot run --stage-list "GB200-4_GPUs-PyTorch-1,GB200-4_GPUs-PyTorch-2,GB200-4_GPUs-PyTorch-3,GB200-4_GPUs-PyTorch-4,GB200-4_GPUs-PyTorch-5,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-
GPU1-GEN1-NODE1-GPU4-Post-Merge-,GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-" --disable-fail-fast

github-actions · 2026-09-23T09:47:54Z

⚠️ Bot command ignored: The /bot command must appear at the very beginning of the comment (no leading blank lines or spaces). Please post a new comment with /bot as the first character.

chuangz0 · 2026-09-23T11:05:55Z

/bot run --stage-list "GB200-4_GPUs-PyTorch-1,GB200-4_GPUs-PyTorch-2,GB200-4_GPUs-PyTorch-3,GB200-4_GPUs-PyTorch-4,GB200-4_GPUs-PyTorch-5,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-
GPU1-GEN1-NODE1-GPU4-Post-Merge-,GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-"

tensorrt-cicd · 2026-09-23T11:13:53Z

PR_Github #75247 [ run ] triggered by Bot. Commit: 69b4596 Link to invocation

tensorrt-cicd · 2026-09-23T11:46:42Z

PR_Github #75247 [ run ] completed with state FAILURE. Commit: 69b4596
/LLM/main/L0_MergeRequest_PR pipeline #61996 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chuangz0 · 2026-09-23T12:38:29Z

/bot run --stage-list "GB200-4_GPUs-PyTorch-1,GB200-4_GPUs-PyTorch-2,GB200-4_GPUs-PyTorch-3,GB200-4_GPUs-PyTorch-4,GB200-4_GPUs-PyTorch-5,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-
GPU1-GEN1-NODE1-GPU4-Post-Merge-,GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-"

tensorrt-cicd · 2026-09-23T12:44:30Z

PR_Github #75252 [ run ] triggered by Bot. Commit: 40aad9e Link to invocation

tensorrt-cicd · 2026-09-23T13:27:34Z

PR_Github #75252 [ run ] completed with state FAILURE. Commit: 40aad9e
/LLM/main/L0_MergeRequest_PR pipeline #62001 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chuangz0 · 2026-10-06T03:32:39Z

/bot run --stage-list "GB200-4_GPUs-PyTorch-1,GB200-4_GPUs-PyTorch-2,GB200-4_GPUs-PyTorch-3,GB200-4_GPUs-PyTorch-4,GB200-4_GPUs-PyTorch-5,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-
GPU1-GEN1-NODE1-GPU4-Post-Merge-,GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-"

tensorrt-cicd · 2026-10-06T03:39:26Z

PR_Github #76301 [ run ] triggered by Bot. Commit: 9d2fcc6 Link to invocation

tensorrt-cicd · 2026-10-06T04:22:14Z

PR_Github #76301 [ run ] completed with state FAILURE. Commit: 9d2fcc6
/LLM/main/L0_MergeRequest_PR pipeline #62909 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chuangz0 · 2026-10-06T05:17:47Z

/bot run

tensorrt-cicd · 2026-10-06T05:25:51Z

PR_Github #76320 [ run ] triggered by Bot. Commit: 9d2fcc6 Link to invocation

tensorrt-cicd · 2026-10-06T06:41:47Z

PR_Github #76320 [ run ] completed with state SUCCESS. Commit: 9d2fcc6
/LLM/main/L0_MergeRequest_PR pipeline #62925 completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Either:

Wait for the PR to be fully approved — the label is added automatically once approval is complete. Having unresolved open comments is fine, or
If needed, ask a member of NVIDIA/trt-llm-ci-approvers to add the label manually.
Then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>

chuangz0 · 2026-10-08T03:40:53Z

/bot run

tensorrt-cicd · 2026-10-08T03:48:51Z

PR_Github #76656 [ run ] triggered by Bot. Commit: 432c777 Link to invocation

tensorrt-cicd · 2026-10-08T06:24:37Z

PR_Github #76656 [ run ] completed with state SUCCESS. Commit: 432c777
/LLM/main/L0_MergeRequest_PR pipeline #63210 completed with status: 'SUCCESS'

CI Report

Link to invocation

chuangz0 · 2026-10-08T07:33:21Z

/bot run --stage-list "GB200-4_GPUs-PyTorch-1,GB200-4_GPUs-PyTorch-2,GB200-4_GPUs-PyTorch-3,GB200-4_GPUs-PyTorch-4,GB200-4_GPUs-PyTorch-5, GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-
GPU1-GEN1-NODE1-GPU4-Post-Merge- , GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge- "

tensorrt-cicd · 2026-10-08T07:40:15Z

PR_Github #76694 [ run ] triggered by Bot. Commit: 432c777 Link to invocation

tensorrt-cicd · 2026-10-08T07:59:24Z

PR_Github #76694 [ run ] completed with state FAILURE. Commit: 432c777
/LLM/main/L0_MergeRequest_PR pipeline #63242 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chuangz0 · 2026-10-08T08:08:48Z

/bot run --stage-list "GB200-4_GPUs-PyTorch-1,GB200-4_GPUs-PyTorch-2,GB200-4_GPUs-PyTorch-3,GB200-4_GPUs-PyTorch-4,GB200-4_GPUs-PyTorch-5,GB200-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU1-GEN1-NODE1-GPU4-Post-Merge-,GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-"

tensorrt-cicd · 2026-10-08T08:15:31Z

PR_Github #76701 [ run ] triggered by Bot. Commit: 432c777 Link to invocation

tensorrt-cicd · 2026-10-08T11:19:11Z

PR_Github #76701 [ run ] completed with state SUCCESS. Commit: 432c777
/LLM/main/L0_MergeRequest_PR pipeline #63250 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

chuangz0 · 2026-10-08T14:56:30Z

/bot skip --comment "all tests have passed"

tensorrt-cicd · 2026-10-08T15:03:12Z

PR_Github #76749 [ skip ] triggered by Bot. Commit: 432c777 Link to invocation

tensorrt-cicd · 2026-10-08T15:15:42Z

PR_Github #76749 [ skip ] completed with state SUCCESS. Commit: 432c777
Skipping testing for commit 432c777

Link to invocation

Main's NVIDIA#19896 (2681ed1) moved PyTorchModelEngine's decoder execution into DecoderRunner (engine/runners/decoder/runner.py) and the encoder-decoder fast path into EncoderDecoderRunner, so this branch's model_engine.py changes conflicted. model_engine.py is main's; the branch's changes move to where that code now lives: - DecoderRunner, from NVIDIA#19816 (U1): the step-copy gather, the staged previous batch / position indices, gather_ids written by copy_to_device_if_changed, no num_accepted_draft_tokens buffer, and the CUDA graphs' static inputs; from NVIDIA#19814 (U3): the gather of vocabulary-sharded logits for post-processors. - EncoderDecoderRunner: its previous_batch_indices_cuda write clears the staged batch indices. - The two tests that called the moved members call the runner. Main's other commits since f891b3d: NVIDIA#19484, NVIDIA#19892, NVIDIA#19409, NVIDIA#19553, NVIDIA#19957, NVIDIA#19749, NVIDIA#19585, NVIDIA#19766, NVIDIA#19963, NVIDIA#19588, NVIDIA#19600, NVIDIA#19825, NVIDIA#19821, NVIDIA#19930, NVIDIA#18485. Signed-off-by: Vasanth Sabavat <vsabavat@nvidia.com>

… bounce The Python KV bounce buffer is no longer used now that the C++ transfer-agent bounce covers the E2E paths (NVIDIA#19588). Route kv_cache_bounce_size_mb straight to the C++ agent arena: - kv_cache_bounce_size_mb > 0 enables the C++ transfer-agent bounce buffer; 0 disables it. - agent_bounce_buffer_enable becomes a deprecated no-op that warns when set to True; True with size 0 no longer raises. - agent_bounce_params now requires kv_cache_bounce_size_mb > 0. - Lift the pipelined-transfer blocker that only applied to the Python bounce, and reword the C++ transceiver, pure-Python agent and FP4-MLA bridge messages for the new semantics. After this commit the Python bounce is unreachable; the next commit deletes it. Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>

chuangz0 requested review from a team as code owners September 23, 2026 08:09

chuangz0 requested review from bo-nv, dc3671, hyukn and reasonsolo September 23, 2026 08:09

github-actions Bot assigned chuangz0 Sep 23, 2026

chuangz0 force-pushed the move_transfer_buffer_to_agent_buffer branch from 69b4596 to 40aad9e Compare September 23, 2026 12:38

chuangz0 force-pushed the move_transfer_buffer_to_agent_buffer branch 2 times, most recently from ac5e177 to 9d2fcc6 Compare October 6, 2026 03:32

Shixiaowei02 added the ci: full pre-merge approved label Oct 8, 2026

chuangz0 added 2 commits October 8, 2026 11:02

[None][test] Migrate disaggregated E2E tests to C++ bounce buffers

5acc5ce

Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>

[None][test] Use power-of-two bounce buffers and default perf admission

432c777

Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>

chuangz0 force-pushed the move_transfer_buffer_to_agent_buffer branch from 9d2fcc6 to 432c777 Compare October 8, 2026 03:03

Shixiaowei02 approved these changes Oct 8, 2026

View reviewed changes

chuangz0 enabled auto-merge (squash) October 8, 2026 14:56

chuangz0 merged commit 147f70f into NVIDIA:main Oct 8, 2026
12 checks passed

chuangz0 mentioned this pull request Oct 9, 2026

[None][chore] Remove the Python KV bounce buffer; kv_cache_bounce_size_mb selects the C++ agent bounce #20049

Open

1 task done

Conversation

chuangz0 commented Sep 23, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Coverage

PR Checklist

Dev Engineer Review

QA Engineer Review

Per-File QA Perspective

Uh oh!

coderabbitai Bot commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chuangz0 commented Sep 23, 2026

Uh oh!

github-actions Bot commented Sep 23, 2026

Uh oh!

chuangz0 commented Sep 23, 2026

Uh oh!

tensorrt-cicd commented Sep 23, 2026

Uh oh!

tensorrt-cicd commented Sep 23, 2026

Uh oh!

chuangz0 commented Sep 23, 2026

Uh oh!

tensorrt-cicd commented Sep 23, 2026

Uh oh!

tensorrt-cicd commented Sep 23, 2026

Uh oh!

chuangz0 commented Oct 6, 2026

Uh oh!

tensorrt-cicd commented Oct 6, 2026

Uh oh!

tensorrt-cicd commented Oct 6, 2026

Uh oh!

chuangz0 commented Oct 6, 2026

Uh oh!

tensorrt-cicd commented Oct 6, 2026

Uh oh!

tensorrt-cicd commented Oct 6, 2026

Uh oh!

chuangz0 commented Oct 8, 2026

Uh oh!

tensorrt-cicd commented Oct 8, 2026

Uh oh!

tensorrt-cicd commented Oct 8, 2026

Uh oh!

chuangz0 commented Oct 8, 2026

Uh oh!

tensorrt-cicd commented Oct 8, 2026

Uh oh!

tensorrt-cicd commented Oct 8, 2026

Uh oh!

chuangz0 commented Oct 8, 2026

Uh oh!

tensorrt-cicd commented Oct 8, 2026

Uh oh!

tensorrt-cicd commented Oct 8, 2026

Uh oh!

chuangz0 commented Oct 8, 2026

Uh oh!

tensorrt-cicd commented Oct 8, 2026

Uh oh!

tensorrt-cicd commented Oct 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

chuangz0 commented Sep 23, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Sep 23, 2026 •

edited

Loading

coderabbitai Bot commented Sep 23, 2026 •

edited

Loading