Skip to content

revert: trtllm routing.priority engine hand-off (#28) - #29

Merged
sopwg612 merged 1 commit into
feat-deepinfra-runtime-07-09from
revert-trtllm-routing-priority
Aug 4, 2026
Merged

revert: trtllm routing.priority engine hand-off (#28)#29
sopwg612 merged 1 commit into
feat-deepinfra-runtime-07-09from
revert-trtllm-routing-priority

Conversation

@sopwg612

@sopwg612 sopwg612 commented Aug 4, 2026

Copy link
Copy Markdown

Reverts #28 (merge commit 82514b574), restoring handler_base.py and its tests to af778016d exactly — verified with git diff af778016d HEAD -- <file> on both files.

Why

#28 fixed the worker to read routing.priority (it reads a top-level key the Rust frontend never sets) so per-request priority reached TRT-LLM's waiting queue.

We're changing direction: priority will live in the dynamo frontend's router queue only, and no priority value will be forwarded to the engines. Reverting leaves the worker on upstream behaviour rather than keeping a plumbed-but-unused engine path around to confuse the next reader.

Blast radius

None. Nothing was built or deployed from the merge — prod decode/prefill still run dynamo-trtllm-runtime:07-09-62fb5b6f-kvcap-v2-pr15737-{lptol-,}unpinfix, whose base sha predates it.

What the revert restores

# health-check-only path: TrtllmHealthCheckPayload pins 1.0; the Rust
# PreprocessedRequest has no top-level `priority`, so real traffic gets 0.5
priority = request.get("priority", DEFAULT_REQUEST_PRIORITY)

Note for the frontend-only design

nvext.agent_hints.priority still feeds priority_jump in the router (max(0, priority) seconds of arrival credit under fcfs) — that is frontend-side and unaffected by this revert. strict_priority likewise. Only the engine hand-off is removed.

Worth knowing before building on it: on prod gpt-oss-120b the router queue is effectively always empty — dynamo_frontend_router_queue_pending_requests max over 6h is 0, and 24h backpressure is 2056 events all in the no-queue class, zero in standard. At --router-queue-threshold=3 the router dispatches straight through, so a router-side priority queue has nothing to reorder until that threshold drops.

🤖 Generated with Claude Code

https://claude.ai/code/session_01QqzQrZR2qrvxM6DsxqGciu

…ority"

This reverts commit 82514b5, restoring
handler_base.py and its tests to af77801 exactly.

#28 wired routing.priority through to generate_async so per-request priority
reached the TRT-LLM waiting queue. We are changing direction: priority will be
implemented in the dynamo frontend's router queue only, and no priority value
will be forwarded to the engines. Reverting keeps the worker on upstream
behaviour (top-level `priority` remains the health-check-only path, pinning
1.0) rather than leaving a plumbed-but-unused engine path around.

Nothing was built or deployed from the merge, so no image is affected.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QqzQrZR2qrvxM6DsxqGciu
@sopwg612
sopwg612 deployed to external_collaborator August 4, 2026 18:05 — with GitHub Actions Active
@sopwg612 sopwg612 changed the title Revert "trtllm: forward routing.priority to the engine waiting queue" revert: trtllm routing.priority engine hand-off (#28) Aug 4, 2026
@github-actions github-actions Bot added the revert label Aug 4, 2026
@sopwg612
sopwg612 merged commit e5c2896 into feat-deepinfra-runtime-07-09 Aug 4, 2026
16 of 21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant