[Bug]: multi-node trtllm-serve with the MPI orchestrator (trtllm-llmapi-launch) hangs during executor startup; cause under investigation #19634

System Info

• CPU architecture: x86_64
• GPU: 2× node, 8× NVIDIA H200 each (16 GPUs total), AWS p5en.48xlarge
• Networking: AWS EFA (32 NICs/node), EFA 1.50 userspace + aws-ofi-nccl (GIN); Kubernetes StatefulSet with hostNetwork: true (pod IP == node IP on a 10.x.0.0/16 primary CIDR), sshd on :2222 for the mpirun bootstrap.
• Libraries:
  • Container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc9
  • TensorRT-LLM: v1.3.0rc9 line (source paths below verified against this line)
  • NCCL 2.30.4, PyTorch as shipped in the container
• OS: Ubuntu 24.04 (container)
• Orchestrator: MPI (trtllm-llmapi-launch → mpirun), i.e. the MpiCommSession/IPC-executor path (not Ray).

Who can help?

Leaving blank for a core maintainer to route (executor / IPC / multi-node serving).

The bug

On a multi-node trtllm-serve launched with the MPI orchestrator (ranks spanning two hosts),
startup hangs at IPC-executor init. Rank-0's orchestrator creates its ZMQ IPC queues bound to
loopback and advertises that tcp://127.0.0.1:PORT endpoint to all workers. The 8 worker ranks on
the second host try to connect to 127.0.0.1:PORT, which resolves to their own loopback, never
reach rank-0, and exit — so MpiSession/IPC init blocks forever waiting for them to connect.

This is specifically an IPC-executor problem, not an MPI / NCCL / EFA problem: the mpirun
cross-node bootstrap is clean (all 16 ranks start; orted launches on the worker node over ssh), and
the hang is before NCCL init.

The hang (reproduced, frozen 8+ minutes, GPUs idle)

trtllm-serve freezes right after these lines and never proceeds:

[TRT-LLM] [W] Orchestrator is creating IPC executor
rank 0 using MpiCommSession to bind to external MPI processes
[TRT-LLM] [I] Generating a new HMAC key for server proxy_request_queue
[TRT-LLM] [I] Generating a new HMAC key for server worker_init_status_queue
[TRT-LLM] [I] Generating a new HMAC key for server proxy_result_queue

Rank-0 waits for the 16 workers to connect to worker_init_status_queue; the 8 worker-node ranks
never connect (they start, fail to reach rank-0, exit → orted defunct), so init hangs.

Root cause (source, v1.3.0rc line)

1. tensorrt_llm/executor/ipc.py:58 — the queue bind address defaults to a hardcoded loopback:

   self.address_endpoint = address[
       0] if address is not None else "tcp://127.0.0.1:*"

2. tensorrt_llm/executor/proxy.py:494 — worker_init_status_queue = IpcQueue(is_server=True, name="worker_init_status_queue") is constructed with no address, so it uses that loopback
   default; proxy.py:520 then advertises that tcp://127.0.0.1:PORT endpoint to the workers.
   Cross-host workers connecting to 127.0.0.1:PORT hit their own loopback → fail.
3. No routable-IP override exists. A whole-tree grep of tensorrt_llm/executor/ for
   TLLM_*ADDR|*IP|*HOST, 0.0.0.0, get_local_ip, use_hostname finds nothing that sets the
   orchestrator IPC bind address. (TLLM_SPAWN_PROXY_PROCESS_IPC_ADDR is an ipc:// unix socket for
   the RPC-proxy path — even more node-local — and MASTER_ADDR is for the Ray backend, not the MPI
   IPC executor.)

So on a single node everything binds and connects on loopback and works; the moment ranks span hosts,
the loopback-bound orchestrator queues are unreachable by the remote workers.

Expected behavior

A multi-node MPI-orchestrated trtllm-serve should let the orchestrator IPC queues bind to a
routable address (or 0.0.0.0) and advertise rank-0's routable IP (not 127.0.0.1) to the
workers, so cross-host workers can connect and /health comes up.

Reproduction

mpirun -np 16 -H <node0>:8,<node1>:8 \
  --mca plm_rsh_args "-p 2222 -o GSSAPIAuthentication=no" \
  -x TRTLLM_FORCE_COMM_METHOD=... \
  trtllm-llmapi-launch trtllm-serve <a-MoE-model> \
  --tp_size 16 --ep_size 16 --pp_size 1 ...
# hangs at "Orchestrator is creating IPC executor" (see log above); GPUs idle.

(GSSAPIAuthentication=no is only needed so ssh doesn't stall on GSSAPI KEX against a no-KDC node —
unrelated to the IPC bug.)

A candidate direction — and why this is filed as an issue, not a "fix" PR

An env-overridable bind address gets past the freeze once, but is not a reliable fix, so I am
filing this as a defensive-note issue rather than presenting a patch as done:

--- a/tensorrt_llm/executor/ipc.py
+++ b/tensorrt_llm/executor/ipc.py
-        self.address_endpoint = address[
-            0] if address is not None else "tcp://127.0.0.1:*"
+        self.address_endpoint = address[
+            0] if address is not None else ("tcp://%s:*" % os.environ.get("TLLM_IPC_BIND_IP", "127.0.0.1"))

Run with -x TLLM_IPC_BIND_IP=<rank-0 routable node IP>.

Caveat (measured): with this change + TLLM_IPC_BIND_IP=<node0 IP>, one attempt advanced all
the way to launch_server (past the IPC freeze); a subsequent clean relaunch (stale procs + port
reaped) re-hung at the same IPC freeze. So a routable-IP bind is necessary but not sufficient.
Unconfirmed which of: (a) the env did not thread to rank-0 on the relaunch (env propagation), (b) a
second worker-connected queue still binds loopback, or (c) an ephemeral-port node↔node reachability
issue. A complete fix should (i) bind the orchestrator IPC to a routable/0.0.0.0 address, (ii)
advertise the routable IP (not 0.0.0.0/127.0.0.1) to workers, and (iii) ensure every
worker-connected queue (not just worker_init_status_queue) uses it — then prove a clean multi-host
trtllm-serve reaches /health.

Proven workaround

Switching the orchestrator from MPI to Ray (--orchestrator_type ray / the Ray-backed launch
path, which uses MASTER_ADDR for a routable rendezvous) brings multi-node trtllm-serve up cleanly
on the same 2-node substrate. So the defect is specific to the MPI IPC-executor bind path, and
there is a working alternative today.

Before submitting

I searched existing and past issues for the IPC/loopback/multi-node-orchestrator hang and did not find
this reported. Happy to provide more logs or test a maintainer-proposed patch on the 2-node EFA
substrate.