Leader broadcast corrupts noncontiguous tensor values

[Alvorecer721]

Describe the bug

_broadcast_batched_data_dict does not preserve logical values when an ordinary tensor field is noncontiguous. With Gloo, a transposed tensor is silently rearranged on the receiving rank. The broadcast descriptor contains shape/dtype/device but not strides; the receiver allocates contiguous storage, while the sender can retain its original strides.

This is in the helper used by fetch_policy="leader_broadcast". The local data-plane path can preserve tensor views, so contiguity is not guaranteed before the helper. The packed-tensor branch already makes its payload contiguous; this report concerns the ordinary-tensor branch.

Observed at NeMo-RL commit 0aa7bf5b2e33ef78b50cb9494ed580dd5ee7b84d.

Steps/Code to reproduce bug

In a configured NeMo-RL checkout at that commit, save the following as repro.py and run torchrun --standalone --nproc-per-node=2 repro.py:

import torch
import torch.distributed as dist

from nemo_rl.data_plane.worker_mixin import _broadcast_batched_data_dict
from nemo_rl.distributed.batched_data_dict import BatchedDataDict


dist.init_process_group("gloo")
rank = dist.get_rank()
x = torch.arange(6, dtype=torch.float32).reshape(2, 3).T
expected = torch.tensor([[0., 3.], [1., 4.], [2., 5.]])
data = BatchedDataDict({"x": x}) if rank == 0 else None
out = _broadcast_batched_data_dict(
    data, is_leader=rank == 0, src=0, group=dist.group.WORLD
)
if rank == 1:
    print("expected:", expected.tolist(), flush=True)
    print("received:", out["x"].tolist(), flush=True)
failed = torch.tensor(int(not torch.equal(out["x"], expected)))
dist.all_reduce(failed, op=dist.ReduceOp.MAX)
dist.destroy_process_group()
assert failed.item() == 0, "broadcast changed logical tensor values"

The receiver prints:

expected: [[0.0, 3.0], [1.0, 4.0], [2.0, 5.0]]
received: [[0.0, 1.0], [2.0, 3.0], [4.0, 5.0]]

Both processes then fail the assertion.

Expected behavior

Every rank receives the sender's logical tensor values, including for transposes and strided slices. Staging a contiguous transport buffer should not replace or alter the leader's original tensor view.

Additional context

• Reproduced with real two-rank Gloo collectives using PyTorch 2.11.0+cu130. Because my installed application dependencies differ from this upstream checkout, I executed the exact helper and complete BatchedDataDict/PackedTensor class source with isolated imports. The reproduction above uses the normal package imports for a configured checkout; I have not validated full-package imports against the full upstream dependency stack.
• NCCL requires contiguous input as well. This is an input-preparation issue in the NeMo-RL wrapper, rather than a report of a defect in the collective libraries.
• Proposed fix: materialize ordinary tensors in contiguous logical order on the transport device. For the adjacent int16 transport path, a byte view can replace int32 widening while preserving every bit and halving payload bytes. I have prepared separate fix and optimization commits, with regression tests, for a linked PR.