[BUG]: tileiras SIGSEGV compiling dynamic MMA loops around a 128x64 panel

[robobryce]

cuTile Python version

1.5.0. The same reproducer also fails with 1.4.0.

CUDA Toolkit version

13.3 (tileiras V13.3.36)

Which installation method does this occur on?

Pip

Describe the bug

tileiras terminates with SIGSEGV while compiling the standalone kernel below
at optimization level 1 or 2. The reduced trigger combines two runtime TF32 MMA
loops, a 128x32 panel transformation, an intervening store, and a runtime
conditional. Compilation fails before the deliberately small tensors execute.

I expected the program to compile. If its live tile/resource combination is not
supported, I expected a compiler diagnostic rather than a native process crash.

This was reduced from a fused 64-column Cholesky superpanel. The original used
two float32[1,32768,32768] operands and a large grid to reach the specialization;
the compiler-only example needs two float32[1,4,4] operands and one program.

Minimum reproducible example

import os
import torch
import cuda.tile as ct


ConstInt = ct.Constant[int]
OPT_LEVEL = int(os.environ.get("CUTILE_OPT_LEVEL", "2"))


@ct.kernel(opt_level=OPT_LEVEL)
def _left_superpanel(a, out, super_step, block: ConstInt, rows_per_tile: ConstInt):
    batch_id = 0
    panel_width = 2 * block
    row_block = ct.bid(0)
    work = ct.load(
        a,
        (batch_id, row_block, 0),
        shape=(1, rows_per_tile, panel_width),
        padding_mode=ct.PaddingMode.ZERO,
    ).reshape((rows_per_tile, panel_width))
    cross_panel = ct.load(
        a,
        (batch_id, 0, 0),
        shape=(1, block, block),
        padding_mode=ct.PaddingMode.ZERO,
    ).reshape((block, block))
    cols = ct.arange(block, dtype=ct.int32)[None, :]
    right_block = ct.load(
        out,
        (batch_id, 0, 0),
        shape=(1, panel_width, panel_width),
        padding_mode=ct.PaddingMode.ZERO,
    ).reshape((panel_width, panel_width))

    for prior_super in range(super_step):
        left_block = ct.load(
            out,
            (batch_id, 0, prior_super),
            shape=(1, rows_per_tile, panel_width),
            padding_mode=ct.PaddingMode.ZERO,
        ).reshape((rows_per_tile, panel_width))
        left = left_block.astype(ct.tfloat32)
        right = right_block.astype(ct.tfloat32)
        work = ct.mma(left, right, work)

    first_panel = ct.extract(work, (0, 0), shape=(rows_per_tile, block))
    second_panel = ct.extract(work, (0, 1), shape=(rows_per_tile, block))
    second_panel = ct.mma(
        first_panel.astype(ct.tfloat32),
        cross_panel.astype(ct.tfloat32),
        second_panel,
    )

    for p in range(block):
        solved = ct.extract(second_panel, (0, p), shape=(rows_per_tile, 1))
        second_panel = ct.where(cols == p, solved, second_panel)

    ct.store(
        out,
        (batch_id, row_block, 0),
        second_panel.reshape((1, rows_per_tile, block)),
    )

    if super_step:
        next_diagonal = cross_panel
        for prior_super in range(super_step):
            factor = ct.load(
                out,
                (batch_id, 0, 0),
                shape=(1, block, panel_width),
                padding_mode=ct.PaddingMode.ZERO,
            ).reshape((block, panel_width))
            next_diagonal = ct.mma(
                factor.astype(ct.tfloat32),
                factor.transpose(0, 1).astype(ct.tfloat32),
                next_diagonal,
            )
        ct.store(
            out,
            (batch_id, 0, 0),
            next_diagonal.reshape((1, block, block)),
        )


def main():
    a = torch.empty((1, 4, 4), device="cuda", dtype=torch.float32)
    out = torch.empty_like(a)
    ct.launch(
        torch.cuda.current_stream(),
        (1,),
        _left_superpanel,
        (a, out, 0, 32, 128),
    )
    torch.cuda.synchronize()


if __name__ == "__main__":
    main()

Run at either original optimization level in a fresh compiler cache. Crash dumps
are disabled only to avoid the separate masking problem in #92.

run=$(mktemp -d)
CUDA_TILE_CACHE_DIR=off \
CUDA_TILE_TEMP_DIR="$run" \
CUDA_TILE_ENABLE_CRASH_DUMP=0 \
CUTILE_OPT_LEVEL=2 \
python repro.py

Set CUTILE_OPT_LEVEL=1 for the other original configuration.

Relevant log output

Both optimization levels report:

subprocess.CalledProcessError: Command '['/usr/local/cuda/bin/tileiras',
  '/tmp/.../kernel....bytecode', '-o',
  '/tmp/.../kernel....cubin', '--gpu-name', 'sm_120',
  '-O2', '--lineinfo']' died with <Signals.SIGSEGV: 11>.

cuda.tile._exception.TileCompilerExecutionError: Return code -11
Unknown location

For the O1 run, the compiler command contains -O1 and has the same result.
Directly invoking tileiras on the emitted input exits 139 at both O1 and O2 and
emits no cubin. Four independent cold processes, two at each level,
reproduced. The same script still fails at O2 when importing cuTile Python 1.5.0.

Environment

OS: Ubuntu 22.04.5 LTS, Linux 6.8.0-90-generic x86_64
GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition, compute capability 12.0
Driver: 580.126.09
CUDA toolkit: 13.3; nvcc 13.3.33; tileiras V13.3.36
Python: 3.13.14
PyTorch: 2.12.0+cu130 (bundled CUDA runtime 13.0)
cuTile Python: 1.5.0; also reproduced on 1.4.0
CPU: AMD EPYC 9355, 16 vCPUs

Other details

The two original source variants were:

• O2;
• its direct O1 variant, which changed only the kernel decorator.

The two sources differ only by changing the decorator from O2 to O1. Their retained
12,226-byte TileIR files are byte-for-byte identical (SHA-256
162fae627522e4be144e959fd1594d0979f133a4b5677f89523f8511ff08928a),
so they are two optimization settings for one compiler input.

The reduced trigger no longer contains Cholesky initialization, factorization,
triangular solves, block-count logic, input-dependent values, or a multi-program
grid. These changes make the reduced form compile:

• remove either runtime MMA loop;
• remove the 32-iteration panel loop;
• remove the store between the two dynamic regions;
• remove the runtime conditional;
• reduce rows_per_tile from 128 to 64;
• reduce block from 32 to 16.

The minimized bytecode compiles when replayed at O0 without --lineinfo, although the unreduced source also crashed at
O0. This suggests the larger original contains a broader trigger, while the small
example isolates a deterministic O1/O2 boundary.

Contributing Guidelines

• I agree to follow cuTile Python's contributing guidelines
• I searched the open bugs and found no duplicate for this report