PyTorch 2.13 Full Inductor Unit Tests

[naromero77amd]

gfx1250 PyTorch Inductor Outcome — 2026-08-21

Important

Complete primary coverage plus latest P1/P2 follow-up. Completed/planned coverage is 27732/27732 exact nodes; pending 0; intentional exclusions 0. The current totals and both result tables include the supplemental MISSED reruns and the 2026-08-29 P1/P2 rerun described below.

Supplemental rerun update — 2026-08-21

Note

This historical checkpoint records the three MISSED-node reruns. The 2026-08-29 P1/P2 rerun below supersedes the prior unresolved-node states in the latest-results view.

• Supplemental result: 3/3 passed with HSA_HOTSWAP_ENABLE=1 and fresh TorchInductor/Triton caches.
• Post-supplemental outcomes before the P1/P2 rerun: passed 22369, skipped 4916, xfailed 319, failed 101, error 13, timed out 14, missed 0.
• Post-supplemental non-failing rate: 99.54%; unresolved rate: 0.46%.
• test_paged_attention_page_size_float16_score_mod8_head_dims2_page_size_128_cuda_float16: PASSED (33.65s pytest session).
• test_windowed_no_mask_vs_sdpa_paged_attention_cuda: PASSED (27.15s pytest session).
• test_windowed_partial_block_vs_sdpa_paged_attention_cuda: PASSED (27.13s pytest session).
• GPU visibility remained healthy, each post-test HIP smoke passed, and no residual GPU clients or D-state test processes remained.
• Evidence: /home/niromero/docker_workspace/inductor_runs/inductor_all_20260814T221517Z_e9c329c8/diagnostics/missed_flex_decoding_rerun_20260821T151701Z/metadata.json.

P1/P2 rerun update — 2026-08-29

Important

The latest-results view overlays the terminal outcome from each exact rerun node onto its prior unresolved state. Nodes not selected for rerun retain their previous latest state.

• Re-ran 324/324 previously unresolved nodes: primary 128, Inductor-wrapped 196.
• Initial 2026-08-29 rerun outcomes: passed 33, skipped 0, xfailed 1, failed 190, error 88, timed out 12, missed 0.
• A 2026-09-02 follow-up reran the six profiler device-event ERROR nodes after removing HSA_TOOLS_DISABLE_REGISTER=1; all 6/6 passed.
• A second 2026-09-02 follow-up reran the five max-pool TIMEOUT nodes with a 900-second per-test limit; all 5/5 passed.
• Latest selected-node outcomes after both follow-ups: passed 44, skipped 0, xfailed 1, failed 190, error 82, timed out 7, missed 0.
• 45 nodes have moved to a non-failing state: primary 40 and wrapped 5.
• Primary latest outcomes: passed 22408, skipped 4916, xfailed 320, failed 33, error 48, timed out 7, missed 0.
• Wrapped latest outcomes: passed 21851, skipped 21732, xfailed 328, failed 157, error 34, timed out 0, missed 0.
• Primary explicit pass rate: 80.80%; non-failing rate: 99.68%; unresolved rate: 0.32%.
• Wrapped explicit pass rate: 49.55%; non-failing rate: 99.57%; unresolved rate: 0.43%.
• Detailed execution provenance and comparability limits are recorded under Execution, Improvement, and Provenance Notes.

Overall Result

• Discovered exact nodes: 27732
• Intentional exclusions: 0
• Included exact nodes: 27732
• Explicit PASSED outcomes: 22408
• Explicit pass rate: 80.80% = 22408 passed / 27732 included
• Latest non-failing rate: 99.68% = (22408 passed + 4916 skipped + 320 xfailed) / 27732 included
• Unresolved rate: 0.32% = (33 failed + 48 error + 7 timed out + 0 missed) / 27732 included
• The latest non-failing and unresolved rates reconcile to 100.00%.
• Immutable campaign journals remain unchanged; supplemental outcomes and the 2026-08-29 exact-node rerun are overlays in this latest-results view.

Suite Summary

Priority	Test Suite	Total	Passed	Skipped	Xfailed	Failed	Error	Timed Out	Missed
	test/inductor/test_alignment.py	12	12	0	0	0	0	0	0
	test/inductor/test_analysis.py	30	30	0	0	0	0	0	0
🟡 P2	test/inductor/test_aot_inductor.py	1028	490	534	3	1	0	0	0
🟡 P2	test/inductor/test_aot_inductor_arrayref.py	340	136	200	3	1	0	0	0
	test/inductor/test_aot_inductor_custom_ops.py	41	41	0	0	0	0	0	0
🟡 P2	test/inductor/test_aot_inductor_package.py	92	63	25	0	4	0	0	0
	test/inductor/test_aot_inductor_utils.py	0	0	0	0	0	0	0	0
	test/inductor/test_aoti_cache_dir.py	1	1	0	0	0	0	0	0
	test/inductor/test_aoti_torchbind_constants.py	3	3	0	0	0	0	0	0
	test/inductor/test_async_compile.py	42	14	28	0	0	0	0	0
	test/inductor/test_augmented_graph_helper.py	20	20	0	0	0	0	0	0
🟡 P2	test/inductor/test_auto_chunker.py	14	12	0	0	2	0	0	0
	test/inductor/test_auto_functionalize.py	44	43	1	0	0	0	0	0
	test/inductor/test_autoheuristic.py	12	6	6	0	0	0	0	0
	test/inductor/test_b2b_gemm.py	9	6	3	0	0	0	0	0
	test/inductor/test_benchmark_fusion.py	16	12	4	0	0	0	0	0
🟡 P2	test/inductor/test_benchmarking.py	27	19	0	6	2	0	0	0
	test/inductor/test_best_config.py	1	1	0	0	0	0	0	0
	test/inductor/test_binary_folding.py	6	5	1	0	0	0	0	0
	test/inductor/test_block_analysis.py	10	10	0	0	0	0	0	0
	test/inductor/test_block_ptr_store_dtype.py	5	5	0	0	0	0	0	0
	test/inductor/test_cache.py	728	728	0	0	0	0	0	0
	test/inductor/test_cache_dir_utils.py	3	3	0	0	0	0	0	0
	test/inductor/test_caching.py	212	212	0	0	0	0	0	0
	test/inductor/test_ck_backend.py	41	0	41	0	0	0	0	0
🟡 P2	test/inductor/test_codecache.py	317	257	58	1	0	1	0	0
	test/inductor/test_codegen_triton.py	16	16	0	0	0	0	0	0
	test/inductor/test_collective_autotuning.py	2	0	2	0	0	0	0	0
🟡 P2	test/inductor/test_combo_kernels.py	125	105	18	0	2	0	0	0
	test/inductor/test_comm_analysis.py	3	3	0	0	0	0	0	0
	test/inductor/test_compile.py	13	13	0	0	0	0	0	0
🔴 P1	test/inductor/test_compile_subprocess.py	1244	984	258	1	0	0	1	0
	test/inductor/test_compile_worker.py	23	16	7	0	0	0	0	0
	test/inductor/test_compiled_autograd.py	944	917	22	5	0	0	0	0
	test/inductor/test_compiled_fx_graph_serialization.py	3	3	0	0	0	0	0	0
	test/inductor/test_compiled_optimizers.py	740	679	61	0	0	0	0	0
	test/inductor/test_config.py	16	16	0	0	0	0	0	0
	test/inductor/test_control_deps.py	6	6	0	0	0	0	0	0
	test/inductor/test_control_flow.py	747	649	98	0	0	0	0	0
	test/inductor/test_cooperative_reductions.py	167	167	0	0	0	0	0	0
	test/inductor/test_coordinate_descent_tuner.py	14	14	0	0	0	0	0	0
	test/inductor/test_cpp_wrapper_custom_ops.py	1	1	0	0	0	0	0	0
	test/inductor/test_cpp_wrapper_hipify.py	3	3	0	0	0	0	0	0
	test/inductor/test_cpu_cpp_wrapper.py	0	0	0	0	0	0	0	0
🔴 P1	test/inductor/test_cpu_repro.py	789	779	7	0	1	1	1	0
	test/inductor/test_cpu_select_algorithm.py	0	0	0	0	0	0	0	0
	test/inductor/test_cuda_repro.py	113	105	7	1	0	0	0	0
	test/inductor/test_cudacodecache.py	3	0	3	0	0	0	0	0
	test/inductor/test_cudagraph_trees.py	220	212	8	0	0	0	0	0
	test/inductor/test_cudagraph_trees_expandable_segments.py	182	176	6	0	0	0	0	0
	test/inductor/test_custom_lowering.py	6	5	1	0	0	0	0	0
	test/inductor/test_custom_op_autotune.py	23	21	2	0	0	0	0	0
	test/inductor/test_custom_op_out_lowering.py	6	6	0	0	0	0	0	0
	test/inductor/test_custom_partitioner_fn.py	1	1	0	0	0	0	0	0
	test/inductor/test_custom_post_grad_passes.py	8	8	0	0	0	0	0	0
	test/inductor/test_cutedsl_grouped_mm.py	24	0	24	0	0	0	0	0
	test/inductor/test_cutedsl_template.py	21	0	21	0	0	0	0	0
	test/inductor/test_cutlass_backend.py	194	0	194	0	0	0	0	0
	test/inductor/test_cutlass_evt.py	8	0	8	0	0	0	0	0
	test/inductor/test_cutlass_fallback.py	12	8	4	0	0	0	0	0
	test/inductor/test_debug_graph_dump.py	6	6	0	0	0	0	0	0
	test/inductor/test_debug_trace.py	4	4	0	0	0	0	0	0
	test/inductor/test_decompose_mem_bound_mm.py	37	35	2	0	0	0	0	0
	test/inductor/test_dependencies.py	7	7	0	0	0	0	0	0
	test/inductor/test_deterministic.py	34	10	24	0	0	0	0	0
	test/inductor/test_device_assert.py	8	8	0	0	0	0	0	0
	test/inductor/test_distributed_patterns.py	20	20	0	0	0	0	0	0
	test/inductor/test_dropout_align_random_eager.py	13	10	0	3	0	0	0	0
	test/inductor/test_efficient_conv_bn_eval.py	6	6	0	0	0	0	0	0
	test/inductor/test_embedding.py	3	3	0	0	0	0	0	0
	test/inductor/test_exc_lowering_stack_trace.py	2	2	0	0	0	0	0	0
	test/inductor/test_extension_backend.py	3	3	0	0	0	0	0	0
	test/inductor/test_external_callables.py	3	3	0	0	0	0	0	0
🟡 P2	test/inductor/test_flex_attention.py	800	790	8	1	1	0	0	0
	test/inductor/test_flex_aux_vectorization.py	17	17	0	0	0	0	0	0
	test/inductor/test_flex_decoding.py	570	554	14	2	0	0	0	0
	test/inductor/test_flex_flash.py	251	7	244	0	0	0	0	0
	test/inductor/test_flex_gemm_runtime.py	7	1	6	0	0	0	0	0
	test/inductor/test_foreach.py	615	589	26	0	0	0	0	0
🟡 P2	test/inductor/test_fp8.py	224	126	55	1	0	42	0	0
	test/inductor/test_fused_attention.py	133	131	2	0	0	0	0	0
	test/inductor/test_fusion_regions.py	7	7	0	0	0	0	0	0
	test/inductor/test_fuzzer.py	11	10	1	0	0	0	0	0
	test/inductor/test_fx_fusion.py	5	5	0	0	0	0	0	0
	test/inductor/test_fxir_backend.py	77	77	0	0	0	0	0	0
	test/inductor/test_gpu_cpp_wrapper.py	312	306	6	0	0	0	0	0
	test/inductor/test_gpu_select_algorithm.py	58	58	0	0	0	0	0	0
	test/inductor/test_graph_transform_observer.py	1	1	0	0	0	0	0	0
	test/inductor/test_grid_sampler_codegen.py	1	1	0	0	0	0	0	0
	test/inductor/test_group_batch_fusion.py	15	13	2	0	0	0	0	0
	test/inductor/test_halide.py	4	0	4	0	0	0	0	0
	test/inductor/test_helion_kernels.py	2	0	2	0	0	0	0	0
	test/inductor/test_indexing.py	56	56	0	0	0	0	0	0
	test/inductor/test_inductor_annotations.py	2	2	0	0	0	0	0	0
	test/inductor/test_inductor_freezing.py	48	44	4	0	0	0	0	0
	test/inductor/test_inductor_scheduler.py	24	24	0	0	0	0	0	0
	test/inductor/test_inductor_utils.py	4	4	0	0	0	0	0	0
	test/inductor/test_inplace_padding.py	9	9	0	0	0	0	0	0
	test/inductor/test_inplacing_pass.py	26	26	0	0	0	0	0	0
	test/inductor/test_interval_mask_packing.py	12	12	0	0	0	0	0	0
	test/inductor/test_kernel_benchmark.py	20	20	0	0	0	0	0	0
	test/inductor/test_kernel_optimization.py	1	1	0	0	0	0	0	0
🔴 P1	test/inductor/test_layout_optim.py	11	6	0	0	0	0	5	0
	test/inductor/test_lookup_table.py	37	28	9	0	0	0	0	0
	test/inductor/test_loop_ordering.py	90	90	0	0	0	0	0	0
	test/inductor/test_low_contention_collectives.py	1	1	0	0	0	0	0	0
🟡 P2	test/inductor/test_max_autotune.py	577	411	158	0	8	0	0	0
	test/inductor/test_max_autotune_blackwell.py	127	3	124	0	0	0	0	0
	test/inductor/test_mem_estimation.py	4	4	0	0	0	0	0	0
🟡 P2	test/inductor/test_memory.py	9	5	0	0	0	4	0	0
	test/inductor/test_memory_planning.py	4	2	2	0	0	0	0	0
	test/inductor/test_metrics.py	6	6	0	0	0	0	0	0
	test/inductor/test_minifier.py	14	14	0	0	0	0	0	0
	test/inductor/test_minifier_isolate.py	2	1	1	0	0	0	0	0
	test/inductor/test_minifier_utils.py	4	4	0	0	0	0	0	0
	test/inductor/test_mix_order_reduction.py	498	277	221	0	0	0	0	0
	test/inductor/test_mkldnn_pattern_matcher.py	21	17	4	0	0	0	0	0
	test/inductor/test_mmdecomp.py	31	31	0	0	0	0	0	0
	test/inductor/test_move_constructors_to_gpu.py	8	7	1	0	0	0	0	0
	test/inductor/test_mps_basic.py	57	0	57	0	0	0	0	0
	test/inductor/test_multi_kernel.py	19	17	2	0	0	0	0	0
	test/inductor/test_native_matmul.py	14	14	0	0	0	0	0	0
	test/inductor/test_needs_exact_strides.py	5	5	0	0	0	0	0	0
	test/inductor/test_nested_reduction.py	180	180	0	0	0	0	0	0
	test/inductor/test_nv_universal_gemm.py	72	3	69	0	0	0	0	0
	test/inductor/test_online_softmax.py	35	35	0	0	0	0	0	0
	test/inductor/test_op_completeness.py	5	4	1	0	0	0	0	0
	test/inductor/test_op_dtype_prop.py	624	624	0	0	0	0	0	0
	test/inductor/test_optimize_indexing.py	15	15	0	0	0	0	0	0
	test/inductor/test_ordered_set.py	401	386	15	0	0	0	0	0
	test/inductor/test_origami.py	9	0	9	0	0	0	0	0
	test/inductor/test_pad_as_cat.py	4	4	0	0	0	0	0	0
	test/inductor/test_pad_mm.py	19	17	2	0	0	0	0	0
	test/inductor/test_pad_mm_utils.py	1	1	0	0	0	0	0	0
	test/inductor/test_padding.py	57	48	9	0	0	0	0	0
	test/inductor/test_pallas.py	0	0	0	0	0	0	0	0
	test/inductor/test_pattern_matcher.py	98	96	2	0	0	0	0	0
	test/inductor/test_perf.py	68	67	1	0	0	0	0	0
	test/inductor/test_profiler.py	8	8	0	0	0	0	0	0
	test/inductor/test_provenance_tracing.py	17	17	0	0	0	0	0	0
	test/inductor/test_quantization.py	3	3	0	0	0	0	0	0
	test/inductor/test_remote_cache.py	7	7	0	0	0	0	0	0
	test/inductor/test_scatter_optimization.py	9	9	0	0	0	0	0	0
	test/inductor/test_segmented_tree.py	12	12	0	0	0	0	0	0
	test/inductor/test_select_algorithm.py	40	38	2	0	0	0	0	0
	test/inductor/test_selective_lowering.py	2	2	0	0	0	0	0	0
	test/inductor/test_simd_range_trees.py	3	3	0	0	0	0	0	0
	test/inductor/test_smoke.py	3	3	0	0	0	0	0	0
	test/inductor/test_snode_runtime.py	27	27	0	0	0	0	0	0
	test/inductor/test_split_cat_fx_aten_passes.py	5	5	0	0	0	0	0	0
	test/inductor/test_split_cat_fx_passes.py	11	11	0	0	0	0	0	0
	test/inductor/test_static_triton_launcher.py	28	21	7	0	0	0	0	0
	test/inductor/test_subgraph_choice.py	2	2	0	0	0	0	0	0
	test/inductor/test_symm_mem_registry.py	19	19	0	0	0	0	0	0
	test/inductor/test_template_heuristics_registry.py	8	8	0	0	0	0	0	0
	test/inductor/test_torchbind.py	16	16	0	0	0	0	0	0
	test/inductor/test_torchinductor.py	1372	1306	65	1	0	0	0	0
	test/inductor/test_torchinductor_codegen_config_overrides.py	6	6	0	0	0	0	0	0
	test/inductor/test_torchinductor_codegen_dynamic_shapes.py	2480	1659	613	208	0	0	0	0
🟡 P2	test/inductor/test_torchinductor_dynamic_shapes.py	2551	2028	520	2	1	0	0	0
🟡 P2	test/inductor/test_torchinductor_opinfo.py	3689	3027	612	47	3	0	0	0
🟡 P2	test/inductor/test_torchinductor_opinfo_properties.py	1134	1025	70	35	4	0	0	0
	test/inductor/test_torchinductor_strided_blocks.py	323	101	222	0	0	0	0	0
	test/inductor/test_triton_cpu_backend.py	0	0	0	0	0	0	0	0
	test/inductor/test_triton_extension_backend.py	3	3	0	0	0	0	0	0
	test/inductor/test_triton_helpers.py	16	16	0	0	0	0	0	0
🟡 P2	test/inductor/test_triton_heuristics.py	53	44	8	0	1	0	0	0
	test/inductor/test_triton_kernels.py	407	358	49	0	0	0	0	0
	test/inductor/test_triton_launcher_integration.py	6	6	0	0	0	0	0	0
	test/inductor/test_triton_syntax.py	1	1	0	0	0	0	0	0
	test/inductor/test_triton_wrapper.py	3	3	0	0	0	0	0	0
	test/inductor/test_unbacked_symints.py	55	55	0	0	0	0	0	0
🟡 P2	test/inductor/test_user_streams.py	71	61	8	0	2	0	0	0
	test/inductor/test_utils.py	22	21	1	0	0	0	0	0
	test/inductor/test_xpu_basic.py	4	4	0	0	0	0	0	0
	Included total	27732	22408	4916	320	33	48	7	0

Execution, Improvement, and Provenance Notes

Max-pool timeout follow-up — 2026-09-02

• Scope: the five primary TestInductorOpInfoCUDA max-pool nodes that timed out at the 300-second watchdog.
• Re-ran each node once with --per-test-timeout 900, no retries, fresh isolated TorchInductor and Triton caches, and primary mode (PYTORCH_TEST_WITH_INDUCTOR unset).
• Preserved HSA_HOTSWAP_ENABLE unset, HSA_HOTSWAP_DISABLE unset, PyTorch 4be323cffdd92aeb272b15958ebafe9a5e6c6a33, and Triton 8500ee1cf43d2ff3967c02129510792a5fac29f6; HSA_TOOLS_DISABLE_REGISTER was also unset following the profiler fix.
• Runner HEAD: 9e0c41ac5ccd803fc3bc251ad14cfffa80680ffb; executed runner SHA-256: a7e564ee184b475c7eb6cd29e912669f3db03336ac9592a1ac34a28369576109.
• Result: 5 passed, 0 failed, 0 error, 0 timed out in 1838.46s. Individual durations were 333.60–401.33s, all above the former 300-second limit and below 900 seconds.
• Post-run HIP smoke passed on gfx1250; no test process or GPU client remained.
• Evidence: /home/niromero/docker_workspace/inductor_runs/issue_6_maxpool_900s_20260902T200835Z.
• Interpretation: the five max-pool outcomes are resolved with a per-test timeout limit of 900 seconds and do not indicate a Triton or external-library defect.

Profiler device-event follow-up — 2026-09-02

• Scope: the six primary nodes with Failed to capture device events for inductor_do_bench_using_profiling from the 2026-08-29 rerun.
• Removed the unconditional HSA_TOOLS_DISABLE_REGISTER=1 injection from pytorch/run_tests.py; the launcher also explicitly unset it.
• Preserved HSA_HOTSWAP_ENABLE unset, HSA_HOTSWAP_DISABLE unset, primary mode (PYTORCH_TEST_WITH_INDUCTOR unset), PyTorch 4be323cffdd92aeb272b15958ebafe9a5e6c6a33, and Triton 8500ee1cf43d2ff3967c02129510792a5fac29f6.
• Re-inserting only the removed environment line reconstructs the 2026-08-29 runner SHA-256 exactly (b7803e1177686ebc070b2e58dccb780b21dc2268bd7895a984fd9ef23649885c).
• Result: 6 passed, 0 failed, 0 error, 0 timed out in 86.05s, using fresh isolated TorchInductor and Triton caches.
• Post-run HIP smoke passed on gfx1250; no test process or GPU client remained.
• Evidence: /home/niromero/docker_workspace/inductor_runs/issue_6_profiler_no_hsa_tools_20260902T170456Z.
• Interpretation: strong A/B evidence that HSA_TOOLS_DISABLE_REGISTER=1 disrupted the ROCm profiler activity required by do_bench_using_profiling; this six-node cluster does not point to Triton. This remains one follow-up attempt per node.

Latest P1/P2 rerun provenance — 2026-08-29

• Selection source: all P1 nodes (TIMED OUT or MISSED) and P2 nodes (FAILED or ERROR) from both pre-rerun tables; 324 unique exact nodes.
• Primary selection manifest: primary_p1_p2_nodes.csv, SHA-256 89f9760ad4bba0a8eb88f9cf9272b2194fdd9ac98e528b11922551bf2b8da140, 128 nodes.
• Wrapped selection manifest: wrapped_p1_p2_nodes.csv, SHA-256 628b26ed33f479ecd6131bb8addc4b54166a29cb4d7784c505d12e3c86a72417, 196 nodes.
• Corrected execution window: 2026-08-29T02:30:27Z to 2026-08-29T05:08:30Z (2h 38m 03s wall time); primary runner 5995.95s, wrapped runner 3476.27s.
• Rerun policy: one attempt per node, 300-second pytest timeout, fresh and separate primary/wrapped TorchInductor and Triton caches, one visible GPU.
• Environment: HSA_HOTSWAP_ENABLE unset, HSA_HOTSWAP_DISABLE unset, PYTORCH_TEST_WITH_ROCM=1; wrapped nodes additionally used PYTORCH_TEST_WITH_INDUCTOR=1.
• HSA state was checked before launch, in the live primary and wrapped runner processes, and in both post-run HIP smoke processes; it remained unset.
• PyTorch remained at 4be323cffdd92aeb272b15958ebafe9a5e6c6a33. Triton was 8500ee1cf43d2ff3967c02129510792a5fac29f6, newer than the original campaign's e9c329c8cb49d3f83b87cce3e4fa1bc7f4a04776.
• Therefore this is a latest-outcome refresh, not an HSA-only A/B comparison: both the HSA setting and Triton revision differ from the original campaign.
• Both post-run HIP smoke tests passed on AMD Radeon Graphics / gfx1250 / HIP 7.16.26315; final GPU use was 0% with no KFD clients.
• A preliminary 22-node launch with HSA_HOTSWAP_ENABLE=1 was stopped, marked invalid, and excluded from every total in this issue.
• Corrected artifacts: /home/niromero/docker_workspace/inductor_runs/issue_6_p1_p2_rerun_20260828/hsa_unset_20260829T0228Z.
• Primary log: /home/niromero/docker_workspace/inductor_runs/issue_6_p1_p2_rerun_20260828/hsa_unset_20260829T0228Z/primary_rerun.log.
• Wrapped log: /home/niromero/docker_workspace/inductor_runs/issue_6_p1_p2_rerun_20260828/hsa_unset_20260829T0228Z/wrapped_rerun.log.
• Environment fingerprint: /home/niromero/docker_workspace/inductor_runs/issue_6_p1_p2_rerun_20260828/hsa_unset_20260829T0228Z/rerun_environment.json.

Completed single primary run

• Classification: completed_with_test_failures
• Primary single-pass outcomes before follow-up: passed 22366, skipped 4916, xfailed 319, failed 101, error 13, timed out 14, missed 3.
• Manifest SHA-256: e23a104881686e468f85b24c6164d7d42f18f5ae6777e6c74d0deed2973efc92
• Coverage: 27732/27732; pending 0
• Batching: file; default per-node timeout: 300 seconds; file timeout: 43200 seconds; retries: 0.
• Scheduled pauses: 1; resumes: 3.
• Active-test duration: 1d 3h 48m 53s; wall duration: 6d 3h 58m 42s.
• Final stop reason: completed_with_test_failures.
• Runner/monitor exits: 1 / 0.
• Post-run HIP smoke: True.

Exactly comparable improvements or regressions

• No HSA-hot-swap-only causal comparison is reported for the 2026-08-29 rerun because that run also used a newer Triton revision and fresh caches.
• The 2026-09-02 profiler follow-up kept the PyTorch and Triton revisions fixed; the only runner-code change was removal of the HSA_TOOLS_DISABLE_REGISTER=1 injection, and fresh isolated caches were used.
• The max-pool follow-up also kept the PyTorch and Triton revisions fixed. Its passing runtimes of 333.60–401.33s directly show why the former 300-second watchdog expired and why the 900-second limit resolves these nodes.
• Descriptively, primary unresolved outcomes decreased from 128 to 88 (40 resolved to PASSED/XFAILED).
• Descriptively, wrapped unresolved outcomes decreased from 196 to 191 (5 resolved to PASSED).
• Across both latest views, unresolved outcomes decreased from 324 to 279.

Exact result provenance

• Started-node journal: /home/niromero/docker_workspace/inductor_runs/inductor_all_20260814T221517Z_e9c329c8/started_nodes.jsonl
• Terminal/correction journal: /home/niromero/docker_workspace/inductor_runs/inductor_all_20260814T221517Z_e9c329c8/terminal_results.jsonl
• Corrections applied to latest terminal state: 1
• Runner log: /home/niromero/docker_workspace/inductor_runs/inductor_all_20260814T221517Z_e9c329c8/inductor_all_20260814T221517Z_e9c329c8_runner.log
• Checkpoint: /home/niromero/docker_workspace/inductor_runs/inductor_all_20260814T221517Z_e9c329c8/inductor_all_20260814T221517Z_e9c329c8_runner.log.checkpoint

Unresolved-outcome interpretation

• P1 is reporting-only for TIMED OUT or MISSED outcomes; P2 is reporting-only for FAILED or ERROR outcomes.
• MISSED means the one-attempt policy could not prove a terminal result, or the primary pass ended before the node started.

Environment

• Container/image identity: {"container_environment_file": false, "docker_environment_file": true, "hostname": "a0594b7e1970", "kernel": "6.14.0-37-generic", "machine": "x86_64", "os_release": {"ID": "ubuntu", "PRETTY_NAME": "Ubuntu 24.04.4 LTS", "VERSION_ID": "24.04"}, "pid1_cgroup_sha256": "e57fd7b21104f091de366ace1464596378a498b0a5e63f1b21147a8998c20107", "selected_environment": {"BUILD_ENVIRONMENT": "rocm", "HOSTNAME": "a0594b7e1970"}}
• GPU: AMD Radeon Graphics / gfx1250 / 0001:01:00.0 / /dev/dri/renderD128
• Python: 3.12.3 (main, Jun 19 2026, 12:46:00) [GCC 13.3.0] at /usr/bin/python3.12
• PyTorch: 2.13.0+rocm10.1.0a20260813; git 4be323cffdd92aeb272b15958ebafe9a5e6c6a33
• ROCm/HIP: 7.16.26315
• Optional dependency versions: {"PyYAML": "6.0.3", "expecttest": "0.3.0", "filelock": "3.29.7", "hypothesis": "6.156.6", "numpy": "2.2.6", "pandas": "2.2.3", "parameterized": "0.8.1", "psutil": "7.2.2", "pytest": "7.3.2", "pytest-rerunfailures": "14.0", "pytest-timeout": "2.4.0", "scipy": "1.14.1", "sympy": "1.14.0", "tqdm": "4.68.4", "transformers": "5.15.0"}
• Schedule: continuous using America/Chicago
• Scheduler functions/settings: {"compile_threads": 32, "cpp_wrapper": "torch._inductor.codegen.cpp_wrapper_gpu.CppWrapperGpu", "cuda_backend": "triton", "fx_wrapper": "torch._inductor.codegen.wrapper_fxir.WrapperFxCodegen", "python_wrapper": "torch._inductor.codegen.wrapper.PythonWrapperCodegen", "scheduling": "torch._inductor.codegen.common.init_backend_registration.<locals>.<lambda>", "triton_amd": {"libhip_path": null, "use_async_copy": null, "use_block_pingpong": null, "use_buffer_atomics": true, "use_buffer_ops": true, "use_coexec_scheduler": null, "use_expert_scheduling": null, "use_in_thread_transpose": null}, "worker_start_method": "subprocess"}
• Visibility, scheduler, and cache environment: {"AMD_SERIALIZE_KERNEL": {"state": "unset"}, "CPLUS_INCLUDE_PATH": {"state": "set", "value": "/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/rocm_sysdeps/include"}, "CUDA_VISIBLE_DEVICES": {"state": "set", "value": "0"}, "C_INCLUDE_PATH": {"state": "set", "value": "/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/rocm_sysdeps/include"}, "HIP_FORCE_DEV_KERNARG": {"state": "unset"}, "HIP_VISIBLE_DEVICES": {"state": "set", "value": "0"}, "HOME": {"state": "set", "value": "/root"}, "HSA_ENABLE_SDMA": {"state": "unset"}, "HSA_HOTSWAP_DISABLE": {"state": "unset"}, "HSA_HOTSWAP_ENABLE": {"state": "set", "value": "1"}, "HSA_OVERRIDE_GFX_VERSION": {"state": "unset"}, "HSA_XNACK": {"state": "unset"}, "LANG": {"state": "set", "value": "C.UTF-8"}, "LD_LIBRARY_PATH": {"state": "set", "value": "/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/host-math/lib:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/rocm_sysdeps/lib:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_core/lib:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_core/lib/rocm_sysdeps/lib"}, "LIBRARY_PATH": {"state": "set", "value": "/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/host-math/lib:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/rocm_sysdeps/lib"}, "PATH": {"state": "set", "value": "/opt/venv/bin:/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"}, "PKG_CONFIG_PATH": {"state": "set", "value": "/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel/lib/rocm_sysdeps/lib/pkgconfig"}, "PYTEST_ADDOPTS": {"state": "unset"}, "ROCM_HOME": {"state": "set", "value": "/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel"}, "ROCM_PATH": {"state": "set", "value": "/opt/venv/lib/python3.12/site-packages/_rocm_sdk_devel"}, "ROCR_VISIBLE_DEVICES": {"state": "set", "value": "0"}, "TMPDIR": {"state": "set", "value": "/tmp"}, "TORCHINDUCTOR_AUTOGRAD_CACHE": {"state": "unset"}, "TORCHINDUCTOR_CACHE_DIR": {"state": "set", "value": "/home/niromero/docker_workspace/inductor_runs/inductor_all_20260814T221517Z_e9c329c8/cache_preflight_torchinductor"}, "TORCHINDUCTOR_COMPILE_THREADS": {"state": "unset"}, "TORCHINDUCTOR_COORDINATE_DESCENT_TUNING": {"state": "unset"}, "TORCHINDUCTOR_CPP_WRAPPER": {"state": "unset"}, "TORCHINDUCTOR_FORCE_DISABLE_CACHES": {"state": "unset"}, "TORCHINDUCTOR_FREEZING": {"state": "unset"}, "TORCHINDUCTOR_FX_GRAPH_CACHE": {"state": "unset"}, "TORCHINDUCTOR_MAX_AUTOTUNE": {"state": "unset"}, "TORCHINDUCTOR_MAX_AUTOTUNE_GEMM": {"state": "unset"}, "TORCHINDUCTOR_WORKER_START": {"state": "unset"}, "TORCH_COMPILE_DEBUG": {"state": "unset"}, "TORCH_LOGS": {"state": "unset"}, "TRITON_CACHE_DIR": {"state": "set", "value": "/home/niromero/docker_workspace/inductor_runs/inductor_all_20260814T221517Z_e9c329c8/cache_preflight_triton"}, "TRITON_HIP_USE_ASYNC_COPY": {"state": "unset"}, "TRITON_HIP_USE_COEX…
• Cache policy: separate initially empty preflight and execution TorchInductor/Triton caches.
• Triton mode: special
• Triton imported path/version: /opt/venv/lib/python3.12/site-packages/triton/__init__.py / 3.8.0
• Triton source commit: e9c329c8cb49d3f83b87cce3e4fa1bc7f4a04776 (verified True)
• Triton LLVM hash: b010a18d2b648cab83c83967ff26b8fde11acdc6 (verified True)
• Triton direct source: {"dir_info": {}, "url": "file:///home/niromero/docker_workspace/triton"}
• Triton build helper: /home/niromero/docker_workspace/framework_scripts/triton/build.sh SHA-256 d932597db672979c09f147234d767a9fc2068463ea9c9610e0904d285341598f
• Triton build log: /home/niromero/docker_workspace/inductor_runs/inductor_all_20260814T221517Z_e9c329c8/triton_build.log SHA-256 ce73455b84492c02b9fd1b874a28847757d3b619baa034c87bfe2b38b4c66c98
• Triton patches/dirty state: {"commit": "e9c329c8cb49d3f83b87cce3e4fa1bc7f4a04776", "dirty": false, "path": "/home/niromero/docker_workspace/triton", "status_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "tracked_diff_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "untracked": []}

Failure Clusters and Current Triage

Note

This section reflects the 2026-09-02 latest-results overlay. P1 is reporting-only for TIMED OUT or MISSED; P2 is reporting-only for FAILED or ERROR. P1 takes precedence in a suite-table label when both are present.

• Current unresolved outcomes across both views: 279 = primary 88 + wrapped 191.
• The rerun plus profiler and max-pool follow-ups resolved 45/324 previously unresolved nodes to PASSED/XFAILED.
• Current P1: primary 7; wrapped 0.
• Current P2: primary 81; wrapped 191.

Priority 1 queue — primary

• timeout at configured 300-second node watchdog: 7 node(s)

Complete current primary Priority 1 exact-node list

• test/inductor/test_compile_subprocess.py::GPUTests::test_avg_pool3d_backward2_cuda — timedout
• test/inductor/test_cpu_repro.py::CPUReproTests::test_vec_compare_op_cpu_only — timedout
• test/inductor/test_layout_optim.py::TestLayoutOptim::test_2conv_with_graph_break — timedout
• test/inductor/test_layout_optim.py::TestLayoutOptim::test_3conv_with_graph_break — timedout
• test/inductor/test_layout_optim.py::TestLayoutOptim::test_mutate_base_for_conv_output — timedout
• test/inductor/test_layout_optim.py::TestLayoutOptim::test_mutate_view_for_conv_output — timedout
• test/inductor/test_layout_optim.py::TestLayoutOptim::test_training_acc — timedout

Priority 1 queue — wrapped

• None; the wrapped P1 queue is clear.

Priority 2 queue — primary

• test/inductor/test_aot_inductor.py: failed 1, error 0
• test/inductor/test_aot_inductor_arrayref.py: failed 1, error 0
• test/inductor/test_aot_inductor_package.py: failed 4, error 0
• test/inductor/test_auto_chunker.py: failed 2, error 0
• test/inductor/test_benchmarking.py: failed 2, error 0
• test/inductor/test_codecache.py: failed 0, error 1
• test/inductor/test_combo_kernels.py: failed 2, error 0
• test/inductor/test_cpu_repro.py: failed 1, error 1
• test/inductor/test_flex_attention.py: failed 1, error 0
• test/inductor/test_fp8.py: failed 0, error 42
• test/inductor/test_max_autotune.py: failed 8, error 0
• test/inductor/test_memory.py: failed 0, error 4
• test/inductor/test_torchinductor_dynamic_shapes.py: failed 1, error 0
• test/inductor/test_torchinductor_opinfo.py: failed 3, error 0
• test/inductor/test_torchinductor_opinfo_properties.py: failed 4, error 0
• test/inductor/test_triton_heuristics.py: failed 1, error 0
• test/inductor/test_user_streams.py: failed 2, error 0

Priority 2 queue — wrapped

• test/test_ops.py: failed 151, error 34
• test/test_torch.py: failed 6, error 0

Dominant rerun clusters

• Primary unresolved nodes by suite: test/inductor/test_fp8.py 42; test/inductor/test_max_autotune.py 8; test/inductor/test_layout_optim.py 5; test/inductor/test_aot_inductor_package.py 4; test/inductor/test_memory.py 4; test/inductor/test_torchinductor_opinfo.py 3.
• Wrapped unresolved nodes by suite: test/test_ops.py 185; test/test_torch.py 6.
• Primary unresolved nodes by test class: TestFP8LoweringCUDA 42; TestTemplateConfigPruning 7; TestLayoutOptim 5; TestOperatorReorderForPeakMemory 4; TestInductorOpInfoCUDA 3.
• Wrapped unresolved nodes by test class: TestMathBitsCUDA 181; TestTorch 6; TestCommonCUDA 2; TestFakeTensorCUDA 2.
• Primary latest signatures (one dominant signature per unresolved node): no valid hipBLASLt solution 42; other assertion/failure 13; 300-second watchdog timeout 7; False-is-not-true assertion 7; SIGABRT 6; other test error 6; SIGSEGV 4; OpenBLAS thread/resource termination 2; scalar mismatch 1.
• Wrapped rerun signatures (one dominant signature per unresolved node): False-is-not-true assertion 123; no-grad view modified in grad mode 34; SIGSEGV 25; other assertion/failure 7; scalar mismatch 2.
• Two primary nodes report the OpenBLAS message Program is Terminated. Because you tried to allocate too many memory regions; treat those as environment/resource failures until reproduced with a bounded OPENBLAS_NUM_THREADS.
• Most current primary errors are FP8 cases reporting could not find valid hipblaslt solution. The six profiler device-event errors resolved to PASSED after HSA_TOOLS_DISABLE_REGISTER was unset.
• The wrapped ERROR cluster is the runtime guard A view was created in no_grad mode and is being modified inplace with grad mode enabled.
• Wrapped test_ops.py remains dominated by TestMathBitsCUDA conjugate/negative-view failures; the two matrix-rank out tests report scalar mismatches.

Inductor Wrapped Results — latest view updated 2026-08-29

Note

This remains a separate Inductor-wrapped campaign. Its totals and pass rate are independent of the primary Inductor campaign above.

• Exact wrapped coverage: 44102/44102 nodes across the four requested suites.
• Explicit pass rate: 49.55% = 21851 passed / 44102 total.
• Non-failing rate: 99.57% = (21851 passed + 21732 skipped + 328 xfailed) / 44102 total.
• Unresolved rate: 0.43% = (157 failed + 34 error + 0 timed out + 0 missed) / 44102 total.
• Original wrapped campaign nodes ran with PYTORCH_TEST_WITH_INDUCTOR=1, PYTORCH_TEST_WITH_ROCM=1, and HSA_HOTSWAP_ENABLE=1.
• The 196 previously unresolved wrapped nodes were rerun with PYTORCH_TEST_WITH_INDUCTOR=1, PYTORCH_TEST_WITH_ROCM=1, and HSA_HOTSWAP_ENABLE unset; those exact outcomes now supersede their prior states.
• The supplemental MISSED-node rerun remains included: that node passed and supersedes MISSED in this latest-results view.
• Wrapped priority classification mirrors the primary campaign: P1 for TIMED OUT or MISSED, P2 for FAILED or ERROR, with P1 taking precedence.

Priority	Test Suite	Total	Passed	Skipped	Xfailed	Failed	Error	Timed Out	Missed	Pass Rate
	test/test_modules.py (WRAPPED)	3694	3257	391	46	0	0	0	0	88.17%
🟡 P2	test/test_ops.py (WRAPPED)	33963	15729	17824	225	151	34	0	0	46.31%
	test/test_ops_gradients.py (WRAPPED)	5423	2039	3339	45	0	0	0	0	37.60%
🟡 P2	test/test_torch.py (WRAPPED)	1022	826	178	12	6	0	0	0	80.82%
	Total (WRAPPED)	44102	21851	21732	328	157	34	0	0	49.55%

Wrapped Skip Analysis

• Every one of the 21732 final SKIPPED nodes was matched one-to-one with its recorded pytest reason.
• Explicitly ROCm-specific: 65 (0.30%) of wrapped skips.
• Non-ROCm-specific: 21667 (99.70%) of wrapped skips.
• Classification boundary: ROCm-specific means the recorded reason explicitly names ROCm or uses a ROCm-only guard. Ten NVIDIA-only Requires CUDA SM >= 8.9 skips are classified as non-ROCm and included under other named exclusions.

Test Suite	Total Skipped	ROCm-Specific	Non-ROCm-Specific	ROCm Share
test/test_modules.py (WRAPPED)	391	0	391	0.00%
test/test_ops.py (WRAPPED)	17824	47	17777	0.26%
test/test_ops_gradients.py (WRAPPED)	3339	16	3323	0.48%
test/test_torch.py (WRAPPED)	178	2	176	1.12%
Total (WRAPPED)	21732	65	21667	0.30%

Explicit ROCm-Specific Skip Reasons

Recorded Pytest Reason	Nodes	Share of ROCm-Specific Skips
skipCUDAIfRocm: test doesn't currently work on the ROCm stack	31	47.69%
Efficient attention on ROCM doesn't support custom_mask_type==2	17	26.15%
Skipped on ROCm (regression in ROCm 6.4)	10	15.38%
skipIfRocm: test doesn't currently work on the ROCm stack	4	6.15%
Skipped for ROCm!	1	1.54%
stale sparse.sampled_addmm OpInfo dtypes on ROCm 7.14	1	1.54%
test_cow_input does not work with efficient attention on ROCM	1	1.54%
Total	65	100.00%

Non-ROCm-Specific Skip Reasons

Reason Group	Nodes	Share of Non-ROCm Skips	Interpretation
Explicit Inductor time guard	14113	65.14%	Exact reason: Takes too long for inductor.
Autograd / OpInfo applicability	3327	15.36%	Unsupported autograd dtype, gradgrad premise, inplace autograd, inplace variant, or backward dtype.
CPU-only	1681	7.76%	Exact reason: Only runs on cpu.
Requires two devices	1102	5.09%	The campaign exposed one GPU; these nodes require at least two.
Generic upstream skip metadata	545	2.52%	Exact generic reason: Skipped! from per-op or per-module metadata.
Other named exclusions	899	4.15%	Known failures, non-comparable or nondeterministic outputs, slow-test gates, Dynamo/Triton issues, and hardware/build constraints.
Total	21667	100.00%

Dominant Skip Reasons by Wrapped Suite

Test Suite	Dominant Recorded Reasons
test/test_modules.py (WRAPPED)	CPU-only 266; generic upstream Skipped! 123; two targeted one-node exclusions.
test/test_ops.py (WRAPPED)	Takes too long for inductor 14113; CPU-only 1390; fewer than two devices 1097; generic Skipped! 347; no autograd support 133; other named reasons 744.
test/test_ops_gradients.py (WRAPPED)	Operation supports gradgrad 849; dtype lacks autograd 811; no inplace autograd 664; no inplace variant 663; unsupported backward dtype 207; other named reasons 145.
test/test_torch.py (WRAPPED)	PyTorch issue 113707 53; CPU-only 23; slow-test gate 12; TorchDynamo exclusion 12; other named reasons 78.

• Wrapped artifact directory: /home/niromero/docker_workspace/inductor_runs/inductor_wrapped_20260821T154016Z
• Wrapped manifest SHA-256: a79ad7bd77a356cc1521c33edca246d64c78b1e0a12ebd4d070adcc2402bd1ef
• Supplemental rerun evidence: /home/niromero/docker_workspace/inductor_runs/inductor_wrapped_20260821T154016Z/supplemental_missed_rerun_20260822.json

Reproduction and Artifacts

• Latest P1/P2 rerun artifacts: /home/niromero/docker_workspace/inductor_runs/issue_6_p1_p2_rerun_20260828/hsa_unset_20260829T0228Z

• Latest reconciliation summary: /home/niromero/docker_workspace/inductor_runs/issue_6_p1_p2_rerun_20260828/hsa_unset_20260829T0228Z/reconciliation_summary.json

• Latest generated issue body: /home/niromero/docker_workspace/inductor_runs/issue_6_p1_p2_rerun_20260828/hsa_unset_20260829T0228Z/issue_body_after_update.md

• Profiler follow-up artifacts: /home/niromero/docker_workspace/inductor_runs/issue_6_profiler_no_hsa_tools_20260902T170456Z.

• Profiler follow-up summary: /home/niromero/docker_workspace/inductor_runs/issue_6_profiler_no_hsa_tools_20260902T170456Z/rerun_summary.md.

• Profiler follow-up log: /home/niromero/docker_workspace/inductor_runs/issue_6_profiler_no_hsa_tools_20260902T170456Z/profiler_rerun.log.

• Max-pool timeout follow-up artifacts: /home/niromero/docker_workspace/inductor_runs/issue_6_maxpool_900s_20260902T200835Z.

• Max-pool timeout follow-up summary: /home/niromero/docker_workspace/inductor_runs/issue_6_maxpool_900s_20260902T200835Z/rerun_summary.md.

• Max-pool timeout follow-up log: /home/niromero/docker_workspace/inductor_runs/issue_6_maxpool_900s_20260902T200835Z/maxpool_900s.log.

• PyTorch checkout: /workspace/pytorch at 4be323cffdd92aeb272b15958ebafe9a5e6c6a33; dirty False; diff SHA-256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

• Runner/framework checkout: /home/niromero/docker_workspace/framework_scripts at 487f9705f8471f5a42826cddd3eaca3d7465c88e; dirty True; diff SHA-256 7b71b9d43dcfa948f8347be5c0766a34652eabff1c02bd727ab5aceaa5c4a787

• Primary command: /opt/venv/bin/python /home/niromero/docker_workspace/framework_scripts/pytorch/run_tests.py --include-inductor-all-tests --pytorch-path /workspace/pytorch --batch-mode file --num-gpus 1 --retry-attempts 0 --per-file-timeout 43200 --log-file /home/niromero/docker_workspace/inductor_runs/inductor_all_20260814T221517Z_e9c329c8/inductor_all_20260814T221517Z_e9c329c8_runner.log

• Environment override: HSA_HOTSWAP_ENABLE=1 (plus the recorded visibility/cache settings above).

• Intentional exclusions: none.

• Resume history: [{"active_node": "test/inductor/test_flex_decoding.py::TestFlexDecodingCUDA::test_paged_attention_page_size_float16_score_mod8_head_dims2_page_size_128_cuda_float16", "at": "2026-08-15T13:03:50+00:00", "checkpoint": {"csv_file": null, "last_index": 9693, "last_test": "test/inductor/test_flex_aux_vectorization.py::TestFlexAuxVectorization::test_score_mod_vec_size_selector_rejects_score_placeholder_index_True", "mode": "full_suite", "next_test": "test/inductor/test_flex_decoding.py::TestFlexDecodingCUDA::test_builtin_score_mods_bfloat16_score_mod0_head_dims0_cuda_bfloat16", "pytorch_path": "/workspace/pytorch", "total": 27732, "updated": "2026-08-15T09:14:27.616423"}, "command": ["/opt/venv/bin/python", "/home/niromero/docker_workspace/framework_scripts/pytorch/run_tests.py", "--include-inductor-all-tests", "--pytorch-path", "/workspace/pytorch", "--batch-mode", "file", "--num-gpus", "1", "--retry-attempts", "0", "--per-file-timeout", "43200", "--log-file", "/home/niromero/docker_workspace/inductor_runs/inductor_all_20260814T221517Z_e9c329c8/inductor_all_20260814T221517Z_e9c329c8_runner.log"], "deadline": "2026-08-15T15:00:00+00:00", "ended_at": "2026-08-15T13:03:51+00:00", "environment_fingerprint_sha256": "747149d9709f490343ed7b5f0536f0b213357e4d0dcdf9b06d332e3487362dc2", "existing_log_offset": 0, "final_log_offset": 6043680, "index": 0, "manifest_sha256": "e23a104881686e468f85b24c6164d7d42f18f5ae6777e6c74d0deed2973efc92", "monitor_exit": null, "monitor_pid": 25108, "outcome": "mandatory_monitor_stop", "reason": "monitor_did_not_exit_after_runner", "resume": false, "runner_count_confirmed": true, "runner_exit": -15, "runner_pgid": 24945, "runner_pid": 24945, "runner_summary_confirmed": false, "started_at": "2026-08-15T05:00:08+00:00"}, {"active_node": "test/inductor/test_flex_decoding.py::TestFlexDecodingCUDA::test_windowed_no_mask_vs_sdpa_paged_attention_cuda", "at": "2026-08-16T15:00:00+00:00", "checkpoint": {"csv_file": null, "last_index": 9693, "last_test": "test/inductor/test_flex_aux_vectorization.py::TestFlexAuxVectorization::test_score_mod_vec_size_selector_rejects_score_placeholder_index_True", "mode": "full_suite", "next_test": "test/inductor/test_flex_decoding.py::TestFlexDecodingCUDA::test_builtin_score_mods_bfloat16_score_mod0_head_dims0_cuda_bfloat16", "pytorch_path": "/workspace/pytorch", "total": 27732, "updated": "2026-08-15T09:14:27.616423"}, "command": ["/opt/venv/bin/python", "/home/niromero/docker_workspace/framework_scripts/pytorch/run_tests.py", "--include-inductor-all-tests", "--pytorch-path", "/workspace/pytorch", "--batch-mode", "file", "--num-gpus", "1", "--retry-attempts", "0", "--per-file-timeout", "43200", "--log-file", "/home/niromero/docker_workspace/inductor_runs/inductor_all_20260814T221517Z_e9c329c8/inductor_all_20260814T221517Z_e9c329c8_runner.log", "--resume"], "deadline": "2026-08-16T15:00:00+00:00", "ended_at": "2026-08-16T15:00:05+00:00", "environment_fingerprint_sha256": "747149d9709f490343ed7b5f0536f0b213357e4d0dcdf9b06d332e3487362dc2", "existing_log_offset": 6043680, "final_log_offset": 6053969, "index": 1, "manifest_sha256": "e23a104881686e468f85b24c6164d7d42f18f5ae6777e6c74d0deed2973efc92", "monitor_exit": 0, "monitor_pid": 3436, "outcome": "scheduled_pause", "resume": true, "runner_count_confirmed": true, "runner_exit": -15, "runner_pgid": 3434, "runner_pid": 3434, "runner_summary_confirmed": false, "started_at": "2026-08-16T05:44:59+00:00"}, {"active_node": "test/inductor/test_flex_decoding.py::TestFlexDecodingCUDA::test_windowed_partial_block_vs_sdpa_paged_attention_cuda", "at": "2026-08-20T15:51:53+00:00", "checkpoint": {"csv_file": null, "last_index": 9693, "last_test": "test/inductor/test_flex_aux_vectorization.py::TestFlexAuxVectorization::test_score_mod_vec_size_selector_rejects_score_placeholder_index_True", "mode": "full_suite", "next_test": "test/inductor/test_flex_decoding.py::TestFlexDecodingCUDA::test_builtin_score_mods_bfloat16_score_mod0_head_dims0_cuda_bfloat16", "pytorch_path": "/workspace/pytorch", "total": 27732, "updated": "2026-08-15T09:14:27.616423"}, "command": ["/opt/venv/bin/python", "/home/niromero/docker_workspace/framework_scripts/pytorch/run_tests.py", "--include-inductor-all-tests", "--pytorch-path", "/workspace/pytorch", "--batch-mode", "file", "--num-gpus", "1", "--retry-attempts", "0", "--per-file-timeout", "43200", "--log-file", "/home/niromero/docker_workspace/inductor_runs/inductor_all_20260814T221517Z_e9c329c8/inductor_all_20260814T221517Z_e9c329c8_runner.log", "--resume"], "deadline": null, "ended_at": "2026-08-20T15:51:54+00:00", "environment_fingerprint_sha256": "890f4fc2e40fce1e1d072d48971ecfc1f48257a1b11754e2b3a0943401bf1bd6", "existing_log_offset": 6053969, "final_log_offset": 6055663, "index": 2, "manifest_sha256": "e23a104881686e468f85b24c6164d7d42f18f5ae6777e6c74d0deed2973efc92", "monitor_exit": 0, "monitor_pid": 3849, "outcome": "infrastructure_failure", "reason": "runner_full_suite_summary_not_confirmed", "resume"…

• Artifact directory: /home/niromero/docker_workspace/inductor_runs/inductor_all_20260814T221517Z_e9c329c8

• Manifest: /home/niromero/docker_workspace/inductor_runs/inductor_all_20260814T221517Z_e9c329c8/exact_manifest.csv

• Preflight: /home/niromero/docker_workspace/inductor_runs/inductor_all_20260814T221517Z_e9c329c8/preflight.json

• Environment fingerprint: /home/niromero/docker_workspace/inductor_runs/inductor_all_20260814T221517Z_e9c329c8/environment_fingerprint.json

• Predictions: /home/niromero/docker_workspace/inductor_runs/inductor_all_20260814T221517Z_e9c329c8/predictions.jsonl

Comments below this description are intermediate Cursor checkpoints and can be ignored.

[naromero77amd]

Inductor campaign inductor_all_20260814T221517Z_e9c329c8

• Updated: 2026-08-21T08:58:59+00:00
• Progress: 27732 / 27732 (100.00%)
• Outcomes: passed 22366, skipped 4916, xfailed 319, failed 101, error 13, timed out 14, missed 3
• Current: (none)
• Next: (none)
• Elapsed: active 1d 3h 48m 53s, wall 6d 3h 58m 51s
• Schedule: continuous / active_window; local 2026-08-21T03:58:59-05:00; next boundary none
• ETA range: fast active 0s (2026-08-21T08:58:59+00:00 wall), conservative active 0s (2026-08-21T08:58:59+00:00 wall)
• ETA confidence: high from 27732 nodes / 170 suites
• Point ETA: active 0s, wall completion 2026-08-21T08:58:59+00:00
• Resumes: 3; tmux: inductor_all_20260814T221517Z_e9c329c8
• Artifacts: /home/niromero/docker_workspace/inductor_runs/inductor_all_20260814T221517Z_e9c329c8

Final classification

• Classification: completed_with_test_failures
• Runner exit: 1
• Monitor exit: 0
• Post-run smoke: True

[naromero77amd]

ERROR-cluster follow-up — 2026-09-01

Scope: the 88 latest rerun outcomes labeled ERROR—primary 54 and Inductor-wrapped 34—from the corrected HSA-unset rerun artifacts under /home/niromero/docker_workspace/inductor_runs/issue_6_p1_p2_rerun_20260828/hsa_unset_20260829T0228Z.

Classification caveat

These are not pytest collection/setup errors and should not all be interpreted as tests that could not execute. All 88/88 nodes were recorded as FAILED in the original campaigns. The exact-node rerun runner reclassifies a nonzero pytest result as ERROR whenever its output contains RuntimeError; pytest itself reports these nodes as F / FAILED. The clusters below therefore describe runtime-exception signatures in the current ERROR bucket.

Common clusters and likely ownership

1. FP8 row-wise scaled GEMM: 42 nodes (47.7% of all ERROR nodes; 77.8% of primary ERROR nodes)

   • Signature: could not find valid hipblaslt solution.
   • Coverage: 30 test_rowwise_scaling_acceptable_input_dims variants plus 12 test_rowwise_scaling variants.
   • The exception occurs in the eager torch.nn.functional.scaled_mm reference call, before torch.compile or Triton runs. PyTorch first receives no hipBLASLt heuristic result, queries all algorithms, and still finds none valid.
   • Best guess: missing hipBLASLt row-wise FP8 algorithm coverage for gfx1250, with overly broad PyTorch FP8 platform gating as a possible contributing test/API issue. A less likely alternative is incorrect hipBLASLt descriptors constructed by PyTorch.
   • Attribution: hipBLASLt/ROCm, not Triton. Confidence: very high that this is not Triton; high for library/architecture coverage.
   • 📌 LIKELY RESOLUTION: Jira ROCM-30074 — This likely resolves the FP8 row-wise scaled GEMM issue.

2. Complex conjugate/negative-conjugate views: 34 wrapped nodes (38.6%)

   • Coverage: 18 test_conj_view_*_cuda_complex64 and 16 test_neg_conj_view_*_cuda_complex128 nodes.
   • Signature: A view was created in no_grad mode and is being modified inplace with grad mode enabled.
   • The trace goes through torch_dynamo_resume_in_test_conj_view... before the in-place OpInfo variant. The test source intentionally places conjugate-view creation outside no_grad to avoid this exact guard, indicating that whole-test Dynamo/Inductor wrapping loses or crosses the grad-mode boundary.
   • The exact failure is also documented in an upstream discussion as unrelated to Inductor code generation.
   • Attribution: PyTorch Dynamo/autograd/functionalization or wrapped-test semantics, not an external GPU library and not Triton. Confidence: high.

3. Profiler device-event capture: 6 primary nodes (6.8%) — Resolved when HSA_TOOLS_DISABLE_REGISTER=1 was removed from the test environment.

4. Memory-planner FileCheck mismatch: 4 primary nodes (4.5%)

   • All four TestOperatorReorderForPeakMemory variants expect buf5 =; generated wrapper code instead contains storage reuse such as buf6 = buf4; del buf4.
   • FileCheck aborts before the test's correctness comparison.
   • Best guess: stale/brittle test expectation after an Inductor memory-planner or buffer-numbering change. Not a Triton compiler failure.

5. CPU upsample layout FileCheck mismatch: 1 primary node (1.1%)

   • test_upsample_layout expects exactly two transpose_mxn calls, but the generated CPU/oneDNN path has a different count.
   • Best guess: CPU Inductor/oneDNN layout-codegen change or stale exact-count expectation. Triton is not involved in this CPU path.

6. Standalone artifact C++ compile error: 1 primary node (1.1%)

   • A fresh process loading the standalone artifact compiles generated CPU C++ that references at::vec::Vectorized, but the standalone compilation context does not declare at::vec.
   • Best guess: PyTorch Inductor standalone-header/artifact packaging bug. g++ is reporting the missing generated-code dependency; neither g++ nor Triton appears to be the root cause.

Overall assessment

The unresolved ERROR population is 82 nodes: hipBLASLt/ROCm (42), PyTorch Dynamo/autograd semantics (34), and six isolated PyTorch Inductor test/codegen expectation issues. The two dominant unresolved signatures explain 76/82 ERROR nodes (92.7%). I do not see a dominant unresolved ERROR signature pointing primarily to Triton.

[naromero77amd]

P1 TIMEOUT cluster analysis

The corrected primary rerun originally contained 12 P1 TIMEOUT nodes. The wrapped campaign contained 0. The max-pool cluster below has since resolved, leaving 7 current primary P1 TIMEOUT nodes.

1. Max-pool comprehensive OpInfo workload: 5 nodes (41.7%) — Resolved with a per-test timeout limit of 900 seconds.

2. Layout optimization / convolution / DDP: 5 nodes (41.7%)

• test_2conv_with_graph_break
• test_3conv_with_graph_break
• test_mutate_base_for_conv_output
• test_mutate_view_for_conv_output
• test_training_acc

These are exactly the five tests in test_layout_optim.py that use the shared convolution/DDP verify_accuracy() path. Unlike the ordinary pytest timeouts, these processes did not respond to pytest's 300-second SIGALRM and were finally killed by the runner's outer safety timeout at approximately 361 seconds. That behavior points to a native-library call rather than Python-side compilation.

Each process's last output is:

MIOpen: Warning [ParseAndLoadDb] File is unreadable: ".../miopen/db/gfx1250100.HIP.fdb.txt"

The installed MIOpen system database directory has no gfx1250 database, while a generated gfx1250 user database contains fallback/JIT solver records. MIOpen has documented that RDNA devices without a system database can experience long first-run JIT/search times: ROCm/TheRock#5105.

Preliminary assessment: most likely an external MIOpen/ROCm issue or first-run performance limitation. DDP/NCCL is the secondary native-library possibility. There is no direct Triton evidence. Confidence: medium.

Note

DDP-disabled full-cluster A/B follow-up — 2026-09-02: All five layout nodes were rerun together with USE_DDP_WRAPPER=0, isolated caches, and the same 300-second per-test limit. All 5 passed in 26.08s: test_2conv_with_graph_break 16.63s, test_3conv_with_graph_break 0.57s, test_mutate_base_for_conv_output 0.52s, test_mutate_view_for_conv_output 0.49s, and test_training_acc 2.28s. The accuracy checks reported exact expected/actual sums where printed.

The missing gfx1250 MIOpen system-database warning still appeared during the passing test_2conv_with_graph_break, confirming that warning is incidental. With DDP enabled, the representative test instead times out in DDP(mod) -> _verify_param_shape_across_processes -> dist._verify_params_across_processes. init_process_group() returns, but DDP's parameter verification triggers the first RCCL collective and lazy communicator initialization. Direct librccl.so probes independently reproduced the hang in single-rank ncclCommInitRank and single-device ncclCommInitAll.

Updated assessment: RCCL communicator initialization is the high-confidence root cause; not Triton or MIOpen. This A/B is causal isolation, not a resolution of the intended coverage: the five nodes still time out with DDP enabled and remain P1 pending an RCCL fix or validation on a newer RCCL/ROCm build.

3. AvgPool3D subprocess compilation: 1 node (8.3%)

• test_compile_subprocess.py::GPUTests::test_avg_pool3d_backward2_cuda

The parent process timed out while waiting in compile_fx_ext._send_to_child().result(), with Inductor subprocess reader threads blocked waiting for a response. The identical non-subprocess test passed in 35.37 seconds in the same rerun, and the adjacent AvgPool2D subprocess test passed in 32.05 seconds.

Assessment: likely a PyTorch Inductor compile-worker/IPC liveness problem, not the pooling operation or an external library. A similar unresolved-future failure mode is described in pytorch/pytorch#189290. A Triton compilation may have triggered the child path, but the observed hang is in PyTorch's subprocess protocol. Confidence: high.

4. CPU vector comparison expression explosion: 1 node (8.3%)

• test_cpu_repro.py::CPUReproTests::test_vec_compare_op_cpu_only

This test explicitly disables CUDA, so Triton and the ROCm GPU libraries are not involved. The timeout stack remains in Inductor's C++ loop-body/code-generation path. Upstream pytorch/pytorch#192498 identifies this exact test and attributes the behavior to exponential expression expansion after the CPU realization threshold increased from 30 to 50. Their causal test reduced compilation from more than 240 seconds and 3.36 GiB to approximately 10 seconds by restoring the threshold to 30.

Assessment: confirmed PyTorch Inductor CPU compile-time bug; neither Triton nor an external GPU library. Confidence: very high.

Overall attribution

• 5/12: known slow tests exceeding the campaign timeout; no product defect established — Resolved with a per-test timeout limit of 900 seconds.
• 5/12: RCCL communicator-initialization defect (high confidence); all five pass with DDP disabled, while direct raw RCCL initialization probes hang. Not Triton or MIOpen.
• 2/12: PyTorch Inductor compiler/infrastructure issues
• 0/12: compelling evidence of a Triton-specific defect

The next layout check is the standalone reproducer on a newer RCCL/ROCm build containing later gfx1250 fixes. The other remaining causal checks are the AvgPool3D subprocess test with the compile-worker pool bypassed and the CPU vector test with realize_cpu_opcount_threshold=30.