Skip to content

Fix invalid concat in find_conv_concat_split_fuse for Topaz flow models - #5156

Open
urpetkov-amd wants to merge 1 commit into
developfrom
fix_thf_models_concat
Open

Fix invalid concat in find_conv_concat_split_fuse for Topaz flow models#5156
urpetkov-amd wants to merge 1 commit into
developfrom
fix_thf_models_concat

Conversation

@urpetkov-amd

Copy link
Copy Markdown
Collaborator

Motivation

Topaz flow U-Net ONNX models (e.g. thf-v4-fnet-fp16-256x352, thd-v3-fnet-fp16-256x352) fail GPU compile in MIGraphX during the simplify_algebra pass. The failure blocks compilation via migraphx-driver and any execution provider that uses the MIGraphX GPU pipeline (including the AMDGPU EP). The only workaround today is disabling the entire pass with MIGRAPHX_DISABLE_PASSES=simplify_algebra.

This PR fixes the compile crash so these models compile without disabling simplify_algebra.

Technical Details

Root cause: In find_conv_concat_split_fuse (src/simplify_algebra.cpp), horizontal conv fusion through a channel concat can build a suffix concat(axis=1) over the remaining concat inputs without verifying that spatial dimensions match. On Topaz-style U-Net graphs—decoder feature + encoder skip concat, conv on the concat, parallel stride-2 conv on the skip branch—this can produce an invalid concat after rewrite_convolution has rewritten transposed convs, triggering:

Error simplify_algebra: .../concat.hpp:97: normalize_compute_shape:
CONCAT: all input dimensions should match in axis 2

Changes:

  • Hoist is_valid_concat() to file scope (same logic already used by find_concat_op) so both transforms share one helper.
  • In find_conv_concat_split_fuse, compute suffix remaining inputs before inserting any fusion IR.
  • Skip fusion when remaining.size() > 1 and is_valid_concat(remaining, 1) is false.

Valid fusion paths (e.g. existing conv_horizontal_fuse in test/simplify_algebra_test.cpp) are unchanged. Invalid paths skip fusion instead of aborting compile.

Changelog Category

Add a CHANGELOG.md entry for any option other than Not Applicable

  • Added: New functionality.
  • Changed: Changes to existing functionality.
  • Removed: Functionality or support that has been removed. (Compared to a previous release)
  • Optimized: Component performance that has been optimized or improved.
  • Resolved Issues: Known issues from a previous version that have been resolved.
  • Not Applicable: This PR is not to be included in the changelog.

Follow the LLVM AI Tool Use Policy for contributions using AI.

@urpetkov-amd
urpetkov-amd requested a review from causten as a code owner August 20, 2026 14:44
@urpetkov-amd urpetkov-amd added the Windows Related changes for Windows Environments label Aug 20, 2026
@urpetkov-amd
urpetkov-amd requested review from TedThemistokleous, pfultz2 and shivadbhavsar and removed request for causten August 20, 2026 14:44
@codecov

codecov Bot commented Aug 20, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Additional details and impacted files
@@           Coverage Diff            @@
##           develop    #5156   +/-   ##
========================================
  Coverage    93.13%   93.13%           
========================================
  Files          625      625           
  Lines        33252    33254    +2     
========================================
+ Hits         30967    30969    +2     
  Misses        2285     2285           
Files with missing lines Coverage Δ
src/simplify_algebra.cpp 97.48% <100.00%> (+<0.01%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@gh-app-migraphx-bot-pr-write

Copy link
Copy Markdown
Test Batch New Rate (16fba2) Old Rate (3a503c)* Diff Status
torchvision-resnet50 64 3,322.17 3,264.92 1.75%
torchvision-resnet50_fp16 64 7,872.47 7,548.67 4.29%
torchvision-densenet121 32 2,489.48 2,483.99 0.22%
torchvision-densenet121_fp16 32 5,023.22 5,004.24 0.38%
torchvision-inceptionv3 32 2,072.70 2,058.51 0.69%
torchvision-inceptionv3_fp16 32 4,463.06 4,416.99 1.04%
cadene-inceptionv4 16 817.23 820.61 -0.41%
cadene-resnext64x4 16 781.35 782.78 -0.18%
slim-mobilenet 64 8,380.78 8,386.36 -0.07%
slim-nasnetalarge 64 228.18 228.86 -0.30%
slim-resnet50v2 64 3,238.03 3,180.91 1.80%
bert-mrpc-onnx 8 1,168.94 1,168.84 0.01%
bert-mrpc-tf 1 496.95 498.63 -0.34%
pytorch-examples-wlang-gru 1 497.86 473.35 5.18% 🔆
pytorch-examples-wlang-lstm 1 409.57 384.83 6.43% 🔆
torchvision-resnet50_1 1 1,052.59 1,046.63 0.57%
cadene-dpn92_1 1 441.41 437.32 0.94%
cadene-resnext101_1 1 365.55 365.89 -0.09%
onnx-taau-downsample 1 847.39 844.09 0.39%
dlrm-criteoterabyte 1 32.24 32.42 -0.56%
dlrm-criteoterabyte_fp16 1 51.56 51.80 -0.46%
agentmodel 1 13,953.05 9,209.12 51.51% 🔆
unet_fp16 2 58.33 58.80 -0.80%
resnet50v1_fp16 1 1,431.14 1,366.11 4.76%
resnet50v1_int8 1 1,811.04 1,883.96 -3.87%
bert_base_cased_fp16 64 1,097.39 1,098.16 -0.07%
bert_large_uncased_fp16 32 345.86 345.59 0.08%
bert_large_fp16 1 206.17 206.59 -0.20%
distilgpt2_fp16 16 2,090.66 2,092.89 -0.11%
yolov5s 1 566.27 558.33 1.42%
tinyllama 1 45.82 45.83 -0.01%
vicuna-fastchat 1 44.33 44.20 0.30%
whisper-tiny-encoder 1 412.74 411.87 0.21%
whisper-tiny-decoder 1 410.73 408.48 0.55%
llama2_7b 1 20.88 20.84 0.19%
qwen1.5-7b 1 23.65 23.58 0.31%
phi3-3.8b 1 26.74 26.72 0.10%
llama3-8b 1 21.72 21.80 -0.37%
whisper-large-encoder 1 10.17 10.18 -0.08%
whisper-large-decoder 1 107.29 105.30 1.89%
mistral-7b 1 23.78 23.78 -0.00%
FLUX.1-schnell 1 780.87 755.22 3.40%

Check flagged results 🔆

* No develop baseline was found for this PR's branch point; compared against the latest available develop run instead.

@gh-app-migraphx-bot-pr-write

Copy link
Copy Markdown
Test Status Result
bert-mrpc-onnx PASSED: MIGraphX meets tolerance
bert-mrpc-tf ERROR - check error output
traceback
Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 377, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 313, in main
import tensorflow as tf
File "/usr/local/lib/python3.12/dist-packages/tensorflow/init.py", line 40, in
from tensorflow.python import pywrap_tensorflow as _pywrap_tensorflow # pylint: disable=unused-import
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 37, in
self_check.preload_check()
File "/usr/local/lib/python3.12/dist-packages/tensorflow/python/platform/self_check.py", line 63, in preload_check
from tensorflow.python.platform import _pywrap_cpu_feature_guard
ImportError: libnuma.so.1: cannot open shared object file: No such file or directory
pytorch-examples-wlang-gru PASSED: MIGraphX meets tolerance
pytorch-examples-wlang-lstm PASSED: MIGraphX meets tolerance
dlrm-criteoterabyte PASSED: MIGraphX meets tolerance
agentmodel PASSED: MIGraphX meets tolerance
unet PASSED: MIGraphX meets tolerance
resnet50v1 PASSED: MIGraphX meets tolerance
bert_base_cased_fp16 PASSED: MIGraphX meets tolerance
bert_large_uncased_fp16 🔴 FAILED: MIGraphX is not within tolerance - check verbose output
bert_large PASSED: MIGraphX meets tolerance
yolov5s PASSED: MIGraphX meets tolerance
tinyllama PASSED: MIGraphX meets tolerance
vicuna-fastchat PASSED: MIGraphX meets tolerance
whisper-tiny-encoder PASSED: MIGraphX meets tolerance
whisper-tiny-decoder PASSED: MIGraphX meets tolerance
distilgpt2_fp16 🔴 FAILED: MIGraphX is not within tolerance - check verbose output
llama2_7b PASSED: MIGraphX meets tolerance
qwen1.5-7b PASSED: MIGraphX meets tolerance
phi3-3.8b PASSED: MIGraphX meets tolerance
llama3-8b PASSED: MIGraphX meets tolerance
whisper-large-encoder PASSED: MIGraphX meets tolerance
whisper-large-decoder PASSED: MIGraphX meets tolerance
mistral-7b PASSED: MIGraphX meets tolerance
FLUX.1-schnell PASSED: MIGraphX meets tolerance

@TedThemistokleous TedThemistokleous left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Add a unit test that triggers the invalid concat so we can capture the failure that this change fixes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Windows Related changes for Windows Environments

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants