fix(attention): fallback for GPUs without flash MMA kernels (sm70) - #423
Open
Th-Underscore wants to merge 9 commits into
Open
fix(attention): fallback for GPUs without flash MMA kernels (sm70)#423Th-Underscore wants to merge 9 commits into
Th-Underscore wants to merge 9 commits into
Conversation
Th-Underscore
changed the title
Th-Underscore
force-pushed
the
sm70-attention-fallback
branch
from
September 3, 2026 01:49
b2a888b to
ca67a82
Compare
Owner
|
@Th-Underscore Thanks! I was just about to rebase it myself. |
…g0#393) * breeze: pack qkv and gate/up projection weights, document weight_type * ggml, qwen_decoder: fuse bf16 activation rounding into a single kernel nsys on the 2080 Ti shows the f32 -> bf16 -> f32 cast pairs behind every activation rounding point cost ~19% of GPU time on the bf16 path and ~29% on the q4_k path (144k tiny cpy kernels per 20-token run), because ggml has no fused round-to-bf16 op and the CUDA backend runs each ggml_cast as a separate kernel. Add GGML_UNARY_OP_ROUND_BF16 (CPU + CUDA implementations; HIP shares the ggml-cuda sources) that rounds f32 values to bf16 precision in one pass, bit-identical to the cast round trip (same __float2bfloat16 / __bfloat162float sequence as cpy). The qwen decoder activation cast policy gains a fused_round flag, enabled for CUDA/HIP only; Vulkan keeps the round trip. Non-contiguous views also keep the round trip, as the unary op requires contiguous input. Verified on RTX 2080 Ti with Breeze-TTS 2: generated codes are bit-identical to the round trip build in all four test cases (bf16/q4_k, fixed 100-token case and both Chinese regression prompts). RTF on the fixed 100-token case: bf16 0.760 -> 0.695, q4_k 0.484 -> 0.419; Chinese regression q4_k 0.861 -> 0.736 (short) and 0.556 -> 0.464 (long). * ggml, breeze: support row-strided inputs in fused bf16 rounding Rounding points fed by non-contiguous views (rope/cache paths) still used the cast round trip: a strided f32 -> bf16 cpy plus a contiguous bf16 -> f32 cpy, ~10% of GPU time on the q4_k path. Add a row-strided variant of the round_bf16 kernel (dst is contiguous by construction) and relax the backend/framework gates from ggml_is_contiguous to ggml_is_contiguous_rows, so those points fuse too. Codes remain bit-identical in all four test cases. RTF on RTX 2080 Ti, q4_k: 100-token 0.419 -> 0.399, Chinese long 0.464 -> 0.441; bf16 100-token 0.695 -> 0.677. * ggml-cuda: allow CUDA graphs on pre-Ampere GPUs via GGML_CUDA_GRAPHS_PRE_AMPERE Upstream disables CUDA graphs below sm_80. Keep that default, but add an env-var escape hatch so pre-Ampere behavior can be tested without recompiling. On the RTX 2080 Ti (sm_75) Breeze-TTS 2 decode the graphs do capture and replay correctly (bit-identical codes), but RTF is neutral to slightly worse (0.399 without vs 0.408 with on the q4_k 100-token case), so the upstream default stands for this workload. * ggml, breeze: generalize fused bf16 rounding to f16/bf16 inputs ggml_round_bf16 now always produces a contiguous f32 result regardless of input type (f32/f16/bf16), matching the cast round trip bit for bit: bf16 input is already rounded so the op degenerates to an exact widening, f16 input rounds through bf16 and widens, both landing on the same real values as cast -> bf16 -> cast -> f32. This fixes a HIP crash where rounding points fed by the bf16 KV cache hit an f32/f16-only assert in the unary kernel, and recovers the fusion for f16 inputs (CUDA f16 KV cache paths) that the previous f32-only gate skipped. The activation cast no longer needs per-type special cases. Verified bit-identical codes in all 8 cases (CUDA + HIP x q4_k/bf16 x 100-token + 2 Chinese regression prompts). RTF, q4_k 100-token: CUDA 0.417 -> 0.405, HIP 0.73 (unchanged); HIP q4_k vs pre-fusion baseline: 0.84 -> 0.73, long 0.92 -> 0.80, bf16 1.50 -> 1.37. * conv_transpose1d: enable col2im fast path on Vulkan The col2im path (mul_mat + ggml_col2im_1d) only ran on CUDA/HIP/Metal; Vulkan fell back to ggml_conv_transpose_1d, whose Vulkan shader is a naive per-element kernel. All ops the col2im path needs are already supported by the Vulkan backend, including col2im_1d (f32/f16 pipelines). Breeze-TTS 2 speech decoder on Radeon 8060S: 190 ms -> 98 ms; greedy output codes identical to the generic path (wav correlation 0.99998). * breeze: skip the unconditional branch when guidance_scale == 1 CFG combines logits as uncond + scale * (cond - uncond), which is exactly cond at the default guidance_scale of 1. Running the unconditional backbone there is pure waste: skipping it removes half the backbone prefill and decode work. The depth projector's logits_cfg also gets a scale == 1 shortcut that copies the conditional half directly, avoiding an inexact uncond + 1 * (cond - uncond) round trip. guidance_scale = 0 (pure unconditional) is now accepted as well. On an RTX 2080 Ti, Breeze-TTS 2 fixed 100-token case, native weights: RTF 0.705 -> 0.605; greedy output is bit-identical with and without the skip. guidance_scale = 1.5 still runs the full CFG path unchanged. * ggml-vulkan: add bf16<->f32/f16 cpy pipelines * breeze: round activations to bf16 on GPU backends to match reference The official Breeze-TTS 2 inference runs the backbone and depth decoder with bf16 activations and a bf16 KV cache. A pure fp32 AR loop drifts into degenerate trajectories on some prompts (mispronounced tokens, repetition collapse, missing EOS), so round activations to bf16 at every op boundary via the qwen decoder activation_cast policy, mirroring the reference torch bf16 semantics. CUDA/HIP use the fused round-to-bf16 op; Vulkan uses the cast round trip. KV cache stays F16 on CUDA and Vulkan: bf16 flash attention is only accelerated with native bf16 MMA (sm_80+) and is ~3x slower on older GPUs. HIP uses a bf16 KV cache like the reference. (Ported onto the perf branch; fused_round requires the ROUND_BF16 op from the preceding commits.)
Owner
|
@Th-Underscore The Qwen eager path is essentially dead in the current code and is not reached by any model’s default path, so that change is safe. Interestingly, I could not reproduce the issue locally on RTX 5090, SM 120, and CUDA runtime 13.2. Higgs TTS works correctly with eager. Would you like to share the exact log? The concern is PR is broad. It exposes a global user-facing knob for what is basically a backend compatibility workaround for old CUDA devices. On Ampere/Ada/Blackwell, users may never need it. Another option is to remove the auto fallback too and make this fully explicit (with clear doc). For example, users on affected old CUDA GPUs can set Another issue: The documented prefixed option |
* breeze: chunk the speech-encoder conv stack to bound clone VRAM The encoder graph was built at the exact reference-audio length, so conv activations grew linearly (~45 MiB/s of reference) and every new length triggered a full graph rebuild; a 60 s reference cost ~2.5 GB extra over a 6 s one. Split the encoder into two graphs. The conv stack now runs on fixed 5 s chunks (120000 samples) preceded by a 9600-sample left overlap that covers the stack's exact 5240-sample receptive field; chunk lengths are multiples of the 960x transformer stride, so no per-stage right padding occurs and the discarded overlap frames absorb the zero left pads that represent audio start in the first chunk. Stitched outputs are bit-identical to a single-pass encode of the same input (verified over 68 frames x 16 codebooks). The transformer, downsample, and projections run once over the full frame sequence at frame scale, where even minute-long references cost only tens of MiB. Measured on a 2080 Ti (Vulkan, native q8_0 GGUF, peak minus idle baseline): the VRAM slope over reference length drops from ~45 MiB/s to ~11 MiB/s (remaining slope is the frame-scale transformer graph and the longer AR prefill from reference codes), and a 60 s reference peaks ~1.4 GB lower. Encode time for 60 s improves from 3561 ms to 2197 ms. * breeze: bucket speech-encoder transformer graph capacity The transformer graph was rebuilt at the exact frame count for every distinct reference length. Round the capacity up to 125-frame (5 s) buckets so lengths within a bucket share one graph. Unused bucket frames are replicate-padded to match the downsample conv's Replicate right pad; causal attention keeps padding frames invisible to real frames. Verified bit-identical reference codes vs exact-length graphs at 6 s and 15 s; odd lengths show sub-1% last-frame diffs from flash-attention tiling, the same accepted noise class as the pre-existing length sensitivity. Single-run peak VRAM is unchanged. * ggml-vulkan, breeze: fused round-to-bf16 unary op on Vulkan Vulkan previously paid a cast round trip (f32->bf16->f32, two kernels, a bf16 intermediate tensor) at every activation-rounding point of the breeze decoder. Add a round_bf16 compute shader (f32/f16/bf16 in, always f32 out, round-to-nearest-even via the same fp32_to_bf16 bit trick the cpy shaders use), register pipelines indexed by source type, handle the widened f32 dst in the unary pipeline selection and op-support checks, and enable fused_round for Vulkan in the breeze activation-cast policy. Verified bit-identical breeze reference codes vs the cast round trip at 6 s and 15 s references. Peak VRAM on a 2080 Ti drops ~250 MiB at a 60 s reference (5491 -> 5239 MiB); no measurable change at 6 s. * ggml-vulkan: handle row-strided inputs in fused round-to-bf16 The breeze activation-rounding policy admits row-strided views into ggml_round_bf16 (ggml_is_contiguous_rows gate in qwen_decoder). The Vulkan port dispatched every input to the flat shader, which indexes the source as a contiguous array, so row-strided views read garbage and clone output degenerated into noise. Route non-contiguous inputs to a new round_bf16_strided shader built on generic_unary_head (same pattern as sigmoid_strided), keeping the flat fast path for contiguous inputs.
Auto-resolve flash vs eager attention from CUDA compute capability:
Volta/Turing (700 <= cc < 800) fall back to eager, since large prefill
shapes select the MMA kernel which has no usable device code there
('flash_attn_ext_f16 has no device code compatible with CUDA arch 700').
- New engine::core::attention_fallback unit: preference parsing
(per-model '<family>.attention' session option + AUDIOCPP_ATTENTION),
CC gating via the CUDA driver (supports_op cannot detect this: it
returns true on sm70 for shapes that later crash at launch).
- Wire auto fallback + session options into higgs_audio_tts and
breeze_tts (backbone, depth, encoder, decoder); process-wide
AUDIOCPP_ATTENTION=eager backstop in the shared SDPA/GQA/QwenDecoder
modules; trace logging of the resolved path.
- Fix latent QwenDecoder prefix-concat dtype assert exposed by the
eager path (cast cached prefix KV on every path, not just flash).
- Unit test, breeze_tts model-spec entry, docs.
Th-Underscore
force-pushed
the
sm70-attention-fallback
branch
from
September 4, 2026 05:11
ca67a82 to
7f62b32
Compare
Author
Ah you're right, for Higgs it was an old bug I initially attributed to fattn but turned out to be something wrong with my environment. Higgs works fine with upstream |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Volta/Turing GPUs (700 <= cc < 800) crash on TTS prefills that select the CUDA MMA flash-attention kernel, which has no usable device code there ("flash_attn_ext_f16 has no device code compatible with CUDA arch 700", followed by MUL_MAT failures). This PR auto-resolves flash vs eager attention from CUDA compute capability and falls back to the exact repeat-KV + matmul/softmax graph:
engine::core::attention_fallbackunit: preference parsing (per-model<family>.attentionsession option +AUDIOCPP_ATTENTIONenv), CC gating via the CUDA driver.ggml_backend_supports_op()cannot be used for detection: on Volta it returns true for shapes that later crash at launch.higgs_audio_ttsandbreeze_tts(backbone, depth, encoder, decoder); process-wideAUDIOCPP_ATTENTION=eagerbackstop in the shared SDPA/GQA/QwenDecoder modules;--logtrace lines for the resolved path.attention_fallback_test),breeze_ttsmodel-spec entry, docs.The change is additive: unknown backends fail open to flash, constructor defaults keep
allow_flash_attention=true, and the flash graph is untouched, so families that do not opt in keep byte-identical behavior.Validation
Build:
scripts/build_linux.sh --backend cuda --with-tests --cuda-arch "70"Backend: CUDA, Tesla V100-SXM2-16GB (sm70, underclocked to 765 MHz — wall times understate full-clock performance). Models:
higgs-audio-v3-tts-4b-q8_0.gguf,breeze-tts-2-q8_0.gguf,fish-audio-s2-pro-q8_0.gguf. Voice ref: VCTK p225 15 s concat preset.--task tts, auto, short textallow_flash=0, 3.0 s audio in 2.8 s wall, no CUDA errorsattention=flash, short textmax_tokensbefore EOC — reproduced on flash too; bracket tags (no Higgs parser) derail AR, unrelated to this PR--task clon, auto--backend cpuallow_flash=1), generates/v1/audio/speech, Higgs)flash_attn/MUL_MATerrorsctestattention_fallback_testChecked families/routes:
higgs_audio_tts(CLI + server),breeze_tts(CLI streaming + offline),fish_audio(CLI regression), CPU backend. All other families keep default behavior (fail-open).Eager vs flash is the same op with ulp-level logit differences (AR trajectories can diverge, e.g. 2.32 s vs 2.56 s output for one short sentence); eager is ~1.5x slower on short samples.
Known limitations