escha: four bit exact kernel changes for the dense path - #2
Open
realugbun wants to merge 2 commits into
Open
Conversation
…path The dense generation path used R = ESCHA_ROWS_DENSE_GEN = 1, so each block owned exactly one row and a 2 to 16 row batch decoded the entire weight payload once per row. R now rounds n_rows up to a power of two, capped at ESCHA_GEN_MAX_ROWS, so one block covers the batch. R == 1 keeps the original staging and grid unchanged. n_slices stays derived from n_rows rather than n_rb, because slice boundaries decide how the fp32 partials are grouped before escha_finalize_dense sums them. Deriving them from n_rb regroups the sum and changes results. Measured on an RTX 5060 Ti (sm_120), llama-bench -p N -n 0, tok/s: batch 1 2 4 8 16 32 64 before 17.9 20.6 23.1 23.6 23.5 175.3 315.1 after 17.8 35.6 57.7 77.7 87.3 174.9 321.3 Output is bit identical. Greedy generation over 5 prompts at 160 tokens hashes the same before and after, both with and without speculative decoding.
…ad generation Three further changes on top of the row amortisation already in this PR. All bit exact against the base kernel.
realugbun
changed the title
Owner
|
@realugbun sorry i completely missed it, let me take a look, meawhile i'll start the action checks. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds three changes on top of the row amortisation this PR already carried. All four are bit exact against the base kernel: identical greedy output and identical speculative acceptance, verified on both cards below.
Each new change has a runtime kill switch, so a variant and its control can be measured from one binary in adjacent runs. Run to run drift on these cards is a few percent, wider than some of the deltas being judged.
ESCHA_NO_ROTWARP: warp wide activation rotate instead of one block per rowESCHA_NO_ADAPT: mma row tile picked from the batch instead of fixed at 128, plus a register B kernel for narrow tilesESCHA_NO_GENTILE: rows per thread refactor of the generation pathMeasurements
RTX 5060 Ti (sm_120), same binary, kill switches toggled, 2 replicates with order reversed:
Tesla V100 (sm_70), where both mma kernels compile out and
use_mmais false at every width:A 2^3 factorial over the three new changes, 2 replicates with the order reversed on the second pass to balance monotonic drift: all 16 cells bit exact, and each effect is estimated from 4 independent contrasts rather than a single comparison.
Separate finding, not fixed here
On pre Turing cards
use_mmais false at every width, so batches above 16 rows fall through to the register tiled kernel atESCHA_BM128 and pad heavily. On the V100 throughput drops from 86.6 tok/s at 16 rows to 29.1 at 24 rows, and does not recover by 32. The adaptive row tile added here only covers the mma path, so it does not help there. Flagging rather than fixing, since the right shape for that path is a separate question.