ggml: additive INT8/ternary types, fused CPU ops, and kernels for VibeASR (1/2)

[XsquirrelC]

Per your request in microsoft/VibeASR.cpp#10, split into two PRs: this one is the additive ggml side, #448 is the model integration. This one changes no existing behaviour and adds no model.

This replaces #438 and #445, which are now closed — same code, squashed into one PR to match what you asked for.

Additive, not a rewrite of ggml_conv_1d

You flagged that VibeASR.cpp's ggml fork modifies existing ggml_conv_1d behaviour, which would affect nearly every model in audio.cpp. That change is not here and is not needed. What the fork gets out of the modified ggml_conv_1d is asymmetric (causal) padding on an INT8 im2col; this PR gets the same thing from a new op, ggml_im2col_asym, leaving ggml_conv_1d and ggml_im2col untouched.

Every entry point added here is new. No existing op changes behaviour, and no existing type changes meaning. The two places existing files are touched at all are:

• ggml_compute_forward_mul_mat and ggml_graph_plan get a branch guarded by src0->type == GGML_TYPE_I2_S (8 lines and 10 lines). Unreachable for every existing type.
• ggml_compute_forward_dup / _cont learn to carry the I8_S in-band scale. Also guarded on the type.

Two new types

Type	Id	Layout
GGML_TYPE_I8_S	42	per-tensor int8: nelements bytes, then one padded F32 scale
GGML_TYPE_I2_S	43	ternary {-1,0,+1} as codes {0,1,2}, 128 values per 32-byte group, then one padded F32 absmax

Ids 42/43 rather than the 36/37 VibeASR.cpp's fork uses, because upstream ggml had already spent 36/37 on the retired IQ4_NL_4_4 / IQ4_NL_4_8 slots. The conversion tool in the model PR rewrites the 4-byte type field; the payload bytes are identical, so no requantization.

Both carry their scale in band, after the payload, rather than in a ggml_tensor field. That is what the published VibeASR checkpoints contain, and it means an I8_S tensor is self-describing — which matters because activations get requantized between every stage and a scale in a side channel would have to be threaded through every node.

That choice has one consequence worth calling out, and it is the dup/cont commit: the encoder flips activations between channel-major and length-major constantly, and a plain byte copy would move the values but leave the scale bytes behind. test_i8_s_cont_permute covers it — without the fix the values are right and everything downstream is off by an arbitrary factor.

The scale is stored as a multiplier (amax/127, dequantize by multiplying), not as VibeASR.cpp's reciprocal (127/amax, dequantize by dividing). Same number to within the last float bit, but the multiplier convention cannot divide by zero, so an all-zero tensor dequantizes to zeros instead of NaNs. test_degenerate_scales pins that.

Five fused INT8 ops

INT8-activation inference is only a win if requantization happens inside the op that produced the values. Splitting matmul → add bias → relu into three ggml nodes means dequantizing to F32 and requantizing twice for nothing, so these are fused:

Op	What it does
ggml_mul_mat_add	I8_S × I8_S → I8_S, bias added and output requantized in the epilogue
ggml_mul_mat_add_relu	same plus ReLU before requantization
ggml_add_scaled	I8_S + I8_S → I8_S with the two input scales reconciled
ggml_rms_norm_scaled	RMSNorm straight from I8_S to I8_S
ggml_im2col_asym	im2col with independent left/right padding, so a causal conv needs no separate pad node

Plus ggml_mul_mat support for I2_S, which needs no new op at all — a ternary weight in a plain ggml_mul_mat just works, so a model can use it without opting into anything.

Arithmetic is exact where it can be

For I2_S the unsigned code trick falls out algebraically: sum(code*q) = sum((w+1)*q) = sum(w*q) + sum(q), so the kernel accumulates sum(code*q) in int32 and the epilogue subtracts the row's int8 sum. Everything up to the final multiply is integer, so i2_s_mul_mat_test compares bit-exactly against a plain-loop reference rather than with a tolerance — a wrong packing that happens to be numerically close would otherwise pass.

ggml_mul_mat with I2_S cannot use the vec_dot_type path: that contract hands the kernel two row pointers and a fixed row_size, while the epilogue here needs the per-row activation scale and the per-row int8 sum. Those live in a sidecar after the quantized rows in params->wdata. Upstream's own fork solves this by patching the body of the generic ggml_compute_forward_mul_mat inline, including a src1_col_de = wdata + i11*nb11/4 pointer reinterpretation — which is exactly the kind of change to a shared path this PR avoids.

Kernels

ggml_vec_dot_i8_i8 and ggml_vec_dot_i2_i8 in vec.cpp, each with AVX2, NEON/aarch64, and scalar paths. The I2_S packing (byte gp of a 32-byte group holds group-relative positions gp, 32+gp, 64+gp, 96+gp in bit pairs 6/4/2/0) is chosen so a group's four code lanes line up with four consecutive 32-value slices of the activation row — no shuffling on either side.

One correctness note on the AVX2 I2_S path: it widens to int32 every eight groups, not at the end of the row. A lane holds a sum of two code*int8 products, at most 2*2*127 = 508; eight groups contribute 32 of them, 32*508 = 16256, inside int16 with room — 16 groups would not be. Upstream accumulates 32 groups in int16, which overflows on activations near full scale. test_accumulator_headroom drives exactly that case (every code 2, every activation +127).

NEON uses vdotq_s32 under __ARM_FEATURE_DOTPROD with a vmull_s8 + vpadalq_s16 fallback, so ARMv8.0 targets without dotprod still work.

Tests

Two new unit tests, no model weights needed, both registered in ctest. Every case runs at nth=1 and nth=4, since a missing barrier or an overlapping output split shows up as a thread-count-dependent result.

i8_s_fused_ops_test — round trips for both types, then add_scaled, rms_norm_scaled, mul_mat_add (with and without ReLU, several IC/OC), depthwise mul_mat_add, im2col_asym, and the cont/permute scale-carrying case, each against a plain-loop reference.

i2_s_mul_mat_test — test_pack_layout decodes the packing by hand (not via ggml_i2_s_to_float) so the layout the kernel reads is pinned independently of the dequantizer written alongside it; test_mul_mat over K = 128 / 1024 / 1152 and N = 1 / 2 / 64 / 70 / 100 (the int16-flush boundary, unaligned tails, and fewer output features than threads); test_mul_mat_batched for 3-D src1; test_accumulator_headroom; test_degenerate_scales.

Build

Release, gcc, x86-64 AVX2, 24 vCPU EPYC 7V13: ctest 42/42 pass, 0 failures, on this branch alone with no model code and no checkpoints. Both new tests pass at nth=1 and nth=4.

No behaviour change for any existing type or op — every new code path is guarded on GGML_TYPE_I8_S / GGML_TYPE_I2_S, and the ggml_graph_plan work-size addition is inside the same guard.

Upstream

• https://github.com/microsoft/VibeASR.cpp
• Weights: https://huggingface.co/XsquirrelC/VibeVoice-ASR-BitNet