feat: simd double unary math ops by clintropolis · Pull Request #19768 · apache/druid

clintropolis · 2026-07-27T06:04:04Z

Description

Adds a bunch of simd optimizations of the remaining double unary operators. The operators do not have direct intrinsics but are backed by a vectorized math library that depends on platform and jdk build, so performance may vary.

Along the way I found sinh, cosh, and tanh were not wired up to SQL, so have added those
Testing with a M1 mac I generally saw a pretty nice improvement:

SELECT SUM(LN(double1)) FROM expressions
SELECT SUM(LN(long1)) FROM expressions
SELECT SUM(EXP(double1)) FROM expressions
SELECT SUM(EXP(long1)) FROM expressions
SELECT SUM(LOG10(double1)) FROM expressions
SELECT SUM(LOG10(long1)) FROM expressions
SELECT SUM(CBRT(double1)) FROM expressions
SELECT SUM(CBRT(long1)) FROM expressions
SELECT SUM(SIN(double1)) FROM expressions
SELECT SUM(SIN(long1)) FROM expressions
SELECT SUM(COS(double1)) FROM expressions
SELECT SUM(COS(long1)) FROM expressions
SELECT SUM(TAN(double1)) FROM expressions
SELECT SUM(TAN(long1)) FROM expressions
SELECT SUM(ASIN(double1)) FROM expressions
SELECT SUM(ASIN(long1)) FROM expressions
SELECT SUM(ACOS(double1)) FROM expressions
SELECT SUM(ACOS(long1)) FROM expressions
SELECT SUM(ATAN(double1)) FROM expressions
SELECT SUM(ATAN(long1)) FROM expressions
SELECT SUM(SINH(double1)) FROM expressions
SELECT SUM(SINH(long1)) FROM expressions
SELECT SUM(COSH(double1)) FROM expressions
SELECT SUM(COSH(long1)) FROM expressions
SELECT SUM(TANH(double1)) FROM expressions
SELECT SUM(TANH(long1)) FROM expressions

Benchmark                        (complexCompression)  (deferExpressionDimensions)  (jsonObjectStorageEncoding)  (query)  (rowsPerSegment)  (schemaType)  (storageType)  (stringEncoding)  (useVectorApi)  (vectorize)  Mode  Cnt   Score   Error  Units
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       71           1500000      explicit           MMAP              UTF8           false        force  avgt    5  30.285 ± 1.411  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       71           1500000      explicit           MMAP              UTF8            true        force  avgt    5  12.606 ± 2.843  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       72           1500000      explicit           MMAP              UTF8           false        force  avgt    5  28.189 ± 1.549  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       72           1500000      explicit           MMAP              UTF8            true        force  avgt    5  14.588 ± 0.554  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       73           1500000      explicit           MMAP              UTF8           false        force  avgt    5  29.777 ± 1.107  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       73           1500000      explicit           MMAP              UTF8            true        force  avgt    5  10.653 ± 0.623  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       74           1500000      explicit           MMAP              UTF8           false        force  avgt    5  22.288 ± 4.846  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       74           1500000      explicit           MMAP              UTF8            true        force  avgt    5  11.983 ± 0.744  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       75           1500000      explicit           MMAP              UTF8           false        force  avgt    5  35.076 ± 1.976  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       75           1500000      explicit           MMAP              UTF8            true        force  avgt    5  13.469 ± 0.812  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       76           1500000      explicit           MMAP              UTF8           false        force  avgt    5  33.361 ± 0.889  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       76           1500000      explicit           MMAP              UTF8            true        force  avgt    5  14.979 ± 0.777  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       79           1500000      explicit           MMAP              UTF8           false        force  avgt    5  33.913 ± 1.415  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       79           1500000      explicit           MMAP              UTF8            true        force  avgt    5  14.416 ± 1.236  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       80           1500000      explicit           MMAP              UTF8           false        force  avgt    5  28.261 ± 1.315  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       80           1500000      explicit           MMAP              UTF8            true        force  avgt    5  17.651 ± 1.072  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       81           1500000      explicit           MMAP              UTF8           false        force  avgt    5  34.117 ± 1.413  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       81           1500000      explicit           MMAP              UTF8            true        force  avgt    5  15.238 ± 0.762  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       82           1500000      explicit           MMAP              UTF8           false        force  avgt    5  28.001 ± 1.855  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       82           1500000      explicit           MMAP              UTF8            true        force  avgt    5  18.244 ± 0.971  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       83           1500000      explicit           MMAP              UTF8           false        force  avgt    5  46.805 ± 2.455  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       83           1500000      explicit           MMAP              UTF8            true        force  avgt    5  17.212 ± 0.605  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       84           1500000      explicit           MMAP              UTF8           false        force  avgt    5  47.951 ± 1.824  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       84           1500000      explicit           MMAP              UTF8            true        force  avgt    5  23.149 ± 0.962  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       85           1500000      explicit           MMAP              UTF8           false        force  avgt    5  28.384 ± 0.810  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       85           1500000      explicit           MMAP              UTF8            true        force  avgt    5  11.907 ± 1.053  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       86           1500000      explicit           MMAP              UTF8           false        force  avgt    5  22.639 ± 1.056  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       86           1500000      explicit           MMAP              UTF8            true        force  avgt    5  13.509 ± 0.615  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       87           1500000      explicit           MMAP              UTF8           false        force  avgt    5  29.227 ± 5.817  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       87           1500000      explicit           MMAP              UTF8            true        force  avgt    5  12.091 ± 0.466  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       88           1500000      explicit           MMAP              UTF8           false        force  avgt    5  22.917 ± 0.659  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       88           1500000      explicit           MMAP              UTF8            true        force  avgt    5  14.683 ± 1.046  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       89           1500000      explicit           MMAP              UTF8           false        force  avgt    5  34.433 ± 2.669  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       89           1500000      explicit           MMAP              UTF8            true        force  avgt    5  15.296 ± 0.497  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       90           1500000      explicit           MMAP              UTF8           false        force  avgt    5  27.284 ± 0.832  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       90           1500000      explicit           MMAP              UTF8            true        force  avgt    5  16.977 ± 1.177  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       91           1500000      explicit           MMAP              UTF8           false        force  avgt    5  38.460 ± 1.404  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       91           1500000      explicit           MMAP              UTF8            true        force  avgt    5  16.669 ± 0.656  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       92           1500000      explicit           MMAP              UTF8           false        force  avgt    5  22.888 ± 0.748  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       92           1500000      explicit           MMAP              UTF8            true        force  avgt    5  17.944 ± 1.868  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       93           1500000      explicit           MMAP              UTF8           false        force  avgt    5  35.114 ± 2.037  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       93           1500000      explicit           MMAP              UTF8            true        force  avgt    5  16.522 ± 1.252  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       94           1500000      explicit           MMAP              UTF8           false        force  avgt    5  22.599 ± 0.924  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       94           1500000      explicit           MMAP              UTF8            true        force  avgt    5  17.941 ± 0.951  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       95           1500000      explicit           MMAP              UTF8           false        force  avgt    5  35.517 ± 0.967  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       95           1500000      explicit           MMAP              UTF8            true        force  avgt    5  37.003 ± 1.007  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       96           1500000      explicit           MMAP              UTF8           false        force  avgt    5  20.983 ± 0.902  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       96           1500000      explicit           MMAP              UTF8            true        force  avgt    5  16.690 ± 0.919  ms/op

FrankChen021

Severity	Findings
P0	0
P1	1
P2	0
P3	0
Total	1

Severity	Findings
P0	0
P1	1
P2	0
P3	0
Total	1

Reviewed 36 of 36 changed files.

This is an automated review by Codex GPT-5.6-Sol

FrankChen021 · 2026-07-27T13:07:23Z

+    final int upperBound = SPECIES.loopBound(currentSize);
+    int i = 0;
+    for (; i < upperBound; i += laneCount) {
+      DoubleVector.fromArray(SPECIES, input, i).lanewise(VectorOperators.SIN).intoArray(outValues, i);


[P1] Keep transcendental results stable across JIT warmup

VectorOperators.SIN initially matches Math.sin, but compiled SLEEF/SVML output can differ. On JDK 25/AArch64, this exact loop changed 665 of 8,192 lanes after warmup; most other added VO_MATHLIB operations also diverged. Compilation mid-query can therefore give identical inputs different bits, affecting exact filters or group keys. Keep these operations scalar unless stable semantics are established, and add warmed parity tests.

hmm, fair. It does seem pretty unlikely that someone is going to be using sin(x) for grouping or equality filtering, so as a compromise I've split these functions out into a separate config useVectorMathApi (which also requires useVectorApi) and documented it to call out this behavior is possible. These configs are both off by default, and this at least makes them available to operators who know they are safe since I am unsure how to reliably pre-warm compilation like this reliably at this time. Added a test that shows that after-warm-up eventually stabilizes.

FrankChen021

Follow-up complete: the stability concern is addressed. VO_MATHLIB operations now require the separate, default-off useVectorMathApi gate in addition to useVectorApi, and the documentation explicitly warns against exact-equality use cases. The warmed parity test also captures the expected stabilization behavior.

Reviewed 41 of 41 changed files.

Validation: git diff --check passed.

This is an automated review by Codex GPT-5.6-Sol

FrankChen021

I have reviewed the updated code for correctness, edge cases, concurrency, and integration risks; no issues found.

Reviewed 42 of 42 changed files. The default-off vector-math gate remains intact; the latest update only adds required spelling exceptions.

This is an automated review by Codex GPT-5.6-Sol

FrankChen021

Severity	Findings
P0	0
P1	1
P2	0
P3	0
Total	1

Severity	Findings
P0	0
P1	1
P2	0
P3	0
Total	1

Reviewed 42 of 42 changed files. The prior default-off stability finding is resolved; this updated review reports one new test-gating regression.

This is an automated review by Codex GPT-5.6-Luna(max)

FrankChen021 · 2026-08-14T12:40:14Z

  public static void initializeForVectorApiTests()
  {
-    INSTANCE = new ExpressionProcessingConfig(null, null, null, true);
+    INSTANCE = new ExpressionProcessingConfig(null, null, null, true, true);


[P1] Keep exact vector-math tests from enabling approximate SIMD

initializeForVectorApiTests() enables approximate math SIMD, but the inherited exact-comparison suite uses Assert.assertEquals for functions such as sin and exp. This PR documents VO_MATHLIB ULP differences and tier-transition changes, so supported JDK/hardware combinations can now fail these tests. Keep this helper's math flag off, or make the consistency assertions tolerance-aware.

This seems like it could be a legit issue with VectorExprResultConsistencyVectorApiTest, possibly dependent on the order in which things are run.

The issue remains at current HEAD: initializeForVectorApiTests() still enables useVectorMathApi at line 60, while the inherited consistency suite compares SIMD and scalar math with exact equality. Please disable this flag in that helper or make the affected assertions tolerance-aware.

Reviewed 42 of 42 changed files.

gianm · 2026-08-14T19:46:17Z

        "true"
    );
    this.useVectorApi = getWithPropertyFallbackFalse(useVectorApi, USE_VECTOR_API);
+    this.useVectorMathApi = getWithPropertyFallbackFalse(useVectorMathApi, USE_VECTOR_MATH_API);


IMO this should be true by default, since I believe most people that enable useVectorApi will want this. Setting it to false is a niche thing.

gianm · 2026-08-14T19:50:20Z

+ * SIMD specialization of {@code (double[]) -> double[]} arc cosine. See {@link SimdSupportedUnaryOp#ACOS} for
+ * the shared VO_MATHLIB performance notes.
+ */
+public final class SimdDoubleAcosProcessor extends SimdDoubleUnaryProcessor


There are so many of these. I wonder if generating the classes at runtime would be better than having all the sources here. Could be something to explore in a follow-up.

gianm · 2026-08-14T19:57:28Z

+    }
+    // VO_MATHLIB ops (SVML/SLEEF-backed transcendentals) sit behind the additional useVectorMathApi opt-in
+    // because they can produce different bits than Math.<op> once JIT tier transitions the loop from C1 to C2.
+    return !simdOp.isMathLib() || ExpressionProcessing.useVectorMathApi();


Should probably have a version of this check in SimdSupportedBinaryOp and SimpleVectorMathBivariateProcessorFactory too. Maybe nothing would return true today, but it seems useful to have it there so it's not forgotten in the future when more ops are added.

Or, possibly even better, move this all into simdOp.isSimdEnabled().

gianm · 2026-08-14T19:58:54Z

+   * {@link #useVectorApi()}.
+   *
+   * <p>These ops route through Intel SVML / Arm SLEEF once the JIT compiles the vector loop to C2; before that
+   * compilation, they fall back to per-lane {@link Math} calls. The two paths can differ by up to 1 ulp, so a


This says up to 1 ulp, but configuration/index.md says "a few ulps" and the test SimdVoMathlibParityTest.MAX_ULPS is checking for 4 ulps. Please make all the commentary about accuracy consistent.

gianm · 2026-08-14T19:59:16Z

  }

+  /**
+   * Whether SIMD dispatch is allowed for math ops backed by the JDK's VO_MATHLIB path (LOG, EXP, SIN, etc). Off by


IMO should be on by default (see other comment).

gianm · 2026-08-14T20:00:38Z

 |`druid.expressions.homogenizeNullMultiValueStringArrays`|If true, multi-value string expression input values of `null`, `[]`, and `[null]` are all coerced to `[null]`. Provided for backwards compatibility with Druid 0.22 and earlier. If false (the default), this coercion only happens when single-value expressions are implicitly mapped across multi-value rows, so the single-valued expression is evaluated with an input of `null`.|false|
 |`druid.expressions.allowVectorizeFallback`|If true, the vectorized query engine handles expressions without a native vectorized implementation using a fallback processor that invokes the scalar expression evaluator in a loop. If false, such expressions cannot be vectorized and the query falls back to the non-vectorized engine.|true|
 |`druid.expressions.useVectorApi`|If true, vectorized expression vector processors and numeric vector aggregators dispatch to SIMD specializations backed by the JDK incubator Vector API (`jdk.incubator.vector`) where available. Requires `--add-modules=jdk.incubator.vector` on the JVM command line (see [strong encapsulation](../operations/java.md#strong-encapsulation)). Off by default while the Vector API remains an incubator JDK feature.|false|
+|`druid.expressions.useVectorMathApi`|If true (and `useVectorApi` is also true), math expressions whose SIMD path is backed by the JDK's VO_MATHLIB routing (Intel SVML on x86, SLEEF on Arm) dispatch to that SIMD path. Results can differ from the scalar `Math.<op>` by up to a few ulps, and the exact bits can shift once the JIT promotes the SIMD loop from C1 to C2, meaning a long-running query can produce different bits for the same input across the tier transition. Safe for `SUM`/`MIN`/`MAX`/range-filter workloads; **do not enable** if queries rely on exact-bit equality of these functions (for example `GROUP BY sin(x)` where all rows with the same `x` must land in the same group, or `WHERE sin(x) = sin(y)`). Off by default.|false|


We definitely shouldn't encourage users to rely on exact-bit equality for floating point numbers. It's better to suggest that people truncate or round for comparison.

gianm · 2026-08-14T20:01:48Z

  public static void initializeForVectorApiTests()
  {
-    INSTANCE = new ExpressionProcessingConfig(null, null, null, true);
+    INSTANCE = new ExpressionProcessingConfig(null, null, null, true, true);


This seems like it could be a legit issue with VectorExprResultConsistencyVectorApiTest, possibly dependent on the order in which things are run.

gianm · 2026-08-14T20:05:08Z

+      DoubleVector.fromArray(SPECIES, input, i).lanewise(VectorOperators.ACOS).intoArray(outValues, i);
+    }
+    for (; i < currentSize; i++) {
+      outValues[i] = scalarFallback.process(input[i]);


This approach of doing scalar fallbacks means that the accuracy discrepancies are more pervasive than the docs in configuration/index.md suggest. The final vector in a cursor (which is likely to be shorter than the max size) will be using different math ops than the other vectors. Probably the best way to fix it is to use a vector op here too, with a mask.

FrankChen021

Severity	Findings
P0	0
P1	0
P2	1
P3	0
Total	1

Reviewed 46 of 46 changed files. The prior test-gating concern appears resolved; this updated review found one test-helper edge case.

Validation: focused git diff --no-ext-diff --check passed. Builds and tests were not run.

This is an automated review by Codex GPT-5.6-Luna(max)

FrankChen021 · 2026-08-27T12:32:44Z

+    }
+    final double ulp = Math.ulp(expected);
+    final double diff = Math.abs(actual - expected);
+    if (Double.isNaN(diff) || diff > (double) maxUlps * ulp) {


P2 Handle non-finite expected values before ULP comparison

When expected is infinite and actual is finite or has the opposite sign, Math.ulp(expected) and the comparison threshold are infinite, so this assertion can incorrectly pass. Handle non-finite values with exact equality before calculating the ULP threshold so parity tests fail when an infinite expected result is not reproduced.

feat: simd double unary math ops

3251e17

clintropolis added the Performance label Jul 27, 2026

github-actions Bot added the Area - Querying label Jul 27, 2026

FrankChen021 reviewed Jul 27, 2026

View reviewed changes

clintropolis added 2 commits July 29, 2026 23:32

add druid.expressions.useVectorMathApi to gate mathlib ops

18228ec

Merge remote-tracking branch 'upstream/master' into simd-double-math-ops

c7203e1

github-actions Bot added the Area - Documentation label Jul 30, 2026

FrankChen021 reviewed Jul 30, 2026

View reviewed changes

spelling exceptions

570a9b2

FrankChen021 reviewed Aug 5, 2026

View reviewed changes

Merge remote-tracking branch 'upstream/master' into simd-double-math-ops

4367685

FrankChen021 reviewed Aug 14, 2026

View reviewed changes

gianm reviewed Aug 14, 2026

View reviewed changes

clintropolis added 3 commits August 15, 2026 15:04

Merge remote-tracking branch 'upstream/master' into simd-double-math-ops

1f7647b

adjust for review

83a7e8b

Merge remote-tracking branch 'upstream/master' into simd-double-math-ops

2b81c4e

FrankChen021 reviewed Aug 27, 2026

View reviewed changes

Uh oh!

Conversation

clintropolis commented Jul 27, 2026

Description

Uh oh!

FrankChen021 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FrankChen021 left a comment

Choose a reason for hiding this comment

Uh oh!

FrankChen021 left a comment

Choose a reason for hiding this comment

Uh oh!

FrankChen021 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FrankChen021 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants