Repository navigation
WaterLily performance regression on CPU multi-threading backend for Julia 1.13.0 #696
Description
The following tests show the performance of WaterLily on Julia 1.11.5 vs 1.13.0-rc1 under different backends. There seems to be a significant performance regression on the KA CPU multi-threading backend, CPUx04. The CPUx01 backend uses SIMD instead of KA and is used for reference. A regression was also observed on Julia 1.12.0, #644.
Benchmark environment: tgv sim_step! (max_steps=25, samples=5)
▶ log2p = 6
┌────────────┬───────────┬────────────┬─────────────┬────────┬─────────┬─────────┬─────────┬──────────────────┬──────────┐
│ Backend │ WaterLily │ Julia │ Allocations │ GC [%] │ Min [s] │ Med [s] │ Max [s] │ Cost [ns/DOF/dt] │ Speed-up │
├────────────┼───────────┼────────────┼─────────────┼────────┼─────────┼─────────┼─────────┼──────────────────┼──────────┤
│ CPUx01 │ v1.6.1 │ 1.11.5 │ 1056 │ 0.00 │ 0.99 │ 1.08 │ 1.18 │ 151.63 │ 1.00 │
│ CPUx01 │ v1.6.1 │ 1.13.0-rc1 │ 1055 │ 0.00 │ 1.04 │ 1.15 │ 1.21 │ 158.71 │ 0.96 │
│ CPUx04 │ v1.6.1 │ 1.11.5 │ 473231 │ 0.00 │ 0.66 │ 0.70 │ 0.71 │ 101.06 │ 1.50 │
│ CPUx04 │ v1.6.1 │ 1.13.0-rc1 │ 473030 │ 0.00 │ 1.00 │ 1.17 │ 1.29 │ 152.73 │ 0.99 │
│ GPU-NVIDIA │ v1.6.1 │ 1.11.5 │ 519531 │ 0.00 │ 0.09 │ 0.10 │ 0.11 │ 14.41 │ 10.52 │
│ GPU-NVIDIA │ v1.6.1 │ 1.13.0-rc1 │ 345454 │ 0.00 │ 0.10 │ 0.10 │ 0.11 │ 15.59 │ 9.73 │
└────────────┴───────────┴────────────┴─────────────┴────────┴─────────┴─────────┴─────────┴──────────────────┴──────────┘
▶ log2p = 7
┌────────────┬───────────┬────────────┬─────────────┬────────┬─────────┬─────────┬─────────┬──────────────────┬──────────┐
│ Backend │ WaterLily │ Julia │ Allocations │ GC [%] │ Min [s] │ Med [s] │ Max [s] │ Cost [ns/DOF/dt] │ Speed-up │
├────────────┼───────────┼────────────┼─────────────┼────────┼─────────┼─────────┼─────────┼──────────────────┼──────────┤
│ CPUx01 │ v1.6.1 │ 1.11.5 │ 1155 │ 0.00 │ 7.03 │ 7.08 │ 7.14 │ 134.01 │ 1.00 │
│ CPUx01 │ v1.6.1 │ 1.13.0-rc1 │ 1155 │ 0.00 │ 7.44 │ 7.47 │ 7.58 │ 141.83 │ 0.94 │
│ CPUx04 │ v1.6.1 │ 1.11.5 │ 508180 │ 0.00 │ 4.25 │ 4.63 │ 5.35 │ 81.14 │ 1.65 │
│ CPUx04 │ v1.6.1 │ 1.13.0-rc1 │ 507980 │ 0.00 │ 6.28 │ 6.76 │ 7.47 │ 119.69 │ 1.12 │
│ GPU-NVIDIA │ v1.6.1 │ 1.11.5 │ 560755 │ 0.00 │ 0.66 │ 0.66 │ 0.67 │ 12.59 │ 10.64 │
│ GPU-NVIDIA │ v1.6.1 │ 1.13.0-rc1 │ 378728 │ 0.00 │ 0.71 │ 0.72 │ 0.72 │ 13.49 │ 9.94 │
└────────────┴───────────┴────────────┴─────────────┴────────┴─────────┴─────────┴─────────┴──────────────────┴──────────┘
Benchmark environment: sphere
▶ log2p = 3
┌────────────┬───────────┬────────────┬─────────────┬────────┬─────────┬─────────┬─────────┬──────────────────┬──────────┐
│ Backend │ WaterLily │ Julia │ Allocations │ GC [%] │ Min [s] │ Med [s] │ Max [s] │ Cost [ns/DOF/dt] │ Speed-up │
├────────────┼───────────┼────────────┼─────────────┼────────┼─────────┼─────────┼─────────┼──────────────────┼──────────┤
│ CPUx01 │ v1.6.1 │ 1.11.5 │ 955 │ 0.00 │ 1.01 │ 1.06 │ 1.18 │ 137.47 │ 1.00 │
│ CPUx01 │ v1.6.1 │ 1.13.0-rc1 │ 955 │ 0.00 │ 1.06 │ 1.08 │ 1.21 │ 143.59 │ 0.96 │
│ CPUx04 │ v1.6.1 │ 1.11.5 │ 437230 │ 0.00 │ 0.60 │ 0.61 │ 0.62 │ 81.01 │ 1.70 │
│ CPUx04 │ v1.6.1 │ 1.13.0-rc1 │ 437030 │ 0.00 │ 1.01 │ 1.11 │ 1.16 │ 137.22 │ 1.00 │
│ GPU-NVIDIA │ v1.6.1 │ 1.11.5 │ 480879 │ 0.00 │ 0.09 │ 0.09 │ 0.09 │ 11.97 │ 11.48 │
│ GPU-NVIDIA │ v1.6.1 │ 1.13.0-rc1 │ 327128 │ 0.00 │ 0.09 │ 0.10 │ 0.11 │ 12.84 │ 10.71 │
└────────────┴───────────┴────────────┴─────────────┴────────┴─────────┴─────────┴─────────┴──────────────────┴──────────┘
▶ log2p = 4
┌────────────┬───────────┬────────────┬─────────────┬────────┬─────────┬─────────┬─────────┬──────────────────┬──────────┐
│ Backend │ WaterLily │ Julia │ Allocations │ GC [%] │ Min [s] │ Med [s] │ Max [s] │ Cost [ns/DOF/dt] │ Speed-up │
├────────────┼───────────┼────────────┼─────────────┼────────┼─────────┼─────────┼─────────┼──────────────────┼──────────┤
│ CPUx01 │ v1.6.1 │ 1.11.5 │ 1055 │ 0.00 │ 9.65 │ 9.80 │ 9.88 │ 163.55 │ 1.00 │
│ CPUx01 │ v1.6.1 │ 1.13.0-rc1 │ 1055 │ 0.00 │ 9.99 │ 10.08 │ 10.18 │ 169.30 │ 0.97 │
│ CPUx04 │ v1.6.1 │ 1.11.5 │ 472180 │ 0.00 │ 4.28 │ 4.38 │ 4.39 │ 72.60 │ 2.25 │
│ CPUx04 │ v1.6.1 │ 1.13.0-rc1 │ 471980 │ 0.00 │ 5.81 │ 7.59 │ 8.79 │ 98.43 │ 1.66 │
│ GPU-NVIDIA │ v1.6.1 │ 1.11.5 │ 522130 │ 0.00 │ 0.70 │ 0.71 │ 0.71 │ 11.89 │ 13.76 │
│ GPU-NVIDIA │ v1.6.1 │ 1.13.0-rc1 │ 365427 │ 0.00 │ 0.76 │ 0.77 │ 0.79 │ 12.90 │ 12.68 │
└────────────┴───────────┴────────────┴─────────────┴────────┴─────────┴─────────┴─────────┴──────────────────┴──────────┘
Benchmark environment: jelly
▶ log2p = 5
┌────────────┬───────────┬────────────┬─────────────┬────────┬─────────┬─────────┬─────────┬──────────────────┬──────────┐
│ Backend │ WaterLily │ Julia │ Allocations │ GC [%] │ Min [s] │ Med [s] │ Max [s] │ Cost [ns/DOF/dt] │ Speed-up │
├────────────┼───────────┼────────────┼─────────────┼────────┼─────────┼─────────┼─────────┼──────────────────┼──────────┤
│ CPUx01 │ v1.6.1 │ 1.11.5 │ 1390 │ 0.00 │ 0.87 │ 0.89 │ 0.92 │ 264.49 │ 1.00 │
│ CPUx01 │ v1.6.1 │ 1.13.0-rc1 │ 1390 │ 0.00 │ 0.92 │ 0.93 │ 0.97 │ 279.29 │ 0.95 │
│ CPUx04 │ v1.6.1 │ 1.11.5 │ 1296258 │ 0.00 │ 0.70 │ 0.73 │ 0.74 │ 214.50 │ 1.23 │
│ CPUx04 │ v1.6.1 │ 1.13.0-rc1 │ 1296258 │ 0.00 │ 1.04 │ 1.23 │ 1.66 │ 316.02 │ 0.84 │
│ GPU-NVIDIA │ v1.6.1 │ 1.11.5 │ 795072 │ 0.00 │ 0.11 │ 0.11 │ 0.13 │ 32.80 │ 8.06 │
│ GPU-NVIDIA │ v1.6.1 │ 1.13.0-rc1 │ 531130 │ 0.00 │ 0.12 │ 0.14 │ 0.15 │ 35.45 │ 7.46 │
└────────────┴───────────┴────────────┴─────────────┴────────┴─────────┴─────────┴─────────┴──────────────────┴──────────┘
▶ log2p = 6
┌────────────┬───────────┬────────────┬─────────────┬────────┬─────────┬─────────┬─────────┬──────────────────┬──────────┐
│ Backend │ WaterLily │ Julia │ Allocations │ GC [%] │ Min [s] │ Med [s] │ Max [s] │ Cost [ns/DOF/dt] │ Speed-up │
├────────────┼───────────┼────────────┼─────────────┼────────┼─────────┼─────────┼─────────┼──────────────────┼──────────┤
│ CPUx01 │ v1.6.1 │ 1.11.5 │ 1852 │ 0.00 │ 6.42 │ 6.68 │ 6.79 │ 245.02 │ 1.00 │
│ CPUx01 │ v1.6.1 │ 1.13.0-rc1 │ 1852 │ 0.00 │ 6.68 │ 6.88 │ 7.13 │ 254.88 │ 0.96 │
│ CPUx04 │ v1.6.1 │ 1.11.5 │ 3027277 │ 0.00 │ 4.02 │ 4.28 │ 4.63 │ 153.52 │ 1.60 │
│ CPUx04 │ v1.6.1 │ 1.13.0-rc1 │ 3027277 │ 0.00 │ 5.96 │ 6.36 │ 6.88 │ 227.45 │ 1.08 │
│ GPU-NVIDIA │ v1.6.1 │ 1.11.5 │ 1011053 │ 0.00 │ 0.47 │ 0.50 │ 0.51 │ 18.08 │ 13.55 │
│ GPU-NVIDIA │ v1.6.1 │ 1.13.0-rc1 │ 675930 │ 0.00 │ 0.53 │ 0.54 │ 0.55 │ 20.26 │ 12.09 │
└────────────┴───────────┴────────────┴─────────────┴────────┴─────────┴─────────┴─────────┴──────────────────┴──────────┘
Activity
b-fg commentedon Oct 19, 2026
For reference, I am tracked the performance of KA 0.10 CPU backend (POCL) on WaterLily in WaterLily-jl/WaterLily.jl#365. I can open a new issue in KA if preferred.
maleadt commentedon Oct 20, 2026
Try testing with #906
b-fg commentedon Oct 21, 2026
Tested #906 with WaterLily (its KA 0.10 branch, WaterLily-jl/WaterLily.jl#365) at 4 threads as in the original report. I also applied #912 on top of it: WaterLily's loops over vector fields are 4-d kernels, which KA 0.10 launches with a LinearLaunch that is very slow on PoCL with a tuned workgroup size. #912 launches them on the N-d grid (it cherry-picks cleanly onto #906).
The slowdown reported here is gone with the current WaterLily, but not because of Julia or KA: with WaterLily v1.6.1 it is still there on Julia 1.13.2 (tgv 64³ +57%), and KA 0.9.45 only reduces it (+20% on 1.13.0-rc1). Something WaterLily changed since v1.6.1 avoids it (second table below). With the current WaterLily, 1.13 is on par with or faster than 1.11, except for the smallest cylinder (+12%).
Time per step [ms], julia -t 4, KA 0.9.45 with workgroup 64, KA 0.10.1 with workgroup 128 (WaterLily's defaults), #906 + #912 with a tuned workgroup size:
JULIA_KA_POCL_HOST_KERNELS=0, POCL: run kernels on Julia's threads with PoCL host kernels #906 + Launch ndranges with more than three dimensions on the N-d grid #912 is about as fast as Launch ndranges with more than three dimensions on the N-d grid #912 alone (2 threads, below). POCL: run kernels on Julia's threads with PoCL host kernels #906 alone with WaterLily's workgroup of 128 is on par with KA 0.9.45 (2 threads, Julia 1.11.5).The slowdown reported here, changing one thing at a time from the original setup (4 threads, time per step [ms], Julia 1.11.5 → 1.13)
The original setup on Julia 1.11.5 gives 26.0 ms for tgv 64³, against 28 ms in the original report on this laptop. "Current WaterLily" is master plus the KA 0.10 changes of WaterLily-jl/WaterLily.jl#365, which don't change the KA 0.9 path.
2 threads, Julia 1.11.5
Launch cost of a trivial kernel, 2 threads, Julia 1.11.5 (median µs per launch including the wait, allocations in parentheses)
Each number is a single run (minimum of three 25-step means after 50 warm-up steps), and some spreads reach 10-23%, so differences under about 10% are noise. #906 was at 7eef96c with
pocl_standalone_next_jll7.2.2+2, and the host kernels were active.From WaterLily's side this is resolved, so I'm closing this issue.