Cache sorted async dependency vectors

[Mac-Huang]

Summary

AsyncDepsInfo::getOpDeps() and getConsumerOps() materialize and sort a DenseSet on every call. The feasible-allocation, prefetch, and scheduling paths query many stable dependency sets repeatedly, reconstructing and sorting the same vectors.

This change:

• lazily caches sorted dependency and consumer vectors;
• invalidates only entries whose underlying set changes;
• clears the relevant caches when dependency maps are optimized or consumer maps are rebuilt;
• routes initial dependency insertion through the same cache-invalidating path; and
• preserves the existing deterministic, sorted, by-value public API.

Performance evidence

The previous validation campaign used Qwen2.5-7B INT8 on NPU.5010 with context windows 1024, 2048, 4096, and 8192. It alternated baseline/candidate ordering across three adjacent pairs per window, excluded warmups, and bypassed the UMD cache.

Context window	Baseline median	Candidate median	Median paired improvement
1024	351.715 s	339.112 s	3.583%
2048	241.834 s	232.774 s	3.746%
4096	253.098 s	243.770 s	3.678%
8192	306.146 s	297.526 s	2.816%

The candidate won all 12 pairs, with a 1.0359x geometric-mean speedup. A gated W2048 CPU profile measured:

• qsort-family CPU time: 7.960 s to 0.747 s (90.6% reduction);
• FeasibleAllocation: 18.471 s to 7.381 s (60.0% reduction); and
• full compile_model: 242.647 s to 234.255 s (3.46% improvement).

Those measurements predate the current develop base and are retained as historical evidence only.

Current develop refresh

The final branch was benchmarked directly against 0b38f7d42113ff329ac2bdd33583d123de4ccf2f with a version-matched OpenVINO 2026.4 C++ harness. The run used one excluded warm-up per arm/window, three adjacent pairs, balanced AB/BA ordering across windows, fresh processes, cold UMD cache, exact artifact hashes, and /usr/bin/time -v RSS.

Window	Baseline median	Candidate median	Median wall improvement	Candidate wins	FeasibleAllocation improvement	Median RSS change
2048	244.907 s	244.982 s	-0.031%	1/3	2.255%	-0.483%
1024	307.716 s	309.224 s	-0.490%	0/3	0.912%	+0.250%

All six measured baseline/candidate pairs had identical 458-pass inventories and report counts. Across both windows, the candidate won 1/6 pairs; its geometric-mean improvement was -0.368%. The RSS gate passed, but the required 2% wall-time improvement did not.

A compile-interval diagnostic profile also did not reproduce the provisional 50% global qsort-family reduction. For one Qwen2.5-7B INT8 W2048 compile per arm at 49 Hz, baseline recorded 24 inclusive qsort-family samples (0.490 sampled CPU seconds) and the candidate recorded 34 (0.694 seconds), an observed reduction of -41.7%. Total sampled CPU and compile wall time were effectively unchanged, pass inventories matched exactly, and compiler-reported FeasibleAllocation time improved by 1.88%. Because this is a low-count, fixed-order, global attribution from stripped binaries, it does not isolate AsyncDepsInfo or establish a regression; it shows that the historical global-qsort gate was not met on current develop.

The current evidence therefore does not justify promoting this draft for review. A redesign should begin with targeted cache hit/miss, invalidation, and sort-call instrumentation; repeat AB/BA profiles with symbolization before spending the larger model-matrix budget.

Correctness and validation

Validated on NUC16 from final commit fce7e0e737e3370b9067fa3adecc201753540aed, based directly on develop at 0b38f7d42113ff329ac2bdd33583d123de4ccf2f:

• clang-format 18.1.8 and git diff --check: passed.

• Focused MLIR_AsyncDepsInfo.*: 5/5 passed.

• Complete npuUnitTests: 7,085/7,085 enabled tests passed; 31 disabled.

• All-platform LIT validation completed with exit status 0 for every invocation:

  Platform	Passed	Expected failures (XFAIL)	Unsupported	Total
  NPU3720	967	3	682	1,652
  NPU4000	1,225	4	423	1,652
  NPU5010	1,237	7	408	1,652
  NPU5020	9	0	1,643	1,652

  UNSUPPORTED is the expected per-platform filtering of the shared corpus; there were no unexpected failures, unresolved tests, timeouts, or unexpected passes.

• ov_npu_unit_tests: 1,500 total; 1,463 passed and 37 hardware-dependent tests skipped.

• Qwen2.5-7B INT8 W2048 PLUGIN real-model correctness smoke: version-matched OpenVINO 2026.4 C++ baseline and candidate compiles both succeeded, each loading exactly its intended compiler artifact. Baseline wall time was 244.495 s; candidate wall time was 245.818 s. These excluded warm-ups are a correctness gate, not performance evidence.

• Paired current-upstream performance and peak-RSS campaign completed with exact pass-inventory agreement and acceptable RSS, but the wall-time acceptance gate was not met. The excluded smoke timings above are not performance evidence.

• Transient fresh-sort cache-coherence oracle: recomputed and compared the sorted dependency/consumer vector on every lookup; 50 focused test executions, all 7,085 enabled unit tests, and a Qwen2.5-7B W2048 compile passed without a stale-cache finding. The source, unit test, compiler, and runtime compiler were restored to their exact pre-run hashes.

• Targeted current-base qsort CPU profile completed with exact compiler/runtime provenance and pass-inventory agreement. The broad global-qsort gate was not met; feature-specific attribution remains inconclusive without symbols or targeted counters.

GitHub-hosted checks awaiting repository approval

The following workflows currently report action_required for the final SHA and remain unchecked until they are approved and complete:

• Linux (Ubuntu 22.04) workflow.
• Linux (Ubuntu 24.04) build/test workflow.
• PLUGIN and DRIVER compilation matrices for NPU.3720, NPU.4000, NPU.5010, and NPU.5020.
• Windows (2022) workflow.
• clang-format workflow (the manual formatting check above has passed).
• Scorecard supply-chain security workflow.

Expanded profiling plan

The work is split into a bounded merge gate and a broader scale study so that the PR is not justified by one model family:

1. Current-base merge gate: Qwen2.5-7B INT8 at W1024 and W2048 is complete and did not meet the wall-time or global-qsort gates. Before proceeding, add cache hit/miss, invalidation, and sort-call counters; repeat symbolized AB/BA profiles. Only if those identify a material feature-specific hotspot, add Qwen2.5-0.5B W128 and ResNet-18 INT8 at 224 as small negative controls.
2. Required compiler compatibility matrix: the repository's 156 public-model configurations in both PLUGIN and DRIVER modes across NPU.3720, NPU.4000, NPU.5010, and NPU.5020. Existing families include MobileNetV2, ResNet-50, YOLOv4, BERT, ViT, CLIP, and SAM components.
3. Synthetic scale controls: dependency DAGs with 128, 1K, 5K, 10K, and 25K async operations at fan-in 2, 8, and 32. Measure construction, cold reads, 100 hot sweeps, 1% mutation plus touched-entry reads, consumer rebuild, and dependency optimization; assert fresh-sort equality after every phase.
4. Installed-model scale study: Qwen2.5 INT8/FP16 at 0.5B, 1.5B, 3B, and 7B; Llama-3.2 INT8/FP16 at 1B and 3B; ResNet-18/34/50/101/152 across 128-640 resolutions; MobileNetV3, EfficientNet-B0, DenseNet-121, RegNetY-040, Inception-v3, ConvNeXt-Tiny, and ViT Base/Large. Screen the 115 currently registered model/shape cells with three pairs, then promote representative improvements, neutral cases, and regressions to five-pair runs.
5. Future architecture coverage: stage TinyLlama, Phi-3 Mini, Gemma-2, Mistral, Qwen2.5-14B, and MoE models only after a separate storage/export preflight; these are follow-up coverage rather than a merge gate.
6. Measurement method: identical Release builds and dependency pins; one fresh process per compile; cold UMD cache; recorded compiler/runtime/model hashes; paired wall time with read time kept separate; and /usr/bin/time -v RSS. Keep perf stat, compile-interval perf record, pass timing, qsort samples, async vertices/edges, degree, cache hit/miss, invalidation, allocation, and retained-capacity counters in separate diagnostic runs so instrumentation cannot perturb the wall-time samples.
7. Acceptance gate: dependency and consumer order matches a diagnostic fresh-sort oracle on every lookup and compilation succeeds; at least 50% less qsort-family CPU time and at least 2% lower median compile_model wall time on both high-task graphs; no median regression above 3% on either small control; no median peak-RSS growth at or above 5% on the largest graph; and explicit reporting of the O(V + E) cache-memory cost. Blob hashes are recorded for provenance, not compared as a correctness gate, because repeated identical compiler runs are not bit-deterministic.

Memory tradeoff

The cache retains sorted vectors alongside the dependency sets, adding O(V + E) storage plus per-entry optional/vector overhead. On the current-base paired run, median maximum RSS changed by -0.483% at W2048 and +0.250% at W1024; the worst paired increase was 2.063%, within the 5% gate.

JIRA ticket

Fixes #342

Target Platform For Release Notes

• NPU37XX
• NPU40XX
• NPU50XX
• NONE (Not included in release notes)

The implementation is architecture-independent. Hardware performance profiling is performed on NPU.5010; the CI compilation and LIT matrices cover all supported architecture families.

Classification of this Pull Request

• Maintenance
• BUG
• Feature

AI Assistance

• AI assistance used: yes
• If yes: repository and checklist review, test planning, and editing suggestions.
• Human validation performed: contributor code review is complete; formatting, focused and full unit tests, all-platform LIT, OpenVINO NPU unit tests, version-matched real-model smoke compiles, the paired current-base campaign, the fresh-sort oracle, and the targeted current-base profile completed on the current head. Correctness passed, but the wall-time and broad global-qsort acceptance gates were not met. Repository CI and maintainer review remain pending while the draft is performance-blocked.

[Mac-Huang]

Update after completing current-base validation at fce7e0e737e3370b9067fa3adecc201753540aed:

All manual correctness gates now pass, including formatting, focused/full compiler unit tests, all-platform LIT, OpenVINO NPU unit tests, version-matched real-model compiles, and a transient fresh-sort cache-coherence oracle.

The performance gate did not pass on current develop: median wall-time changes were -0.031% at W2048 and -0.490% at W1024, with 1/6 candidate wins and -0.368% geometric-mean improvement. A separate compile-interval diagnostic did not reproduce the provisional 50% global qsort-family reduction. Its low-count global attribution is not feature-specific, so it does not establish a regression, but it also does not justify promotion.

I am keeping the PR in draft and withdrawing the immediate CI request while the change is performance-blocked. The description contains the complete evidence and a targeted-instrumentation/redesign plan; there is no need to spend hosted CI capacity on this head yet.