Problem cache follow-on: pluggable backends, layered cache paths, compile-options API, and offline tooling

[danieyan-amd]

Summary

Follow-on to #4835 (hardware-provenance problem cache). This adds the deployment and tooling layer around the problem cache so pre-tuned caches can be shipped, discovered, and combined without environment variables.

• Pluggable cache backends — a type-erased problem_cache_backend with a JSON backend and an optional SQLite backend (built on system SQLite), selectable through compile_options.
• Layered cache paths — compile_options gains an ordered list of problem-cache paths (problem_cache_paths, with a single-path convenience). Paths are searched in priority order (first hit wins) and loaded read-only, so an application-provided cache can take precedence over a shipped one without mutating either.
• Compile-options API (C / C++ / Python) — migraphx_compile_options_set_problem_cache_paths and the matching C++/Python wrappers, so the paths can be set programmatically rather than via an environment variable.
• Offline aggregator — a library API to merge, validate, and convert problem-cache files across devices, with duplicate/conflict detection and configurable conflict policies (error_on_conflict / first_wins / last_wins).
• Driver subcommands — command-line frontends for the aggregator (merge / validate / convert).

Testing

New GPU unit tests cover the backend abstraction, the SQLite backend round-trip, the layered path override, and the aggregator (merge / validate / convert plus conflict and legacy-device policies), along with an API-level compile-options test.

Notes

• Opening as a draft to run CI against develop and for review.
• The runtime engine and the hardware-provenance device key land in Extend problem cache with hardware provenance metadata #4835; this builds on top of that.

[github-actions]

Thank you for your contribution! Since this is an external pull request, a maintainer must review PR and add the "ok-to-test" label if it is approved for testing.

[danieyan-amd]

I realized that not all of the runtime functionality is there for the backed abstraction layer is there, thus why it was in a draft so more changes are going to be committed.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #5117   +/-   ##
========================================
  Coverage    93.09%   93.09%           
========================================
  Files          623      623           
  Lines        33073    33073           
========================================
  Hits         30789    30789           
  Misses        2284     2284           

Files with missing lines	Coverage Δ
src/include/migraphx/compile_options.hpp	85.71% <ø> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

This PR builds out the “deployment + tooling” layer around the GPU problem cache by introducing a type-erased backend interface, adding layered (priority-ordered) cache paths, and wiring the cache configuration through GPU backend options so applications can ship and override pre-tuned caches.

Changes:

• Introduces problem_cache_backend type-erasure plus a JSON backend and a new SQLite backend implementation.
• Adds layered cache loading (multiple read-only caches searched in priority order; single cache is writable) and threads this through GPU target/context/compile paths.
• Adds GPU unit tests covering backend behavior, SQLite round-trip, and layered-path override; updates generator plumbing and build files.

Reviewed changes

Copilot reviewed 23 out of 23 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tools/include/gpu/problem_cache_backend.hpp	te.py DSL definition for the type-erased backend interface.
tools/generate.py	Routes tools/include/gpu/* generated headers into the GPU target include tree.
test/gpu/problem_cache_backend.cpp	Tests type-erased backend forwarding and any_cast for JSON backend.
test/gpu/sqlite_problem_cache.cpp	Tests SQLite backend round-trip and type-erasure integration.
test/gpu/problem_cache_path_override.cpp	Tests explicit path override and layered (priority) cache behavior.
src/targets/gpu/include/migraphx/gpu/problem_cache_backend.hpp	Generated type-erased backend wrapper header.
src/targets/gpu/include/migraphx/gpu/json_problem_cache.hpp	Declares JSON backend (including legacy migration support).
src/targets/gpu/json_problem_cache.cpp	Implements JSON backend load/save + legacy-format migration + key normalization.
src/targets/gpu/include/migraphx/gpu/sqlite_problem_cache.hpp	Declares SQLite backend.
src/targets/gpu/sqlite_problem_cache.cpp	Implements SQLite backend load/save and in-memory operations.
src/targets/gpu/include/migraphx/gpu/cache_device_key.hpp	Extracts cache_device_key into its own header.
src/targets/gpu/include/migraphx/gpu/problem_cache.hpp	Reworks problem_cache to use backend + layered read-only backends + path override.
src/targets/gpu/problem_cache.cpp	Implements layered lookup semantics and path override behavior.
src/targets/gpu/include/migraphx/gpu/context.hpp	Adds layered cache configuration and narrow cache get/insert/mark/save API.
src/targets/gpu/target.cpp	Adds problem_cache_files backend option and configures cache loading from it.
src/targets/gpu/compile_ops.cpp	Uses layered cache lookup and writes only to the writable cache.
src/targets/gpu/include/migraphx/gpu/gemm_impl.hpp	Adjusts default-solution API to take const context&.
src/targets/gpu/gemm_impl.cpp	Uses new context cache API for gemm solution load/store.
src/targets/gpu/include/migraphx/gpu/hip_gemm_impl.hpp	Adjusts default-solution API to take const context&.
src/targets/gpu/hip_gemm_impl.cpp	Uses new context cache API for hipBLASLt solution load/store.
src/targets/gpu/CMakeLists.txt	Adds new backend implementation sources to GPU library build.
src/include/migraphx/compile_options.hpp	Adds <vector> include (compile options now transport path lists elsewhere in the PR).
CHANGELOG.md	Notes the new layered cache priority list delivered via GPU backend options.

Suppressed comments (1)

src/targets/gpu/sqlite_problem_cache.cpp:157

• sqlite_problem_cache::get()/has() perform lookups using the raw key value, which requires callers to already canonicalize the key. Normalizing the lookup key here keeps behavior aligned with json_problem_cache and with problem_cache (which normalizes problems before querying).

optional<value> sqlite_problem_cache::get(const cache_device_key& dk, const value& key) const
{
    auto bucket_it = cache.find(dk);
    if(bucket_it == cache.end())
        return nullopt;
    auto it = bucket_it->second.find(key);
    if(it == bucket_it->second.end())
        return nullopt;
    return it->second;
}

bool sqlite_problem_cache::has(const cache_device_key& dk, const value& key) const
{
    auto bucket_it = cache.find(dk);
    if(bucket_it == cache.end())
        return false;
    return contains(bucket_it->second, key);
}

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.