Refactor OpenVINO backend: isolate op support, manage buffers, and document API - #278
Open
zhaixuejun1993 wants to merge 16 commits into
Open
Conversation
Move OpenVINO op support / unsupported-case policy logic out of ggml-openvino.cpp into a dedicated implementation pair: ggml-openvino-op-support.cpp and ggml-openvino-op-support.h. Keep behavior unchanged by retaining the original device callback shape and delegating through ggml_openvino_device_supports_op_impl(). This reduces ggml-openvino.cpp size and keeps support-policy code isolated for easier maintenance and future policy updates.
Expand the developer-facing OP/Limitation comment block to include all operators currently registered in openvino/op_table.cpp. For each registered op, document either a concrete runtime policy gate or explicitly mark that no extra restriction exists beyond the global type/rank gates.
Move OpenVINO backend buffer allocation into a small storage abstraction that owns either host memory or GPU remote USM tensors. The buffer context now keeps a unique_ptr to this storage and only exposes the raw data pointer and ov::Tensor wrapper for ggml/OpenVINO integration. Store tensor extras in unique_ptrs owned by the OpenVINO buffer context instead of manually deleting raw pointers at each replacement and during context teardown. This makes tensor->extra a non-owning view while the context remains responsible for lifetime, reducing leak and double-delete risks when extras are replaced or buffers are destroyed. Remove the obsolete raw-pointer tensor extra factory and keep the unique_ptr-returning factory as the only creation API so new call sites cannot accidentally reintroduce ambiguous ownership.
Move the GGML_OPENVINO_RELEASE_WEIGHTS host weight-buffer registry out of ggml-openvino.cpp and into a dedicated ggml-openvino-weight-buffer-release module. Keep ggml-openvino.cpp focused on backend buffer/device glue while the new helper owns registration, release state, and MADV_DONTNEED handling for host weight pages. Include the helper explicitly from the backend and utils call sites instead of exposing these declarations through ggml-openvino-extra.h.
Keep ggml_backend_openvino_buffer_context from caching data, size, and ov_buffer fields that are already owned by ggml_openvino_buffer_storage. Expose small data() and size() accessors on the context so callers continue to read the buffer base and allocation size through the storage owner. This leaves storage as the single source of truth for host and remote buffer state. Also compute the KV-cache buffer offset before replacing the old context during host-to-remote migration, so the offset calculation no longer depends on a pointer after its owning storage has been destroyed.
Remove the unused name field from ggml_backend_openvino_buffer_context. Buffer type and device contexts still keep their names for get_name callbacks, but concrete buffer instances only need device, id, remote state, storage, and tensor extras.
Move the OpenVINO backend buffer context and ggml_backend_buffer_i callbacks out of ggml-openvino.cpp into ggml-openvino-buffer.cpp/.h. Keep ggml-openvino.cpp focused on buffer types, backend/device registration, and high-level OpenVINO backend entry points. Preserve the existing public C ABI for ggml_backend_buffer_is_openvino and ggml_backend_openvino_buffer_get_ctx_id by defining them with GGML_BACKEND_API from the new buffer implementation file. This avoids hidden or C++-mangled symbols when ggml-openvino is built as a shared backend. Fold the host/remote buffer storage helpers into the new buffer implementation file because they are only used by the concrete buffer context. This removes the separate buffer-storage files while keeping host aligned allocation and GPU USM remote allocation behavior unchanged. Add Doxygen-style descriptions for the new internal buffer helpers and the host weight-buffer release helpers.
Add Doxygen-style descriptions for the public OpenVINO backend API declarations, including backend initialization, buffer type queries, device count, and registry access.
Avoid rebuilding the host buffer type name through a mutable static string in get_name. Store the _HOST-suffixed name in the per-device buffer type context so returned name pointers have stable storage and do not depend on shared mutable state.
Add explicit host/device metadata to OpenVINO buffer type contexts and use it when querying OpenVINO buffer type kinds. This avoids identifying buffer types by comparing get_name callback pointers while still checking that the buffer type belongs to the OpenVINO registry before reading its context.
Move the GGML_OPENVINO_STATEFUL_EXECUTION environment check into a shared ggml_openvino_is_stateful_enabled helper. This keeps the backend runtime context and buffer initialization paths aligned on the same stateful execution policy.
Move GGML_UNUSED markers ahead of their return statements so they are reachable and keep the intent clear to readers and compilers.
Factor the duplicated device and host buffer type initialization loops into a shared helper with separate static state for each buffer type kind. This keeps returned buffer type pointers and context storage stable while reducing repeated setup logic.
Remove OpenVINO runtime, quantization, and C library includes from ggml-openvino.cpp that are no longer needed after moving the concrete buffer implementation into its own source file.
Document that the OpenVINO backend currently exposes one logical ggml device and selects the actual OpenVINO plugin/device through configuration.
Clarify that OpenVINO registry and device contexts are intentionally owned by the process-lifetime backend registry singleton.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Additional information
Requirements