Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs/onnx-pack.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,12 @@ execution provider, with weights pre-quantised into that provider's own layout.
opaque: the pack decides the quantisation, the kernel, the tiling and the schedule, and the operator
that runs on the device is a black box the vendor compiles.

This holds for every `*.onnx` in a pack, not only the transformer. A component whose inputs and outputs
are plain tensors can still carry custom-domain nodes. For example, a 1.9 pack's `vision.onnx` holds
`com.ryzenai:DynamicPad`, and its strategy file lists more rewrites of the same kind. Under the plain
ONNX Runtime EPs it fails at session create, so "the pack ships it as ONNX" does not mean "`ort-run`
can run it" (engine#2000).

We do not execute those operators. Three concrete reasons, in the order they bind:

1. **Licence and footprint.** The VitisAI EP arrives in the AMD wheel, which the contract forbids us
Expand Down
31 changes: 26 additions & 5 deletions docs/onnx.md
Original file line number Diff line number Diff line change
Expand Up @@ -484,17 +484,38 @@ build/ort_run --model m.onnx --ep build/ep/libonebit_ep.so \
| `--output NAME=FILE` | repeatable; write that output to that path |
| `--dump-all DIR` | write every output as `DIR/<name>.bin` plus `DIR/outputs.txt`, one line per tensor with its dtype, shape, byte count and path |

**Two entry points, one parser.** `1bit ort-run …` and the standalone `build/ort_run …` are the same
function (`onebit_ort_run_main` in `onnx/ort_run.cpp`; `app/main.cpp` hands it the subcommand's
arguments unchanged), so they accept the same flags, `--input`/`--output`/`--dump-all` included
(since #1406). They differ only in how they are built:

- `1bit ort-run` exists only in a build configured with `-DONEBIT_ORT=ON`. Without that, `1bit` says
`1bit: unknown command 'ort-run'`.
- the standalone binary is its own target. `--target onebit` does not build it, so run
`ninja -C build ort_run`.

If `1bit ort-run` answers `onebit-ort-run: unknown argument --input`, that `1bit` is older than #1406.
Rebuild it with `--target onebit` (the target is `onebit`, while `1bit` is only its output file name:
engine#1993) and check the binary's mtime against the checkout.

Both kinds of graph keep the same EP ordering story: `--ep` libraries are registered in the order
given and the CPU EP is always appended last, so our EP is asked first and whatever it refuses falls
through. A graph the EP cannot help is still served — and the EP's `ONEBIT_EP_TRACE=1` log says how
many nodes it was offered and how many it claimed, which is the difference between "the plugin was
loaded" and "the plugin did work".

Two model-file traps are worth knowing before blaming the runner: a graph exported by python `onnx`
1.23 defaults to **IR version 14** and ONNX Runtime 1.23.2 refuses anything above **11**
(`Unsupported model IR version: 14, max supported IR version: 11`), and a graph with any `MatMul`
whose `K`/`N` no NPU ELF covers simply falls through to the CPU EP — which is the designed behaviour,
not an error.
Three model-file traps are worth knowing before blaming the runner:

- A graph exported by python `onnx` 1.23 defaults to **IR version 14**, and ONNX Runtime 1.23.2
refuses anything above **11** (`Unsupported model IR version: 14, max supported IR version: 11`).
- A graph with any `MatMul` whose `K`/`N` no NPU ELF covers simply falls through to the CPU EP. That
is the designed behaviour, not an error.
- **A Ryzen AI pack's `*.onnx` component is not a plain graph, even when its interface is.** For
example, `vision.onnx` in `Qwen3.5_4B_rai_1.9.0_npu` (inputs `pixel_values` f32 and
`image_grid_thw` i64) fails at session create with
`com.ryzenai:DynamicPad(-1) is not a registered function/op`. Its strategy passes rewrite it into
the vendor's custom op domain ([onnx-pack.md](onnx-pack.md)). No plain EP loads it, so such a
component has no CPU-EP oracle here (engine#2000).

`onnx/tests/ort_run_generic.sh` (ctest `ort_run_generic`, registered when python `onnx` is present)
writes a two-node `MatMul`+`Relu` model, runs this path on it, compares the tensor it writes with
Expand Down
Loading