From 6c3e747857a060ed9f08b6067b2cdcfc55d4e896 Mon Sep 17 00:00:00 2001 From: agent Date: Tue, 10 Nov 2026 15:13:38 -0400 Subject: [PATCH] docs: a 1.9 pack's ONNX components are vendor-EP graphs, and ort_run is the onnx/ component (engine#2000) Two doc gaps, both checked against current main rather than taken from the issue. 1. docs/onnx-pack.md: a 1.9 pack's *.onnx component can require vendor op domains (com.ryzenai:DynamicPad, matmul_add_to_sd_gemm, mha_to_sd_mha, conv_to_nhwc_conv) and will not load under the plain ONNX Runtime EPs, so there is no CPU-EP oracle for it -- only the compiled partitioned/ route. Placed as the exception to the existing 'No vendor operators, ever' bullet, which is about OUR exports. 2. docs/onnx.md: the any-graph section now says which binary carries --input, that ort_run lives in the onnx/ component (add_executable in onnx/CMakeLists.txt), that ninja onebit does not build it, and where it lands. Measured here: cmake -B /tmp/bort -DONEBIT_ORT=ON && ninja ort_run -> /tmp/bort/onnx/ort_run, so the page's build/ort_run examples were one directory out and are corrected. Correction to the issue's second claim, which the docs now reflect: '1bit ort-run' does NOT reject --input on current main. app/main.cpp dispatches ort-run to onebit_ort_run_main inside #ifdef ONEBIT_ORT, and onnx/ort_run.cpp handles --input; with ONEBIT_ORT off the subcommand does not exist at all (1bit: unknown command 'ort-run'). So the gotcha is the configure guard, not the flag set. --- docs/onnx-pack.md | 19 +++++++++++++++++++ docs/onnx.md | 41 +++++++++++++++++++++++++++++++++++------ 2 files changed, 54 insertions(+), 6 deletions(-) diff --git a/docs/onnx-pack.md b/docs/onnx-pack.md index 9cb8b645a..c8502387b 100644 --- a/docs/onnx-pack.md +++ b/docs/onnx-pack.md @@ -114,6 +114,25 @@ model. - **No vendor operators, ever.** If a shape has no kernel it goes to the CPU EP. That keeps the failure mode boring: a slower session, never a session that refuses to load because a custom domain is missing. +- **…and the vendor's own components are the exception that proves it** (engine#2000). The bullet above + is about what *our* export emits. A 1.9 pack also carries the vendor's leftover graphs, and those are + written for their execution provider, not for ours: `Qwen3.5_4B_rai_1.9.0_npu`'s + `partitioned_patch1_sdgemm_leftover/vision.onnx` (84.7 MiB, with `.data` and `vision_strategy.yaml`) + has a clean ONNX interface — `pixel_values [num_patches,1536] f32` + `image_grid_thw [1,3] i64` → + `image_features [num_logical_patches,2560] f32` — and still will not load here. Its first strategy pass + is `qwen35_vision_stx.dynamic_pad`, and the graph is built on custom domains + (`com.ryzenai:DynamicPad`, then `matmul_add_to_sd_gemm`, `mha_to_sd_mha`, `conv_to_nhwc_conv`), so the + plain ONNX Runtime EPs reject it outright: + + ``` + Fatal error: com.ryzenai:DynamicPad(-1) is not a registered function/op + ``` + + with inputs built to the declared interface at bucket 1 of the three in `vision_strategy.yaml` — so + the failure is the operator, not the shape. **"The pack contains `vision.onnx`" does not mean there is + a portable graph to compare against.** For those components the compiled `partitioned/` route is the + only path, and there is no CPU-EP oracle for them: a correctness check on the vision tower has to be + built against the pack's own component, not against a reference run of the `.onnx` file. ## Where the numbers live diff --git a/docs/onnx.md b/docs/onnx.md index 48955cc7a..eabff2270 100644 --- a/docs/onnx.md +++ b/docs/onnx.md @@ -463,18 +463,47 @@ silent fall back cannot be mistaken for a device pass. ## Any graph: `ort-run --input` -`ort-run` speaks two interfaces. With `--ids` it is the Qwen3 language model of -[Qwen3-0.6B](#qwen3-06b) below. With `--input` it is **any** ONNX graph: nothing about the graph's -names, shapes or roles is assumed, so a vision model, an audio model and a one-op toy all take the -same path. +`ort-run` speaks two interfaces, and **which binary you are holding decides which one you get** +(engine#2000): + +* **`--ids`** is the Qwen3 language model of [Qwen3-0.6B](#qwen3-06b) below. +* **`--input`** (with `--output`, `--dump-all`) is **any** ONNX graph: nothing about the graph's names, + shapes or roles is assumed, so a vision model, an audio model and a one-op toy all take the same path. + +Both are implemented once, in `onnx/ort_run.cpp`, and that file backs two entry points: + +| entry point | built by | notes | +|---|---|---| +| `1bit ort-run` | `ninja onebit` **with `-DONEBIT_ORT=ON`** | dispatches to the same `onebit_ort_run_main` (`app/main.cpp`), whose parser handles `--input` (`onnx/ort_run.cpp`) | +| `build/onnx/ort_run` | `ninja -C build ort_run` | the standalone binary; **not** part of `ninja onebit` | + +The trap is the guard, not the flags. `app/main.cpp` dispatches `ort-run` inside `#ifdef ONEBIT_ORT`, and +`onnx/` is added with `add_subdirectory(onnx)` under the same option — so in a configure without +`ONEBIT_ORT` there is no standalone binary *and* the subcommand does not exist: + +```sh +$ 1bit ort-run --model m.onnx --input x=x.bin:float32:1x4x4 # ONEBIT_ORT not configured +1bit: unknown command 'ort-run' +``` + +Build the ONNX component before reaching for either form, and note where the standalone binary lands — +`add_executable(ort_run …)` lives in `onnx/CMakeLists.txt` and the root does not override +`CMAKE_RUNTIME_OUTPUT_DIRECTORY`, so a Ninja or Makefiles build puts it in the subdirectory (measured): + +```sh +cmake -B build -DONEBIT_ORT=ON +ninja -C build ort_run # -> build/onnx/ort_run +``` + +Earlier revisions of this page wrote the examples below as `build/ort_run`, which is not that path. ```sh # one declared input, raw little-endian float32, NHWC 1x640x640x3 -build/ort_run --model yolov8m.onnx \ +build/onnx/ort_run --model yolov8m.onnx \ --input DetectionModel::input_0=x.bin:float32:1x640x640x3 \ --dump-all out/ # the same, writing only what you name, and with our plugin EP asked first -build/ort_run --model m.onnx --ep build/ep/libonebit_ep.so \ +build/onnx/ort_run --model m.onnx --ep build/ep/libonebit_ep.so \ --input x=x.bin:float32:1x4x4 --output y=y.bin ```