diff --git a/docs/onnx-pack.md b/docs/onnx-pack.md index 9cb8b645a..c8502387b 100644 --- a/docs/onnx-pack.md +++ b/docs/onnx-pack.md @@ -114,6 +114,25 @@ model. - **No vendor operators, ever.** If a shape has no kernel it goes to the CPU EP. That keeps the failure mode boring: a slower session, never a session that refuses to load because a custom domain is missing. +- **…and the vendor's own components are the exception that proves it** (engine#2000). The bullet above + is about what *our* export emits. A 1.9 pack also carries the vendor's leftover graphs, and those are + written for their execution provider, not for ours: `Qwen3.5_4B_rai_1.9.0_npu`'s + `partitioned_patch1_sdgemm_leftover/vision.onnx` (84.7 MiB, with `.data` and `vision_strategy.yaml`) + has a clean ONNX interface — `pixel_values [num_patches,1536] f32` + `image_grid_thw [1,3] i64` → + `image_features [num_logical_patches,2560] f32` — and still will not load here. Its first strategy pass + is `qwen35_vision_stx.dynamic_pad`, and the graph is built on custom domains + (`com.ryzenai:DynamicPad`, then `matmul_add_to_sd_gemm`, `mha_to_sd_mha`, `conv_to_nhwc_conv`), so the + plain ONNX Runtime EPs reject it outright: + + ``` + Fatal error: com.ryzenai:DynamicPad(-1) is not a registered function/op + ``` + + with inputs built to the declared interface at bucket 1 of the three in `vision_strategy.yaml` — so + the failure is the operator, not the shape. **"The pack contains `vision.onnx`" does not mean there is + a portable graph to compare against.** For those components the compiled `partitioned/` route is the + only path, and there is no CPU-EP oracle for them: a correctness check on the vision tower has to be + built against the pack's own component, not against a reference run of the `.onnx` file. ## Where the numbers live diff --git a/docs/onnx.md b/docs/onnx.md index 48955cc7a..eabff2270 100644 --- a/docs/onnx.md +++ b/docs/onnx.md @@ -463,18 +463,47 @@ silent fall back cannot be mistaken for a device pass. ## Any graph: `ort-run --input` -`ort-run` speaks two interfaces. With `--ids` it is the Qwen3 language model of -[Qwen3-0.6B](#qwen3-06b) below. With `--input` it is **any** ONNX graph: nothing about the graph's -names, shapes or roles is assumed, so a vision model, an audio model and a one-op toy all take the -same path. +`ort-run` speaks two interfaces, and **which binary you are holding decides which one you get** +(engine#2000): + +* **`--ids`** is the Qwen3 language model of [Qwen3-0.6B](#qwen3-06b) below. +* **`--input`** (with `--output`, `--dump-all`) is **any** ONNX graph: nothing about the graph's names, + shapes or roles is assumed, so a vision model, an audio model and a one-op toy all take the same path. + +Both are implemented once, in `onnx/ort_run.cpp`, and that file backs two entry points: + +| entry point | built by | notes | +|---|---|---| +| `1bit ort-run` | `ninja onebit` **with `-DONEBIT_ORT=ON`** | dispatches to the same `onebit_ort_run_main` (`app/main.cpp`), whose parser handles `--input` (`onnx/ort_run.cpp`) | +| `build/onnx/ort_run` | `ninja -C build ort_run` | the standalone binary; **not** part of `ninja onebit` | + +The trap is the guard, not the flags. `app/main.cpp` dispatches `ort-run` inside `#ifdef ONEBIT_ORT`, and +`onnx/` is added with `add_subdirectory(onnx)` under the same option — so in a configure without +`ONEBIT_ORT` there is no standalone binary *and* the subcommand does not exist: + +```sh +$ 1bit ort-run --model m.onnx --input x=x.bin:float32:1x4x4 # ONEBIT_ORT not configured +1bit: unknown command 'ort-run' +``` + +Build the ONNX component before reaching for either form, and note where the standalone binary lands — +`add_executable(ort_run …)` lives in `onnx/CMakeLists.txt` and the root does not override +`CMAKE_RUNTIME_OUTPUT_DIRECTORY`, so a Ninja or Makefiles build puts it in the subdirectory (measured): + +```sh +cmake -B build -DONEBIT_ORT=ON +ninja -C build ort_run # -> build/onnx/ort_run +``` + +Earlier revisions of this page wrote the examples below as `build/ort_run`, which is not that path. ```sh # one declared input, raw little-endian float32, NHWC 1x640x640x3 -build/ort_run --model yolov8m.onnx \ +build/onnx/ort_run --model yolov8m.onnx \ --input DetectionModel::input_0=x.bin:float32:1x640x640x3 \ --dump-all out/ # the same, writing only what you name, and with our plugin EP asked first -build/ort_run --model m.onnx --ep build/ep/libonebit_ep.so \ +build/onnx/ort_run --model m.onnx --ep build/ep/libonebit_ep.so \ --input x=x.bin:float32:1x4x4 --output y=y.bin ```