Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions docs/onnx-pack.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,25 @@ model.
- **No vendor operators, ever.** If a shape has no kernel it goes to the CPU EP. That keeps the
failure mode boring: a slower session, never a session that refuses to load because a custom domain
is missing.
- **…and the vendor's own components are the exception that proves it** (engine#2000). The bullet above
is about what *our* export emits. A 1.9 pack also carries the vendor's leftover graphs, and those are
written for their execution provider, not for ours: `Qwen3.5_4B_rai_1.9.0_npu`'s
`partitioned_patch1_sdgemm_leftover/vision.onnx` (84.7 MiB, with `.data` and `vision_strategy.yaml`)
has a clean ONNX interface — `pixel_values [num_patches,1536] f32` + `image_grid_thw [1,3] i64` →
`image_features [num_logical_patches,2560] f32` — and still will not load here. Its first strategy pass
is `qwen35_vision_stx.dynamic_pad`, and the graph is built on custom domains
(`com.ryzenai:DynamicPad`, then `matmul_add_to_sd_gemm`, `mha_to_sd_mha`, `conv_to_nhwc_conv`), so the
plain ONNX Runtime EPs reject it outright:

```
Fatal error: com.ryzenai:DynamicPad(-1) is not a registered function/op
```

with inputs built to the declared interface at bucket 1 of the three in `vision_strategy.yaml` — so
the failure is the operator, not the shape. **"The pack contains `vision.onnx`" does not mean there is
a portable graph to compare against.** For those components the compiled `partitioned/` route is the
only path, and there is no CPU-EP oracle for them: a correctness check on the vision tower has to be
built against the pack's own component, not against a reference run of the `.onnx` file.

## Where the numbers live

Expand Down
41 changes: 35 additions & 6 deletions docs/onnx.md
Original file line number Diff line number Diff line change
Expand Up @@ -463,18 +463,47 @@ silent fall back cannot be mistaken for a device pass.

## Any graph: `ort-run --input`

`ort-run` speaks two interfaces. With `--ids` it is the Qwen3 language model of
[Qwen3-0.6B](#qwen3-06b) below. With `--input` it is **any** ONNX graph: nothing about the graph's
names, shapes or roles is assumed, so a vision model, an audio model and a one-op toy all take the
same path.
`ort-run` speaks two interfaces, and **which binary you are holding decides which one you get**
(engine#2000):

* **`--ids`** is the Qwen3 language model of [Qwen3-0.6B](#qwen3-06b) below.
* **`--input`** (with `--output`, `--dump-all`) is **any** ONNX graph: nothing about the graph's names,
shapes or roles is assumed, so a vision model, an audio model and a one-op toy all take the same path.

Both are implemented once, in `onnx/ort_run.cpp`, and that file backs two entry points:

| entry point | built by | notes |
|---|---|---|
| `1bit ort-run` | `ninja onebit` **with `-DONEBIT_ORT=ON`** | dispatches to the same `onebit_ort_run_main` (`app/main.cpp`), whose parser handles `--input` (`onnx/ort_run.cpp`) |
| `build/onnx/ort_run` | `ninja -C build ort_run` | the standalone binary; **not** part of `ninja onebit` |

The trap is the guard, not the flags. `app/main.cpp` dispatches `ort-run` inside `#ifdef ONEBIT_ORT`, and
`onnx/` is added with `add_subdirectory(onnx)` under the same option — so in a configure without
`ONEBIT_ORT` there is no standalone binary *and* the subcommand does not exist:

```sh
$ 1bit ort-run --model m.onnx --input x=x.bin:float32:1x4x4 # ONEBIT_ORT not configured
1bit: unknown command 'ort-run'
```

Build the ONNX component before reaching for either form, and note where the standalone binary lands —
`add_executable(ort_run …)` lives in `onnx/CMakeLists.txt` and the root does not override
`CMAKE_RUNTIME_OUTPUT_DIRECTORY`, so a Ninja or Makefiles build puts it in the subdirectory (measured):

```sh
cmake -B build -DONEBIT_ORT=ON
ninja -C build ort_run # -> build/onnx/ort_run
```

Earlier revisions of this page wrote the examples below as `build/ort_run`, which is not that path.

```sh
# one declared input, raw little-endian float32, NHWC 1x640x640x3
build/ort_run --model yolov8m.onnx \
build/onnx/ort_run --model yolov8m.onnx \
--input DetectionModel::input_0=x.bin:float32:1x640x640x3 \
--dump-all out/
# the same, writing only what you name, and with our plugin EP asked first
build/ort_run --model m.onnx --ep build/ep/libonebit_ep.so \
build/onnx/ort_run --model m.onnx --ep build/ep/libonebit_ep.so \
--input x=x.bin:float32:1x4x4 --output y=y.bin
```

Expand Down
Loading