Add operator benchmark utility scripts

scripts/README.md

@@ -1,14 +1,15 @@
 # Testing Scripts
 
-Unified test execution scripts for InfiniMetrics.
+Utilities for running InfiniMetrics tests, generating operator inputs, and
+aggregating benchmark results.
 
 ## Quick Start
 
 ```bash
-# Run tests with input file(s)
+# Run one input file
 ./scripts/run_tests.sh test.json
 
-# Run tests in a directory
+# Run every input in a directory
 ./scripts/run_tests.sh test_dir/
 
 # Run multiple input files
@@ -17,99 +18,148 @@ Unified test execution scripts for InfiniMetrics.
 
 ## Structure
 
-```
+```text
 scripts/
-├── run_tests.sh               # Unified test execution script
-└── common/                    # Shared utilities
-    ├── install_deps.sh        # Dependency management (check + install)
-    └── prepare_env.sh         # Environment preparation functions
+|-- run_tests.sh
+|-- generate_operator_inputs.py
+|-- aggregate_results.py
+`-- common/
+    |-- install_deps.sh
+    `-- prepare_env.sh
 ```
 
-## Script Organization
-
-### Main Script: `run_tests.sh`
+## Test Runner
 
-Unified test execution script with automatic dependency management.
+`run_tests.sh` is the unified entry point with optional dependency checks.
 
-**Usage (Direct Execution):**
 ```bash
 ./scripts/run_tests.sh [OPTIONS] <input_paths...>
 ```
 
-**Usage (Source Mode - Environment Variables Persist):**
+Options:
+
+- `--check <types>` checks comma-separated dependency groups: `hardware`,
+  `operator`, or `all`.
+- `--no-check` skips dependency checks.
+- `--help`, `-h` prints help.
+
+The script can also be sourced when environment changes must remain in the
+current shell:
+
 ```bash
 source scripts/run_tests.sh
-run_tests [OPTIONS] <input_paths...>
+run_tests --check operator test.json
 ```
 
-**Options:**
-```bash
---check <types>   Check specific dependencies before running (comma-separated)
-                   Types: hardware, operator, all
---no-check        Skip dependency checking
---help, -h        Show help message
-```
+## InfiniOps Adapter
 
-**Input paths:**
-- Can be JSON files or directories
+Operator testcases whose framework segment is `InfiniOps` are dispatched to
+the InfiniOps adapter. Cambricon and Ascend input configurations generated by
+this directory use that framework automatically.
 
-**Examples:**
-```bash
-# Direct execution (recommended for CI/automation)
-./scripts/run_tests.sh test.json
-./scripts/run_tests.sh test_dir/
-./scripts/run_tests.sh test1.json test2.json
-./scripts/run_tests.sh --check hardware test.json
+Runtime prerequisites depend on the selected device:
 
-# Source mode (recommended for development)
-source scripts/run_tests.sh
-run_tests test.json
-run_tests --check all test.json
-```
+- InfiniOps Python bindings (`infini.ops`)
+- PyTorch
+- `torch_mlu` for Cambricon MLU
+- `torch_npu` for Ascend NPU
 
-### Common Functions (`common/`)
+The adapter benchmarks the first implementation registered by InfiniOps,
+including an ATen fallback when the runtime reports one. If the installed
+InfiniOps build has no implementation for an operator and device, the test
+fails with an actionable error instead of invoking an unregistered slot. It
+reports latency, tensor accuracy, estimated TFLOPS, and estimated memory bandwidth.
+An accuracy mismatch produces a nonzero `result_code`, so failed correctness
+checks are also counted as failed runs by the result aggregator.
 
-**`install_deps.sh`**: Unified dependency management (check + install)
+## Operator Input Generator
 
-Can be used standalone or sourced by other scripts.
+`generate_operator_inputs.py` creates InfiniMetrics-compatible JSON files and
+NumPy tensor data for small, medium, and large shape groups.
 
-**Standalone usage:**
 ```bash
-# Install specific component
-export INFINICORE_PATH="/path/to/InfiniCore"
-source scripts/common/install_deps.sh operator   # Install InfiniCore
-source scripts/common/install_deps.sh hardware   # Build CUDA benchmark
-source scripts/common/install_deps.sh all        # Install everything
+python scripts/generate_operator_inputs.py [OPTIONS]
 ```
 
-**Components:**
-- `operator` - InfiniCore (operator testing)
-- `hardware` - CUDA memory benchmark (hardware testing)
+| Option | Default | Description |
+|---|---|---|
+| `--output`, `-o` | `./operator_test_inputs` | Output directory |
+| `--operators` | `matmul add sub mul div` | Operators to generate |
+| `--dtypes` | `float16 float32 bfloat16` | Tensor data types |
+| `--scales` | `small medium large` | Shape groups |
+| `--device` | `nvidia` | Target platform |
+| `--seed` | `42` | Random seed |
+| `--warmup` | `10` | Non-negative warmup iterations |
+| `--measured` | `100` | Positive measured iterations |
+| `--dry-run` | disabled | Print the plan without writing files |
 
-**Checking functions** (available when sourced):
-- `check_cuda` - Check NVIDIA CUDA toolkit
-- `check_infinicore` - Check InfiniCore package
+Examples:
 
-**Installation functions** (available when sourced):
-- `install_infinicore` - Install InfiniCore from source
-- `install_hardware` - Build CUDA memory benchmark
+```bash
+# Preview Cambricon testcases without writing data
+python scripts/generate_operator_inputs.py \
+  --device cambricon --operators matmul add --dry-run
+
+# Generate selected operators and data types
+python scripts/generate_operator_inputs.py \
+  --output ./test_inputs \
+  --device cambricon \
+  --operators matmul add sub mul div \
+  --dtypes float16 float32
+
+# Run the generated configurations
+python main.py ./test_inputs/configs/
+```
+
+Generated layout:
+
+```text
+operator_test_inputs/
+|-- configs/
+|   `-- opbench.<operator>.<scale>.<shape>.<dtype>.json
+|-- data/
+|   `-- <operator>_<scale>_<shape>_<dtype>_<input>.npy
+|-- all_test_inputs.json
+`-- _generation_metadata.json
+```
+
+Supported generator operators are `matmul`, `mm`, `add`, `sub`, `mul`, `div`,
+and `linear`. The adapter also accepts InfiniOps configurations for the common
+`cast`, `cat`, and `gemm` operators.
 
-**`prepare_env.sh`**: Environment preparation functions
-- `log_test_start` - Log test start message with timestamp
-- `log_test_end` - Log test completion with exit code
-- `cleanup_on_error` - Error trap handler
-- `get_timestamp` - Get current timestamp
+## Result Aggregator
 
-## Output
+`aggregate_results.py` discovers `*_results.json` files recursively and builds
+pass/fail and metric summaries by operator, scale, and dtype.
 
-All test results are saved to:
+```bash
+python scripts/aggregate_results.py [OPTIONS]
 ```
-output/
+
+| Option | Default | Description |
+|---|---|---|
+| `--input`, `-i` | `./output` | Result directory |
+| `--output`, `-o` | `./aggregated_results.json` | Summary JSON path |
+| `--print` | disabled | Print a console summary |
+| `--filter-operator` | none | Include one operator |
+| `--filter-scale` | none | Include one scale |
+| `--filter-dtype` | none | Include one dtype |
+
+Examples:
+
+```bash
+python scripts/aggregate_results.py --print
+python scripts/aggregate_results.py -i ./output -o ./summary.json --print
+python scripts/aggregate_results.py --filter-operator matmul --print
+python scripts/aggregate_results.py \
+  --filter-scale large --filter-dtype float16 --print
 ```
 
-## Requirements
+Malformed JSON files are skipped with a warning. Metrics with nonnumeric values
+remain in detailed records but are excluded from min/max/average calculations.
+
+## Common Helpers
 
-- Python 3.10+
-- Bash 4.0+
-- CUDA toolkit (for CUDA hardware tests)
-- InfiniCore source (for operator tests)
+`common/install_deps.sh` provides dependency checks and installers for operator
+and hardware tests. `common/prepare_env.sh` provides test logging, timestamps,
+and cleanup helpers used by the runner.