diff --git a/docs/issue-821-before-and-after-report.md b/docs/issue-821-before-and-after-report.md new file mode 100644 index 0000000..f1e3d80 --- /dev/null +++ b/docs/issue-821-before-and-after-report.md @@ -0,0 +1,36 @@ +# Soroban Before-and-After Report (#821) + +## Problem +After an optimization pass runs against a Soroban contract, developers have no +consolidated view of what actually changed. They must manually diff findings +and resource metrics between the original and optimized source, which is slow +and error-prone. + +## Design +Add a comparison reporter that consumes two analysis runs (pre- and +post-optimization) produced by `packages/rules/soroban` (see +`packages/rules/soroban/src/analyzer/wasm-inspector.ts` for the existing +finding shape) and the resource metrics already emitted by +`packages/benchmark`. + +Proposed layout: +- `packages/reporting/soroban/comparison/` — report builder and formatters + (JSON + Markdown output, mirroring the style of `docs/RULE_TESTING_FRAMEWORK.md`). +- `packages/benchmark/soroban/` — resource snapshot capture used as report + input (CPU instructions, storage read/write bytes). + +The builder diffs two `Finding[]` arrays keyed by rule id + location to +classify each as resolved, new, or unchanged, and diffs the resource +snapshots numerically (absolute + percentage change). + +## Report Sections +1. Changed findings (resolved / newly introduced) +2. Resource differences (instructions, storage I/O, before vs. after) +3. Applied optimizations (rule id, description, location) +4. Remaining issues (findings still present after optimization) + +## Acceptance Criteria +- [ ] Comparison report implemented in `packages/reporting/soroban/comparison/` +- [ ] Resource changes displayed with before/after/delta values +- [ ] Applied optimizations listed with rule id and source location +- [ ] Remaining findings included, grouped by severity diff --git a/docs/issue-822-optimization-quality-gate.md b/docs/issue-822-optimization-quality-gate.md new file mode 100644 index 0000000..19b4853 --- /dev/null +++ b/docs/issue-822-optimization-quality-gate.md @@ -0,0 +1,41 @@ +# Soroban Optimization Quality Gate (#822) + +## Problem +An automated optimization can reduce one resource metric (e.g. instruction +count) while silently regressing another, breaking compilation, or +reintroducing a security finding. Reducing a single metric is not sufficient +grounds to accept a transformation. + +## Design +Introduce a quality gate that runs after `packages/autofix` produces a +candidate transformation and before it is applied, consuming the comparison +data described in `docs/issue-821-before-and-after-report.md` plus the +compilation/test results from #823 and #824. + +Proposed layout: +- `packages/quality-gates/soroban/` — gate orchestration and checks. +- Reuses `packages/autofix` transformation output and `packages/benchmark` + resource snapshots as inputs; does not duplicate their logic. + +## Checks (each returns pass/fail with a reason) +1. **Resource improvement** — net resource delta from the before/after + report must be non-negative (no metric regresses beyond a configurable + tolerance). +2. **Analysis regressions** — no `rules/stellar/optimization` or + `rules/security` findings present after the change that were absent + before. +3. **Compilation status** — the compilation validator (#823) result must be + success. +4. **Security findings** — no new/increased-severity findings from + `rules/security/*`. + +## Result +Gate returns an aggregate pass/fail plus the list of individual check +results, so callers (CLI, CI) can report exactly which check failed. + +## Acceptance Criteria +- [ ] Quality gate implemented in `packages/quality-gates/soroban/` +- [ ] Resource improvements validated against before/after report +- [ ] Regressions detected via rule re-analysis +- [ ] Security findings considered as a blocking check +- [ ] Aggregate pass/fail result generated with per-check reasons diff --git a/docs/issue-823-optimization-compilation-validator.md b/docs/issue-823-optimization-compilation-validator.md new file mode 100644 index 0000000..adcbd43 --- /dev/null +++ b/docs/issue-823-optimization-compilation-validator.md @@ -0,0 +1,36 @@ +# Soroban Optimization Compilation Validator (#823) + +## Problem +Automated source transformations produced by `packages/autofix` can +introduce syntax or type errors in the rewritten Rust contract. Nothing +today confirms an optimized contract still compiles before it is offered +back to the developer. + +## Design +Add a compilation validator invoked by `packages/autofix` immediately after +a candidate transformation is generated, using the existing Rust toolchain +integration point. + +Proposed layout: +- `packages/autofix/validation/compile/` — validator entry point, invoked + per candidate fix. +- `packages/integrations/rust/` — thin wrapper around the Rust compiler + invocation (`cargo check` against the Soroban contract's crate), kept + separate so other validators (e.g. #824's test runner) can reuse it + instead of shelling out independently. + +## Flow +1. Write the transformed source to a temporary build directory alongside + the contract's existing `Cargo.toml`. +2. Invoke the compiler check via `packages/integrations/rust`. +3. Capture stdout/stderr and exit code; parse diagnostics into structured + `{ file, line, column, message, level }` records rather than raw text. +4. If exit code is non-zero, mark the transformation rejected and attach + the diagnostics to the autofix result so `packages/quality-gates/soroban` + (#822) can block it. + +## Acceptance Criteria +- [ ] Compilation validator implemented in `packages/autofix/validation/compile/` +- [ ] Optimized contracts compiled via `packages/integrations/rust` +- [ ] Compiler failures detected and diagnostics preserved in structured form +- [ ] Failed transformations rejected before reaching the quality gate diff --git a/docs/issue-824-optimization-test-runner.md b/docs/issue-824-optimization-test-runner.md new file mode 100644 index 0000000..aa0cf82 --- /dev/null +++ b/docs/issue-824-optimization-test-runner.md @@ -0,0 +1,37 @@ +# Soroban Optimization Test Runner (#824) + +## Problem +Passing compilation (#823) does not guarantee an optimized contract still +behaves correctly. Existing contract tests need to run against the +transformed source to catch behavioral regressions before a fix is accepted. + +## Design +Add a test runner invoked by `packages/autofix` after the compilation +validator succeeds, using the same Rust integration point described in +`docs/issue-823-optimization-compilation-validator.md`. + +Proposed layout: +- `packages/autofix/validation/tests/` — test runner entry point. +- `packages/testing/soroban/` — baseline test result capture and + comparison logic, reusable outside the autofix flow (e.g. manual CLI + runs against `packages/cli`). + +## Flow +1. Detect available tests for the contract crate (`cargo test` targets + declared in the crate's `Cargo.toml`, mirroring how + `packages/integrations/rust` locates the crate for #823). +2. Record a baseline result by running the same tests against the + pre-optimization source first. +3. Execute the same test set against the optimized source via + `packages/integrations/rust`. +4. Capture per-test pass/fail/error and duration; diff against baseline by + test name. +5. Any test that passed in baseline and fails post-optimization is + reported as a regression and fails the run, feeding into the quality + gate (#822). + +## Acceptance Criteria +- [ ] Optimization test runner implemented in `packages/autofix/validation/tests/` +- [ ] Contract tests executed against the optimized source +- [ ] Test failures detected and reported per test +- [ ] Baseline (pre-optimization) comparison supported via `packages/testing/soroban/`