Skip to content

docs(benchmarks): drop superseded results and development records - #25

Open
Longxmas wants to merge 1 commit into
mainfrom
docs/benchmark-data-cleanup
Open

Longxmas wants to merge 1 commit into
mainfrom
docs/benchmark-data-cleanup

Conversation

@Longxmas

@Longxmas Longxmas commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Keep one current, reproducible result set per model and device. Remove measurements that newer runs replaced, and notes that only recorded how a result was produced. H200 numbers are not kept in the published benchmark index.

Summary

  • Replace the September 7 aggregate in docs/{en,zh}/benchmark.md with a latest-results table (one row per model × device), refresh coverage gaps, and drop the H200 engine-mechanism section and the October 14 prefix GELU note.
  • Remove superseded pi0.5 Thor / RTX 4090 quantization replays and early 1,600-frame rows; keep AGX Orin quant results and add the Orin tuned-kernel latency row.
  • Delete the August 3B-navigation evaluation reports; latency for those models now points at qwenvl-benchmark.
  • Condense ActiveVLN, GR00T, OFT, LingBot, DM0.5, Cosmos, StreamVLN, Qwen and related READMEs to results, conditions and commands (no rejected candidates, local reports/ paths, snapshot tars or checksum lists).
  • Rename VLN device sections by protocol/date; sync VLN and RLinf tables into README.zh-CN.md; fix related anchors in docs/{en,zh}/api.md.

Type of change

  • Bug fix
  • New model adapter or policy-contract change
  • Engine, scheduler, session, or rollout change
  • Kernel, layer, or numerical change
  • Documentation
  • Build, dependency group, or lock-file change

Verification

Docs-only change; no CUDA or weight runs.

$ python scripts/check_commit_policy.py
# passed for d60e49b

$ mkdocs build --strict -f mkdocs.yml   # en
$ mkdocs build --strict -f mkdocs.zh.yml  # zh
# both passed (Windows: temp copy / symlink workaround)

$ git diff --check
# clean
  • Hardware: not applicable (documentation only)
  • Weights: none
  • Parity: not applicable (no numerical re-measurement in this PR; published numbers are the retained latest runs already on the branch)

Engine invariants

  • No model-name branch was added to the engine; capabilities are declared
    through ActionDecoder / RLDecoder and the shared metadata.
  • Parity tests under the affected layers or backend still pass, or the
    intentional difference is documented with numbers.
  • New dependencies are declared in the correct profile group, and
    uv.lock is consistent.
  • No deployment-runtime code was imported across the boundary; integrations
    use the versioned API.

Checklist

  • My PR title and every introduced commit follow the
    commit policy,
    and every author/coauthor has supplied their own DCO Signed-off-by.
  • New models follow the flow in
    CONTRIBUTING.md
    and have a design document under docs/proposals/ (N/A: no new model).
  • Documentation reflects the change, and parity or runtime numbers in
    docs/en/benchmark.md
    are updated when they change.
  • No checkpoints, datasets, benchmark outputs, recordings, credentials,
    addresses, private paths, or machine-specific scripts are included.
  • I agree that this contribution is licensed under Apache-2.0, per
    CONTRIBUTING.md.

Keep one current, reproducible result set per model and device, and
remove the measurements that newer runs replaced and the notes that
only recorded how a result was produced.

- Replace the September 7 aggregate in the benchmark index with a
  latest-results table per model and device, refresh the coverage
  gaps, and drop the H200 engine-mechanism section and the October 14
  prefix GELU note.
- Remove the pi0.5 Thor and RTX 4090 quantization replays and the
  early 1,600-frame rows that the November presets supersede, and add
  the AGX Orin tuned-kernel result.
- Delete the August 3B-navigation evaluation reports; qwenvl-benchmark
  holds the newer latency results for the same models.
- Condense the ActiveVLN, GR00T and other model READMEs to results,
  conditions and commands, without rejected candidates, local report
  paths, snapshot archives or checksum lists.
- Name the VLN device sections by protocol and date, and add the VLN
  and RLinf tables to the Chinese README.

Signed-off-by: longxmas <1185267696@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant