Document public DexBench result submissions

.gitignore

@@ -5,6 +5,7 @@ __pycache__/
 runs/
 results/*.json
 !results/README.md
+!results/submission.schema.json
 dist/
 build/
 *.egg-info/

README.md

@@ -99,7 +99,7 @@ The replay command checks the task, calibration ID, and hardware metadata agains
 
 ## Results
 
-Publish each run's task, policy and model, success rate, completion time, hardware and calibration metadata, and links to its `results.json` and MCAP trial recordings. Include failed trials and give a reason for each invalid hardware trial. Upload and view recorded MCAP data at [DexData](https://dexdata.roboticscenter.ai); view published benchmark results at [Dexterity Benchmark Results](https://dexterity.roboticscenter.ai/benchmark). See [results publishing](results/README.md).
+Keep every trial in the run's `results.json`, including failures and hardware-invalid trials. Upload MCAPs to [DexData](https://dexdata.roboticscenter.ai), then submit a complete real-robot run to the [Dexterity benchmark leaderboard](https://dexterity.roboticscenter.ai/benchmark). The page accepts standard tasks and custom protocols that follow the submission format, validates the per-trial outcomes, and recalculates the public score. See the [submission instructions](results/README.md) and the [submission JSON Schema](results/submission.schema.json) for the exact required fields and format.
 
 | Recorded example | Label in source run | MCAP |
 | --- | --- | --- |

results/README.md

@@ -1,3 +1,37 @@
-# Publishing results
+# Publish a result
 
-For each run, report the task, policy and model, success rate, completion time, hardware configuration, calibration ID, and links to the group `results.json` and trial MCAPs. Keep all trials, including failures. Mark hardware faults invalid and include the reason. Publish larger recordings as a GitHub release or an archive and link them here.
+The Dexterity leaderboard accepts complete, non-mock DexBench runs. Upload the `results.json` file written by `dexbench run`, provide the hardware and policy metadata below, and link public MCAP trial recordings in DexData. The public site validates the fixed protocol and recalculates the score from the per-trial outcomes.
+
+## Submit on the website
+
+1. Finish the complete task protocol with a real adapter. The standard counts are 20 trials for `keypress-ldr`, 20 for `piano-seq`, and 30 for `pick-place-ab`.
+2. Upload each trial MCAP to [DexData](https://dexdata.roboticscenter.ai) and create a public viewer link covering the run's recordings.
+3. Open the [Dexterity benchmark page](https://dexterity.roboticscenter.ai/benchmark), select the matching task, and upload that run's `results.json`.
+4. Enter the hand manufacturer and model, the arm manufacturer and model when used (required for `pick-place-ab`), and policy type and name. LLM results must include provider and model.
+5. Add the MCAP viewer URL and, optionally, a public URL for `results.json`. Choose a public display name or organization if desired.
+6. Consent to publish the result. Contact email is optional and is kept private. To be contacted about a possible future purchase discount for a top-ranked contribution, provide an email and opt in.
+
+The result appears on the public leaderboard after validation. Published rows are community-reported and are not labeled as independently verified.
+
+## Submission format
+
+The website accepts the `dexbench.submission.v1` JSON envelope defined by [submission.schema.json](submission.schema.json). The form builds this envelope from `results.json` and the fields above. Its top-level fields are:
+
+| Field | Required values |
+| --- | --- |
+| `schema_version` | `dexbench.submission.v1` |
+| `benchmark` | `task_id`, `protocol_version`, and `custom_task` (`null` for the three standard tasks) |
+| `run` | Scoring fields copied from `results.json`: task, mock status, protocol count/completeness, and each trial's `success`, `invalid`, and optional completion time |
+| `hardware` | `hand.manufacturer`, `hand.model`, and `arm` (`null` if not used) |
+| `policy` | `type`, `name`, `provider`, `model`, and `version`; provider and model are required for `llm` |
+| `artifacts` | Public DexData `mcap_viewer_url`; optional HTTPS `results_json_url` |
+| `contributor` | Optional public display name and organization; contact email and explicit incentive opt-in are private |
+| `publish` | Must be `true` to enter the public leaderboard |
+
+For custom tasks, set the benchmark task ID in both `benchmark.task_id` and `run.task`, and supply the task name, protocol version, complete trial count, and HTTPS protocol/specification link in `benchmark.custom_task`. Custom results rank only against the same task ID and protocol version.
+
+The site rejects mock runs, partial protocols, mismatched task IDs or trial counts, malformed trial outcomes, and runs with no valid trials. Success rate is recomputed as successful valid trials divided by valid trials; hardware-invalid trials stay in the uploaded record but are excluded from the denominator. Mean completion time is calculated from valid trials that report a completion time and breaks success-rate ties.
+
+The form reads the complete `results.json` locally, then sends only the scoring fields to the submission service. Calibration IDs, session metadata, endpoint URLs, and local file paths are stripped before upload. The leaderboard stores aggregate scores and public metadata, not the trial records. The public MCAP viewer link and optional public JSON link are displayed to help readers inspect the result; make sure an optional JSON link contains only information you intend to publish.
+
+Top-ranked contributors may receive a discount on a future Robotics Center purchase. The amount and eligibility details are provided directly to eligible contributors; submitting a result does not guarantee a discount.

results/submission.schema.json

@@ -0,0 +1,123 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://github.com/RoboticsCenter/dexbench/blob/main/results/submission.schema.json",
+  "title": "DexBench public leaderboard submission",
+  "description": "Public, community-reported result envelope accepted by the Robotics Center Dexterity leaderboard.",
+  "type": "object",
+  "additionalProperties": false,
+  "required": ["schema_version", "benchmark", "run", "hardware", "policy", "artifacts", "contributor", "publish"],
+  "properties": {
+    "schema_version": { "const": "dexbench.submission.v1" },
+    "benchmark": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["task_id", "protocol_version", "custom_task"],
+      "properties": {
+        "task_id": { "type": "string", "minLength": 1, "maxLength": 80, "pattern": "^[a-zA-Z0-9][a-zA-Z0-9._-]*$" },
+        "protocol_version": { "type": "string", "minLength": 1, "maxLength": 40 },
+        "custom_task": {
+          "oneOf": [
+            { "type": "null" },
+            {
+              "type": "object",
+              "additionalProperties": false,
+              "required": ["id", "name", "protocol_version", "protocol_trials", "spec_url"],
+              "properties": {
+                "id": { "type": "string", "minLength": 1, "maxLength": 80 },
+                "name": { "type": "string", "minLength": 1, "maxLength": 120 },
+                "protocol_version": { "type": "string", "minLength": 1, "maxLength": 40 },
+                "protocol_trials": { "type": "integer", "minimum": 1, "maximum": 500 },
+                "spec_url": { "type": "string", "format": "uri", "pattern": "^https://" }
+              }
+            }
+          ]
+        }
+      }
+    },
+    "run": {
+      "type": "object",
+      "description": "Scoring-only fields copied from the uploaded results.json. Do not include session metadata, serial numbers, calibration IDs, policy endpoints, or file paths.",
+      "additionalProperties": false,
+      "required": ["task", "mock", "protocol_trials", "trial_count", "protocol_complete", "trials"],
+      "properties": {
+        "task": { "type": "string", "minLength": 1 },
+        "mock": { "type": "boolean", "const": false },
+        "protocol_trials": { "type": "integer", "minimum": 1 },
+        "trial_count": { "type": "integer", "minimum": 1 },
+        "protocol_complete": { "type": "boolean", "const": true },
+        "trials": {
+          "type": "array",
+          "minItems": 1,
+          "maxItems": 500,
+          "items": {
+            "type": "object",
+            "additionalProperties": false,
+            "required": ["success", "invalid"],
+            "properties": {
+              "success": { "type": "boolean" },
+              "invalid": { "type": "boolean" },
+              "metrics": {
+                "type": "object",
+                "additionalProperties": false,
+                "properties": { "completion_time_s": { "type": "number", "minimum": 0 } }
+              }
+            }
+          }
+        }
+      }
+    },
+    "hardware": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["hand", "arm"],
+      "properties": {
+        "hand": { "$ref": "#/$defs/device" },
+        "arm": { "oneOf": [{ "type": "null" }, { "$ref": "#/$defs/device" }] }
+      }
+    },
+    "policy": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["type", "name", "provider", "model", "version"],
+      "properties": {
+        "type": { "enum": ["llm", "learned", "scripted", "teleop", "replay", "other"] },
+        "name": { "type": "string", "minLength": 1, "maxLength": 120 },
+        "provider": { "type": ["string", "null"], "maxLength": 100 },
+        "model": { "type": ["string", "null"], "maxLength": 120 },
+        "version": { "type": ["string", "null"], "maxLength": 120 }
+      }
+    },
+    "artifacts": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["mcap_viewer_url", "results_json_url"],
+      "properties": {
+        "mcap_viewer_url": { "type": "string", "format": "uri", "pattern": "^https://dexdata\\.roboticscenter\\.ai(/|$)" },
+        "results_json_url": { "type": ["string", "null"], "format": "uri", "pattern": "^https://" }
+      }
+    },
+    "contributor": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["display_name", "organization", "contact_email", "contact_for_incentive"],
+      "properties": {
+        "display_name": { "type": ["string", "null"], "maxLength": 80 },
+        "organization": { "type": ["string", "null"], "maxLength": 100 },
+        "contact_email": { "type": ["string", "null"], "format": "email" },
+        "contact_for_incentive": { "type": "boolean" }
+      }
+    },
+    "publish": { "const": true }
+  },
+  "$defs": {
+    "device": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["manufacturer", "model"],
+      "properties": {
+        "manufacturer": { "type": "string", "minLength": 1, "maxLength": 80 },
+        "model": { "type": "string", "minLength": 1, "maxLength": 100 }
+      }
+    }
+  }
+}