A virtual testing engineer for autonomous robots — it learns where your robot is weak and probes there.
Intelligent Virtual Robotics Testing & Failure Discovery Platform
Documentation · Roadmap · PRD · Architecture · Contributing
Important
RoboProbe is in the design phase. This repository currently contains the complete engineering specification (35 documents + 16 ADRs). There is no runnable code yet — implementation starts with milestone M1. Star/watch the repo to follow progress, and feedback on the design is very welcome.
- Why RoboProbe?
- What it is
- Key features (planned)
- How it works
- Architecture overview
- Example: the warehouse robot
- Tech stack
- Project status & roadmap
- Documentation map
- Quick start (planned)
- Repository structure
- Contributing · Code of conduct · Security · License
- Citation · Acknowledgements & prior art · Author
Testing autonomous robots on real hardware is slow, expensive and sometimes unsafe, and rare failures are hard to reproduce. Simulators solve the "virtual robot" part — but deciding which scenarios to simulate is still done by hand or at random. Random testing spends most of its budget on ordinary, passing cases, and one-factor-at-a-time test plans miss the most dangerous class of bugs: interaction failures, where a robot handles high speed, sensor noise and low light individually but collides when they occur together.
RoboProbe is a software-only, open-source testing layer on top of an established robotics simulator (Gazebo Harmonic + ROS 2 Jazzy + Nav2). It is not a simulator and not a robot. It:
- generates test scenarios from a declared parameter space,
- runs them headless on the simulator with parallel workers,
- judges every run with robustness-valued failure oracles,
- learns from each outcome to choose the next, more informative scenarios,
- clusters failures into explained failure modes, and
- maps the safe, uncertain and failure-prone regions of the robot's operating envelope.
| Feature | Spec | |
|---|---|---|
| 🎯 | Intelligent test selection — S-PROBE: Gaussian-process Bayesian optimisation on a continuous robustness score, diversity-seeking failure-mode search, level-set boundary learning |
07 |
| ⚖️ | Fair baselines — random, Latin hypercube, Sobol, grid and one-factor-at-a-time under identical budgets and seeds | 07 §3 |
| 🚨 | Failure oracles — collision, timeout, stuck, localisation divergence, planner abort, instability, near-miss/oscillation warnings, each with an STL-style robustness value | 08 |
| 🧭 | Failure-mode analysis — HDBSCAN clustering, SHAP importance & interaction values, human-readable rules, confirmed interaction failures | 09 |
| 🗺️ | Risk atlas — calibrated p_fail maps classifying SAFE / UNCERTAIN / FAILURE regions, robustness score per robot configuration |
09 §5 |
| 🔁 | Reproducibility — hierarchical seeds, provenance manifests, one-click replays, failure → regression-scenario export | 21 |
| 🧪 | Built-in fast simulator — deterministic kinematic2d backend so the whole loop runs in CI and large experiments without ROS |
10 §7 |
| 📊 | Experiment harness — 30-seed comparisons with Mann–Whitney U, Vargha–Delaney Â12, bootstrap CIs, Holm correction | 19 |
| 📓 | "Field Notebook" dashboard — an original UI: live campaign console, casebook of failure modes, risk atlas, run replays | 13 |
| 🧩 | Pluggable by design — simulators, strategies, oracles, analyzers and stores are entry-point plugins | 34 |
| 💸 | 100 % free & open source — runs on a student laptop, no paid services | 15 |
flowchart LR
SP["Scenario space<br/>speed, clearance, light,<br/>noise, friction, delay"] --> SEL["Intelligent selector<br/>GP surrogate + acquisition"]
SEL -->|"next scenarios"| GEN["Scenario generator<br/>world + params"]
GEN --> SIM["Simulation runner<br/>Gazebo or kinematic2d"]
SIM --> DET["Failure detector<br/>verdict + robustness"]
DET -->|"learn from result"| SEL
DET --> ANA["Failure analysis<br/>modes, rules, SHAP"]
ANA --> RISK["Risk atlas<br/>safe / uncertain / failure"]
ANA --> UI["Dashboard and report"]
RISK --> UI
Each finished run updates a surrogate model of how close to failure the robot is across the scenario space. The selector then picks scenarios that are likely to fail in a new way, sit on the uncertain pass/fail boundary, or refine a freshly found failure — so simulation time is concentrated where it is informative.
flowchart TB
WEB["Dashboard SPA<br/>React + TypeScript"] -->|"REST + SSE"| API["API<br/>FastAPI"]
API --> DB[("PostgreSQL<br/>data + job queue")]
ORCH["Orchestrator<br/>campaign loop, selection,<br/>analysis, risk"] --> DB
W1["Workers: kinematic2d"] -->|"claim runs"| DB
W2["Workers: Gazebo + ROS 2 + Nav2"] -->|"claim runs"| DB
W1 --> ART[("Artifact store<br/>traces, bags, reports")]
W2 --> ART
API --> ART
Hexagonal core with ports for simulators, storage and plugins; stateless pull-based workers scale horizontally (more processes or more machines) with configuration only. Details: 04 Architecture, ADRs.
A TurtleBot 4 must deliver a package across the Nav2 warehouse world while a dropped pallet lies near its path.
RoboProbe varies speed, obstacle clearance, illuminance, lidar noise, floor friction and command delay
(plus obstacle position and lidar dropout). The robot passes with high speed alone, noise alone and
low light alone — but RoboProbe is designed to discover that high speed + low light + sensor noise makes
the pallet visible too late and ends in a collision, then confirm it is a genuine interaction, report the
rule (speed > 0.38 m/s ∧ light < 32 lux ∧ noise > 0.04 m ⇒ collision, illustrative), the minimal
reproducing scenario, a replay, and the region of the envelope where the risk lives.
See 06 for the exact parameter space.
| Layer | Choice |
|---|---|
| Language | Python 3.12 (core), TypeScript (web) |
| Simulator | Gazebo Harmonic + ROS 2 Jazzy + Nav2, TurtleBot 4 (Docker, headless) · built-in kinematic2d |
| Intelligence | PyTorch (CPU), GPyTorch, BoTorch · scikit-learn (HDBSCAN, gradient boosting) · SHAP · SciPy QMC |
| Backend | FastAPI, Pydantic v2, pydantic-settings, SQLAlchemy 2, Alembic, Typer |
| Data | PostgreSQL 17 (store + queue + notifications), Parquet, MCAP |
| Frontend | React 19, Vite, TanStack Query, Radix UI, Plotly.js, CSS Modules with original design tokens |
| Ops | Docker Compose, GitHub Actions, GHCR, structlog, Prometheus (optional) |
| Quality | pytest, Hypothesis, Testcontainers, ruff, mypy, import-linter, Vitest, Playwright, axe-core |
Rationale and alternatives for each choice: 15 Tech stack & ADRs.
| Milestone | Scope | Status |
|---|---|---|
| M0 | Specification (this repo) | ✅ done |
| M1 | Foundation: skeleton, settings, logging, CI, compose | ⏳ next |
| M2–M3 | Scenario space, kinematic2d, oracles, runner, storage, CLI |
planned |
| M4–M6 | Baselines, metrics, intelligent selection (S-PROBE), analysis & risk |
planned |
| M7–M8 | REST/SSE API, Field Notebook dashboard | planned |
| M9 | Gazebo + ROS 2 + Nav2 backend | planned |
| M10–M11 | Evaluation experiments, v1.0 release | planned |
Full plan with ~90 agent-sized tasks: 22 Roadmap · risks: 23.
Start at docs/00-index.md for reading paths and conventions.
| # | Document | One-liner |
|---|---|---|
| 00 | Index | Map, reading paths, ID conventions, change control |
| 01 | Vision & product brief | Problem, goals, MVP, success criteria, research question |
| 26 | PRD | KPIs, MoSCoW requirements, journeys, release criteria, sign-off |
| 02 | SRS | IEEE-830-style FR/NFR with IDs and acceptance criteria |
| 03 | Personas & use cases | Personas, user stories, use-case specs |
| 04 | System architecture | C4 views, layers & ports, scalability path |
| 05 | Module design | Normative classes, interfaces and signatures for M1–M8 |
| 06 | Scenario space & schema | Parameters, physical models, YAML + JSON Schema |
| 07 | Selection algorithms | Baselines, GP surrogate, acquisitions, S-PROBE pseudocode |
| 08 | Failure taxonomy & detection | Failure classes, oracle formulas, verdict engine |
| 09 | Failure analysis & risk | Clustering, SHAP, rules, interaction test, risk atlas |
| 10 | Simulator integration | Gazebo/ROS 2/Nav2 adapter, kinematic2d, determinism |
| 11 | Data model & DB | ER diagram, PostgreSQL DDL, queue SQL, artefacts |
| 12 | API specification | REST + SSE endpoints, schemas, errors, versioning |
| 13 | Dashboard UX/UI | Original "Field Notebook" design language, screens, components |
| 14 | Workflows & sequences | End-to-end sequence and state diagrams |
| 15 | Tech stack & ADR summary | Every dependency with rationale; ADR index |
| 16 | Repo structure & standards | Folder tree, layering rules, Python/TS/ROS/git conventions |
| 17 | Dev environment & Docker | Setup, compose services, images, troubleshooting |
| 18 | Testing strategy | Test pyramid, contract suites, statistical algorithm tests |
| 19 | Evaluation plan | Hypotheses, metric formulas, experiments, statistics |
| 20 | CI/CD & DevOps | GitHub Actions pipelines, release flow (free tier) |
| 21 | Security, reproducibility, observability | Controls, seed tree, manifests, logs/metrics |
| 22 | Roadmap & tasks | Milestones M0–M11, task backlog for AI agents |
| 23 | Risks | Risk register with mitigations |
| 24 | Glossary | Terms and acronyms |
| 25 | References & prior art | Bibliography and positioning |
| 27 | Configuration reference | Every setting/env var; no hard-coded values |
| 28 | Traceability matrix | PRD → SRS → design → tasks → tests |
| 29 | Deployment guide | Topologies, install, scaling, backup, rollback |
| 30 | Operations runbook | Incident procedures and maintenance |
| 31 | Threat model | STRIDE analysis |
| 32 | Engineering process | DoR/DoD, PR and code-review rules, RFCs |
| 33 | Versioning & release | SemVer, schema/semantic versions, migrations |
| 34 | Design principles & patterns | SOLID, DI, patterns, error handling, logging |
| ADR | Decision records | ADR-0001 … ADR-0016 |
| — | Templates · RFCs · Evidence | ADR/RFC/task/bug templates; proposals; results |
Note
Not available yet — these commands describe the intended experience once milestone M1 lands (see 17).
git clone https://github.com/CodewithJha/RoboProbe.git && cd RoboProbe
cp .env.example .env
make setup # Python 3.12 via uv, web deps via pnpm
make up # PostgreSQL, API, orchestrator, kinematic2d workers, dashboard
make demo # runs a small S-PROBE campaign
# open http://localhost:8080Requires git, make and uv (uv installs Python 3.12 itself).
cp .env.example .env # environment template (docs/27)
cp roboprobe.example.toml roboprobe.toml # optional config file template (docs/27 §5)
make setup # uv sync --locked --all-extras in backend/ + pre-commit hooks
make lint # ruff check + ruff format --check
make typecheck # mypy --strict on backend/src
make test # pytest (excludes `sim` and `slow` markers); `make test-fast` runs in parallel
make help # list every targetCurrent (design phase):
RoboProbe/
├── docs/ # 35 specification documents, adr/, templates/, rfc/, evidence/
├── .github/ # issue & PR templates, CODEOWNERS
├── AGENTS.md # instructions for AI coding agents
├── README.md CONTRIBUTING.md CODE_OF_CONDUCT.md SECURITY.md SUPPORT.md GOVERNANCE.md
├── CHANGELOG.md CITATION.cff LICENSE .editorconfig .gitignore
Planned (from M1): backend/ (Python package roboprobe), ros_ws/ (ROS 2 package), web/ (dashboard),
docker/, configs/, schemas/, tasks/, scripts/, .github/workflows/ — full tree in 16 §1.
Contributions are welcome — right now especially design reviews of the documents. Please read CONTRIBUTING.md (workflow, conventions, PR process, good first issues). AI coding agents: read AGENTS.md. Questions: SUPPORT.md · Governance: GOVERNANCE.md.
This project follows the Contributor Covenant 2.1. By participating, you agree to uphold it.
Please report vulnerabilities privately — see SECURITY.md.
MIT © 2026 Priyanshu Jha
If RoboProbe or its design helps your work, please cite it (metadata in CITATION.cff):
@software{jha_roboprobe_2026,
author = {Jha, Priyanshu},
title = {RoboProbe: Intelligent Virtual Robotics Testing and Failure Discovery Platform},
year = {2026},
url = {https://github.com/CodewithJha/RoboProbe},
license = {MIT}
}RoboProbe builds on ideas from falsification and scenario-based testing (S-TaLiRo, Breach, Scenic & VerifAI), adaptive stress testing, search-based testing of autonomous systems, and Bayesian optimisation / active learning (BoTorch, LogEI, level-set estimation), and on the open-source robotics ecosystem — ROS 2, Nav2 and Gazebo. Full bibliography and positioning: 25 References & prior art.
Priyanshu Jha — @CodewithJha