Skip to content
CodewithJhaPublic

About

RoboProbe — Intelligent Virtual Robotics Testing & Failure Discovery Platform: AI-driven adaptive scenario selection and automatic failure discovery on top of Gazebo/ROS 2 Nav2

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

110 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

RoboProbe

A virtual testing engineer for autonomous robots — it learns where your robot is weak and probes there.

Intelligent Virtual Robotics Testing & Failure Discovery Platform

License: MIT Status: design phase Python 3.12 ROS 2 Jazzy Docs PRs welcome Contributor Covenant

Documentation · Roadmap · PRD · Architecture · Contributing

Important

RoboProbe is in the design phase. This repository currently contains the complete engineering specification (35 documents + 16 ADRs). There is no runnable code yet — implementation starts with milestone M1. Star/watch the repo to follow progress, and feedback on the design is very welcome.

Table of contents

Why RoboProbe?

Testing autonomous robots on real hardware is slow, expensive and sometimes unsafe, and rare failures are hard to reproduce. Simulators solve the "virtual robot" part — but deciding which scenarios to simulate is still done by hand or at random. Random testing spends most of its budget on ordinary, passing cases, and one-factor-at-a-time test plans miss the most dangerous class of bugs: interaction failures, where a robot handles high speed, sensor noise and low light individually but collides when they occur together.

What it is

RoboProbe is a software-only, open-source testing layer on top of an established robotics simulator (Gazebo Harmonic + ROS 2 Jazzy + Nav2). It is not a simulator and not a robot. It:

  1. generates test scenarios from a declared parameter space,
  2. runs them headless on the simulator with parallel workers,
  3. judges every run with robustness-valued failure oracles,
  4. learns from each outcome to choose the next, more informative scenarios,
  5. clusters failures into explained failure modes, and
  6. maps the safe, uncertain and failure-prone regions of the robot's operating envelope.

Key features (planned)

Feature Spec
🎯 Intelligent test selection — S-PROBE: Gaussian-process Bayesian optimisation on a continuous robustness score, diversity-seeking failure-mode search, level-set boundary learning 07
⚖️ Fair baselines — random, Latin hypercube, Sobol, grid and one-factor-at-a-time under identical budgets and seeds 07 §3
🚨 Failure oracles — collision, timeout, stuck, localisation divergence, planner abort, instability, near-miss/oscillation warnings, each with an STL-style robustness value 08
🧭 Failure-mode analysis — HDBSCAN clustering, SHAP importance & interaction values, human-readable rules, confirmed interaction failures 09
🗺️ Risk atlas — calibrated p_fail maps classifying SAFE / UNCERTAIN / FAILURE regions, robustness score per robot configuration 09 §5
🔁 Reproducibility — hierarchical seeds, provenance manifests, one-click replays, failure → regression-scenario export 21
🧪 Built-in fast simulator — deterministic kinematic2d backend so the whole loop runs in CI and large experiments without ROS 10 §7
📊 Experiment harness — 30-seed comparisons with Mann–Whitney U, Vargha–Delaney Â12, bootstrap CIs, Holm correction 19
📓 "Field Notebook" dashboard — an original UI: live campaign console, casebook of failure modes, risk atlas, run replays 13
🧩 Pluggable by design — simulators, strategies, oracles, analyzers and stores are entry-point plugins 34
💸 100 % free & open source — runs on a student laptop, no paid services 15

How it works

flowchart LR
    SP["Scenario space<br/>speed, clearance, light,<br/>noise, friction, delay"] --> SEL["Intelligent selector<br/>GP surrogate + acquisition"]
    SEL -->|"next scenarios"| GEN["Scenario generator<br/>world + params"]
    GEN --> SIM["Simulation runner<br/>Gazebo or kinematic2d"]
    SIM --> DET["Failure detector<br/>verdict + robustness"]
    DET -->|"learn from result"| SEL
    DET --> ANA["Failure analysis<br/>modes, rules, SHAP"]
    ANA --> RISK["Risk atlas<br/>safe / uncertain / failure"]
    ANA --> UI["Dashboard and report"]
    RISK --> UI
Loading

Each finished run updates a surrogate model of how close to failure the robot is across the scenario space. The selector then picks scenarios that are likely to fail in a new way, sit on the uncertain pass/fail boundary, or refine a freshly found failure — so simulation time is concentrated where it is informative.

Architecture overview

flowchart TB
    WEB["Dashboard SPA<br/>React + TypeScript"] -->|"REST + SSE"| API["API<br/>FastAPI"]
    API --> DB[("PostgreSQL<br/>data + job queue")]
    ORCH["Orchestrator<br/>campaign loop, selection,<br/>analysis, risk"] --> DB
    W1["Workers: kinematic2d"] -->|"claim runs"| DB
    W2["Workers: Gazebo + ROS 2 + Nav2"] -->|"claim runs"| DB
    W1 --> ART[("Artifact store<br/>traces, bags, reports")]
    W2 --> ART
    API --> ART
Loading

Hexagonal core with ports for simulators, storage and plugins; stateless pull-based workers scale horizontally (more processes or more machines) with configuration only. Details: 04 Architecture, ADRs.

Example: the warehouse robot

A TurtleBot 4 must deliver a package across the Nav2 warehouse world while a dropped pallet lies near its path. RoboProbe varies speed, obstacle clearance, illuminance, lidar noise, floor friction and command delay (plus obstacle position and lidar dropout). The robot passes with high speed alone, noise alone and low light alone — but RoboProbe is designed to discover that high speed + low light + sensor noise makes the pallet visible too late and ends in a collision, then confirm it is a genuine interaction, report the rule (speed > 0.38 m/s ∧ light < 32 lux ∧ noise > 0.04 m ⇒ collision, illustrative), the minimal reproducing scenario, a replay, and the region of the envelope where the risk lives. See 06 for the exact parameter space.

Tech stack

Layer Choice
Language Python 3.12 (core), TypeScript (web)
Simulator Gazebo Harmonic + ROS 2 Jazzy + Nav2, TurtleBot 4 (Docker, headless) · built-in kinematic2d
Intelligence PyTorch (CPU), GPyTorch, BoTorch · scikit-learn (HDBSCAN, gradient boosting) · SHAP · SciPy QMC
Backend FastAPI, Pydantic v2, pydantic-settings, SQLAlchemy 2, Alembic, Typer
Data PostgreSQL 17 (store + queue + notifications), Parquet, MCAP
Frontend React 19, Vite, TanStack Query, Radix UI, Plotly.js, CSS Modules with original design tokens
Ops Docker Compose, GitHub Actions, GHCR, structlog, Prometheus (optional)
Quality pytest, Hypothesis, Testcontainers, ruff, mypy, import-linter, Vitest, Playwright, axe-core

Rationale and alternatives for each choice: 15 Tech stack & ADRs.

Project status & roadmap

Milestone Scope Status
M0 Specification (this repo) ✅ done
M1 Foundation: skeleton, settings, logging, CI, compose ⏳ next
M2–M3 Scenario space, kinematic2d, oracles, runner, storage, CLI planned
M4–M6 Baselines, metrics, intelligent selection (S-PROBE), analysis & risk planned
M7–M8 REST/SSE API, Field Notebook dashboard planned
M9 Gazebo + ROS 2 + Nav2 backend planned
M10–M11 Evaluation experiments, v1.0 release planned

Full plan with ~90 agent-sized tasks: 22 Roadmap · risks: 23.

Documentation map

Start at docs/00-index.md for reading paths and conventions.

# Document One-liner
00 Index Map, reading paths, ID conventions, change control
01 Vision & product brief Problem, goals, MVP, success criteria, research question
26 PRD KPIs, MoSCoW requirements, journeys, release criteria, sign-off
02 SRS IEEE-830-style FR/NFR with IDs and acceptance criteria
03 Personas & use cases Personas, user stories, use-case specs
04 System architecture C4 views, layers & ports, scalability path
05 Module design Normative classes, interfaces and signatures for M1–M8
06 Scenario space & schema Parameters, physical models, YAML + JSON Schema
07 Selection algorithms Baselines, GP surrogate, acquisitions, S-PROBE pseudocode
08 Failure taxonomy & detection Failure classes, oracle formulas, verdict engine
09 Failure analysis & risk Clustering, SHAP, rules, interaction test, risk atlas
10 Simulator integration Gazebo/ROS 2/Nav2 adapter, kinematic2d, determinism
11 Data model & DB ER diagram, PostgreSQL DDL, queue SQL, artefacts
12 API specification REST + SSE endpoints, schemas, errors, versioning
13 Dashboard UX/UI Original "Field Notebook" design language, screens, components
14 Workflows & sequences End-to-end sequence and state diagrams
15 Tech stack & ADR summary Every dependency with rationale; ADR index
16 Repo structure & standards Folder tree, layering rules, Python/TS/ROS/git conventions
17 Dev environment & Docker Setup, compose services, images, troubleshooting
18 Testing strategy Test pyramid, contract suites, statistical algorithm tests
19 Evaluation plan Hypotheses, metric formulas, experiments, statistics
20 CI/CD & DevOps GitHub Actions pipelines, release flow (free tier)
21 Security, reproducibility, observability Controls, seed tree, manifests, logs/metrics
22 Roadmap & tasks Milestones M0–M11, task backlog for AI agents
23 Risks Risk register with mitigations
24 Glossary Terms and acronyms
25 References & prior art Bibliography and positioning
27 Configuration reference Every setting/env var; no hard-coded values
28 Traceability matrix PRD → SRS → design → tasks → tests
29 Deployment guide Topologies, install, scaling, backup, rollback
30 Operations runbook Incident procedures and maintenance
31 Threat model STRIDE analysis
32 Engineering process DoR/DoD, PR and code-review rules, RFCs
33 Versioning & release SemVer, schema/semantic versions, migrations
34 Design principles & patterns SOLID, DI, patterns, error handling, logging
ADR Decision records ADR-0001 … ADR-0016
— Templates · RFCs · Evidence ADR/RFC/task/bug templates; proposals; results

Quick start (planned)

Note

Not available yet — these commands describe the intended experience once milestone M1 lands (see 17).

git clone https://github.com/CodewithJha/RoboProbe.git && cd RoboProbe
cp .env.example .env
make setup        # Python 3.12 via uv, web deps via pnpm
make up           # PostgreSQL, API, orchestrator, kinematic2d workers, dashboard
make demo         # runs a small S-PROBE campaign
# open http://localhost:8080

Backend development (available now)

Requires git, make and uv (uv installs Python 3.12 itself).

cp .env.example .env                        # environment template (docs/27)
cp roboprobe.example.toml roboprobe.toml    # optional config file template (docs/27 §5)
make setup        # uv sync --locked --all-extras in backend/ + pre-commit hooks
make lint         # ruff check + ruff format --check
make typecheck    # mypy --strict on backend/src
make test         # pytest (excludes `sim` and `slow` markers); `make test-fast` runs in parallel
make help         # list every target

Repository structure

Current (design phase):

RoboProbe/
├── docs/                 # 35 specification documents, adr/, templates/, rfc/, evidence/
├── .github/              # issue & PR templates, CODEOWNERS
├── AGENTS.md             # instructions for AI coding agents
├── README.md  CONTRIBUTING.md  CODE_OF_CONDUCT.md  SECURITY.md  SUPPORT.md  GOVERNANCE.md
├── CHANGELOG.md  CITATION.cff  LICENSE  .editorconfig  .gitignore

Planned (from M1): backend/ (Python package roboprobe), ros_ws/ (ROS 2 package), web/ (dashboard), docker/, configs/, schemas/, tasks/, scripts/, .github/workflows/ — full tree in 16 §1.

Contributing

Contributions are welcome — right now especially design reviews of the documents. Please read CONTRIBUTING.md (workflow, conventions, PR process, good first issues). AI coding agents: read AGENTS.md. Questions: SUPPORT.md · Governance: GOVERNANCE.md.

Code of conduct

This project follows the Contributor Covenant 2.1. By participating, you agree to uphold it.

Security

Please report vulnerabilities privately — see SECURITY.md.

License

MIT © 2026 Priyanshu Jha

Citation

If RoboProbe or its design helps your work, please cite it (metadata in CITATION.cff):

@software{jha_roboprobe_2026,
  author  = {Jha, Priyanshu},
  title   = {RoboProbe: Intelligent Virtual Robotics Testing and Failure Discovery Platform},
  year    = {2026},
  url     = {https://github.com/CodewithJha/RoboProbe},
  license = {MIT}
}

Acknowledgements & prior art

RoboProbe builds on ideas from falsification and scenario-based testing (S-TaLiRo, Breach, Scenic & VerifAI), adaptive stress testing, search-based testing of autonomous systems, and Bayesian optimisation / active learning (BoTorch, LogEI, level-set estimation), and on the open-source robotics ecosystem — ROS 2, Nav2 and Gazebo. Full bibliography and positioning: 25 References & prior art.

Author

Priyanshu Jha — @CodewithJha

About

RoboProbe — Intelligent Virtual Robotics Testing & Failure Discovery Platform: AI-driven adaptive scenario selection and automatic failure discovery on top of Gazebo/ROS 2 Nav2

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages