testbench

A hardware test platform. A microcontroller sends JSON frames over UART to a bridge host (a Raspberry Pi), the bridge publishes them over MQTT, and a Next.js dashboard stores the results, runs tests on demand, raises alerts, and compares firmware versions.

The only contract with the hardware is the frame format, so any UART-capable board works. The repo has reference firmware for five boards (MSP430FR2355, ESP32, STM32F103C8, RP2040, nRF52840 on Zephyr) and a simulator, so the whole pipeline also runs with no hardware attached.

Architecture

Device under test  (any MCU that emits JSON frames over UART)
  │  UART, newline-terminated JSON (default 9600 baud)
  ▼
Bridge host  ──  pi/bridge.py   (Raspberry Pi Zero 2 W, or anything with a serial port)
  │  MQTT publish
  ▼
Mosquitto broker (Docker, port 1883)
  │
  ├─ scripts/mqtt-subscriber.ts  →  validated with src/lib/frames.ts (Zod)  →  PostgreSQL
  └─ /api/metrics/stream (SSE)  →  browser

pi/replay.py records real MQTT traffic to a file and plays it back with the original timing, for demos and regression runs without a device.

Stack

Layer	Technology
Web app	Next.js 16 (TypeScript, App Router), Tailwind CSS v4
Database	PostgreSQL 16 (Drizzle ORM)
Messaging	MQTT (Eclipse Mosquitto)
AI	IBM watsonx.ai (ibm/granite-4-h-small) for failure analysis and the Edward chat panel
Validation and tests	Zod, Vitest

Quick start

Needs Docker Desktop, Node.js 20+, and an IBM watsonx.ai API key and project (the free Lite plan works).

1. Create .env.local:

   DATABASE_URL=postgres://testbench:testbench@localhost:5432/testbench
   MQTT_URL=mqtt://localhost:1883
   WATSONX_API_KEY=<your key>
   WATSONX_PROJECT_ID=<your project id>
   WATSONX_SERVICE_URL=https://us-south.ml.cloud.ibm.com
   
   # Auth (required)
   NEXTAUTH_SECRET=<run: openssl rand -base64 32>
   NEXTAUTH_URL=http://localhost:3000
   
   # OAuth (optional, leave blank to disable those providers)
   GOOGLE_CLIENT_ID=
   GOOGLE_CLIENT_SECRET=
   GITHUB_CLIENT_ID=
   GITHUB_CLIENT_SECRET=
   

2. Start Postgres (5432) and Mosquitto (1883), then migrate:

   docker compose up -d
   npm run db:migrate

3. Run the app and the MQTT-to-database subscriber in two terminals:

   npm run dev
   npm run subscriber

4. Open http://localhost:3000. With no board attached, start the simulator: python3 pi/bridge-sim.py --scenario normal.

Checks: npm run lint, npm run typecheck, npm test. Load test (with the stack running): npm run stress -- --devices 20 --duration 30 runs simulated devices and API traffic together and reports p50/p95/p99 latency.

Connecting a board

Install the bridge dependencies on the host and point it at the device and broker:

pip3 install paho-mqtt pyserial
MQTT_HOST=<broker-ip> SERIAL_PORT=/dev/ttyS0 BAUD_RATE=9600 HARDWARE_ID=device-01 python3 bridge.py

The device sends newline-terminated JSON frames:

Frame (device → bridge)	Purpose
{"type":"heartbeat","hardwareId":"...","firmwareVersion":"..."}	Liveness and version
{"type":"metrics","temperature":..,"voltage":..,"currentMa":..,"gpio":{...}}	Telemetry (all fields optional; gpio is free-form)
{"type":"run_start","runId":"...","hardwareId":"...","firmwareVersion":"..."}	Begin a test run
{"type":"run_step","runId":"...","sequence":N,"name":"...","status":"passed|failed","startedAt":"...","finishedAt":"...","message":...}	One step result
{"type":"run_end","runId":"...","status":"passed|failed","finishedAt":"..."}	End a run

Frame (bridge → device)	Purpose
{"type":"command_run","runId":"...","hardwareId":"..."}	Start the on-device test sequence
{"type":"command_fault","runId":"...","hardwareId":"...","fault":"vdd_droop|gpio_stuck|current_spike"}	Same sequence, with one step's reading forced out of range

hardwareId is any label you choose per board. Fault injection needs no extra wiring: the firmware swaps in an out-of-range value just before the step's normal threshold check, so the failure travels the same path a real one would.

Reference firmware

All five emit identical frames; nothing in the bridge, backend, or dashboard changed between them.

• firmware/main.c: MSP430FR2355 LaunchPad, bare metal with direct register access (eUSCI_A1, P4.2 TX / P4.3 RX)
• firmware/esp32/: ESP32, Arduino core via PlatformIO (Serial2, GPIO16/17)
• firmware/stm32/: STM32F103C8 "Blue Pill", STM32Cube HAL via PlatformIO
• firmware/rp2040/: Raspberry Pi Pico, pico-sdk with CMake
• firmware/zephyr/: nRF52840 DK, Zephyr RTOS built with west

The ESP32, STM32, RP2040, and Zephyr folders each have a README with wiring and build steps. src/lib/frames.crossplatform.test.ts checks that the backend accepts every board's exact frames.

Testing

• Unit tests (Vitest): frame validation, alert thresholds, firmware-version comparison, and the cross-platform frame fixtures. No database or broker needed.
• Lint and typecheck on every push.
• End-to-end in CI (hardware-validation.yml): starts Postgres and Mosquitto, boots the app and subscriber, runs the simulator as the device, triggers a run through /api/webhook/run the way a CI job would, and checks that every step passes. Logs are uploaded as artifacts on failure. A second job runs the same flow against real hardware when a TESTBENCH_URL secret is set.

Design notes

• The frame format is the hardware interface. A new board only needs firmware. The trade-off: bad frames are only caught at the JSON parse, so the bridge treats every line as untrusted, and the subscriber validates every payload with Zod before it reaches the database.
• The bridge sets timestamps, not the device. Boards report uptime ("t+15000ms") because they have no trustworthy clock. I found this bug when testing the STM32 and RP2040 ports against real firmware output; the simulator had always sent proper timestamps, so it never showed up there.
• Runs are dispatched by long-polling. /api/webhook/run publishes a command and polls the database until the run finishes, so CI can trigger a run with one curl, at the cost of holding a connection open for up to five minutes.
• Firmware versions are compared on telemetry, not just pass/fail. The Firmware Versions page flags a new version whose average voltage, temperature, or current shifts past a threshold compared with the previous version, so a board that still passes but runs hotter gets noticed.
• AI is optional. If watsonx is down or unconfigured, runs, telemetry, and alerts keep working.

Deploying

The Railway deployment runs four services matching docker-compose.yml: Postgres, Mosquitto, the app, and the subscriber. The Mosquitto image in mosquitto/ locks the broker to one user when MQTT_USERNAME and MQTT_PASSWORD are set, and allows anonymous access otherwise (local dev). Set the same variables, plus MQTT_HOST/MQTT_URL, on the app, the subscriber, and any bridge host.

Reference

Topic	Direction	Payload
testbench/command/run	server → bridge	{ runId, hardwareId? }
testbench/command/fault	server → bridge	{ runId, hardwareId?, fault }
testbench/metrics	bridge → broker	{ runId?, temperature, voltage, currentMa, gpioStates }
testbench/run/status	bridge → broker	{ runId, status, finishedAt? }
testbench/run/step	bridge → broker	{ runId, sequence, name, status, startedAt, ... }
testbench/heartbeat	bridge → broker	{ hardwareId, timestamp }

Method	Route	Description
GET / POST	/api/test-runs	List runs / create one (pass fault for fault injection)
GET / PATCH	/api/test-runs/[id]	Get a run / update its status
GET / POST	/api/test-runs/[id]/steps	List steps / add one
GET / POST	/api/metrics	Historical metrics / insert a reading
GET	/api/metrics/stream	Live SSE feed
GET / POST	/api/alerts	List alerts / create one
PATCH	/api/alerts/[id]	Acknowledge an alert
POST	/api/analysis	watsonx analysis of a failed run
POST	/api/chat	Streaming chat with Edward (watsonx)

Edward

The chat panel in the corner of the dashboard. Edward started as a desktop assistant I built at the IBM Bob hackathon; here he's a browser chat that knows the platform's runs, telemetry, and thresholds. He uses the same watsonx credentials as the analysis feature.

Next up

• Generate the simulator, bridge, and Zod schemas from one source instead of keeping them in sync by hand
• Tests for the API routes
• An ESP32 variant that publishes straight to MQTT over Wi-Fi
• A recorded demo of a full run