Voice interface to your codebase, powered by pi, a multi-model AI coding agent. Speak into your phone (or browser), ask about your code, hear the answer, and have it save notes for you.
Product behavior and limitations: product specification. Proposed hands-free mode: continuous-voice research.
A two-part system: an iOS app and a server that runs alongside pi on a machine you control (macOS or Linux, local or remote).
You hold a button and ask about a project: what it does, how a piece works, what changed in the last commit. The server transcribes it with faster-whisper, sends it to pi (via a custom extension bundled in the repo), and returns the answer with audio files for playback. The agent can read files, search, and inspect git history, and it can save notes to a locked folder in the repo, but it does not modify your code. The whole exchange is stored per-session so context accumulates across turns.
Two clients ship with the repo: a native iOS app and a single-file web client served directly by the server at GET /. Both share the same API.
A voice interface is used when you are not looking at the screen: walking, hands busy, phone in a pocket. That single fact changes what a failure is allowed to do: a silent app leaves you waiting on the sidewalk for a reply that never comes, with no way to tell it broke. The intended rule is that a turn produces an audible response when audio is enabled. Supported synthesis and agent failures have fallback paths; network loss, permissions and audio routing still limit what can be guaranteed. The decisions that follow from it were easy to leave implicit until voice forced them, so they are written down as architecture decision records and tested against, not buried in the code:
- Audio is best-effort, never required. The transcript and reply text are the permanent record; spoken audio is a disposable rendering. Synthesis runs a one-way fallback chain (Google Cloud TTS → local
say/espeak/piper→ a pre-recorded "audio unavailable" clip) and flags when it had to fall back. - Failures speak. If pi times out or transcription fails, the turn is still saved and a short spoken notice comes back through that same chain. The technical detail stays in the turn's text; what you hear is a plain sentence.
- Overlapping turns are serialized. Requests on the same chat wait on an in-memory server lock and are processed separately. This is not a durable queue; process loss and delivery recovery remain limitations.
- Audio is ephemeral on purpose. Known audio files are deleted on the next turn in that chat; tracking across server restarts remains incomplete. This is a live conversation, not a podcast archive.
iOS app or web client (GET / → web/index.html)
└── Hold to talk / type a message
└── POST /api/talk → server
├── faster-whisper (local transcription)
├── pi --print --mode json --session <id> --extension pi/tools.ts
└── Google Cloud TTS or macOS say (audio response)
└── Completed audio file URLs returned to client
Sessions map one-to-one to directories under ~/Developer. Each session carries a pi session ID so pi maintains context across turns.
- macOS or Linux (the server host, local or remote)
- pi coding agent in
$PATH(pibinary) - uv for Python dependency management
- Node.js (required by pi to load the TypeScript extension)
- faster-whisper (bundled via uv, no manual install)
- Google Cloud service account with Text-to-Speech API enabled (optional; falls back to a local voice: macOS
say, orespeak/piperon Linux)
cd server
uv sync
uv run uvicorn app:app --host 127.0.0.1 --port 31552Or use the convenience wrapper:
cd server
./run.shRuns on port 31552 by default. Configure with environment variables:
| Variable | Default | Description |
|---|---|---|
VOICE_HOST |
127.0.0.1 |
Bind address |
VOICE_PORT |
31552 |
Port |
VOICE_AUTH_TOKEN |
(empty, auth off) | Optional bearer token required on /api/* when set |
PI_BIN |
pi |
Path to pi binary |
PI_PROVIDER |
litellm |
pi model provider |
PI_MODEL |
small |
pi model alias |
DEVELOPER_DIR |
~/Developer |
Root for project sessions |
WHISPER_MODEL |
base.en |
faster-whisper model size |
TTS_VOICE |
Samantha |
macOS say voice name |
TTS_SPEAKING_RATE |
1.0 |
Default speaking rate (0.5–2.0) |
GOOGLE_TTS_VOICE |
en-US-Chirp3-HD-Schedar |
Google Cloud TTS voice |
GOOGLE_TTS_SERVICE_ACCOUNT_JSON |
(from pass) | Google service account JSON string |
GOOGLE_TTS_SERVICE_ACCOUNT_FILE |
(unset) | Path to the Google service-account JSON file (for Linux boxes with no pass) |
LOCAL_TTS_ENGINE |
(auto) | Force the local engine: say, piper, or espeak (auto-detects when unset) |
VOICE_FORCE_MODEL |
(unset) | Pin the pi model server-side, ignoring the client's request (demo / locked-down servers) |
VOICE_FORCE_TTS |
(unset) | Pin the TTS provider server-side (e.g. google), ignoring the client's request |
VOICE_FORCE_COMMIT_BRANCH |
(unset) | Pin the auto-commit branch server-side, ignoring the client's request |
VOICE_ONDEVICE_PROJECTS |
(unset) | Comma-separated project dirs for which the server returns no audio, so a capable client speaks on-device |
VOICE_VERSION |
(git/BUILD_INFO) | Version string reported at /api/version (stamp a deploy that isn't a git checkout) |
To run as a persistent macOS service, copy and load the included plist:
cp com.synodic.patchbay-voice-server.plist ~/Library/LaunchAgents/
launchctl load ~/Library/LaunchAgents/com.synodic.patchbay-voice-server.plistThe server is also available via synodic-studio/homebrew-synodic:
brew tap synodic-studio/synodic
brew install --HEAD synodic-studio/synodic/patchbay-voice-server
brew services start synodic-studio/synodic/patchbay-voice-serverThe server binds 127.0.0.1 by default. To expose it on your LAN or Tailscale network, set VOICE_HOST (e.g. 0.0.0.0, or a Tailscale 100.x.x.x IP for device-specific access):
VOICE_HOST=0.0.0.0 patchbay-voice start # foreground
launchctl setenv VOICE_HOST 0.0.0.0 && \
brew services restart synodic-studio/synodic/patchbay-voice-server # as a service(Plain VOICE_HOST=… brew services start does not work, because shell environment variables don't propagate into launchd-managed services.) Run patchbay-voice urls to print connection URLs.
Pick one approach. The Homebrew formula and the repo-checkout LaunchAgent plist (above) manage the same service. Choose one; do not load both.
Run patchbay-voice qr on the server to print a setup QR that encodes the server URL and token. Scan it with the iPhone's Camera app (it offers "Open in Patchbay Voice") or in the app via Settings → Scan setup code, and the URL and token are filled in for you. Pass a URL to override the auto-detected one: patchbay-voice qr http://my-host:31552.
Auth is off by default. On a loopback or Tailscale-only bind, the network is the boundary. If you expose the server more widely, set a shared token:
VOICE_AUTH_TOKEN=<token> uv run uvicorn app:app --port 31552 # repo checkout
launchctl setenv VOICE_AUTH_TOKEN <token> && \
brew services restart patchbay-voice-server # brew serviceWhen set, every /api/* request must send Authorization: Bearer <token> or it gets a 401. Paste the same token into the iOS app's Settings (Server → Token) and the web client's Settings (Server → Token). GET / and /audio/* (unguessable UUID filenames) stay open so the web page and audio playback work without headers.
By default pi talks to a cloud model (you provide the key). If you'd rather run with no cloud credentials at all, an opt-in helper points pi at a local Gemma model served by Ollama:
scripts/enable-local-llm.sh # default: gemma4:e2b
scripts/enable-local-llm.sh gemma4:e4b # a larger model
scripts/enable-local-llm.sh --disable # revert to the cloud providerIt installs Ollama if needed, pulls the model, registers an ollama provider in pi's config, and restarts the server with PI_PROVIDER=ollama. Gemma 4 supports tool-calling, so write_file works fully offline (Gemma 3 does not; it is conversational only). This is opt-in and never part of the default install: gemma4:e2b is ~7.2GB, and a small local model is a weaker coding agent than a frontier cloud model. Combined with local faster-whisper and the say/espeak TTS engines, this makes the whole stack runnable with zero external accounts.
The file pi/tools.ts is a pi extension that replaces pi's built-in tools entirely. pi runs with --no-builtin-tools --no-extensions --no-skills, so the agent can touch the project only through the twelve parameterized, injection-safe tools this extension registers:
| Group | Tools |
|---|---|
| Read / search | read_file, grep_search, glob_find, list_dir, tree |
| Git history | git_log, git_show, git_blame, git_diff, git_branch, git_show_file |
| Write | write_file |
Every tool takes explicit arguments and shells out through argv arrays (never a shell string), so metacharacters in a transcribed instruction like &&, ;, or $() are inert. Reads and git inspection are scoped inside the project directory; write_file is constrained to a configured save path (VOICE_SAVE_PATH, default docs/patchbay/) at the tool layer, not just in the prompt. There is no shell tool and no way to write outside that path.
pi loads the extension automatically via --extension on every invocation. It has a regression suite (pi/tools.test.mts, node --test) that pins the full tool set and the current pi API shape. See pi/README.md.
A single-file HTML/JS/CSS app at web/index.html, served by the server at GET /. No build step. Mirrors the iOS app: sessions list, talk screen with hold-to-talk mic, keyboard fallback, and a settings panel with the same voice, model, and file options. Spoken replies use the browser's own on-device voice (Web Speech) or a server provider (macOS say / Google Cloud); like the iOS app, a failed turn is never silent — the client voices a short notice. The conversation is kept in the browser (localStorage) so a refresh doesn't lose it. Works in any modern browser.
Microphone requires a secure context. The hold-to-talk feature uses
getUserMedia, which browsers only allow onhttps://orlocalhost. The server serves plainhttp://<lan-or-tailscale-ip>:31552, so voice input will not work over LAN or Tailscale unless you front the server with HTTPS (e.g.tailscale serve). Text input always works.On-device voice needs a tap first. Mobile Safari only unlocks speech synthesis inside a user gesture, so the browser voice is primed on each mic press / Send — the gesture that starts a turn unlocks the voice before that turn's reply comes back.
Built with SwiftUI, targeting iOS 17+. Managed with Tuist.
cd ios
tuist generate --no-open
tuist xcodebuild build -workspace PatchbayVoice.xcworkspace -scheme PatchbayVoice -destination 'generic/platform=iOS Simulator'Ships to TestFlight via Fastlane:
fastlane bump_build # syncs with TestFlight, increments
fastlane beta # build + upload + add to internal testers- Voice input: hold the mic button, release to send; or switch to keyboard mode
- TTS responses: spoken replies via Google Cloud, macOS
say, or the localespeak/piperengines, toggleable per-session - Speaking rate: adjustable 0.5×–2.0× slider in Settings, applied to both providers
- Sessions: one per repo, persisted across app launches; swipe to reset or delete
- Turn history: stored on the server, fetched on open, and cached locally until reset
- Write directory: server constrains pi to a single save path per project
- Auto-commit / auto-push: optionally commit and push saved notes to a side branch, each toggle enabled only when the repo actually supports it (a Git repo for commit, a remote for push), so neither is ever a silent no-op
- Dark graphite UI: designed around the Patchbay Voice design system (1A Graphite)
- Model switching: any LiteLLM alias, switchable from Settings
Two subtitle-first review videos can be captured and rendered with demo/video. See the product spec and launch-readiness packet for current behavior and public-demo limitations.
scripts/demo.sh narrates a live turn from the terminal: the hardened pi invocation and the twelve tools it is left with, the tool calls streaming out of the server log while the agent works, and the branch it pushed afterward. Ask the question from the phone, or pass --drive to ask it over the API.
cp scripts/demo.env.example scripts/demo.env # gitignored; every value has a default
./scripts/demo.sh # you ask from the phone
./scripts/demo.sh --auto # end to end, no interaction
./scripts/demo.sh --cleanup # delete the demo branch and noteIt reads tool calls from the server's log (~/Library/Logs/patchbay-voice-server.log for both the brew service and the LaunchAgent), so a server started in the foreground needs its stderr redirected there.
./scripts/test.sh # all non-iOS suites (brew wrapper + server + pi)
cd server && uv run pytest # server tests
node --test pi/tools.test.mts # pi extension regression suite
bash brew/tests/discover-urls.test.sh # brew CLI wrapper (survives a failing interface probe)
cd ios && tuist xcodebuild test -workspace PatchbayVoice.xcworkspace -scheme PatchbayVoice -destination 'platform=iOS Simulator,name=iPhone 17 Pro' # choose an installed simulatorscripts/test.sh is the gate: it's run by .githooks/pre-push, which blocks a
push if any suite fails. Because git config isn't cloned, a fresh clone must
re-enable it once with git config --local core.hooksPath .githooks.
The server suite covers the turn state machine, the TTS fallback chain, the FIFO queue, failed-turn persistence, and the auto-commit/push git plumbing (against real throwaway repos). The pi extension has its own suite because it is loaded by pi at runtime, not by Python. It pins the full tool set and the pi API shape so the extension can't silently regress. The design decisions above are backed by the ADRs and tested against them.
MIT. See LICENSE.
Real voice conversations are retained as golden cases and candidate captures in evals/. Run ./scripts/eval.sh --all --calibrate for a live experiment through the production agent path, using LiteLLM small for both candidate and grader. The normal ./scripts/test.sh gate stays offline. Conversation review explains the selected cases and remaining candidates.


