TypeWhisper R2T2 Plugin

On-device real-time transcription for TypeWhisper with Confucius4-R2T2, served locally by audio.cpp on Metal. No cloud, no API key.

This repository is retired. The plugin is now an official TypeWhisper community add-on: install Confucius4-R2T2 from Settings > Integrations > Discover (add-on page). It downloads audio.cpp and a model and runs the server itself. The source now lives in typewhisper-mac. The rest of this README describes the old setup and is kept for reference.

The Whisper saga

How this plugin came to be, from the first fork to the official add-on:

1. The Fork I Was Happy to Delete: dictation with Ghost Pepper and OpenWispr, and why deleting my fork was the best outcome.
2. Voice Input Revisited: this repository starts. It's a TypeWhisper plugin that streams to Confucius4-R2T2 on audio.cpp, with a server you run yourself.
3. A Team of Strangers: finalize used to re-transcribe the whole utterance. Fixing that took people who had never met: David Feng's port, NairoDorian's Q4_K_M, and 0xShug0 merging it all into audio.cpp (#618, #696, #702).
4. Who Signs for Claude?: audio.cpp v0.9.0 ships the fixes, and the plugin learns to run its own server. The CLA bot asks @claude to sign. Two review loops follow, one with CodeRabbit and one with the maintainer. Then TypeWhisper#1432 is merged and released as plugin-r2t2-v0.2.0.

It started on 2026-09-20 with a hand-run audiocpp_server and a patched plugin. It ended on 2026-10-01, when the official add-on went live and this repository was retired.

• plugin/ — the TypeWhisper transcription engine plugin (Swift). See plugin/README.md for the wire protocol, settings, and measured latency.
• patches/ — one patch that registers the plugin's SwiftPM and Xcode targets in a typewhisper-mac checkout.
• server/ — audio.cpp server config and launcher for the R2T2 GGUF.

Requirements

• macOS 14+, Apple silicon
• TypeWhisper 1.7.0 or newer
• audio.cpp v0.9.0 or newer

Server

Stock audio.cpp v0.9.0 has everything the plugin needs: the live-route prompt parameter (#618), incremental streaming with an instant finalize (#696), K-quant weights and the empty-transcript crash fix (#712). Use the server archive from this repo's release, which bundles that binary with start.sh, or take audiocpp_server from audio.cpp's own audio-v0.9.0-bin-macos-arm64-metal.tar.gz (run chmod +x audiocpp_server after unpacking), or build it:

git clone --recurse-submodules https://github.com/0xShug0/audio.cpp
cd audio.cpp && scripts/build_metal.sh --build-type Release --deployment-build

Pick a GGUF:

Quant	File	Size	Source	Loads in
Q8_0 (recommended)	r2t2-q8_0.gguf	2.5 GB	davidxifeng/Confucius4-R2T2-gguf	any audio.cpp with R2T2
F16	r2t2-f16.gguf	4.1 GB	davidxifeng/Confucius4-R2T2-gguf	any audio.cpp with R2T2
Q4_K_M	r2t2-q4_k_m.gguf	1.2 GB	Nairod785/Confucius4-R2T2-Q4_K_M-GGUF	audio.cpp v0.9.0 or newer

Q4_K_M is a mixed quant, not plain Q4_K. Its streaming transcripts matched Q8_0 on a 14 s and a 60 s English clip, apart from two British spellings, and the 60 s clip streamed in 26 s instead of 39 s on an M-series Mac. audio.cpp builds before v0.9.0 refuse it with R2T2 ASR supports f32/f16/bf16/q8_0 weights, but tensor ... is q4_k.

Make sure the file the server opens ends in .gguf — HuggingFace cache symlinks resolve to extension-less blobs that audio.cpp cannot identify, so hardlink or copy it:

hf download davidxifeng/Confucius4-R2T2-gguf r2t2-q8_0.gguf
ln "$(readlink -f ~/.cache/huggingface/hub/models--davidxifeng--Confucius4-R2T2-gguf/snapshots/*/r2t2-q8_0.gguf)" \
   models/Confucius4-R2T2-GGUF/r2t2-q8_0.gguf

Copy server/server.json.example to server/server.json and point path at that GGUF. start.sh runs the audiocpp_server sitting next to it (as shipped in the release archive); name your own build with AUDIOCPP_SERVER:

AUDIOCPP_SERVER=/path/to/audio.cpp/build/macos-metal-release/bin/audiocpp_server server/start.sh

Smoke test:

tail -c +45 audio.cpp/assets/resources/sample_16k.wav > /tmp/s.pcm
curl -N -X POST -H 'Expect:' -H 'Transfer-Encoding: chunked' -T /tmp/s.pcm \
  'http://127.0.0.1:8488/v1/audio/transcriptions/live?model=r2t2&sample_rate=16000&channels=1&sample_format=s16le'

Plugin

Prebuilt R2T2Plugin.bundle and a server archive are attached to each release. Both are ad-hoc signed, not notarized, so clear the quarantine flag after unzipping:

xattr -dr com.apple.quarantine R2T2Plugin.bundle
cp -R R2T2Plugin.bundle ~/Library/Application\ Support/TypeWhisper/Plugins/

To build it yourself against a typewhisper-mac checkout:

git clone https://github.com/TypeWhisper/typewhisper-mac
cp -R plugin/ typewhisper-mac/TypeWhisperPluginSDK/Plugins/R2T2Plugin/
cd typewhisper-mac && git apply ../patches/0001-register-r2t2-plugin-targets.patch
xcodebuild -skipPackagePluginValidation -project TypeWhisper.xcodeproj -target R2T2Plugin \
  -configuration Release SYMROOT="$(pwd)/build" \
  CODE_SIGN_IDENTITY="-" CODE_SIGNING_REQUIRED=NO CODE_SIGNING_ALLOWED=NO
cp -R build/Release/R2T2Plugin.bundle ~/Library/Application\ Support/TypeWhisper/Plugins/

Restart TypeWhisper, then pick Confucius4-R2T2 as the transcription engine and confirm the server URL (http://127.0.0.1:8488) and model id (r2t2) with Test Connection.

License

GPL-3.0-or-later, matching TypeWhisper, whose plugin SDK this links against.