Skip to content
BranchesTags

Repository files navigation

wisper

wisper is an audio transcription tool with two modes: batch file transcription and a real-time transcription service. Powered by NVIDIA NeMo ASR models (Parakeet TDT 0.6B).

Modes

wisper has three distinct modes — pick the one that fits your use case:

Mode Entry point Use when
Real-time service serve.py You want live transcription from a mic, desktop audio, or a network stream, queryable via REST/WebSocket
Batch transcription main.py You have video or audio files to transcribe offline
Single-track + diarization main_single_track.py You need speaker labels in a single-speaker recording

Features

  • Real-time transcription — captures live audio from microphone, desktop (WASAPI loopback), or a network stream (RTP/RTSP/UDP) and transcribes continuously
  • Network stream input — pipe audio from any source ffmpeg can decode (IP cameras, remote mics, broadcast feeds) over RTP, RTSP, or UDP
  • REST + WebSocket API — query the latest word, sentence, last N words, or subscribe to a real-time word stream
  • Push and pull models — poll endpoints for latest transcription, consume words from a queue, or receive words via WebSocket as they're spoken
  • Multi-track extraction — automatically detects and extracts all audio tracks from video files via ffprobe/ffmpeg
  • NVIDIA NeMo ASR — transcription powered by the Parakeet TDT 0.6B model for high-accuracy speech recognition
  • Speaker diarization — optional single-track mode with streaming diarization and multitalker support
  • Flexible CLI — process specific files, entire directories, or audio-only workflows
  • Multiple output formats — JSON with full word/segment/char timestamps, plain text with timestamps (txt), or plain text with no timestamps (plain)
  • Batch processing — process all files in a directory with one command

Requirements

GPU required. wisper uses NVIDIA NeMo ASR models which require a CUDA-capable GPU. It will not run on CPU.

  • Python 3.12 (recommended). Python 3.13 has no prebuilt editdistance wheel yet, so on Windows it must compile from source — see the note below.
  • NVIDIA GPU with CUDA 12.8 support and at least 8GB VRAM (10GB+ recommended)
  • CUDA 12.8 toolkit installed
  • ffmpeg and ffprobe installed and on PATH
  • uv package manager
  • Windows + Python 3.13 only: Microsoft C++ Build Tools are required to compile editdistance from source (no 3.13 wheel exists), and Windows 11 Smart App Control may block that build (ImportError: DLL load failed ... An Application Control policy has blocked this file). Using Python 3.12 avoids both — the editdistance wheel is prebuilt, so no compiler is needed.

Installation

  1. Clone the repository:

    git clone https://github.com/yourusername/wisper.git
cd wisper

  2. Install dependencies (with CUDA 12.8 PyTorch):

    uv sync --extra cu128

    The cu128 extra installs PyTorch built for CUDA 12.8. This is required — there is no CPU fallback.

    Build error with editdistance? If you see error: Microsoft Visual C++ 14.0 or greater is required, install the Microsoft C++ Build Tools (select "Desktop development with C++"), then re-run uv sync --extra cu128.

Quick Start

Real-Time Service

Start the real-time transcription service:

# Transcribe from your microphone
uv run serve.py

# Transcribe desktop audio (Windows WASAPI loopback)
uv run serve.py --source desktop

# Both mic + desktop as separate streams
uv run serve.py --source both

# Both mic + desktop mixed into one stream
uv run serve.py --source both --mix mixed

# Network audio stream (RTP, RTSP, UDP — requires ffmpeg on PATH)
uv run serve.py --source network --stream-url rtp://0.0.0.0:5004
uv run serve.py --stream-url rtsp://192.168.1.10/live   # auto-detects --source=network
uv run serve.py --stream-url udp://0.0.0.0:5005

To send audio to the network source from another machine:

# Stream a microphone over RTP using ffmpeg
ffmpeg -f dshow -i audio="Microphone" -acodec pcm_s16le -ar 16000 -ac 1 -f rtp rtp://SERVER_IP:5004

# Stream from a file in a loop
ffmpeg -re -stream_loop -1 -i audio.mp3 -acodec pcm_s16le -ar 16000 -ac 1 -f rtp rtp://SERVER_IP:5004

Then query the API:

# Get the last word spoken
curl http://localhost:8000/api/last-word

# Get the last sentence
curl http://localhost:8000/api/last-segment

# Get the last 50 words
curl http://localhost:8000/api/last-words?n=50

# Consume the next word from the queue (deleted after read)
curl http://localhost:8000/api/pop-word

# Connect via WebSocket for real-time push
websocat ws://localhost:8000/ws/stream

When running with --source both (default --mix separate), add ?source=mic or ?source=desktop to any endpoint.

Batch File Transcription

Place video files in the video/ directory and run:

uv run main.py

Transcripts are saved to transcript/ as JSON files with word, segment, and character-level timestamps.

Usage

Real-Time Service (serve.py)

uv run serve.py [OPTIONS]
Flag Default Description
--source {mic,desktop,both,network} mic Audio source to capture
--stream-url URL — Network stream URL (implies --source=network). Supports rtp://, rtsp://, udp://, tcp://
--mix {separate,mixed} separate When --source=both: keep separate transcripts or mix into one
--host HOST 127.0.0.1 Bind address
--port PORT 8000 Port
--chunk-seconds N 3.0 Seconds of audio to accumulate before ASR inference
--device-index N auto Override audio device index (mic/desktop only)
--model NAME nvidia/parakeet-tdt-0.6b-v3 ASR model
--log-level INFO Logging level

API Endpoints

Endpoint Description
GET /api/status Service status, sources, word counts
GET /api/last-word Most recent word
GET /api/last-segment Most recent sentence/segment
GET /api/last-words?n=100 Last N words (rolling buffer)
GET /api/pop-word Next word from queue (consumed on read)
GET /api/pop-words?n=10 Next N words from queue (consumed on read)
WS /ws/stream Real-time push of words as recognized

All endpoints accept ?source=mic or ?source=desktop when running in separate mode.

Batch Transcription (main.py)

uv run main.py [OPTIONS]

Input Selection

Flag Description
--input, -i PATH Process a specific file or directory
--audio-only Transcribe audio files directly, skip video extraction
--tracks N [N ...] Only extract/transcribe specific track numbers (1-indexed)
--youtube URL Download a YouTube video and transcribe it (cannot combine with --input or --audio-only)

Directories

Flag Default Description
--video-dir PATH video/ Video input directory
--audio-dir PATH audio/ Audio working directory
--output-dir PATH transcript/ Transcript output directory

Output

Flag Description
--format {json,txt,plain} Output format (default: json). txt = segments with timestamps; plain = segment text only, no timestamps
--model NAME ASR model (default: nvidia/parakeet-tdt-0.6b-v3)
--verbose, -v Show segment-level timestamps during transcription
--quiet, -q Suppress all output except errors and final file paths

Behavior

Flag Description
--dry-run List files that would be processed without running
--skip-existing Skip files that already have transcripts in the output directory
--chunk-minutes N Split audio into N-minute chunks before transcribing to avoid CUDA OOM on long files (default: 5.0). Use 0 to disable and process the full file in one pass.

Examples

# Process all videos in the default video/ directory
uv run main.py

# Process a single video file
uv run main.py --input video/interview.mp4

# Transcribe an existing audio file directly
uv run main.py --audio-only --input audio/podcast.mp3

# Transcribe all audio files in a custom directory
uv run main.py --audio-only --audio-dir /path/to/recordings

# Extract and transcribe only tracks 1 and 3
uv run main.py --input video/recording.mkv --tracks 1 3

# Output as plain text (segments with timestamps)
uv run main.py --format txt

# Output as plain text WITHOUT timestamps (segment text only)
uv run main.py --format plain

# Preview what would be processed
uv run main.py --dry-run

# Skip videos that already have transcripts
uv run main.py --skip-existing

# Verbose mode with a custom model
uv run main.py --verbose --model nvidia/parakeet-tdt-1.1b

# Download and transcribe a YouTube video
uv run main.py --youtube "https://www.youtube.com/watch?v=..."

# Transcribe a long file without chunking (caution: may OOM on large GPUs)
uv run main.py --chunk-minutes 0 --input video/longfile.mp4

# Use smaller chunks if still hitting OOM
uv run main.py --chunk-minutes 3 --input video/longfile.mp4

Output Format

JSON (default)

Each transcript is a JSON array with one entry per audio track. Each entry contains word, segment, and char arrays with start/end timestamps:

[
  {
    "word": [{ "start": 0.0, "end": 0.32, "word": "Hello" }, ...],
    "segment": [{ "start": 0.0, "end": 2.5, "segment": "Hello world" }, ...],
    "char": [{ "start": 0.0, "end": 0.08, "char": "H" }, ...]
  }
]

Plain Text with timestamps (--format txt)

--- Track 1 ---
0.0s - 2.5s : Hello world
2.8s - 5.1s : This is a transcript

Plain Text without timestamps (--format plain)

Just the segment text, one segment per line — no timestamps (track headers still appear only when there are multiple tracks):

Hello world
This is a transcript

Note: txt and plain both write a .txt file, so they share the same output filename — the newer run replaces the older one. To keep both, point --output-dir somewhere else for one of them.

Project Structure

wisper/
├── serve.py                         # Real-time service CLI entry point
├── server.py                        # FastAPI app (REST + WebSocket endpoints)
├── audio_capture.py                 # Audio capture: mic, desktop (WASAPI), or network stream (ffmpeg)
├── realtime_asr.py                  # Chunked Parakeet ASR worker for real-time
├── transcript_store.py              # Thread-safe transcript accumulator
├── models.py                        # Shared data types (timestamps, LiveWord)
├── main.py                          # Batch CLI entry point (multi-track pipeline)
├── main_single_track.py             # Single-track pipeline with speaker diarization
├── multitalker_transcript_config.py # Configuration for diarization and streaming ASR
├── test_transcript.py               # Utility to replay transcripts with timing
├── pyproject.toml                   # Project metadata and dependencies
├── video/                           # Place input video files here
├── audio/                           # Extracted audio tracks (auto-generated)
└── transcript/                      # Output transcripts (auto-generated)

Pipelines

  • main.py — Multi-track pipeline. Extracts all audio tracks from video via ffprobe/ffmpeg, transcribes each with the Parakeet TDT model, and saves structured JSON or text output.
  • main_single_track.py — Single-track pipeline with advanced features including streaming speaker diarization (Sortformer) and multitalker ASR for conversations with multiple speakers.

Models

Model Purpose
nvidia/parakeet-tdt-0.6b-v3 Primary ASR transcription
nvidia/diar_streaming_sortformer_4spk-v2.1 Speaker diarization (single-track mode)
nvidia/multitalker-parakeet-streaming-0.6b-v1 Multitalker streaming ASR (single-track mode)

Contributing

Contributions are welcome! Please open issues or submit pull requests for improvements and bug fixes.

About

Audio Transcription tool for ripping videos

Resources

Readme
Activity

Stars

2 stars

Watchers

0 watching

Forks

1 fork
Report repository

Releases

Packages

Contributors

Languages