Voice-RAG MSMARCO-XI

A voice-driven, multilingual Retrieval-Augmented Generation engine built for sub-second, fully cited answers.

---

Overview

Voice-RAG MSMARCO-XI converts a spoken question in Hindi or English into a grounded, citation-backed answer.

The system transcribes speech with Whisper, embeds the resulting query using an int8-quantized ONNX model on CPU, retrieves the most semantically relevant passages from the ai4bharat/MSMARCO-XI corpus with FAISS, and asks Llama 3.1 to answer using only the retrieved evidence.

Every stage is guarded. When retrieval confidence is low, or when the generated response is not supported by the retrieved passages, the system declines to answer instead of guessing.

Key capabilities

• Voice-first interface. Gradio handles microphone capture; Groq-hosted Whisper handles low-latency speech-to-text.
• Cross-lingual retrieval. Sentence-Transformers embeddings map Hindi and English queries into a shared vector space, so a Hindi question can retrieve English evidence and the reverse.
• CPU-only inference. The embedding model is exported through Optimum to ONNX and quantized to int8, removing the need for a GPU at query time.
• In-process vector search. A FAISS FlatIP index is held in memory, eliminating network round trips from the hot path.
• Enforced groundedness. Answers must cite retrieved passages. Unsupported output is blocked before it reaches the user.
• Predictable latency. The full retrieval and generation path completes in roughly 100 ms at the median on the reference setup.

Architecture

flowchart TD
    A[Voice input] -->|Whisper STT| B[Text query]
    B --> D[Input safety guardrail]
    D --> C[ONNX embedder<br/>MiniLM int8, CPU]
    C --> E[FAISS vector search<br/>in-process FlatIP]
    E --> F[Retrieval gate]
    F --> G[Groq generator<br/>Llama 3.1 8B]
    G --> H[Groundedness check]
    H --> I[Answer with citations]

Loading

Pipeline stages

1. Capture and transcribe. Audio is recorded in the browser and transcribed by Whisper via the Groq API.
2. Screen. The transcript passes through concurrent safety routines that reject malicious or out-of-scope input.
3. Embed. The query is encoded by the quantized MiniLM ONNX model into a dense vector.
4. Retrieve. FAISS performs inner-product similarity search across the indexed MSMARCO-XI passages.
5. Gate. Results below the configured similarity threshold stop the request before any tokens are generated.
6. Generate. Llama 3.1 8B, served by Groq, composes an answer constrained to the retrieved context.
7. Verify and return. The response is checked for citation coverage, then returned alongside its source passages.

Technology stack

Layer	Technology	Role
API framework	FastAPI and Uvicorn	Fully async request handling for high throughput
Validation	Pydantic v2	Strict schema enforcement on every request and response
Interface	Gradio	Browser-based UI with built-in microphone support
Data processing	Datasets and Polars	Out-of-core Parquet operations during corpus preparation
Embeddings	Sentence-Transformers	Cross-lingual encoder covering English and Hindi
Optimization	Optimum with ONNX Runtime	int8 quantization for fast CPU inference
Vector search	FAISS (CPU)	In-memory similarity search with no network overhead
Reliability	Tenacity	Retry and backoff logic around external API calls

Guardrails

Guardrail	Trigger	Behaviour
Input safety	Malicious or unsafe query content	Request is rejected before embedding
Retrieval gate	Top similarity score below 0.35	Request is refused rather than answered speculatively
Groundedness check	Answer lacks a citation to a retrieved passage	Output is blocked and not returned

The retrieval threshold is deliberately conservative. A refusal is treated as a correct outcome when the corpus does not contain a supporting passage.

Performance

Measured over 200 consecutive requests on the reference configuration.

Stage	P50	P99
Query embedding	15.62 ms	23.36 ms
FAISS search	1.00 ms	2.00 ms
LLM generation	81.33 ms	95.84 ms
Total hot path	98.96 ms	120.30 ms

Figures exclude speech-to-text and depend on host hardware, corpus size, and network conditions between the service and the Groq API. Re-run the benchmark in your own environment before quoting these numbers.

Getting started

Prerequisites

• Python 3.10 or newer
• A Groq API key
• A working microphone and a browser that permits microphone access

Installation

python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install -r requirements.txt

Configuration

export GROQ_API_KEY="your_groq_api_key"

Variable	Required	Description
GROQ_API_KEY	Yes	Authenticates both the Whisper transcription and Llama 3.1 generation calls

Additional tunables, including the retrieval similarity threshold (0.35 by default) and the number of passages retrieved per query, are defined in the application configuration.

Running locally

python run.py

Open the URL printed in the terminal, grant microphone access, and ask a question.

Deployment on Hugging Face Spaces

1. Create a new Space and select the Gradio SDK.
2. Push this repository to the Space.
3. Add GROQ_API_KEY under Settings > Repository secrets.
4. Set run.py as the entry point.
5. Wait for the build to finish, then open the Space.

The first request after a cold start includes index loading time and will be slower than the benchmarks above.

Acknowledgements

• ai4bharat/MSMARCO-XI for the multilingual passage corpus
• Groq for low-latency Whisper and Llama 3.1 inference
• FAISS and ONNX Runtime for the retrieval and embedding layers

License

Add a LICENSE file at the repository root and reference it here.