Skip to content
samadplsPublic

About

Urdu end-of-turn detection for voice agents

Topics

Resources

Contributing

Stars

5 stars

Watchers

1 watching

Forks

baari: knows when an Urdu speaker has finished talking

Model on Hugging Face by samadpls.com Code license: BSD-2-Clause Data: IndicVoices, CC BY 4.0 PyPI Downloads Python 3.10+

End-of-turn detection for Urdu voice agents.
At every pause, it decides whether the speaker has finished (reply now) or is only pausing mid-sentence (keep listening).


baari (باری) means "turn" in Urdu. The table below shows that today's turn detectors often cut Urdu speakers off mid-sentence, or wait after they have finished. baari is a small audio model trained to get this right. It follows the Smart Turn recipe and takes the same input and output, so it can stand in for Smart Turn.

Status: the first model is out, on Hugging Face and as the baari package on PyPI. On the Urdu test set below, it beats every existing turn detector.

🎧 Results on Urdu

1,588 clips of real Urdu phone conversations, cut from the official held-out split of IndicVoices: 1,470 where the speaker had finished and 118 where they were pausing mid-sentence. Because most clips are "finished", a model that always says "finished" gets 92.6% plain accuracy, so the fair score is the average of the last two columns (balanced accuracy).

Model Balanced accuracy (95% CI) Waits when unfinished Replies when finished
baari int8 (default) 77.0% (72.9–81.0) 62.7% 91.4%
baari fp32 76.8% (72.4–80.7) 63.6% 89.9%
Always says "finished" (reference) 50.0% 0.0% 100.0%
Smart Turn v3.2 (CPU, int8) 63.4% (58.8–68.1) 57.6% 69.2%
Smart Turn v3.2 (GPU, fp32) 58.8% (54.8–63.0) 76.3% 41.3%
Indic Smart Turn tiny (int8) 73.4% (69.0–77.8) 61.0% 85.9%
Indic Smart Turn tiny (fp32) 70.0% (65.8–74.4) 51.7% 88.4%
Indic Smart Turn base (fp32) 70.9% (66.5–75.4) 47.5% 94.4%
PuristanLabs1/urdu-turn-v2 (Urdu) 51.8% (48.4–55.4) 16.9% 86.7%

Scored on a MacBook Pro (M3 Pro), with the same audio preparation Pipecat uses. On the same clips, baari fp32 beats Indic Smart Turn tiny int8, the best existing model, by 3.3 points (paired bootstrap, 95% CI +0.3 to +6.4). int8 models are sensitive: their scores shift by a few points between Arm and x86 CPUs (baari int8 gets 77.0% on Arm and 74.9% on x86), and even tiny changes in how the audio is prepared flip about 1 answer in 100. fp32 results stay the same.

How the test set was made
  • Source: the conversation recordings in the valid split of IndicVoices Urdu (CC BY 4.0): 1,654 segments, 81 calls, 80 speakers, 1.7 hours. Each segment ends at a pause.
  • Labels: two LLM agents each read every call's transcript in order and marked whether the speaker had finished at the end of each segment (prompt). They agreed on 96.0%. Agreed segments form the test set; the 66 others are kept apart as test_ambiguous.
  • Clips: the last 8 s of 16 kHz audio ending at the segment, with the speaker's preceding segments as context, the same recipe as Indic Smart Turn.
  • Checked by ear: a native speaker listened to 50 random clips. All 25 "finished" labels were right, and 19 of 25 "unfinished" labels were. In the wrong ones the words stop mid-sentence but the voice sounds finished, which a transcript cannot show. So roughly 1 in 4 "unfinished" clips may really be finished, and the "waits when unfinished" column is a rough measure.
  • Leakage: 70 of the 80 test speakers, and the other side of 57 of the 81 test calls, also appear in the IndicVoices train split. Any training data will leave out those speakers and calls.

Use it

pip install baari
from baari import TurnDetector

detector = TurnDetector(threads=4)  # downloads baari's 9 MB int8 model once
detector.is_finished(audio, sample_rate=16000)  # mono audio (float or int16) that ends at the pause
detector.probability(audio, sample_rate=16000)  # the same decision as a number from 0 to 1

It needs only numpy and onnxruntime. With the default int8 file, one call takes about 12 ms on 4 CPU threads and 36 ms on one; TurnDetector(precision="fp32") takes 23 and 77 ms (MacBook Pro, M3 Pro). Use threads=1 on a server that handles many calls at once. baari needs 16 kHz audio; for 8 kHz phone audio, resample first, for example with soxr.resample(audio, 8000, 16000).

In Pipecat, baari replaces Smart Turn with one line, because it takes the same input and gives the same output:

from baari import model_path
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3

turn_analyzer = LocalSmartTurnAnalyzerV3(smart_turn_model_path=model_path())

TurnDetector(path=...) runs a local ONNX file instead of downloading one.

🛠️ Run it

uv sync --extra pipeline
echo "HF_TOKEN=..." > .env          # accept the IndicVoices terms on Hugging Face first
uv run baari scan --split valid     # index the conversation segments (text only)
uv run baari batches                # label batches for the LLM agents
# the agents write data/labels/valid/results/{a,b}/batch-*.jsonl
uv run baari build                  # cut the 8 s clips
uv run baari evaluate               # score the existing models
Training

The same steps on the IndicVoices train split, with one labeler. Test speakers and both sides of test calls are left out, and about 5% of the remaining speakers form a dev split that picks the best epoch.

uv run baari scan --split train
uv run baari batches --split train
# the agents write data/labels/train/results/a/batch-*.jsonl
uv run baari build --split train --labelers a --delete-shards
uv run baari train                  # fine-tune Indic Smart Turn tiny, export fp32 and int8 ONNX

Training starts from Indic Smart Turn tiny. That model ships only as ONNX, so baari train first reads its weights back into PyTorch. The kaggle folder runs the clip build and the training on Kaggle.


Credits · Contributing · BSD-2-Clause license
Made by Abdul Samad (abdulsamadsid1@gmail.com)

About

Urdu end-of-turn detection for voice agents

Topics

Resources

Contributing

Stars

5 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages