Skip to content

feat: locale packs so non-English calls are not silent passes - #10

Open
AshSgDe29071999 wants to merge 1 commit into
AgentPostmortem:mainfrom
AshSgDe29071999:feat/3-locale-packs
Open

feat: locale packs so non-English calls are not silent passes#10
AshSgDe29071999 wants to merge 1 commit into
AgentPostmortem:mainfrom
AshSgDe29071999:feat/3-locale-packs

Conversation

@AshSgDe29071999

Copy link
Copy Markdown
Contributor

Summary

Mishearing and confirmation vocabularies were hardcoded English. A Spanish or Hindi call returned empty findings from the two most important checks — the same silent-pass shape this project exists to catch.

Changes

  • `voiceeval/locale_packs/{en,es}.json` with confusable pairs + confirmation phrases
  • `Interaction.language` (default `en`); `analyse(..., packs=..., locale_pack=...)`
  • Missing pack → `locale_unavailable` (high) so the suite does not pass
  • English default pack matches the previous in-code lists
  • Spanish example pack + `fixtures/misheard_es.json`
  • Suites can pass their own pack

Test plan

  • `python3 -m pytest -q tests/test_voiceeval.py` (22 passed)
  • `good_call.json` still has zero findings under `en`

Fixes #3

Hardcoded English vocabularies made Spanish/Hindi calls look clean.
Load packs from locale_packs/, default en (same pairs/phrases as
before), and emit locale_unavailable instead of an empty pass when
no pack exists. Ships an es example with a fixture.

Fixes AgentPostmortem#3

@royalpinto007 royalpinto007 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Locale packs are the right call, and locale_unavailable at high severity instead of an empty result is better than I would have asked for. Verified: language=hi gives [(locale_unavailable, high)], en clean call gives nothing.

Same matching bug as #9, written independently here:

FLAGGED  "I ordered fifteen items for fifty pounds"       <- transcript == truth
FLAGGED  "we have thirteen in stock and thirty on order"  <- transcript == truth

Only the -teen/-ty pairs are in en today so it is rare, but once #9 adds to/two and a/eight it fires constantly.

Suggest landing this one first with a positional matcher (difflib opcodes, pairs taken from the pack), then rebasing #9 so it only adds pairs and a fixture. One matcher, one place.

@royalpinto007 royalpinto007 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing locale packs currently stop all analysis, hiding language-independent findings like policy violations, latency, and incomplete calls. Please add locale_unavailable, skip only locale-dependent checks, and continue running the others.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The mishearing and confirmation vocabularies are hardcoded English, so a non-English call reports zero findings

2 participants