pii_scrubber is never applied on the LLM verification path: raw evidence text is sent to OpenAI and Groq

[kilodesodiq-arch]

Problem

The PII scrubber is only reachable as an explicit /anonymize (and /v1/ai/anonymize) endpoint; the LLM verification path that sends free-text evidence to an external provider never calls it. HumanitarianVerificationService.verify_claim builds prompts from raw inputs and posts them straight to OpenAI or Groq:

# app/ai-service/services/humanitarian_verification.py
primary_prompt = self.prompt_engine.build_primary_prompt(
    aid_claim=aid_claim, supporting_evidence=evidence, context_factors=context)  # raw text
...
raw_content = self._call_provider(provider=provider, model=model,
    system_prompt=prompt["system"], user_prompt=prompt["user"], ...)            # sent verbatim

PIIScrubberService (app/ai-service/services/pii_scrubber.py) exists and is instantiated in main.py, but a search for its usage in app/ai-service/services/ finds it only in its own file. Nothing in humanitarian_verification.py, ocr.py, or fraud_detection.py scrubs before the provider call.

Consequence: the README's claim that the platform performs "PII anonymization before external LLM processing" holds only if a caller separately remembers to invoke /anonymize first. On /v1/ai/humanitarian/verify, supporting_evidence — which the schema accepts as free text and which is described as recipient-provided evidence — is transmitted unredacted to a third-party LLM. Recipient names, locations, dates, ID numbers, and contact details can leave the trust boundary with no enforcement. The backend-side issue that scrubbed metadata: Json fields (#213) does not cover this AI-service path, because the evidence reaches the provider from inside the AI service itself.

Root cause

Anonymization is modelled as a separate, opt-in endpoint rather than as a mandatory preprocessing stage of the verification pipeline, so the pipeline and the scrubber are decoupled and the decoupling is invisible in the response.

Why this is architecturally hard

1. It must be enforced, not just added. Calling pii_scrubber.anonymize in verify_claim is the easy part; the hard part is making it impossible to skip — e.g. a pipeline stage that runs before any provider call, with a fail-closed policy when scrubbing itself fails.
2. Anonymized vs. raw evidence changes verification semantics. Names/locations are often material to a humanitarian claim. The design must decide whether the LLM sees masked tokens (and whether the verdict degrades), and how the unredacted text is retained for the human reviewer without leaking it to the provider.
3. The scrubber's own precision is imperfect. pii_scrubber.py relies on a small regex set plus a spaCy blank en entity ruler; its NAME_REGEXES includes \b[A-Z][a-z]+\s+[A-Z][a-z]+\b, which matches generic capitalized phrases. Relying on it as a hard gate requires a measurable detection bar and a documented residual-risk posture.
4. Retention/audit already exists and must stay correct. persistence/pii_decisions.py records aggregate scrub metadata (never text); wiring the scrubber into the hot path must feed that store without accidentally persisting raw evidence.

Proposed design

Introduce a single preprocessing step in the verification service that scrubs aid_claim/supporting_evidence/context_factors before prompt construction, with a config flag that fails closed (returns an error) when scrubbing is unavailable rather than silently sending raw text. Emit the scrub summary into pii_decisions and expose the scrubbed/raw distinction in the response envelope so the backend can keep the raw text on the human-review side only.

Acceptance criteria

Service

• A /v1/ai/humanitarian/verify request whose evidence contains a name/phone/ID transmits only masked tokens to the provider.
• If scrubbing fails or is disabled, the request is rejected (or routed to a no-provider path) rather than sent unredacted.

Tests

• A test asserts the provider payload (mocked) contains no raw PII for a seeded evidence string.
• A test pins the fail-closed behavior when the scrubber is unavailable.

Documentation

• The AI service README states which endpoints scrub PII and which do not, and the residual-risk posture for regex-based detection.

Out of scope

Improving the scrubber's regex/spaCy precision and the backend metadata scrubbing (#213) are separate concerns.

Getting started

Files: app/ai-service/services/humanitarian_verification.py, app/ai-service/services/pii_scrubber.py, app/ai-service/services/humanitarian_prompt.py, app/ai-service/persistence/pii_decisions.py.

cd app/ai-service
pytest

Good first files to read: services/humanitarian_verification.py (the raw-to-provider flow) and services/pii_scrubber.py (the anonymize contract to call).

[Devadakene]

I am experienced enough to implement this issue having worked on similar issues in the past. Please assign

[Flames4fun]

Hi! Nice to meet you — I’m @Flames4fun, a software engineer working mainly around AI infrastructure, security boundaries, and regression testing in open source.

I’d be very interested in taking this issue if it’s still available. I’ve worked on a closely related class of problem in OWASP/Agent-Security-Regression-Harness#107, where I hardened the boundary between externally supplied execution data and host-owned evidence, added regression coverage for forged evidence shapes, and followed the issue through maintainer security review and merge. I’ve also contributed deterministic policy work around sensitive data and PII handling in CSL Core.

For this issue, I’d treat the scrubber as an enforced preprocessing invariant rather than an optional helper: raw evidence should never reach the provider path, scrubber failure should remain fail-closed, and the tests should assert against the actual mocked provider payload so the guarantee cannot regress later. I’d also keep the persistence/audit behavior and the raw-vs-scrubbed boundary aligned with the scope described here rather than broadening the PII detector itself.

Given the security and verification semantics involved, I’d estimate about 6–8 focused hours for a PR I’m comfortable submitting, including regression tests and a full pass over the affected persistence/provider paths.

If you’re happy with that approach, I’d appreciate being assigned to the issue. Thanks, and nice to meet you!

[Degentle12]

Hi Maintainer
May I fix this issue please

[grantfox-oss]

🦊 GrantFox — @Degentle12 has been assigned to this issue as part of the Third Campaign campaign!

Next steps:

1. Open a Pull Request referencing this issue (e.g., Closes #430)
2. Your PR will be reviewed by the ChainForgee maintainers

Good luck! Track your progress on GrantFox.