Skip to content

Leg 28 llm responses - #9

Open
Ankita-Advitot wants to merge 2 commits into
stagingfrom
LEG_28_llm_responses
Open

Leg 28 llm responses#9
Ankita-Advitot wants to merge 2 commits into
stagingfrom
LEG_28_llm_responses

Conversation

@Ankita-Advitot

Copy link
Copy Markdown
Contributor
  • Add three new files that make up the LLM semantic-validation step: Extraction/validate_llm/llm_field_checks.py (prompt + parsing), Extraction/validate_llm/ollama_client.py (Ollama HTTP client), Extraction/validate_llm/test_prompt.py (standalone tester).
  • llm_field_checks.py sends one Ollama call per case, covering only fields with real semantic risk (SEMANTIC_FIELDS whitelist: name, role, address, office_address, designation, district, state, purpose, nature_of_disposal) across persons, judges, lawyers, organizations, and hearings. Case numbers, dates, IDs, and other rule-checkable fields are excluded — those are already handled by the existing rule engine.
  • The model returns invalid-only JSON ({"checked": N, "invalid": [{"n", "reason"}, ...]}) instead of a verdict per field, with a checked-count guard (one retry, then skip on persistent mismatch) so an incomplete response can't be silently trusted.
  • A small closed reason-code vocabulary (wrong_content, missing_location, missing_designation, narrative_text) — an unrecognized/hallucinated code is treated as valid rather than kept under a relabeled reason.
  • ollama_client.py provides ask_json() for this JSON-object response shape, plus _call_ollama()/ask_valid()/ask_lines() as supporting/alternate helpers.
  • Extraction/main.py: wire the new check into the pipeline — build the per-entity field lists (persons, judges, lawyers from missing_advocates, organizations from new_parties, hearings, case-level district/state) and add primary/secondary drop logic: name is primary for persons, judges, lawyers, and organizations — if missing or flagged wrong, the whole entity is dropped before the Neo4j insert phase runs. Every other checked field is secondary — a flag there only nulls that field.
  • Extraction/validate_llm/engine.py / init.py: expose validate_case_llm()/check_case() as the entry point main.py calls.
  • shared/config.py: add the Ollama URL/model env vars this step needs.
  • requirements.txt: fix two pins that don't exist on PyPI — onnxruntime==1.27.0 → 1.23.2, pandas==3.0.3 → 2.3.3.

@Ankita-Advitot Ankita-Advitot self-assigned this Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant