Skip to content

About

Customer-service agent for bank transaction disputes: the LLM labels intent; policy, confirmation and human handoff run in code. Continues the Factored AI & Data Hackathon 2026 project.

Topics

Resources

Contributing

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

Sentinel Engine

Status: active development. There is no public demo: run it locally (Run it locally). The project site has the overview, the diagrams and the slides. The history is in the changelog.

The AI talks. The rules decide.

A bank customer says "I do not recognize this charge". Sentinel finds the charge, checks the policy, opens a verified dispute in one conversation, or gives an advisor a complete case file. It speaks Spanish and Portuguese. It acts only when the policy allows it. It never invents a fact.

Transaction disputes for a bank in México, Colombia and Argentina · Prototype

Read more: Project site · Product · What is real · Evidence index · Metrics report

The customer chat in Spanish (es-MX): the customer names a charge, confirms it, and gets a verified dispute. The left panel shows each step. The right panel shows the claims and the recent charges.

Contents

  1. What Sentinel does
  2. How it works
  3. Headline results
  4. Where to find everything
  5. Run it locally
  6. Demo and production
  7. Limitations
  8. Roadmap
  9. Team and rules

What Sentinel does

Sentinel is an assistant, not only a chatbot. It handles the intake and triage of a transaction dispute. It does not decide the outcome of the dispute, and it does not move money.

Step What the code does
Understand Labels the intent. Masks personal data before the model call
Find Looks up the charges of the logged-in customer in Gold and narrows them with the words of the customer
Decide The policy engine checks the status of the charge, the dispute window, a prior dispute, the fraud rule and the amount rule
Confirm Shows the charge in a confirm box. The assistant never opens a dispute without the confirmation of the customer
Act and verify Opens the case once (idempotent) and reads it back before it says that the case exists
Hand off Files a ticket for an advisor with the request, the verified facts, the actions, the evidence and the open questions

The system files a handoff ticket on its own when a rule asks for it:

  • The customer insists on a person.
  • The customer says that the charge is not theirs.
  • A fraud rule or a high-amount rule fires.
  • Required fields are missing.
  • The third question has no answer.

The advisor reads the ticket in a read-only view of the same page, with the trace of each step. The decision on the dispute belongs to the bank.

How it works

The guiding principle: AI understands; code executes and verifies. The loop is: Understand → Decide → Act → Verify → Escalate. Policy, confirmations and the session are in code. The model never sees the customer identifiers.

Architecture of Sentinel, demo view. The model labels the intent. The code masks, decides, acts and verifies. Each part is marked real or mock.

Layer What it is
Data A Delta Lakehouse: S3 raw data → Bronze → Silver → Gold. DuckDB runs it locally. The Azure Databricks mode is implemented and not deployed. Both use the same sentinel_data package.
Service One FastAPI process with the page, the orchestrator, the policy engine and four session-bound tools. One API under /api/v1: auth, transactions, chat, disputes (two steps), handoffs (advisor) and health.
Router router_v2, a prompted GLM 5.3 Flash that only labels the intent. The keyword baseline answers a turn when the model fails. Both are behind one model port.
State Sessions, conversation state, disputes and handoff tickets are in SQLite. A restart keeps them.
Human in the loop Each handoff becomes a ticket with the reason, a summary, the verified facts and every action that the system tried.

Safety in code, before and around the model call:

  • Code masks personal identifiers in the customer text.
  • Code refuses a request to reveal the prompt and records injection attempts.
  • Fraud and high-amount handoffs use synthetic thresholds per account country and currency. A bank replaces them in configuration.
  • A "why?" gets its answer from the stored policy decision, not from the model.

The architecture pages give the target system, the demo with its mocks, and the specification.

No public demo. The earlier demo on Azure Container Apps (019) was retired on 2026-10-26. The last live revision was from 2026-10-05, with /health bundle_hash 2efe5962f9a50d0b4fed8e7b91c10a4c7fd212d5229d74a2e58d1024ee96dfd2, the hash of the sealed v8 measurement.

Headline results

All numbers are simulation on team-written cases, not production measurements (what is real). Each row cites a field of a frozen summary.json.

Result Baseline router_v2 Field
Intent accuracy, sealed held-out set (n = 308) 0.6916 0.8182 eval-v8: candidates.<version>.intent.accuracy
Model latency p50 / p95 (ms) code only 1474.2 / 4317.14 candidates.router_v2.latency_ms
Safe automated resolution, mock store (n = 56) 16 16 resolution-v2: system.<version>.safe_resolution.resolved
Unsafe outcomes, resolution set 0/56 0/56 system.<version>.unsafe_outcomes.rate
Cost per successful resolution (USD) 0 0.000561 system.<version>.cost_usd.per_resolution
Unsafe outcomes, adversarial suite 0/42 adversarial: totals.unsafe_outcome_rate

A safe automated resolution means that an eligible case ends in a verified case with no person. A correct handoff or refusal does not count as resolved. It counts under escalation quality.

The single v8 measurement ran six candidates on two sealed sets. The verdict serves router_v2. router_v3 has the higher kind accuracy (0.974), but it fails the zero-unsafe-wording gate (4/93) and the subtype gate (0.8971 against 0.95). Decision 018 gives the gates and the failed rules.

The metrics report gives the intervals, the breakdown by language and country, and the limits.

Where to find everything

Start with a question

You want to know Go to
Where the project started and what the first brief asked The brief
What the product does for a bank Product · Project site
What it would take to run this at a bank Project site · What is real
Which parts are real and which are mocks What is real · Mocks
Why we chose transaction disputes Flow selection
Why each main choice, with its evidence Rationale · Decisions
How we measured the model and the system Metrics report · Model card · Evidence index
Which requirements are done Requirements
How the assistant talks Conversation rules · Glossary
How safe it is Security · Attack coverage
What it costs Cost · ROI (projection) · Sizing
What the data holds Dataset · Data dictionary · Inventory

The documentation index gives the full reading order.

Repository map

Path What it holds
Product code
sentinel-ai-core/ The service: FastAPI, the orchestrator, the policy engine, the tools, the chat page and the advisor view (app/). The evaluation harness with the development cases, the sealed held-out set and the resolution set (eval/). The tests, with the adversarial suite (tests/).
sentinel-data-engine/ The medallion pipeline (S3 → Bronze → Silver → Gold) on Delta Lake. 13 LATAM Bank tables, about 19 M records.
deploy/azure/ The Azure Container Apps deployment script, the demo users and the log queries. The demo is retired; kept as a reference
Proof
evidence/ Frozen, reproducible runs. Each run has its scripts and a summary.json. The index gives the status and the data type of each run.
docs/build/metrics-report.md The full measurement, with intervals and limits
Documentation
docs/architecture/ System architecture, demo architecture, specification, mocks and what is real
docs/rationale/ Why each choice, with an evidence table and the slide sentence
docs/build/ Conversation rules, decisions, metrics, security, cost and delivery
docs/requirements/ Every requirement, with its status, sources and evidence
docs/data/, docs/glossary/ The dataset, the official data dictionary and the locale vocabulary
Site
sentinel-engine-site The project site in its own repository, in English, Spanish and Portuguese, published at https://rdorta27.github.io/sentinel-engine-site/. The slides (site/slides/) and the diagrams (site/diagrams/).
branding/ The brand tokens, the fonts and the chat styles
Tooling
scripts/ Repository scripts: site numbers, link checks and load runs
openspec/ OpenSpec specs and the archive of each change
.github/workflows/ CI: both test suites, the Markdown links and the requirement status that the specs cite
team/ Record of the first phase: plan, tasks and decisions

Run it locally

You need Python 3.12 or newer.

1. Start the demo with the labelled mock store:

cd sentinel-ai-core
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
SENTINEL_GOLD_SOURCE=mock SENTINEL_REFERENCE_DATE=2026-06-17 uvicorn app.main:app

Open http://localhost:8000/ui. Log in as CUST-0001, CUST-0002 or CUST-0003, with the password Testpass-001. These credentials are false and only for local runs and tests.

2. See the advisor side. Start the service with the demo roles:

SENTINEL_GOLD_SOURCE=mock SENTINEL_DEMO_AUTH=1 uvicorn app.main:app

Ask for a person two times as a customer. Then log in as ADV-0001 with the password Advisor-001.

3. Use the measured router. Without a model, the service uses the keyword baseline.

  1. Copy .env.example to .env.
  2. Fill in the SENTINEL_LLM_* values (.env.example, secrets guide).
  3. Load the file: set -a; source ../.env; set +a.
  4. Start uvicorn again. GET /api/v1/health shows the active model.

4. Run the tests. They need no model keys and no cloud credentials, and they write no evidence:

cd sentinel-ai-core && SENTINEL_WRITE_EVIDENCE=0 python3 -m pytest -q
cd ../sentinel-data-engine && pip install -e ".[dev]" && python3 -m pytest -q

Useful details:

Topic Detail
State sentinel-ai-core/var/sentinel.db (gitignored). Delete it for a clean demo.
Turn log tail -f sentinel-ai-core/var/turns.jsonl shows the structured log of each turn.
Gold source SENTINEL_GOLD_SOURCE is mock, duckdb or auto (the default). auto uses the real Gold file when it is present. The test users exist only in mock. GET /api/v1/health shows the source in gold_source.
Data pipeline Sync the raw tables into data/raw/, then build data/gold_bank.duckdb. See the pipeline quickstart. Both folders are gitignored.
Replay a run The evidence index tells how to replay each frozen run offline, and which runs match their frozen summary.
Deploy deploy/azure/README.md

Demo and production

The same code runs in the demo and in production. Each mock keeps the production contract, so production replaces a backend, not code.

Part Demo (this repository) Production
Gold data A labelled mock store, or the PII-free DuckDB view of the pipeline when you run it locally with the data file Gold on Azure Databricks
Identity Test users with a password The identity provider of the bank
Advisor A demo advisor user with a read-only view A human advisor. Tickets go to the CRM of the bank through a queue
Case store SQLite PostgreSQL
Dispute policy Team-written files marked synthetic: true The policy that the bank approves
Opening a dispute The case store records the case. No bank system receives it The dispute system of the bank

The mocks page gives the reason, the limit and the production backend of each mock.

Limitations

What the prototype does not do (REQ-0013, REQ-0030). Each row links to the detail.

Area Limitation
Mocks A test session, SQLite, a demo advisor user, a synthetic policy, a stand-in model in the test suite, and a labelled Gold mock (mocks).
Charge not yet in Gold The assistant treats it as not found. It names what it searched and, after repeated questions, files a handoff. It does not open a case marked as pending verification, the target behavior in the conversation rules.
Data Synthetic and in Spanish only. The data cannot support a charge investigation. A fraud_score above 30 is always fraud, an artefact of the generator (investigation data support). Real Gold runs locally only.
Policy One 90-day window for the three countries, marked synthetic: true. The sources disagree per country. The engine cannot express a different window start, provisional credit or a response time (021).
Languages A model wrote the Portuguese (pt-BR) cases. No native speaker reviewed them. A Colombian teammate accepted the es-CO lines. A model checked the Mexican and Argentine lines (018). The dataset transcripts are two Spanish templates (evidence).
Site translations A model wrote the Spanish and Portuguese copies of the site and the slides. English is the reference.
Model A turn that falls back to the baseline has the accuracy of the baseline. The confidence cut-offs are descriptive (validation n = 26) and off by default (calibration-v1, cutoff diagnosis). The learned charge selector stays off, because it fails the serving rule (025).
State SQLite and one replica. Login and write-rate counters are per process. PostgreSQL is the path to more than one instance.
Privacy Code masks free text by pattern. It does not detect personal data outside those patterns.
Safety evidence 0/42 unsafe outcomes (adversarial run). The three injection attempts also pass against the real router model (real-model run). Known limitation B4: the service accepts a replayed bearer cookie. Zero unsafe outcomes on a small set does not prove zero risk.
Resolution A simulation over the mock store only, with no pending charge. It is not a field resolution rate (022).
Capacity About 5 requests a second on one replica at 0.5 vCPU and 1 GiB, with recorded answers (load run, capacity and latency).
Monitoring and ROI Country monitoring uses a replayed, simulated workload (evidence). The ROI is a break-even projection, not a measured saving (roi).

Roadmap

What the team did not build, and why. Each row cites its evidence (REQ-0030, REQ-0056). None of these is built yet.

Item What it would do Why we did not build it Evidence
Customer 360 Show balances and account history next to a charge The balance has no usable as-of date. A complaint cannot be tied to a charge. customer-360/dev-v1: balance.safe_to_show, balance.asof_usable, complaints.charge_linkable
Investigation of a charge Say that a charge is unusual for this customer No customer signal predicts fraud in the synthetic data. It needs real bank data. customer-360/dev-signals-v1: investigation.has_signal; investigation data support
Feedback dataset Keep the outcome that an advisor gives to each handoff, to train a model later The dataset cannot train a dispute model. The system has no live traffic yet. flows/2024Q4-v3: disputes.cnr_learnable, disputes.esc_learnable
Spending assistant Answer questions about the spending of the customer It needs balances and history. The data does not support them. customer-360/dev-v1: balance.safe_to_show
Policy retrieval Read the dispute policy from the documents of the bank The policy stays in code. The sources disagree and we have no bank document. policy sources; the plan for the measure is in metrics
Handoff routing Send a handoff to an advisor with the right language and specialty Not started. The demo has one advisor view. REQ-0046

Team and rules

  • No secrets and no data in the repository. The repository is public. Credentials go in .env (gitignored). Teammates share them by direct message.
  • English, in ASD-STE100. Code and documents are in English, in simplified technical English. The assistant answers in Spanish and Portuguese.
  • Label what is real. Each number is real, mock, synthetic, team-generated, simulation or projection. A simulation is never shown as a production measurement.
For Read
A person who contributes CONTRIBUTING.md
An AI agent AGENTS.md
The record of the first phase team/README.md
The change history CHANGELOG.md

About

Customer-service agent for bank transaction disputes: the LLM labels intent; policy, confirmation and human handoff run in code. Continues the Factored AI & Data Hackathon 2026 project.

Topics

Resources

Contributing

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages