Status: active development. There is no public demo: run it locally (Run it locally). The project site has the overview, the diagrams and the slides. The history is in the changelog.
The AI talks. The rules decide.
A bank customer says "I do not recognize this charge". Sentinel finds the charge, checks the policy, opens a verified dispute in one conversation, or gives an advisor a complete case file. It speaks Spanish and Portuguese. It acts only when the policy allows it. It never invents a fact.
Transaction disputes for a bank in México, Colombia and Argentina · Prototype
Read more: Project site · Product · What is real · Evidence index · Metrics report
- What Sentinel does
- How it works
- Headline results
- Where to find everything
- Run it locally
- Demo and production
- Limitations
- Roadmap
- Team and rules
Sentinel is an assistant, not only a chatbot. It handles the intake and triage of a transaction dispute. It does not decide the outcome of the dispute, and it does not move money.
| Step | What the code does |
|---|---|
| Understand | Labels the intent. Masks personal data before the model call |
| Find | Looks up the charges of the logged-in customer in Gold and narrows them with the words of the customer |
| Decide | The policy engine checks the status of the charge, the dispute window, a prior dispute, the fraud rule and the amount rule |
| Confirm | Shows the charge in a confirm box. The assistant never opens a dispute without the confirmation of the customer |
| Act and verify | Opens the case once (idempotent) and reads it back before it says that the case exists |
| Hand off | Files a ticket for an advisor with the request, the verified facts, the actions, the evidence and the open questions |
The system files a handoff ticket on its own when a rule asks for it:
- The customer insists on a person.
- The customer says that the charge is not theirs.
- A fraud rule or a high-amount rule fires.
- Required fields are missing.
- The third question has no answer.
The advisor reads the ticket in a read-only view of the same page, with the trace of each step. The decision on the dispute belongs to the bank.
The guiding principle: AI understands; code executes and verifies. The loop is: Understand → Decide → Act → Verify → Escalate. Policy, confirmations and the session are in code. The model never sees the customer identifiers.
| Layer | What it is |
|---|---|
| Data | A Delta Lakehouse: S3 raw data → Bronze → Silver → Gold. DuckDB runs it locally. The Azure Databricks mode is implemented and not deployed. Both use the same sentinel_data package. |
| Service | One FastAPI process with the page, the orchestrator, the policy engine and four session-bound tools. One API under /api/v1: auth, transactions, chat, disputes (two steps), handoffs (advisor) and health. |
| Router | router_v2, a prompted GLM 5.3 Flash that only labels the intent. The keyword baseline answers a turn when the model fails. Both are behind one model port. |
| State | Sessions, conversation state, disputes and handoff tickets are in SQLite. A restart keeps them. |
| Human in the loop | Each handoff becomes a ticket with the reason, a summary, the verified facts and every action that the system tried. |
Safety in code, before and around the model call:
- Code masks personal identifiers in the customer text.
- Code refuses a request to reveal the prompt and records injection attempts.
- Fraud and high-amount handoffs use synthetic thresholds per account country and currency. A bank replaces them in configuration.
- A "why?" gets its answer from the stored policy decision, not from the model.
The architecture pages give the target system, the demo with its mocks, and the specification.
No public demo. The earlier demo on Azure Container Apps (019) was retired on 2026-10-26. The last live revision was from 2026-10-05, with /health bundle_hash 2efe5962f9a50d0b4fed8e7b91c10a4c7fd212d5229d74a2e58d1024ee96dfd2, the hash of the sealed v8 measurement.
All numbers are simulation on team-written cases, not production measurements (what is real). Each row cites a field of a frozen summary.json.
| Result | Baseline | router_v2 |
Field |
|---|---|---|---|
| Intent accuracy, sealed held-out set (n = 308) | 0.6916 | 0.8182 | eval-v8: candidates.<version>.intent.accuracy |
| Model latency p50 / p95 (ms) | code only | 1474.2 / 4317.14 | candidates.router_v2.latency_ms |
| Safe automated resolution, mock store (n = 56) | 16 | 16 | resolution-v2: system.<version>.safe_resolution.resolved |
| Unsafe outcomes, resolution set | 0/56 | 0/56 | system.<version>.unsafe_outcomes.rate |
| Cost per successful resolution (USD) | 0 | 0.000561 | system.<version>.cost_usd.per_resolution |
| Unsafe outcomes, adversarial suite | 0/42 | adversarial: totals.unsafe_outcome_rate |
A safe automated resolution means that an eligible case ends in a verified case with no person. A correct handoff or refusal does not count as resolved. It counts under escalation quality.
The single v8 measurement ran six candidates on two sealed sets. The verdict serves router_v2. router_v3 has the higher kind accuracy (0.974), but it fails the zero-unsafe-wording gate (4/93) and the subtype gate (0.8971 against 0.95). Decision 018 gives the gates and the failed rules.
The metrics report gives the intervals, the breakdown by language and country, and the limits.
| You want to know | Go to |
|---|---|
| Where the project started and what the first brief asked | The brief |
| What the product does for a bank | Product · Project site |
| What it would take to run this at a bank | Project site · What is real |
| Which parts are real and which are mocks | What is real · Mocks |
| Why we chose transaction disputes | Flow selection |
| Why each main choice, with its evidence | Rationale · Decisions |
| How we measured the model and the system | Metrics report · Model card · Evidence index |
| Which requirements are done | Requirements |
| How the assistant talks | Conversation rules · Glossary |
| How safe it is | Security · Attack coverage |
| What it costs | Cost · ROI (projection) · Sizing |
| What the data holds | Dataset · Data dictionary · Inventory |
The documentation index gives the full reading order.
| Path | What it holds |
|---|---|
| Product code | |
sentinel-ai-core/ |
The service: FastAPI, the orchestrator, the policy engine, the tools, the chat page and the advisor view (app/). The evaluation harness with the development cases, the sealed held-out set and the resolution set (eval/). The tests, with the adversarial suite (tests/). |
sentinel-data-engine/ |
The medallion pipeline (S3 → Bronze → Silver → Gold) on Delta Lake. 13 LATAM Bank tables, about 19 M records. |
deploy/azure/ |
The Azure Container Apps deployment script, the demo users and the log queries. The demo is retired; kept as a reference |
| Proof | |
evidence/ |
Frozen, reproducible runs. Each run has its scripts and a summary.json. The index gives the status and the data type of each run. |
docs/build/metrics-report.md |
The full measurement, with intervals and limits |
| Documentation | |
docs/architecture/ |
System architecture, demo architecture, specification, mocks and what is real |
docs/rationale/ |
Why each choice, with an evidence table and the slide sentence |
docs/build/ |
Conversation rules, decisions, metrics, security, cost and delivery |
docs/requirements/ |
Every requirement, with its status, sources and evidence |
docs/data/, docs/glossary/ |
The dataset, the official data dictionary and the locale vocabulary |
| Site | |
| sentinel-engine-site | The project site in its own repository, in English, Spanish and Portuguese, published at https://rdorta27.github.io/sentinel-engine-site/. The slides (site/slides/) and the diagrams (site/diagrams/). |
branding/ |
The brand tokens, the fonts and the chat styles |
| Tooling | |
scripts/ |
Repository scripts: site numbers, link checks and load runs |
openspec/ |
OpenSpec specs and the archive of each change |
.github/workflows/ |
CI: both test suites, the Markdown links and the requirement status that the specs cite |
team/ |
Record of the first phase: plan, tasks and decisions |
You need Python 3.12 or newer.
1. Start the demo with the labelled mock store:
cd sentinel-ai-core
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
SENTINEL_GOLD_SOURCE=mock SENTINEL_REFERENCE_DATE=2026-06-17 uvicorn app.main:appOpen http://localhost:8000/ui. Log in as CUST-0001, CUST-0002 or CUST-0003, with the password Testpass-001. These credentials are false and only for local runs and tests.
2. See the advisor side. Start the service with the demo roles:
SENTINEL_GOLD_SOURCE=mock SENTINEL_DEMO_AUTH=1 uvicorn app.main:appAsk for a person two times as a customer. Then log in as ADV-0001 with the password Advisor-001.
3. Use the measured router. Without a model, the service uses the keyword baseline.
- Copy
.env.exampleto.env. - Fill in the
SENTINEL_LLM_*values (.env.example, secrets guide). - Load the file:
set -a; source ../.env; set +a. - Start
uvicornagain.GET /api/v1/healthshows the active model.
4. Run the tests. They need no model keys and no cloud credentials, and they write no evidence:
cd sentinel-ai-core && SENTINEL_WRITE_EVIDENCE=0 python3 -m pytest -q
cd ../sentinel-data-engine && pip install -e ".[dev]" && python3 -m pytest -qUseful details:
| Topic | Detail |
|---|---|
| State | sentinel-ai-core/var/sentinel.db (gitignored). Delete it for a clean demo. |
| Turn log | tail -f sentinel-ai-core/var/turns.jsonl shows the structured log of each turn. |
| Gold source | SENTINEL_GOLD_SOURCE is mock, duckdb or auto (the default). auto uses the real Gold file when it is present. The test users exist only in mock. GET /api/v1/health shows the source in gold_source. |
| Data pipeline | Sync the raw tables into data/raw/, then build data/gold_bank.duckdb. See the pipeline quickstart. Both folders are gitignored. |
| Replay a run | The evidence index tells how to replay each frozen run offline, and which runs match their frozen summary. |
| Deploy | deploy/azure/README.md |
The same code runs in the demo and in production. Each mock keeps the production contract, so production replaces a backend, not code.
| Part | Demo (this repository) | Production |
|---|---|---|
| Gold data | A labelled mock store, or the PII-free DuckDB view of the pipeline when you run it locally with the data file | Gold on Azure Databricks |
| Identity | Test users with a password | The identity provider of the bank |
| Advisor | A demo advisor user with a read-only view | A human advisor. Tickets go to the CRM of the bank through a queue |
| Case store | SQLite | PostgreSQL |
| Dispute policy | Team-written files marked synthetic: true |
The policy that the bank approves |
| Opening a dispute | The case store records the case. No bank system receives it | The dispute system of the bank |
The mocks page gives the reason, the limit and the production backend of each mock.
What the prototype does not do (REQ-0013, REQ-0030). Each row links to the detail.
| Area | Limitation |
|---|---|
| Mocks | A test session, SQLite, a demo advisor user, a synthetic policy, a stand-in model in the test suite, and a labelled Gold mock (mocks). |
| Charge not yet in Gold | The assistant treats it as not found. It names what it searched and, after repeated questions, files a handoff. It does not open a case marked as pending verification, the target behavior in the conversation rules. |
| Data | Synthetic and in Spanish only. The data cannot support a charge investigation. A fraud_score above 30 is always fraud, an artefact of the generator (investigation data support). Real Gold runs locally only. |
| Policy | One 90-day window for the three countries, marked synthetic: true. The sources disagree per country. The engine cannot express a different window start, provisional credit or a response time (021). |
| Languages | A model wrote the Portuguese (pt-BR) cases. No native speaker reviewed them. A Colombian teammate accepted the es-CO lines. A model checked the Mexican and Argentine lines (018). The dataset transcripts are two Spanish templates (evidence). |
| Site translations | A model wrote the Spanish and Portuguese copies of the site and the slides. English is the reference. |
| Model | A turn that falls back to the baseline has the accuracy of the baseline. The confidence cut-offs are descriptive (validation n = 26) and off by default (calibration-v1, cutoff diagnosis). The learned charge selector stays off, because it fails the serving rule (025). |
| State | SQLite and one replica. Login and write-rate counters are per process. PostgreSQL is the path to more than one instance. |
| Privacy | Code masks free text by pattern. It does not detect personal data outside those patterns. |
| Safety evidence | 0/42 unsafe outcomes (adversarial run). The three injection attempts also pass against the real router model (real-model run). Known limitation B4: the service accepts a replayed bearer cookie. Zero unsafe outcomes on a small set does not prove zero risk. |
| Resolution | A simulation over the mock store only, with no pending charge. It is not a field resolution rate (022). |
| Capacity | About 5 requests a second on one replica at 0.5 vCPU and 1 GiB, with recorded answers (load run, capacity and latency). |
| Monitoring and ROI | Country monitoring uses a replayed, simulated workload (evidence). The ROI is a break-even projection, not a measured saving (roi). |
What the team did not build, and why. Each row cites its evidence (REQ-0030, REQ-0056). None of these is built yet.
| Item | What it would do | Why we did not build it | Evidence |
|---|---|---|---|
| Customer 360 | Show balances and account history next to a charge | The balance has no usable as-of date. A complaint cannot be tied to a charge. | customer-360/dev-v1: balance.safe_to_show, balance.asof_usable, complaints.charge_linkable |
| Investigation of a charge | Say that a charge is unusual for this customer | No customer signal predicts fraud in the synthetic data. It needs real bank data. | customer-360/dev-signals-v1: investigation.has_signal; investigation data support |
| Feedback dataset | Keep the outcome that an advisor gives to each handoff, to train a model later | The dataset cannot train a dispute model. The system has no live traffic yet. | flows/2024Q4-v3: disputes.cnr_learnable, disputes.esc_learnable |
| Spending assistant | Answer questions about the spending of the customer | It needs balances and history. The data does not support them. | customer-360/dev-v1: balance.safe_to_show |
| Policy retrieval | Read the dispute policy from the documents of the bank | The policy stays in code. The sources disagree and we have no bank document. | policy sources; the plan for the measure is in metrics |
| Handoff routing | Send a handoff to an advisor with the right language and specialty | Not started. The demo has one advisor view. | REQ-0046 |
- No secrets and no data in the repository. The repository is public. Credentials go in
.env(gitignored). Teammates share them by direct message. - English, in ASD-STE100. Code and documents are in English, in simplified technical English. The assistant answers in Spanish and Portuguese.
- Label what is real. Each number is real, mock, synthetic, team-generated, simulation or projection. A simulation is never shown as a production measurement.
| For | Read |
|---|---|
| A person who contributes | CONTRIBUTING.md |
| An AI agent | AGENTS.md |
| The record of the first phase | team/README.md |
| The change history | CHANGELOG.md |
