TI One Voice — Data Analysis API: sample project

A complete, runnable example of how to consume the TI One Voice Data Analysis API: a small client library, nine annotated example scripts, and an exporter that downloads every published data set as both JSON and CSV.

It is written to be read. Every non-obvious line explains why the API behaves the way it does, because most of this API's design is a privacy decision with consequences for your code.

git clone <this repo> && cd onevoice_data_analysis_api_access
python3 -m pip install -r requirements.txt
cp .env.example .env                 # contains a working demo key

python3 examples/01_check_key.py     # is my key alive?
python3 -m onevoice_da.export        # download everything → ./data

---

All documentation

Start wherever your question is. Everything below is in this repository.

In the repository root

Document	What it covers
README.md	This file — quickstart, project layout, what the exporter produces, ground rules for using the data
data-analysis-api.md	The official API specification this project was built against: endpoints, authentication, the anonymization guarantee, pagination, rate limits, errors

In docs/

Document	What it covers
data_analysis_functional_api.md	Start here if you know the app, not the API. Every Data Analysis screen and chart explained in plain language, mapped to the API call that returns it and to the exact JSON + CSV files in data/. Includes the statistics primer, the acronym glossary, and where the API and the screens disagree
API-GUIDE.md	Endpoint-by-endpoint reference with real captured responses, the metric catalogue by theme, timestamps, pagination, rate limits, withheld payloads
CSV-SCHEMA.md	How arbitrary JSON becomes CSV: the flattening rules, the output tree, how to join child tables, Excel encodings, what CSV cannot carry
ERRORS.md	Every status code, its cause, its remedy, the exception it maps to, and real captures of each — plus what this client retries and what it does not
ANONYMIZATION.md	What the de-identification guarantee covers, what it deliberately does not, how to read aliases, and your obligations once the data is on your disk
FIELD-NOTES.md	Where the live deployment differs from the published spec. Read before debugging something that looks broken
TI_One_Voice_Data_Analysis_for_End_Users.pdf	The 209-page end-user manual for the app itself (source for the functional guide above)

---

What you get

onevoice_da/	The client. Auth, retries, cursor pagination, withheld-payload detection, JSON→CSV flattening, the exporter. This is the part to lift into your own project.
examples/	Nine standalone scripts, one idea each, from "does my key work" to "download the whole corpus".
docs/	Detailed documentation — endpoint reference, CSV schema, error catalogue, anonymization notes, and field notes on where this live deployment differs from the published spec.
tests/	Offline tests for the JSON→CSV rules. No network, no key: python3 tests/test_flatten.py.
data/	Export output (git-ignored). Re-create it any time with one command.

The API in one paragraph

Five GET endpoints under https://one.witysk.org/api/da/v1, authenticated with an X-API-Key header. They serve the corpus-level metrics behind the app's Data Analysis screens: 41 metric slugs plus a rolling window of analysis-cycle records. Nothing writes, nothing exports raw records, and every response passes through one de-identification chokepoint before any bytes leave the server — so people appear only as stable opaque aliases like user_DF5343A9, and a payload that cannot be proven anonymous is withheld whole rather than served partially clean.

Quickstart

Requirements: Python 3.9+ and requests (the only dependency).

python3 -m pip install -r requirements.txt
cp .env.example .env

The key is read from ONEVOICE_API_KEY — as an environment variable, or from .env. .env.example ships with a working demo key so the project runs the moment you clone it:

export ONEVOICE_API_KEY="ovda_…"     # alternative to .env

About the demo key. It is read-only (data_analysis:read), rate-limited to 1000 requests/hour like any other key, and shared with everyone reading this repo — so treat its quota as shared too. For real work, mint your own on the Data Analysis page in the app. Keys are shown exactly once at creation; only a SHA-256 digest is stored, so a lost key is re-minted, never recovered. .gitignore excludes .env for that reason.

Download everything

python3 -m onevoice_da.export                        # JSON + CSV → ./data
python3 -m onevoice_da.export --format csv           # CSV only
python3 -m onevoice_da.export --csv-bom              # Excel-friendly UTF-8
python3 -m onevoice_da.export --metrics eqs-dashboard onset-cohorts --skip-cycles
python3 -m onevoice_da.export --out ~/dump --sleep 0.5 --quiet

One run costs about 53 requests of your 1000/hour and takes a few minutes, most of it waiting on one metric that times out server-side. A full run on 2026-08-13 produced 247 files (198 CSV, 49 JSON), 5.7 MB, 15,063 CSV rows — 45 resources served, 4 withheld, 1 errored. It produces:

data/
├── manifest.json                 every resource attempted, its outcome, every file written
├── json/
│   ├── meta.json                 GET /meta
│   ├── metrics-catalogue.json    GET /metrics, all pages merged
│   ├── metrics/<slug>.json       GET /metrics/{slug}  ×41, exactly as served
│   ├── cycles.json               GET /cycles, all pages + window/withheld counts
│   └── cycles/cycle-<id>.json    GET /cycles/{id}, narrative included
└── csv/
    ├── _index.csv                one row per resource: ok / withheld / error
    ├── meta._scalars.csv         meta.metric-slugs.csv, metrics-catalogue.csv
    ├── cycles.csv                cycle-narratives.csv
    ├── metrics/<slug>/           _scalars.csv + one CSV per table found in the payload
    └── cycles/cycle-<id>/        _scalars.csv + one CSV per structured_data table

The JSON is the API's own bytes, untouched. The CSV is derived from it, so anything surprising in a CSV can be checked against the JSON beside it.

How JSON becomes CSV

The metric envelope is identical for all 41 slugs, but the shape inside data is not — flat scalars, nested objects, record lists, records containing record lists, and one 138×138 matrix. Rather than 41 hand-written mappers, the exporter derives tables from the data:

In the JSON	In the CSV
list of objects	its own file, one row per element
object nested in a row	dotted columns — cache_meta.cached
list of scalars in a row	one cell, joined with |
list of objects in a row	child file parent[].child.csv with _parent_index / _parent_key to join back
list of lists of scalars	a matrix file, labelled from a sibling labels list
everything else	_scalars.csv, long format (key,value)

A metric added to the catalogue tomorrow therefore exports correctly today. Full rules and worked examples: docs/CSV-SCHEMA.md.

The examples

Run them in order; each is standalone and self-explaining.

Script	Teaches
01_check_key.py	GET /meta — key validity, expiry, live quota, schema pinning
02_list_metrics.py	GET /metrics — keyset cursors, written out by hand then via the helper
03_fetch_metric.py	GET /metrics/{slug} — the shared envelope, per-metric shapes, freshness
04_list_cycles.py	GET /cycles — the 50-cycle window, withheld_count, naive-UTC timestamps
05_fetch_cycle.py	GET /cycles/{id} — narratives, structured_data, stable aliases
06_handle_errors.py	Every failure mode, provoked on purpose: 400, 401, 404, 422, 503, withheld
07_metric_to_csv.py	The flattening rules on a single metric
08_download_everything.py	Driving the exporter from Python and reading the outcomes
09_cache_and_poll.py	Polling etiquette — one cheap call decides whether anything is stale

Using the client in your own code

from onevoice_da import OneVoiceClient, WithheldPayload, NotFoundError

with OneVoiceClient() as client:                 # key from env or .env
    meta = client.meta()                         # cheapest liveness check

    for entry in client.iter_metric_catalogue(): # follows cursors for you
        try:
            payload = client.metric(entry["metric"])
        except WithheldPayload as exc:           # HTTP 200 + available:false
            print(f"{entry['metric']}: withheld — {exc.reason}")
            continue
        except NotFoundError:
            continue
        print(entry["metric"], payload["generated_at"])

    for cycle in client.iter_cycles():
        print(cycle["id"], cycle["layer"], cycle["status"])

What the client handles so you do not have to:

• Auth — header only. Cookies, bearer tokens and query-string keys are rejected by the API so a key cannot leak through a browser history or an access log. The key is redacted in every repr and log line.
• Schema pinning — schema_version / X-Schema-Version must be 1.0, or it raises. Fail loudly rather than parse an unknown shape.
• 429 — honours Retry-After exactly. Note the second, separate limit: 20 failed auth attempts on one credential in 5 minutes also earns a 429.
• 503 and timeouts — exponential backoff with jitter. Some metrics are computed on demand and one currently never finishes; that is one missing metric, not a dead API.
• Cursors — followed to the end, never parsed or constructed. They are opaque and bound to the collection that minted them.
• Withheld payloads — detected at both levels they occur (top-level, and nested inside data) and raised as WithheldPayload, so a refusal can never be mistaken for empty data.

Documentation

The full index is at the top of this file. Two entry points cover most questions:

• "I know this chart from the app — where is it on the API?" → docs/data_analysis_functional_api.md
• "How do I call this endpoint correctly?" → docs/API-GUIDE.md

Ground rules for anyone using this data

The corpus describes people who report being harmed. The API's design keeps them unidentifiable; your analysis has to keep them that way too.

• Do not attempt re-identification, and do not combine this data with other sources for that purpose. Keys are issued per person and revoked on misuse.
• There is no k-anonymity threshold and no differential-privacy noise on this surface. Small counts come back as they are. A cell of two people is a small cell — do not publish it in a way that narrows down who they are.
• Aliases are stable, which is a feature and a risk. user_DF5343A9 is the same person across every endpoint and over time, so you can follow cohorts longitudinally — and so can anyone you hand a file to.
• perpetrator-registry is the one metric that names third parties, on purpose; the member-identifying columns around them are dropped instead.
• Cite generated_at, not your download date. Figures move when a nightly cycle refreshes them, and your key cannot force a recomputation.

Licence and status

Sample code, provided as-is, for learning how to call this API. It is not an official client library. The API itself is documented in data-analysis-api.md; where this project and that document disagree about observed behaviour, docs/FIELD-NOTES.md records the difference.