A multi-project ML platform template: shared libraries and governance under projects of different kinds.
51 done · 3 partial · 11 absent of 65 tracked components — see implementation status, where every row names the command that proves it. Proven in CI: 39 at L1 (contract) · 12 at L2 (component). Evidence that needs a cluster exists for 5 and is not run in CI. L4 (cloud): 0 — nothing has been deployed to a cloud.
| Project | State | Evidence layer |
|---|---|---|
demand-forecast |
✅ built | L1 |
store-assistant |
✅ built | L1 |
rag-assistant |
✅ built | L1 |
credit-risk |
⬜ planned (Phase 4) | — |
doc-intelligence |
⬜ planned (Phase 5) | — |
agent-ops |
⬜ planned (Phase 6) | — |
Last independent audit: QA-4 round 19, 2026-10-22, at 947c14d — report.
A monorepo where several ML systems share one substrate — data contracts,
features, evaluation, orchestration, serving, observability, governance and
agentic tooling — and each system is a vertical under projects/. The
projects differ in kind (time series, retrieval, agents), not merely in
dataset, because a substrate that serves one shape of problem has not been
shown to be a substrate. Multi-cloud by design (GCP and AWS), with every
non-trivial decision recorded as an ADR.
Its central claim is falsifiable and tested: a second project reuses at least
three shared libraries with no fork (ADR-000,
criterion C1, checked by scripts/check_library_reuse.py). If that fails, this
is a monorepo of unrelated projects.
For teams that run more than one ML system and want them to share what should be shared — and adopters who want a worked example of governed MLOps across kinds. Not for a single tabular service on Kubernetes: that is ml-service-template, smaller and readable in an afternoon, which this repository consumes for its serving layer (ADR-003) rather than replacing. docs/ADOPTION.md is the full decision guide.
git clone --branch main https://github.com/DuqueOM/ml-platform.git # the development line
cd ml-platform
bash scripts/bootstrap.sh # uv, the pinned Python, the locked workspace
uv run pytest projects/demand-forecast -q # one vertical's suite
uv run copier copy --vcs-ref HEAD --trust . projects/my-project # a vertical, with the locked Copiermake verify then runs every gate CI runs, on your machine.
QUICK_START.md continues to a pod answering on a local kind
cluster (needs Docker); docs/PROGRESSION.md takes it
from there to a second vertical and to a cloud.
| Capability | Where it is, and what shows it is real |
|---|---|
| Shared libraries — conformal prediction, data contracts, point-in-time features, retrieval evaluation | libs/; a test fails any import that points upward (ADR-001), and the reuse gate fails a fork |
| Projects of different kinds — demand forecasting on an Iceberg lakehouse, retrieval-augmented generation on pgvector, an agent with a deterministic policy gate | projects/, each with its README; demand-forecast and store-assistant also carry evaluation gates and a model card — rag-assistant's are recorded deviations from the project contract (P6, P7) |
A project generator whose vertical meets six of the seven contract requirements as generated — the seventh, its evaluation thresholds, is left TODO for the adopter to decide |
templates/project/ (Copier); docs/PROJECT_CONTRACT.md, checked by tests/test_project_contract.py |
| Orchestration — Airflow for the business flow, KFP for ML compute, judging promotion by one shared rule | orchestration/dags, orchestration/pipelines; tests/test_promotion_agreement.py |
| Serving from a scaffold generated by ml-service-template, with the model artifact loaded under the image's own requirements in CI | services/; gate P18 in quality gates; ADR-008 |
| Infrastructure as code — Terraform for GCP and AWS, Kustomize overlays per cloud and environment, an ArgoCD ApplicationSet, default-deny NetworkPolicies | platform/; rendered and tested in CI, never applied to a cloud — see the status above |
| A local stack on kind — Postgres with pgvector, an S3-compatible store, OpenTelemetry, Jaeger, Prometheus, Grafana | platform/local/README.md; make local-up, make local-verify |
| Supply chain and security — gitleaks over the full history, Trivy over the whole lock and over the serving image's resolved requirements (no image is built, so none is scanned), Bandit, commit-pinned actions, scanner baselines with owners and expiry | SECURITY.md, .security-baselines/ |
| Agentic governance — one canonical set of rules, skills and workflows, rendered for Claude, Cursor, Codex and Devin, under an AUTO / CONSULT / STOP protocol | AGENTS.md, ADR-005 |
libs/ ml-core · data-contracts · feature-defs · llm-core · serving-core
projects/ one deployable ML system each, generated from templates/project/
orchestration/ Airflow DAGs + KFP pipelines
services/ serving, generated from ml-service-template (not edited here)
platform/ terraform · kubernetes · gitops · policies · observability · local
agentic/ rules · skills · workflows (canonical; per-tool files are pointers)
docs/ decisions · architecture · governance · datasets · runbooks
tests/ repository-level invariants
Dependencies point one way: libs/ never imports projects/, and projects
never import each other — enforced by a test, not by review
(ADR-001). The
technical plan gives each phase its
executable acceptance criteria; the decisions index
lists every ADR with the alternatives it rejected.
- Every published quality claim maps to a command that can fail a build. A metric that is reported but cannot fail is decoration. Quality gates is that mapping, and a gate checks it for gaps.
- Each claim carries the layer its evidence reaches — L1 contract (tests), L2 component (the thing executed), L3 cluster (kind), L4 cloud. The layer is derived from the command that ran, never declared: implementation status.
- An independent audit, enforced by CI. QA-4 is run in a separate session
against a fixed commit; when too many commits land after it, the
documentation gate fails until the next one is recorded. The
procedure, the reports under
docs/governance/qa4/, and the remediation work order. - Gates are watched failing. A mutation catalogue (
tests/mutations.yaml) breaks every implemented gate in quality gates on purpose and expects it to go red; a test fails an implemented gate that no entry breaks. - Measurements carry their method, and wrong claims are corrected in place with a date rather than edited away.
| Purpose | Start with |
|---|---|
| Tutorials — learn by doing | QUICK_START.md, docs/PROGRESSION.md |
| How-to guides — get a task done | RUNBOOK.md, docs/EXPORTING.md, docs/RELEASING.md, docs/environment-promotion.md, docs/runbooks/ |
| Reference — look something up | Quality gates, project contract, implementation status, technology inventory, dataset register, CHANGELOG.md |
| Explanation — understand why | Decisions (ADRs), technical plan, adoption guide, compliance mapping |
| For agents | AGENTS.md, llms.txt |
This README follows the README standard it shares with ml-service-template.
| Repository | Role |
|---|---|
| ML-MLOps-Portfolio | End-to-end tabular ML services where these patterns were first proven; ml-service-template was extracted from it |
| ml-service-template | A governed Copier template for one tabular ML service on Kubernetes, on GKE and EKS |
| ml-platform | A multi-project ML platform template: shared libraries, orchestration and governance under projects of different kinds. It consumes ml-service-template for serving, and where the two disagree on serving, containers, manifests or supply chain, ml-service-template is authoritative |
| agent-local | A local multi-tier LLM agent core with a deterministic policy gate. Its core moved into ml-platform, which is now authoritative; agent-local is a one-way export of it |
- Contributing: CONTRIBUTING.md — the pre-push checks, the derived documents, and the AUTO / CONSULT / STOP protocol every change is classed under. Code of conduct.
- Security: report a vulnerability privately as SECURITY.md describes — never in a public issue.
- Support: open a GitHub issue. Maintained by @DuqueOM (CODEOWNERS).