[dmealing]

What Changed

• Adds examples/head-to-head/: one small app (a TypeScript API plus a Python prompt-rendering worker on Postgres) set up twice. baseline/ is a DIY stack: TypeSpec to OpenAPI, openapi-typescript, datamodel-code-generator, Atlas, a Jinja StrictUndefined render test, git diff --exit-code and oasdiff against the released contract. metaobjects/ uses a single MetaObjects model with meta gen, metaobjects gen, meta migrate and meta verify. Each setup has its own ci.sh, smoke probe and live-drift script.
• Adds a harness (harness/run.py, harness/lines.py, run.sh) that applies six scripted drift scenarios to a throwaway copy of each setup (S1 rename a field, S2 add a required field, S3 change a prompt variable, S4 skip regeneration, S5 hand-edit the live DB, S6 change a field on the partner API), plus a no-change control S0. It records which checks go red and the lines each setup needs, and writes SCORECARD.md.
• Publishes SCORECARD.md and a README. Both state that the result is self-measured by the MetaObjects maintainers, who chose the baseline's tools. The scorecard includes the row MetaObjects loses: partner-breaking changes, which only the baseline's oasdiff flags. The root README.md and docs/README.md link to the demo.

Findings and run notes

• no-mistakes runs: 1 (one review gate; seven findings fixed in a single response, two inline README fixes by the pipeline).
• Result as measured (SCORECARD.md): layer checks go red on 4 of 6 scenarios on both sides (S1, S3, S4, S5). Only the baseline has a partner-contract check (oasdiff), and it flags S1, S2 and S6. MetaObjects has no such check, so that row is the one it loses. The baseline did better on that row; MetaObjects did better on none of the layer rows. Self-measured by the MetaObjects maintainers; the scorecard and README say so and list what is not measured.
• MetaObjects behaviour the scenarios exposed (nothing tuned to avoid it):
  • The runtime prompt renderer drops an unknown variable silently; only meta verify / metaobjects verify report ERR_VAR_NOT_ON_PAYLOAD (S3). The baseline's StrictUndefined fails at render time.
  • meta gen prints a "no prompt generator is wired" warning on every run in a model that declares a template.prompt and renders it only from Python.
  • meta migrate warns when a required column is added to a table that may have rows (S2, S6); the migration is still written.
• Fairness notes: the baseline's handlers use raw SQL (invisible to tsc for column changes) and do not validate request bodies at run time; MetaObjects' handlers validate with the generated Zod schema. Both are listed under known limits in the README.
• Out of scope, as briefed: the site, any link from it to the scorecard, and any release. The pipeline ran with local CI; the hosted checks on this PR are the CI evidence.

Validation

Risk Assessment

✅ Low: Docs-and-example-only change under examples/head-to-head; prior-round fixes check out (line counts recomputed to 6 files/425 and 9 files/376, flags removed, unknown ids rejected, S5 control added, INT8 parser added) and no new defects found.

Testing

The baseline ts-fast/ts-unit command had already passed. I then drove the real harness against Docker Postgres 16, Atlas, oasdiff and the published npm and PyPI packages. A full run completed (exit 0, 123 s). Its regenerated scorecard differs from the committed one by one line, the run duration (224 s committed). S0 is green on both sides. S6 shows the baseline flagging the partner break while MetaObjects reports "no such check". The scorecard carries the self-measured disclaimer. The S5 control runs before the ALTER. Subset runs print the table without rewriting SCORECARD.md. Unknown ids and the removed --json/--out flags are rejected. I restored SCORECARD.md after the full run and left no stray containers. I did not run a separate repo-wide regression suite beyond the configured baseline, because the change is an isolated examples directory.

• Live validation: ✅ go - 7 of 7 scenarios driven live against the product

Scenario	Result	Live	Evidence
Full ./run.sh produces a scorecard for both setups across S0–S6, matching the committed one	✅ pass	live	full.txt; SCORECARD.regenerated.md differs from the committed SCORECARD.md only in the duration line (123 s vs 224 s)
Control S0 is green on both setups (no false alarm)	✅ pass	live	subset.txt and full.txt: S0 'no false alarm' for baseline and metaobjects
Rename a field (S1): the checks that go red differ per setup	✅ pass	live	baseline red on tsc, mypy and prompt-render; MetaObjects red on tsc
Edit the DB by hand (S5): live-drift catches it on both setups, and the control check runs first	✅ pass	live	subset.txt: S5 red on live-drift for both; the S0/S5 text states the control
Partner break (S6): the baseline flags it and MetaObjects has no such check; the scorecard states MetaObjects loses this row	✅ pass	live	subset.txt: S6 baseline 'partner break flagged', MetaObjects 'all green'; partner table shows 'no such check'
Scorecard says it is self-measured	✅ pass	live	The first line of the generated scorecard is the 'Self-measured' note
Adversarial: a subset run does not overwrite SCORECARD.md, and an unknown scenario id or the removed --json flag is rejected	✅ pass	live	git status clean after ./run.sh S1 S5 S6; './run.sh s1' printed 'unknown scenario id(s): s1'; './run.sh --json x' printed 'unrecognized arguments: --json'

Evidence: Subset run S1 S5 S6 transcript and scorecard

[h2h] Postgres postgres:16 up as h2h-pg-fd066191 on 127.0.0.1:48947
[h2h] S0 Control: no change / baseline
[h2h]   -> no false alarm (gate exit 0, red: -, partner break: False, smoke ok) 13s
[h2h] S0 Control: no change / metaobjects
[h2h]   -> no false alarm (gate exit 0, red: -, partner break: False, smoke ok) 7s
[h2h] S1 Rename a field / baseline
[h2h]   -> layers red (gate exit 1, red: tsc, mypy, prompt-render, partner break: True, smoke FAIL) 15s
[h2h] S1 Rename a field / metaobjects
[h2h]   -> layers red (gate exit 1, red: tsc, partner break: False, smoke FAIL) 11s
[h2h] S5 Edit the database by hand / baseline
[h2h]   -> layers red (gate exit 1, red: live-drift, partner break: False, smoke -) 5s
[h2h] S5 Edit the database by hand / metaobjects
[h2h]   -> layers red (gate exit 1, red: live-drift, partner break: False, smoke -) 3s
[h2h] S6 Change a field on the partner API / baseline
[h2h]   -> partner break flagged (gate exit 1, red: -, partner break: True, smoke ok) 12s
[h2h] S6 Change a field on the partner API / metaobjects
[h2h]   -> all green (gate exit 0, red: -, partner break: False, smoke ok) 8s
# Head-to-head scorecard

> **Self-measured.** The MetaObjects maintainers wrote both setups and this harness. The baseline is built to be a competent do-it-yourself stack, but we chose its tools, its scenarios and its app, and a reader who disagrees with any of those should change them and rerun (`./run.sh`). Read [README.md](README.md) for what this does and does not measure.

Generated by `./run.sh` on 2026-10-31; 4 scenarios × 2 setups took 76 s. Every result cell below is computed from the exit codes of the checks that ran; none is typed by hand.

## Results

Each scenario is one change a developer makes where the setup keeps its source of truth, followed by that setup's own generators. The pull-request gate (for S5, the live-database check) is then run, and each check's exit code is recorded. The **probe** is a happy-path run (create a customer through the API, render its prompt in the worker) that is identical in both setups; it is informational and is not part of either gate. Two different questions follow, scored separately.

\### Do the layers stay consistent?

**red** means at least one check other than the partner-contract diff failed: the model, the generated code, the database, the prompt and the hand-written code disagreed and a check said so. **green** means every such check passed. It is not a claim that the change was safe; the probe column says whether the happy path still works.

| # | Scenario | DIY baseline | MetaObjects |
|---|---|---|---|
| S0 | **Control: no change** | ✅ **no false alarm** · exit 0 · probe passed | ✅ **no false alarm** · exit 0 · probe passed |
| S1 | **Rename a field** | 🔴 **red**: `tsc`, `mypy`, `prompt-render` · probe failed | 🔴 **red**: `tsc` · probe failed |
| S5 | **Edit the database by hand** | 🔴 **red**: `live-drift` | 🔴 **red**: `live-drift` |
| S6 | **Change a field on the partner API** | 🟢 **green** · probe passed | 🟢 **green** · probe passed |

\### Does a breaking change for partners get flagged?

The DIY baseline runs `oasdiff` against the OpenAPI document released to partners. MetaObjects 1.2.0 has no equivalent check, so it cannot flag a change that is legal everywhere in the project but breaks an outside caller. This is the row MetaObjects loses.

| # | Scenario | DIY baseline | MetaObjects |
|---|---|---|---|
| S0 | **Control: no change** | 🟢 none flagged | ❌ **no such check** |
| S1 | **Rename a field** | ✅ **breaking change flagged** | ❌ **no such check** |
| S5 | **Edit the database by hand** | n/a | n/a |
| S6 | **Change a field on the partner API** | ✅ **breaking change flagged** | ❌ **no such check** |

- **DIY baseline**: layer checks went red on 2 of 3 scenarios (S1, S5).
- **MetaObjects**: layer checks went red on 2 of 3 scenarios (S1, S5).
- **Partner-break check:** the baseline flagged 2 (S1, S6); MetaObjects has none.
- **Where the baseline did better:** the partner-contract check (S1, S6); layer checks red only on the baseline side: none.
- **Where MetaObjects did better:** layer checks red only on the MetaObjects side: none.

\### What each scenario does

- **S0 Control: no change.** Nothing is edited. Every check must pass, so a red cell elsewhere is not noise from the setup itself. (The live-database check has its own control: S5 runs it on the unedited database first and aborts the run unless it passes.)
- **S1 Rename a field.** `Customer.email` becomes `contactEmail` in the place each setup treats as its source of truth, then that setup's own generators run. Baseline: the TypeSpec model is edited, including the payload's field list, so the payload field is renamed too; `./gen.sh` runs; `atlas/schema.sql`, the migrations, the prompt and the hand-written code are left as they were. MetaObjects: the field and the payload's `extends` are edited, so the payload field keeps the name `email`; `meta gen`, `metaobjects gen` and `meta migrate --rename-column customers.email=contact_email` run; the prompt and the hand-written code are left as they were.
- **S5 Edit the database by hand.** Someone runs `ALTER TABLE customers ADD COLUMN vip boolean NOT NULL DEFAULT false` on a live database. The check is each setup's live-database drift command, the one a team runs on a schedule or at deploy time. It is run first on the unedited database and must pass, so a red result after the edit is the edit, not a broken tool.
- **S6 Change a field on the partner API.** `name` becomes required on `POST /customers`: a request partners send today without it would be rejected. Generators run; the database follows where the setup derives it.

\### Which check said so

**DIY baseline**

| scenario | `generated-in-sync` | `api-compat` | `atlas-migrations` | `atlas-lint` | `tsc` | `mypy` | `prompt-render` | `live-drift` |
|---|---|---|---|---|---|---|---|---|
| S0 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | · |
| S1 | ✓ | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ | · |
| S5 | · | · | · | · | · | · | · | ✗ |
| S6 | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | · |

✓ passed, ✗ failed, · not run in that scenario. `generated-in-sync`: regenerate, then `git diff --exit-code`; `api-compat`: oasdiff against the released OpenAPI document; `atlas-migrations`: Atlas: migrations match `schema.sql`; `atlas-lint`: Atlas migration lint; `tsc`: TypeScript typecheck; `mypy`: mypy with the pydantic plugin; `prompt-render`: pytest render of the prompt with `StrictUndefined`. `live-drift`: `atlas schema diff` of the live database against the migrations, wrapped to exit 1 on a non-empty plan.

**MetaObjects**

| scenario | `meta-verify` | `metaobjects-verify` | `meta-verify-db` | `tsc` | `mypy` | `live-drift` |
|---|---|---|---|---|---|---|
| S0 | ✓ | ✓ | ✓ | ✓ | ✓ | · |
| S1 | ✓ | ✓ | ✓ | ✗ | ✓ | · |
| S5 | · | · | · | · | · | ✗ |
| S6 | ✓ | ✓ | ✓ | ✓ | ✓ | · |

✓ passed, ✗ failed, · not run in that scenario. `meta-verify`: `meta verify` (codegen, templates, requirement links); `metaobjects-verify`: `metaobjects verify` (Python codegen and templates); `meta-verify-db`: CI database from the migrations, then `meta verify --db`; `tsc`: TypeScript typecheck; `mypy`: mypy with the pydantic plugin. `live-drift`: `meta verify --db` against the live database.

\### What the failing checks printed

The informative lines of each failing check's log, picked by a per-check extractor in `harness/run.py` (paths removed); full logs are written to `.ci-logs/` in the scenario's working directory (`--keep DIR`). *authoring* is what the setup's own generators said while the developer made the change, before CI.

- **S1, DIY baseline:**
  - `api-compat`: oasdiff: 5 changes: 4 error, 1 warning, 0 info; first: [response-required-property-removed] API GET /customers: removed the required property `items/email` from the response with the `200` status
  - `tsc`: api/src/app.ts(34,13): error TS2339: Property 'email' does not exist on type '{ contactEmail: string; name?: string | undefined; plan: "free" | "pro"; }'.
  - `mypy`: tests/test_prompts.py:7: error: Unexpected keyword argument "email" for "WelcomePayload" [call-arg]
  - `prompt-render`: pydantic_core._pydantic_core.ValidationError: 1 validation error for WelcomePayload contactEmail
  - smoke: pydantic_core._pydantic_core.ValidationError: 1 validation error for WelcomePayload contactEmail Field required [type=missing, input_value={'name': 'Ada Lovelace', ...lan': <Plan.pro: 'pro'>}, input_…
- **S1, MetaObjects:**
  - `tsc`: api/src/app.ts(16,67): error TS2339: Property 'email' does not exist on type 'PgTableWithColumns<{ name: "customers"; schema: undefined; columns: { id: PgColumn<{ name: "id"; tableName: "customers"; …
  - smoke: Error: POST /customers -> 400 {"error":"invalid_request","issues":[{"expected":"string","code":"invalid_type","path":["contactEmail"],"message":"Invalid input: expected string, received undefined"}]}
  - authoring: `npx meta gen` printed: warnings[1]: "1 declared template.prompt (WelcomePrompt) generated no prompt code: no prompt generator is wired, so there is no render function to send one and nothing read a reply. Add promptRender()
- **S5, DIY baseline:**
  - `live-drift`: atlas would change the live database with: ALTER TABLE "customers" DROP COLUMN "vip";
- **S5, MetaObjects:**
  - `live-drift`: meta verify — schema drift vs postgresql://postgres:***@127.0.0.1:<port>/me_s5_live (1 change(s)):; - column customers.vip
- **S6, DIY baseline:**
  - `api-compat`: oasdiff: 1 changes: 1 error, 0 warning, 0 info; first: [request-property-became-required] API POST /customers: the request property `name` became required
- **S6, MetaObjects:**
  - authoring: `npx meta gen` printed: warnings[1]: "1 declared template.prompt (WelcomePrompt) generated no prompt code: no prompt generator is wired, so there is no render function to send one and nothing read a reply. Add promptRender()
  - authoring: `npx meta migrate --slug name-required --allow nullable-to-not-null` printed: warnings[1]: customers.name becomes NOT NULL — fails if any row holds NULL. Backfill it first.

## Lines each setup needs

One rule for both: every committed file in the setup's directory counts, minus lock files, machine-generated output (listed below), the smoke probe (identical in both) and `.gitignore`. Blank lines and comment-only lines do not count. The numbers are from the files in this repository at the commit that generated this page.

| Category | DIY baseline | MetaObjects |
|---|---:|---:|
| Model and schema (the sources of truth) | 43 | 23 |
| Tool configuration | 38 | 38 |
| Checks and glue scripts | 54 | 21 |
| Hand-written application code and prompt | 64 | 56 |
| Dependency manifests | 25 | 25 |
| **Model + configuration + checks** | **135** | **82** |
| **All counted lines** | **224** | **163** |

Not counted, machine-generated: baseline 6 files / 425 lines (`generated/`, `atlas/migrations/`, the released OpenAPI copy); MetaObjects 9 files / 376 lines (`api/src/generated/`, `worker/generated/`, `.metaobjects/migrations/`, `.metaobjects/.gen-state/`).

\<details><summary>Per-file counts</summary>

**DIY baseline**

- `atlas/schema.sql` (model): 7
- `spec/main.tsp` (model): 36
- `api/tsconfig.json` (config): 12
- `atlas.hcl` (config): 10
- `mypy.ini` (config): 8
- `tspconfig.yaml` (config): 8
- `atlas.sh` (checks): 2
- `ci.sh` (checks): 28
- `db-apply.sh` (checks): 4
- `gen.sh` (checks): 8
- `live-drift.sh` (checks): 7
- `tests/__init__.py` (checks): 0
- `tests/test_prompts.py` (checks): 5
- `api/src/app.ts` (app): 35
- `prompts/welcome/greeting.jinja` (app): 1
- `worker/__init__.py` (app): 0
- `worker/main.py` (app): 18
- `worker/prompts.py` (app): 10
- `package.json` (manifest): 19
- `requirements.in` (manifest): 6

**MetaObjects**

- `metaobjects/meta.crm.yaml` (model): 23
- `.metaobjects/config.json` (config): 5
- `api/tsconfig.json` (config): 12
- `metaobjects.config.ts` (config): 7
- `metaobjects.config.yaml` (config): 6
- `mypy.ini` (config): 8
- `ci.sh` (checks): 15
- `db-apply.sh` (checks): 3
- `live-drift.sh` (checks): 3
- `api/src/app.ts` (app): 28
- `prompts/welcome/greeting.jinja` (app): 1
- `worker/__init__.py` (app): 0
- `worker/main.py` (app): 18
- `worker/prompts.py` (app): 9
- `package.json` (manifest): 21
- `requirements.in` (manifest): 4

</details>

## Versions

| Scope | Tool | Version |
|---|---|---|
| both | Node.js | v24.22.1 |
| both | npm | 11.19.1 |
| both | Python (venv) | 3.12.12 |
| both | uv | 0.9.18 |
| both | Docker | 29.1.3 |
| both | PostgreSQL | 16.15 (Debian 16.15-1.pgdg13+2) (postgres:16) |
| both | TypeScript | 5.9.3 |
| both | Fastify | 5.12.7 |
| both | mypy | 2.4.1 |
| both | pydantic | 2.14.2 |
| baseline | TypeSpec compiler | 1.17.0 |
| baseline | @typespec/openapi3 | 1.17.0 |
| baseline | openapi-typescript | 7.13.0 |
| baseline | datamodel-code-generator | 0.87.0 |
| baseline | Atlas (community) | arigaio/atlas:1.3.6-community (atlas community version v1.3.6) |
| baseline | oasdiff | tufin/oasdiff:v1.35.0 (oasdiff version v1.35.0) |
| baseline | Jinja2 | 3.1.6 |
| baseline | pytest | 9.2.0 |
| metaobjects | @metaobjectsdev/cli | 1.2.0 |
| metaobjects | @metaobjectsdev/codegen-ts | 1.2.0 |
| metaobjects | metaobjects (PyPI) | 1.2.0 |
| metaobjects | Drizzle ORM | 0.45.4 |
| metaobjects | zod | 4.6.7 |

Platform: Linux x86_64.

rc=0

Evidence: Full run transcript

[h2h] Postgres postgres:16 up as h2h-pg-84c565cc on 127.0.0.1:48948
[h2h] S0 Control: no change / baseline
[h2h]   -> no false alarm (gate exit 0, red: -, partner break: False, smoke ok) 12s
[h2h] S0 Control: no change / metaobjects
[h2h]   -> no false alarm (gate exit 0, red: -, partner break: False, smoke ok) 6s
[h2h] S1 Rename a field / baseline
[h2h]   -> layers red (gate exit 1, red: tsc, mypy, prompt-render, partner break: True, smoke FAIL) 13s
[h2h] S1 Rename a field / metaobjects
[h2h]   -> layers red (gate exit 1, red: tsc, partner break: False, smoke FAIL) 7s
[h2h] S2 Add a required field / baseline
[h2h]   -> partner break flagged (gate exit 1, red: -, partner break: True, smoke ok) 10s
[h2h] S2 Add a required field / metaobjects
[h2h]   -> all green (gate exit 0, red: -, partner break: False, smoke FAIL) 8s
[h2h] S3 Change a prompt variable / baseline
[h2h]   -> layers red (gate exit 1, red: prompt-render, partner break: False, smoke FAIL) 12s
[h2h] S3 Change a prompt variable / metaobjects
[h2h]   -> layers red (gate exit 1, red: meta-verify, metaobjects-verify, partner break: False, smoke FAIL) 7s
[h2h] S4 Skip a regen / baseline
[h2h]   -> layers red (gate exit 1, red: generated-in-sync, partner break: False, smoke ok) 9s
[h2h] S4 Skip a regen / metaobjects
[h2h]   -> layers red (gate exit 1, red: meta-verify, metaobjects-verify, meta-verify-db, partner break: False, smoke ok) 7s
[h2h] S5 Edit the database by hand / baseline
[h2h]   -> layers red (gate exit 1, red: live-drift, partner break: False, smoke -) 4s
[h2h] S5 Edit the database by hand / metaobjects
[h2h]   -> layers red (gate exit 1, red: live-drift, partner break: False, smoke -) 3s
[h2h] S6 Change a field on the partner API / baseline
[h2h]   -> partner break flagged (gate exit 1, red: -, partner break: True, smoke ok) 12s
[h2h] S6 Change a field on the partner API / metaobjects
[h2h]   -> all green (gate exit 0, red: -, partner break: False, smoke ok) 8s
[h2h] wrote SCORECARD.md

Evidence: Regenerated SCORECARD.md from the full run

# Head-to-head scorecard

> **Self-measured.** The MetaObjects maintainers wrote both setups and this harness. The baseline is built to be a competent do-it-yourself stack, but we chose its tools, its scenarios and its app, and a reader who disagrees with any of those should change them and rerun (`./run.sh`). Read [README.md](README.md) for what this does and does not measure.

Generated by `./run.sh` on 2026-10-31; 7 scenarios × 2 setups took 123 s. Every result cell below is computed from the exit codes of the checks that ran; none is typed by hand.

## Results

Each scenario is one change a developer makes where the setup keeps its source of truth, followed by that setup's own generators. The pull-request gate (for S5, the live-database check) is then run, and each check's exit code is recorded. The **probe** is a happy-path run (create a customer through the API, render its prompt in the worker) that is identical in both setups; it is informational and is not part of either gate. Two different questions follow, scored separately.

\### Do the layers stay consistent?

**red** means at least one check other than the partner-contract diff failed: the model, the generated code, the database, the prompt and the hand-written code disagreed and a check said so. **green** means every such check passed. It is not a claim that the change was safe; the probe column says whether the happy path still works.

| # | Scenario | DIY baseline | MetaObjects |
|---|---|---|---|
| S0 | **Control: no change** | ✅ **no false alarm** · exit 0 · probe passed | ✅ **no false alarm** · exit 0 · probe passed |
| S1 | **Rename a field** | 🔴 **red**: `tsc`, `mypy`, `prompt-render` · probe failed | 🔴 **red**: `tsc` · probe failed |
| S2 | **Add a required field** | 🟢 **green** · probe passed | 🟢 **green** · probe failed |
| S3 | **Change a prompt variable** | 🔴 **red**: `prompt-render` · probe failed | 🔴 **red**: `meta-verify`, `metaobjects-verify` · probe failed |
| S4 | **Skip a regen** | 🔴 **red**: `generated-in-sync` · probe passed | 🔴 **red**: `meta-verify`, `metaobjects-verify`, `meta-verify-db` · probe passed |
| S5 | **Edit the database by hand** | 🔴 **red**: `live-drift` | 🔴 **red**: `live-drift` |
| S6 | **Change a field on the partner API** | 🟢 **green** · probe passed | 🟢 **green** · probe passed |

\### Does a breaking change for partners get flagged?

The DIY baseline runs `oasdiff` against the OpenAPI document released to partners. MetaObjects 1.2.0 has no equivalent check, so it cannot flag a change that is legal everywhere in the project but breaks an outside caller. This is the row MetaObjects loses.

| # | Scenario | DIY baseline | MetaObjects |
|---|---|---|---|
| S0 | **Control: no change** | 🟢 none flagged | ❌ **no such check** |
| S1 | **Rename a field** | ✅ **breaking change flagged** | ❌ **no such check** |
| S2 | **Add a required field** | ✅ **breaking change flagged** | ❌ **no such check** |
| S3 | **Change a prompt variable** | 🟢 none flagged | ❌ **no such check** |
| S4 | **Skip a regen** | 🟢 none flagged | ❌ **no such check** |
| S5 | **Edit the database by hand** | n/a | n/a |
| S6 | **Change a field on the partner API** | ✅ **breaking change flagged** | ❌ **no such check** |

- **DIY baseline**: layer checks went red on 4 of 6 scenarios (S1, S3, S4, S5).
- **MetaObjects**: layer checks went red on 4 of 6 scenarios (S1, S3, S4, S5).
- **Partner-break check:** the baseline flagged 3 (S1, S2, S6); MetaObjects has none.
- **Where the baseline did better:** the partner-contract check (S1, S2, S6); layer checks red only on the baseline side: none.
- **Where MetaObjects did better:** layer checks red only on the MetaObjects side: none.

\### What each scenario does

- **S0 Control: no change.** Nothing is edited. Every check must pass, so a red cell elsewhere is not noise from the setup itself. (The live-database check has its own control: S5 runs it on the unedited database first and aborts the run unless it passes.)
- **S1 Rename a field.** `Customer.email` becomes `contactEmail` in the place each setup treats as its source of truth, then that setup's own generators run. Baseline: the TypeSpec model is edited, including the payload's field list, so the payload field is renamed too; `./gen.sh` runs; `atlas/schema.sql`, the migrations, the prompt and the hand-written code are left as they were. MetaObjects: the field and the payload's `extends` are edited, so the payload field keeps the name `email`; `meta gen`, `metaobjects gen` and `meta migrate --rename-column customers.email=contact_email` run; the prompt and the hand-written code are left as they were.
- **S2 Add a required field.** `Customer` gains a required `country` (2 characters). Generators run; nothing else is touched.
- **S3 Change a prompt variable.** The prompt template is edited to use `{{ fullName }}`, which the payload does not have. Nothing is regenerated.
- **S4 Skip a regen.** An optional `nickname` field is added to the model and no generator is run, so the committed generated files are stale.
- **S5 Edit the database by hand.** Someone runs `ALTER TABLE customers ADD COLUMN vip boolean NOT NULL DEFAULT false` on a live database. The check is each setup's live-database drift command, the one a team runs on a schedule or at deploy time. It is run first on the unedited database and must pass, so a red result after the edit is the edit, not a broken tool.
- **S6 Change a field on the partner API.** `name` becomes required on `POST /customers`: a request partners send today without it would be rejected. Generators run; the database follows where the setup derives it.

\### Which check said so

**DIY baseline**

| scenario | `generated-in-sync` | `api-compat` | `atlas-migrations` | `atlas-lint` | `tsc` | `mypy` | `prompt-render` | `live-drift` |
|---|---|---|---|---|---|---|---|---|
| S0 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | · |
| S1 | ✓ | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ | · |
| S2 | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | · |
| S3 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | · |
| S4 | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | · |
| S5 | · | · | · | · | · | · | · | ✗ |
| S6 | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | · |

✓ passed, ✗ failed, · not run in that scenario. `generated-in-sync`: regenerate, then `git diff --exit-code`; `api-compat`: oasdiff against the released OpenAPI document; `atlas-migrations`: Atlas: migrations match `schema.sql`; `atlas-lint`: Atlas migration lint; `tsc`: TypeScript typecheck; `mypy`: mypy with the pydantic plugin; `prompt-render`: pytest render of the prompt with `StrictUndefined`. `live-drift`: `atlas schema diff` of the live database against the migrations, wrapped to exit 1 on a non-empty plan.

**MetaObjects**

| scenario | `meta-verify` | `metaobjects-verify` | `meta-verify-db` | `tsc` | `mypy` | `live-drift` |
|---|---|---|---|---|---|---|
| S0 | ✓ | ✓ | ✓ | ✓ | ✓ | · |
| S1 | ✓ | ✓ | ✓ | ✗ | ✓ | · |
| S2 | ✓ | ✓ | ✓ | ✓ | ✓ | · |
| S3 | ✗ | ✗ | ✓ | ✓ | ✓ | · |
| S4 | ✗ | ✗ | ✗ | ✓ | ✓ | · |
| S5 | · | · | · | · | · | ✗ |
| S6 | ✓ | ✓ | ✓ | ✓ | ✓ | · |

✓ passed, ✗ failed, · not run in that scenario. `meta-verify`: `meta verify` (codegen, templates, requirement links); `metaobjects-verify`: `metaobjects verify` (Python codegen and templates); `meta-verify-db`: CI database from the migrations, then `meta verify --db`; `tsc`: TypeScript typecheck; `mypy`: mypy with the pydantic plugin. `live-drift`: `meta verify --db` against the live database.

\### What the failing checks printed

The informative lines of each failing check's log, picked by a per-check extractor in `harness/run.py` (paths removed); full logs are written to `.ci-logs/` in the scenario's working directory (`--keep DIR`). *authoring* is what the setup's own generators said while the developer made the change, before CI.

- **S1, DIY baseline:**
  - `api-compat`: oasdiff: 5 changes: 4 error, 1 warning, 0 info; first: [response-required-property-removed] API GET /customers: removed the required property `items/email` from the response with the `200` status
  - `tsc`: api/src/app.ts(34,13): error TS2339: Property 'email' does not exist on type '{ contactEmail: string; name?: string | undefined; plan: "free" | "pro"; }'.
  - `mypy`: tests/test_prompts.py:7: error: Unexpected keyword argument "email" for "WelcomePayload" [call-arg]
  - `prompt-render`: pydantic_core._pydantic_core.ValidationError: 1 validation error for WelcomePayload contactEmail
  - smoke: pydantic_core._pydantic_core.ValidationError: 1 validation error for WelcomePayload contactEmail Field required [type=missing, input_value={'name': 'Ada Lovelace', ...lan': <Plan.pro: 'pro'>}, input_…
- **S1, MetaObjects:**
  - `tsc`: api/src/app.ts(16,67): error TS2339: Property 'email' does not exist on type 'PgTableWithColumns<{ name: "customers"; schema: undefined; columns: { id: PgColumn<{ name: "id"; tableName: "customers"; …
  - smoke: Error: POST /customers -> 400 {"error":"invalid_request","issues":[{"expected":"string","code":"invalid_type","path":["contactEmail"],"message":"Invalid input: expected string, received undefined"}]}
  - authoring: `npx meta gen` printed: warnings[1]: "1 declared template.prompt (WelcomePrompt) generated no prompt code: no prompt generator is wired, so there is no render function to send one and nothing read a reply. Add promptRender()
- **S2, DIY baseline:**
  - `api-compat`: oasdiff: 1 changes: 1 error, 0 warning, 0 info; first: [new-required-request-property] API POST /customers: added the new required request property `country`
- **S2, MetaObjects:**
  - smoke: Error: POST /customers -> 400 {"error":"invalid_request","issues":[{"expected":"string","code":"invalid_type","path":["country"],"message":"Invalid input: expected string, received undefined"}]}
  - authoring: `npx meta gen` printed: warnings[1]: "1 declared template.prompt (WelcomePrompt) generated no prompt code: no prompt generator is wired, so there is no render function to send one and nothing read a reply. Add promptRender()
  - authoring: `npx meta migrate --slug add-country` printed: warnings[1]: "customers.country is added NOT NULL with no default — fails if customers has rows. Give the field a @default, or add it optional, backfill, then make it required."
- **S3, DIY baseline:**
  - `prompt-render`: jinja2.exceptions.UndefinedError: 'fullName' is undefined
  - smoke: jinja2.exceptions.UndefinedError: 'fullName' is undefined
- **S3, MetaObjects:**
  - `meta-verify`: [WelcomePrompt] (prompt) ERR_VAR_NOT_ON_PAYLOAD: fullName (prompts/welcome/greeting.jinja:1); meta verify — 1 drift error(s) across 1 template(s).
  - `metaobjects-verify`: error: [WelcomePrompt] ref 'welcome/greeting' — ERR_VAR_NOT_ON_PAYLOAD: fullName
  - smoke: probe: the rendered prompt does not contain the customer's name: Write a short, friendly welcome note for <ada@example.test>, who just joined on the pro plan.
- **S4, DIY baseline:**
  - `generated-in-sync`: a fresh regeneration differs from the committed `generated/api.d.ts`, `generated/models.py`, `generated/openapi.yaml`
- **S4, MetaObjects:**
  - `meta-verify`: meta verify — codegen drift (1 file(s) differ from a fresh regen):; ~ api/src/generated/Customer.ts (committed content differs from a fresh regen)
  - `metaobjects-verify`: error: [app] generated code is out of sync with metadata. drifted: Customer.py
  - `meta-verify-db`: meta verify — schema drift vs postgresql://postgres:***@127.0.0.1:<port>/me_s4_ci (1 change(s)):; + column customers.nickname
- **S5, DIY baseline:**
  - `live-drift`: atlas would change the live database with: ALTER TABLE "customers" DROP COLUMN "vip";
- **S5, MetaObjects:**
  - `live-drift`: meta verify — schema drift vs postgresql://postgres:***@127.0.0.1:<port>/me_s5_live (1 change(s)):; - column customers.vip
- **S6, DIY baseline:**
  - `api-compat`: oasdiff: 1 changes: 1 error, 0 warning, 0 info; first: [request-property-became-required] API POST /customers: the request property `name` became required
- **S6, MetaObjects:**
  - authoring: `npx meta gen` printed: warnings[1]: "1 declared template.prompt (WelcomePrompt) generated no prompt code: no prompt generator is wired, so there is no render function to send one and nothing read a reply. Add promptRender()
  - authoring: `npx meta migrate --slug name-required --allow nullable-to-not-null` printed: warnings[1]: customers.name becomes NOT NULL — fails if any row holds NULL. Backfill it first.

## Lines each setup needs

One rule for both: every committed file in the setup's directory counts, minus lock files, machine-generated output (listed below), the smoke probe (identical in both) and `.gitignore`. Blank lines and comment-only lines do not count. The numbers are from the files in this repository at the commit that generated this page.

| Category | DIY baseline | MetaObjects |
|---|---:|---:|
| Model and schema (the sources of truth) | 43 | 23 |
| Tool configuration | 38 | 38 |
| Checks and glue scripts | 54 | 21 |
| Hand-written application code and prompt | 64 | 56 |
| Dependency manifests | 25 | 25 |
| **Model + configuration + checks** | **135** | **82** |
| **All counted lines** | **224** | **163** |

Not counted, machine-generated: baseline 6 files / 425 lines (`generated/`, `atlas/migrations/`, the released OpenAPI copy); MetaObjects 9 files / 376 lines (`api/src/generated/`, `worker/generated/`, `.metaobjects/migrations/`, `.metaobjects/.gen-state/`).

\<details><summary>Per-file counts</summary>

**DIY baseline**

- `atlas/schema.sql` (model): 7
- `spec/main.tsp` (model): 36
- `api/tsconfig.json` (config): 12
- `atlas.hcl` (config): 10
- `mypy.ini` (config): 8
- `tspconfig.yaml` (config): 8
- `atlas.sh` (checks): 2
- `ci.sh` (checks): 28
- `db-apply.sh` (checks): 4
- `gen.sh` (checks): 8
- `live-drift.sh` (checks): 7
- `tests/__init__.py` (checks): 0
- `tests/test_prompts.py` (checks): 5
- `api/src/app.ts` (app): 35
- `prompts/welcome/greeting.jinja` (app): 1
- `worker/__init__.py` (app): 0
- `worker/main.py` (app): 18
- `worker/prompts.py` (app): 10
- `package.json` (manifest): 19
- `requirements.in` (manifest): 6

**MetaObjects**

- `metaobjects/meta.crm.yaml` (model): 23
- `.metaobjects/config.json` (config): 5
- `api/tsconfig.json` (config): 12
- `metaobjects.config.ts` (config): 7
- `metaobjects.config.yaml` (config): 6
- `mypy.ini` (config): 8
- `ci.sh` (checks): 15
- `db-apply.sh` (checks): 3
- `live-drift.sh` (checks): 3
- `api/src/app.ts` (app): 28
- `prompts/welcome/greeting.jinja` (app): 1
- `worker/__init__.py` (app): 0
- `worker/main.py` (app): 18
- `worker/prompts.py` (app): 9
- `package.json` (manifest): 21
- `requirements.in` (manifest): 4

</details>

## Versions

| Scope | Tool | Version |
|---|---|---|
| both | Node.js | v24.22.1 |
| both | npm | 11.19.1 |
| both | Python (venv) | 3.12.12 |
| both | uv | 0.9.18 |
| both | Docker | 29.1.3 |
| both | PostgreSQL | 16.15 (Debian 16.15-1.pgdg13+2) (postgres:16) |
| both | TypeScript | 5.9.3 |
| both | Fastify | 5.12.7 |
| both | mypy | 2.4.1 |
| both | pydantic | 2.14.2 |
| baseline | TypeSpec compiler | 1.17.0 |
| baseline | @typespec/openapi3 | 1.17.0 |
| baseline | openapi-typescript | 7.13.0 |
| baseline | datamodel-code-generator | 0.87.0 |
| baseline | Atlas (community) | arigaio/atlas:1.3.6-community (atlas community version v1.3.6) |
| baseline | oasdiff | tufin/oasdiff:v1.35.0 (oasdiff version v1.35.0) |
| baseline | Jinja2 | 3.1.6 |
| baseline | pytest | 9.2.0 |
| metaobjects | @metaobjectsdev/cli | 1.2.0 |
| metaobjects | @metaobjectsdev/codegen-ts | 1.2.0 |
| metaobjects | metaobjects (PyPI) | 1.2.0 |
| metaobjects | Drizzle ORM | 0.45.4 |
| metaobjects | zod | 4.6.7 |

Platform: Linux x86_64.

Pipeline

Updates from git push no-mistakes

✅ **Intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 2 infos

• ⚠️ examples/head-to-head/harness/run.py:216 - The published S1 description is false for both setups. It says "The prompt payload, the database column and the hand-written code are left as they were." The baseline edit renames the payload field (PickProperties<Customer, "name" | "contactEmail" | "plan">, run.py:157). That rename is what turns mypy and prompt-render red. The MetaObjects edit changes the payload's extends (run.py:163) and renames the column through meta migrate --rename-column customers.email=contact_email (run.py:166). A reader therefore misreads why the baseline shows three reds against one: the MetaObjects payload keeps the name email, the baseline payload does not. Same claim at SCORECARD.md:48. README.md:49-51 says each scenario "leaves anything else alone", and README.md:63 sends readers to these edits. Fix: state per setup what S1 changes, in the Scenario text, then regenerate or edit SCORECARD.md.
• ⚠️ examples/head-to-head/harness/run.py:432 - live-drift is the only scored check with no clean control, and a tool failure counts as a catch. S0 is kind "pr" (run.py:214), so the S0 row shows · for live-drift (SCORECARD.md:61 and 75). The S0 text still says "Every check must pass, so a red cell elsewhere is not noise from the setup itself" (run.py:214, SCORECARD.md:47). Failure sequence: baseline/live-drift.sh:5-8 discards stderr under set -e, so any Atlas or Docker error exits non-zero with empty output. run.py:434-435 then records a red S5 with blank evidence. metaobjects/live-drift.sh:5 has the same exit-code path for a meta verify --db connection failure. The committed run's evidence names only vip, so the committed numbers look right, but the harness would publish the false red without complaint. Fix: run ./live-drift.sh on the freshly provisioned database before the ALTER (run.py:429), and raise SystemExit unless it exits 0. Then state that control in the S0 or S5 text.
• ⚠️ examples/head-to-head/harness/lines.py:107 - The published 'not counted' file totals are wrong, and the counting rule omits an exclusion. generated_files is len(generated), which includes package-lock.json and requirements.txt. generated_lines excludes those two (lines.py:105-106). SCORECARD.md:142 therefore prints 'baseline 8 files / 425 lines' and 'MetaObjects 11 files / 376 lines'. The directories it names hold 6 and 9 files; hand summing the lines of those files gives exactly 425 and 376. Fix: compute the file total from the same filtered list, and update SCORECARD.md:142 (rendered at run.py:681-684). Same class of defect: the rule says 'every committed file ... counts' except locks, generated output and the probe (README.md:103-105, run.py:662, lines.py:3-5), yet .gitignore is silently excluded through the 'ignore' category (lines.py:45 and 59). That hides 6 baseline lines and 8 MetaObjects lines. Add .gitignore to the stated exclusions.
• ⚠️ examples/head-to-head/baseline/api/src/app.ts:12 - The baseline API returns id as a JSON string, which violates its own contract. id is bigint (atlas/schema.sql:2) and node-postgres returns int8 as a string by default. pool.query<Customer> at app.ts:19, 23 and 33 is an unchecked cast, so tsc stays silent. POST /customers then answers {"id":"1",...} while generated/openapi.yaml:86-88 and api.d.ts declare an integer id. The smoke probe hides this: console.log(id) prints 1 either way. The MetaObjects side uses bigint("id", { mode: "number" }), so the comparison carries an accidental baseline bug although README.md:8 claims a competent baseline. Fix: add a one-line int8 parser, for example pg.types.setTypeParser(pg.types.builtins.INT8, Number), then rerun to refresh the line count.
• ℹ️ examples/head-to-head/harness/run.py:725 - A partial run or a mistyped scenario id silently replaces the committed scorecard. ./run.sh S1 S6 is documented at README.md:38, and it rewrites SCORECARD.md with only those rows, with totals such as 'red on 2 of 2'. Only the full-run line at README.md:37 mentions the rewrite. Unknown ids are dropped without error: ./run.sh s1 runs only S0, publishes a control-only scorecard and exits 0 (run.py:725, 751-753). Fix: reject ids not in SCENARIOS, and skip the default SCORECARD.md write when a subset was requested.
• ⚠️ examples/head-to-head/harness/run.py:715 - The --out and --json flags (run.py:715-716, 753, 755-759) are not needed by any intent requirement. The intent requires a published scorecard, and only the default SCORECARD.md write delivers it. Neither flag is documented in the README, and no script calls either one. Recommend removing both flags, along with the now-unused top-level json import. npm_version at run.py:453 re-imports json as _json, and that import can then stay as a plain local or move to the top.
• ℹ️ examples/head-to-head/harness/run.py:527 - The S0 false-alarm rule is defined twice, at run.py:527 in verdict and run.py:539 in layers_cell. Outcome.clean_gate also accepts a None gate_rc that never occurs, because every path sets gate_rc. Fix: extract one control_ok(o) helper for both sites and compare gate_rc to 0.
• ℹ️ examples/head-to-head/README.md:76 - Inline fix: README said every row of the partner table reads "no such check" for MetaObjects, but S5 reads n/a. Changed the wording to "every pull-request row" and noted that S5 is n/a on both sides.
• ℹ️ examples/head-to-head/README.md:37 - Inline fix: README said a full run takes about three minutes, but SCORECARD.md:5 records 232 s. Changed the text to "about four minutes".

🔧 Fix applied.
2 infos still open:

• ℹ️ examples/head-to-head/README.md:76 - Inline fix: README said every row of the partner table reads "no such check" for MetaObjects, but S5 reads n/a. Changed the wording to "every pull-request row" and noted that S5 is n/a on both sides.
• ℹ️ examples/head-to-head/README.md:37 - Inline fix: README said a full run takes about three minutes, but SCORECARD.md:5 records 232 s. Changed the text to "about four minutes".

✅ **Test** - passed

✅ No issues found.

• Live validation: ✅ go - 7 of 7 scenarios driven live against the product

Scenario	Result	Live	Evidence
Full ./run.sh produces a scorecard for both setups across S0–S6, matching the committed one	✅ pass	live	full.txt; SCORECARD.regenerated.md differs from the committed SCORECARD.md only in the duration line (123 s vs 224 s)
Control S0 is green on both setups (no false alarm)	✅ pass	live	subset.txt and full.txt: S0 'no false alarm' for baseline and metaobjects
Rename a field (S1): the checks that go red differ per setup	✅ pass	live	baseline red on tsc, mypy and prompt-render; MetaObjects red on tsc
Edit the DB by hand (S5): live-drift catches it on both setups, and the control check runs first	✅ pass	live	subset.txt: S5 red on live-drift for both; the S0/S5 text states the control
Partner break (S6): the baseline flags it and MetaObjects has no such check; the scorecard states MetaObjects loses this row	✅ pass	live	subset.txt: S6 baseline 'partner break flagged', MetaObjects 'all green'; partner table shows 'no such check'
Scorecard says it is self-measured	✅ pass	live	The first line of the generated scorecard is the 'Self-measured' note
Adversarial: a subset run does not overwrite SCORECARD.md, and an unknown scenario id or the removed --json flag is rejected	✅ pass	live	git status clean after ./run.sh S1 S5 S6; './run.sh s1' printed 'unknown scenario id(s): s1'; './run.sh --json x' printed 'unrecognized arguments: --json'

• scripts/ci-local.sh --only ts-fast --only ts-unit --strict-toolchains
• ./run.sh s1 (unknown scenario id, rejected with an error)
• ./run.sh --json x (removed flag, rejected)
• ./run.sh S1 S5 S6 (subset run; prints the table and leaves SCORECARD.md untouched)
• ./run.sh (full run, 7 scenarios × 2 setups, then diffed against the committed SCORECARD.md)
• Baseline: scripts/ci-local.sh --only ts-fast --only ts-unit --strict-toolchains (already passed before this step)

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.