diff --git a/CHANGELOG.md b/CHANGELOG.md index a5d1e9e3f..e3e2145cc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,20 @@ here.** - **A `pydantic` extra on the Python package.** `pip install "metaobjects[pydantic]"` installs `pydantic>=2.8`, the minimum the generated Python models need, so an adopter can get the enforced floor with the package. The base install is unchanged. +- **A recipe for moving existing models into MetaObjects without a rewrite.** + `docs/features/adopting-existing-models.md` takes Drizzle tables, Zod schemas and Pydantic models + from writing the model through retiring the old code, and proves the generated types match first. + The proof is a test file that `tsc` or pytest runs, so nothing reads your source and there is no + new command. `examples/brownfield/` holds a runnable project for each, and the recipe ships in + the `metaobjects-authoring` skill. + +### Fixed + +- **Python: an array field's `default` generated a string.** A model with + `field.string[]: { default: "{}" }` generated `tags: list[str] = '{}'`, which fails on first use. + It now generates `tags: list[str] = []`, from the same four spellings TypeScript accepts, or + from a native YAML list such as `default: [a, b]`. A default Python cannot spell as a list is + left to the database. ## [1.2.0] — 2026-10-30 diff --git a/agent-context/skills/metaobjects-authoring/SKILL.md b/agent-context/skills/metaobjects-authoring/SKILL.md index 85ac7bc5c..a1e709017 100644 --- a/agent-context/skills/metaobjects-authoring/SKILL.md +++ b/agent-context/skills/metaobjects-authoring/SKILL.md @@ -25,6 +25,7 @@ This file covers what almost every model needs. The topics below live in | File | Open it when | |---|---| | `references/adopting-existing-code.md` | working code or a populated schema already exists for what you are modeling | +| `references/adopting-existing-models.md` | moving existing Zod, Drizzle or Pydantic models into the model: the step-by-step recipe, and how to prove the generated types match before the switch | | `references/choosing-a-shape.md` | no row of "Canonical form for common field needs" fits; you need the full UUID / timestamp / URI / `@autoSet` rule; or you are registering your own vocabulary | | `references/json-columns.md` | a JSON column beyond the ladder below: coverage per port, `field.map` at runtime, version caveats, bag → native array | | `references/identities-and-indexes.md` | declaring a primary, secondary or reference identity, an FK, or a non-unique `index.lookup` | diff --git a/agent-context/skills/metaobjects-authoring/references/adopting-existing-code.md b/agent-context/skills/metaobjects-authoring/references/adopting-existing-code.md index 5d265812a..b611b9d1d 100644 --- a/agent-context/skills/metaobjects-authoring/references/adopting-existing-code.md +++ b/agent-context/skills/metaobjects-authoring/references/adopting-existing-code.md @@ -13,6 +13,9 @@ redefining it. (The metadata is still the durable spine *going forward*; only th **The observable predicate:** does working code or a populated schema already exist for what you're modeling? If yes, you are in adoption mode and these rules apply. +For Zod schemas, Drizzle tables or Pydantic models, `adopting-existing-models.md` is the +step-by-step recipe: generate beside them, prove the types match, move one call site at a time. + **Author metadata to match what the code ALREADY IS — not what you'd design fresh.** Read the existing code and schema *first*, then model to reproduce them: - The **native types the code uses** are the spec — model `field.uuid` when the code diff --git a/agent-context/skills/metaobjects-authoring/references/adopting-existing-models.md b/agent-context/skills/metaobjects-authoring/references/adopting-existing-models.md new file mode 100644 index 000000000..d35d70fe2 --- /dev/null +++ b/agent-context/skills/metaobjects-authoring/references/adopting-existing-models.md @@ -0,0 +1,386 @@ +# Adopting existing models + +> Part of the `metaobjects-authoring` skill. Read it when a project already has Zod schemas, Drizzle tables or Pydantic models and you are moving them into the model. + +MetaObjects never reads the project's source. You read the existing types and write the model, and +the language's own tools compare the two: `tsc` for TypeScript, `model_json_schema()` for Python. + +## Contents +- The recipe +- Drizzle +- Zod +- Pydantic +- Fixing the differences +- What the proof does not cover + +## The recipe + +1. **Write the model from the existing types.** Read the types and model what they are, not what + you would design fresh. `references/adopting-existing-code.md` in the `metaobjects-authoring` + skill has the rules. +2. **Generate beside them.** Send the output to a new directory. Nothing imports it yet. +3. **Write the proof.** It is one test file that names each pair of types. +4. **See it fail once.** Break one field in the model, regenerate, and check that the proof names + that field. Then undo it. +5. **Fix the differences in the model,** not in the existing code, until the proof passes. +6. **Check the database,** if the project has one. +7. **Move one call site at a time.** Run the proof and the project's tests after each move. +8. **Retire the old code,** delete the proof, and keep `meta verify` in CI. + +Step 4 is not optional. A proof that cannot fail looks the same as one that passes. If the proof +stays green with a field broken, its file is outside `include` in `tsconfig.json`, or `strict` is +off, or pytest never collected it. + +### Drizzle + +Install the CLI with `npm i -D @metaobjectsdev/cli @metaobjectsdev/codegen-ts typescript`, then run +`npx meta init` once in the project root. + +Write the model. Field names are the TypeScript property names. `snake_case` is the default for +columns, so `displayName` already means `display_name`. Reuse the names the database has: the name +of a `.unique()` index goes on `identity.secondary`, and an existing foreign key name goes in +`constraintName`. + +```yaml +metadata: + package: shop + children: + - object.entity: + name: Customer + children: + - source.rdb: { table: customers } + - field.uuid: { name: id, required: true } + - field.string: { name: email, required: true, maxLength: 254 } + - field.string: { name: displayName, required: true, maxLength: 120 } + - field.int: { name: loyaltyPoints, required: true, default: 0 } + - field.string: { name: notes } + - field.timestamp: { name: createdAt, required: true, default: now(), mutability: readOnly } + - identity.primary: { fields: id, generation: uuid } + - identity.secondary: { name: customers_email_unique, fields: email } +``` + +Generate beside the existing schema: + +```ts +// metaobjects.config.ts +import { defineConfig } from "@metaobjectsdev/cli"; + +export default defineConfig({ + outDir: "src/generated", // beside the existing code; nothing imports it yet + dialect: "postgres", + timestampMode: "date", // when the existing timestamp columns read as Date + generators: ["entity"], +}); +``` + +Run `npx meta gen` to generate. + +Copy this helper into `proof/differs.ts`: + +```ts +// Names every field that differs between two object types, or `never` when they match. +// +// Mutual assignability alone is not enough. A field that is optional and present on only one +// side passes both ways, and `any` is assignable to and from everything, so this compares +// key by key and treats `any` on one side as a difference. Pick<> keeps each field's optional +// modifier. It looks one level down: give a nested type its own line in equivalence.ts. +type Mutual = [A] extends [B] ? ([B] extends [A] ? true : false) : false; +type IsAny = 0 extends 1 & T ? true : false; + +export type Differs = { + [K in keyof A | keyof B]: K extends keyof A & keyof B + ? [IsAny] extends [IsAny] + ? Mutual, Pick> extends true + ? never + : K + : K + : K; +}[keyof A | keyof B]; + +// tsc reports each differing field as: Type '"name"' does not satisfy the constraint 'never'. +export type Same<_NoDifferences extends never> = true; +``` + +Then name each pair in `proof/equivalence.ts`, inside `include` in `tsconfig.json`, with `strict` +on. Compare what a query reads and what an insert takes: + +```ts +import type { Customer, NewCustomer } from "../src/db/schema.js"; +import type { Customer as GeneratedCustomer, CustomerInsert } from "../src/generated/Customer.js"; +import type { Differs, Same } from "./differs.js"; + +export type Proof = [ + Same>, + Same>, +]; +``` + +`npx tsc --noEmit` is the check. + +Types do not include column names, so `tsc` passes even with a wrong column. `meta verify --db` is +the check for those (it needs `pg` installed): + +```sh +npx meta verify --codegen --db "$DATABASE_URL" +``` + +`--db` runs only the database check unless you name the others, so keep `--codegen` beside it. + +Two Drizzle objects for one table can exist side by side, since each only describes SQL. Move a +call site by changing its import, with an alias where the generated name differs: + +```ts +import { type Customer, customers, type CustomerInsert as NewCustomer } from "./generated/Customer.js"; +``` + +When the last one has moved, replace the old table definitions with re-exports and delete +`proof/`: + +```ts +export { customers } from "../generated/Customer.js"; +export { orders } from "../generated/Order.js"; +``` + +Keep one migration tool per database. To stay on drizzle-kit, point it at this file. To move to +`meta migrate`, adopt the live schema with `meta migrate baseline --from-db`. + +### Zod + +Install the CLI with `npm i -D @metaobjectsdev/cli @metaobjectsdev/codegen-ts typescript`, then run +`npx meta init` once in the project root. + +A Zod project often has no ORM, so generate a contract-only target. An object with no table is an +`object.value`: + +```ts +// metaobjects.config.ts +import { defineConfig } from "@metaobjectsdev/cli"; +import { entityFile } from "@metaobjectsdev/codegen-ts/generators"; + +export default defineConfig({ + outDir: "src/generated", + dialect: "postgres", + targets: { + contract: { outDir: "src/generated", runtime: false }, // Zod schemas and types, no ORM + }, + generators: [entityFile({ target: "contract" })], +}); +``` + +```yaml +metadata: + package: api + children: + - object.value: + name: CreateCustomer + children: + - field.string: { name: email, required: true, maxLength: 254 } + - field.int: { name: seats } + - field.string[]: { name: tags, required: true } +``` + +Run `npx meta gen` to generate. The schema for `CreateCustomer` is `CreateCustomerInsertSchema`. The generated `CreateCustomer` is +the type. Use the same `proof/differs.ts` as above, and compare both `z.input` and `z.output`, +because a default or a transform changes one without the other: + +```ts +import type { z } from "zod"; +import type { CreateCustomer } from "../src/api/schemas.js"; +import type { CreateCustomerInsertSchema } from "../src/generated/CreateCustomer.js"; +import type { Differs, Same } from "./differs.js"; + +export type Proof = [ + Same, z.input>>, + Same, z.output>>, +]; +``` + +`tsc` cannot see `.min()`, `.max()` or `.email()`, because they do not change the type. Zod can +describe them, so add `proof/rules.ts` and run it with your test runner, or with node or bun: + +```ts +import assert from "node:assert/strict"; +import { z } from "zod"; +import { CreateCustomer } from "../src/api/schemas.js"; +import { CreateCustomerInsertSchema } from "../src/generated/CreateCustomer.js"; + +// `required` follows the order of the fields, which does not change what a schema accepts. +const rules = (schema: z.ZodType, io: "input" | "output") => + JSON.parse( + JSON.stringify(z.toJSONSchema(schema, { io, unrepresentable: "any" }), (key, value) => + key === "required" && Array.isArray(value) ? [...value].sort() : value, + ), + ); + +for (const io of ["input", "output"] as const) { + assert.deepEqual(rules(CreateCustomer, io), rules(CreateCustomerInsertSchema, io)); +} +``` + +`z.toJSONSchema` is Zod 4. On Zod 3, sort and compare `zodToJsonSchema(schema)` from the +`zod-to-json-schema` package instead. + +A required string generates `.min(1)`, so a hand-written `z.string()` with no minimum shows up here +as a `minLength` difference. If the existing schema really accepts an empty string, say so in the +model with `validator.length: { min: 0 }` as a child of the field. + +Move a call site by swapping the import and the schema name: + +```ts +import { type CreateCustomer, CreateCustomerInsertSchema } from "./generated/CreateCustomer.js"; + +export function parseSignup(body: unknown): CreateCustomer { + return CreateCustomerInsertSchema.parse(body); +} +``` + +### Pydantic + +Install with `pip install metaobjects "pydantic>=2.8"`. + +The model's field name is the Pydantic attribute, so keep `snake_case` names in the model. A shared +`str` Enum is an abstract `field.enum` that each field `extends`. An enum declared inline on the +field generates a `Literal` instead. + +```yaml +metadata: + package: shop + children: + - field.enum: + name: OrderStatus + abstract: true + values: [pending, paid, shipped] + - object.entity: + name: Order + children: + - source.rdb: { table: orders } + - field.long: { name: id, required: true } + - field.enum: { name: status, extends: OrderStatus, required: true, default: pending } + - field.string[]: { name: tags, required: true, default: "{}" } + - field.date: { name: placed_on, required: true } + - identity.primary: { fields: id, generation: increment } +``` + +Generate beside the existing models: + +```yaml +# metaobjects.config.yaml +metadata: metaobjects +targets: + app: + outDir: generated # beside the existing code; nothing imports it yet + generators: [entity] +``` + +Run `metaobjects gen` from the project root to generate. + +mypy cannot compare two Pydantic models, because a class is its own type, whatever its fields are. +Pydantic can describe a model as JSON Schema, so compare those. Pydantic validates input and writes +output differently, so check both modes. Copy this into `proof/test_equivalence.py` and name each +pair: + +```python +from typing import Any + +import pytest +from pydantic import BaseModel + +import app.models as existing +from generated import Customer, Order, ShippingAddress + + +def shape(model: type[BaseModel], mode: str) -> dict[str, Any]: + """The schema with what does not change behaviour removed: titles and descriptions, the + order of `required`, and `$ref`, which is inlined so a `str` Enum matches a Literal.""" + schema = model.model_json_schema(mode=mode) + defs = schema.pop("$defs", {}) + + def walk(node: Any, seen: tuple[str, ...] = ()) -> Any: + if isinstance(node, list): + return [walk(item, seen) for item in node] + if not isinstance(node, dict): + return node + if "$ref" in node: + name = node["$ref"].rsplit("/", 1)[-1] + if name in seen: # a model that contains itself + return {"$recursive": name} + rest = {key: value for key, value in node.items() if key != "$ref"} + return walk({**defs[name], **rest}, (*seen, name)) + out: dict[str, Any] = {} + for key, value in node.items(): + if key in ("title", "description"): + continue + if key == "properties": + out[key] = {field: walk(prop, seen) for field, prop in value.items()} + elif key == "required": + out[key] = sorted(value) + elif key == "default": + out[key] = value + else: + out[key] = walk(value, seen) + return out + + return walk(schema) + + +@pytest.mark.parametrize("mode", ["validation", "serialization"]) +@pytest.mark.parametrize( + ("hand_written", "generated"), + [ + (existing.Customer, Customer.Customer), + (existing.CustomerCreate, Customer.CustomerCreate), + (existing.ShippingAddress, ShippingAddress.ShippingAddress), + (existing.Order, Order.Order), + ], +) +def test_same_shape(hand_written: type[BaseModel], generated: type[BaseModel], mode: str) -> None: + assert shape(hand_written, mode) == shape(generated, mode) +``` + +Run `python -m pytest proof/ -vv` from the project root. `-vv` prints the whole difference, with +the field name in it. Schemas carry field constraints such as `max_length`, so this also catches a +rule the Python type does not show. + +Move a call site by changing its import: + +```python +from generated.Customer import Customer, CustomerCreate +``` + +Move models that nest together. Pydantic refuses an instance of a different class even when the +shape matches, so a generated `ShippingAddress` passed to the hand-written `Order` raises a +`ValidationError`. If they cannot move together, hand over `address.model_dump()` at the seam. + +The check against a database is `meta verify --db`, which needs the Node CLI, because schema +tooling is TypeScript only. `metaobjects verify` compares the generated code with the model. + +## Fixing the differences + +Fix each one in the model or the config, and leave the existing code alone until it has moved. + +| What you see | Why | Fix | +|---|---|---| +| A timestamp is `Date` on one side and `string` on the other | Generated timestamps read as strings by default | `timestampMode: "date"` in the config (Postgres) | +| A field the database fills is required on insert | The model has no default | Add `default:` to the field | +| A database-set timestamp is accepted on create | The field is writable | `default: now(), mutability: readOnly` | +| An array column's default is reported by `verify --db` | An array `default` must be a string | `default: "{}"`, not `default: []` | +| `verify --db` reports a different foreign key name | The generated name differs | `constraintName` on `identity.reference` | +| `verify --db` reports a different unique index name | The name comes from the identity | Name `identity.secondary` after the existing index | +| A `tsc` failure names a field that is `T \| null` on one side | The existing type allows both null and absent | Model what the code relies on. If the difference is deliberate, keep it visible: `Same, "notes">>` with a comment | + +## What the proof does not cover + +- **Custom logic.** A `.refine()`, a `.transform()` or a Pydantic validator is not in the type or + in the JSON Schema, so neither side of the proof sees it, and the model does not carry it. Keep + it in your code: refine the generated schema, or subclass the generated model. +- **Behaviour.** The proof compares declared shapes. The project's own tests still have to pass + after each move. +- **Native Postgres enums.** A `pgEnum` column cannot be modelled: `field.enum` is text plus a + CHECK. `tsc` passes and `meta verify --db` reports the missing CHECK. Convert the column to text + with a CHECK first, or leave that table out for now. +- **Python enum member names.** JSON Schema sees the values, not the member names. A generated enum + has `UPPER_CASE` members, so `OrderStatus.pending` in a call site must become + `OrderStatus.PENDING`. mypy or pyright finds those call sites. +- **Extra optional fields on a nested TypeScript type.** `Differs` compares a nested field as a + whole, so an optional extra inside it still assigns both ways. Give each nested type its own line. +- **Pairs you did not name.** The proof compares only what it lists, so list every exported type. diff --git a/docs/README.md b/docs/README.md index 1f3a9dc2c..af6e59a15 100644 --- a/docs/README.md +++ b/docs/README.md @@ -73,6 +73,7 @@ this tree is documentation, not the source of truth. | Record what the system is supposed to do, and stop agents reviving retired features | [`features/requirements.md`](features/requirements.md) | | Declare what a dashboard groups by and counts (`dimension`, `measure`, `segment`, `object.report`; load-time checked; a view-backed report becomes a SQL view served by a generated read-only list route and a TypeScript list hook) | [`features/reporting.md`](features/reporting.md) | | Download a report as CSV, JSON or XML (`?format=`, the `report-export` generator, its cap and null forms) | [`features/reporting.md`](features/reporting.md#exporting-a-report) | +| Move existing Zod, Drizzle or Pydantic models into the model, and prove the generated types match before you switch | [`features/adopting-existing-models.md`](features/adopting-existing-models.md) | | Keep Prisma as the ORM, with `schema.prisma` generated from the model and gated by `verify` (TypeScript) | [`recipes/prisma.md`](recipes/prisma.md) | | Export the reporting vocabulary to Cube (`cube-model`: cubes, joins, rollups; the `cube` lane that checks it against a real Cube) | [`features/cube-export.md`](features/cube-export.md) | | Wire prompt construction (FR-004) | [`features/templates-and-payloads.md`](features/templates-and-payloads.md) | diff --git a/docs/features/adopting-existing-models.md b/docs/features/adopting-existing-models.md new file mode 100644 index 000000000..cd9f13a89 --- /dev/null +++ b/docs/features/adopting-existing-models.md @@ -0,0 +1,395 @@ +# Adopting existing models + +> The sections from "The recipe" on are shipped verbatim in the `metaobjects-authoring` skill as +> `references/adopting-existing-models.md`; change them here and copy them there (a test compares +> the two). + +You have Zod schemas, Drizzle tables or Pydantic models in a working project, and you want the +MetaObjects model to own them. You do not need a rewrite. You write the model from the types you +already have, generate beside them, prove the two match, and move call sites one at a time. The +old code keeps working until the last one has moved. + +**MetaObjects never reads your source.** You, or your agent, read the existing types and write the +model. The comparison belongs to your language's own tools: `tsc` for TypeScript, Pydantic's +`model_json_schema()` for Python, and a test file you delete at the end. There is no import command +and no special `verify` mode. + +Three small projects run this recipe end to end, each with deliberate mismatches that must fail: + +| Project | Moves | Gate | +|---|---|---| +| [`examples/brownfield/drizzle`](../../examples/brownfield/drizzle) | Drizzle tables, and a real Postgres built by [`db/schema.sql`](../../examples/brownfield/drizzle/db/schema.sql) | `server/typescript/packages/cli/test/integration/brownfield-recipe.test.ts` | +| [`examples/brownfield/zod`](../../examples/brownfield/zod) | Zod request schemas | the same file | +| [`examples/brownfield/pydantic`](../../examples/brownfield/pydantic) | Pydantic models | `server/python/tests/codegen/test_brownfield_recipe.py` | + +## The recipe + +1. **Write the model from the existing types.** Read the types and model what they are, not what + you would design fresh. `references/adopting-existing-code.md` in the `metaobjects-authoring` + skill has the rules. +2. **Generate beside them.** Send the output to a new directory. Nothing imports it yet. +3. **Write the proof.** It is one test file that names each pair of types. +4. **See it fail once.** Break one field in the model, regenerate, and check that the proof names + that field. Then undo it. +5. **Fix the differences in the model,** not in the existing code, until the proof passes. +6. **Check the database,** if the project has one. +7. **Move one call site at a time.** Run the proof and the project's tests after each move. +8. **Retire the old code,** delete the proof, and keep `meta verify` in CI. + +Step 4 is not optional. A proof that cannot fail looks the same as one that passes. If the proof +stays green with a field broken, its file is outside `include` in `tsconfig.json`, or `strict` is +off, or pytest never collected it. + +### Drizzle + +Install the CLI with `npm i -D @metaobjectsdev/cli @metaobjectsdev/codegen-ts typescript`, then run +`npx meta init` once in the project root. + +Write the model. Field names are the TypeScript property names. `snake_case` is the default for +columns, so `displayName` already means `display_name`. Reuse the names the database has: the name +of a `.unique()` index goes on `identity.secondary`, and an existing foreign key name goes in +`constraintName`. + +```yaml +metadata: + package: shop + children: + - object.entity: + name: Customer + children: + - source.rdb: { table: customers } + - field.uuid: { name: id, required: true } + - field.string: { name: email, required: true, maxLength: 254 } + - field.string: { name: displayName, required: true, maxLength: 120 } + - field.int: { name: loyaltyPoints, required: true, default: 0 } + - field.string: { name: notes } + - field.timestamp: { name: createdAt, required: true, default: now(), mutability: readOnly } + - identity.primary: { fields: id, generation: uuid } + - identity.secondary: { name: customers_email_unique, fields: email } +``` + +Generate beside the existing schema: + +```ts +// metaobjects.config.ts +import { defineConfig } from "@metaobjectsdev/cli"; + +export default defineConfig({ + outDir: "src/generated", // beside the existing code; nothing imports it yet + dialect: "postgres", + timestampMode: "date", // when the existing timestamp columns read as Date + generators: ["entity"], +}); +``` + +Run `npx meta gen` to generate. + +Copy this helper into `proof/differs.ts`: + +```ts +// Names every field that differs between two object types, or `never` when they match. +// +// Mutual assignability alone is not enough. A field that is optional and present on only one +// side passes both ways, and `any` is assignable to and from everything, so this compares +// key by key and treats `any` on one side as a difference. Pick<> keeps each field's optional +// modifier. It looks one level down: give a nested type its own line in equivalence.ts. +type Mutual = [A] extends [B] ? ([B] extends [A] ? true : false) : false; +type IsAny = 0 extends 1 & T ? true : false; + +export type Differs = { + [K in keyof A | keyof B]: K extends keyof A & keyof B + ? [IsAny] extends [IsAny] + ? Mutual, Pick> extends true + ? never + : K + : K + : K; +}[keyof A | keyof B]; + +// tsc reports each differing field as: Type '"name"' does not satisfy the constraint 'never'. +export type Same<_NoDifferences extends never> = true; +``` + +Then name each pair in `proof/equivalence.ts`, inside `include` in `tsconfig.json`, with `strict` +on. Compare what a query reads and what an insert takes: + +```ts +import type { Customer, NewCustomer } from "../src/db/schema.js"; +import type { Customer as GeneratedCustomer, CustomerInsert } from "../src/generated/Customer.js"; +import type { Differs, Same } from "./differs.js"; + +export type Proof = [ + Same>, + Same>, +]; +``` + +`npx tsc --noEmit` is the check. + +Types do not include column names, so `tsc` passes even with a wrong column. `meta verify --db` is +the check for those (it needs `pg` installed): + +```sh +npx meta verify --codegen --db "$DATABASE_URL" +``` + +`--db` runs only the database check unless you name the others, so keep `--codegen` beside it. + +Two Drizzle objects for one table can exist side by side, since each only describes SQL. Move a +call site by changing its import, with an alias where the generated name differs: + +```ts +import { type Customer, customers, type CustomerInsert as NewCustomer } from "./generated/Customer.js"; +``` + +When the last one has moved, replace the old table definitions with re-exports and delete +`proof/`: + +```ts +export { customers } from "../generated/Customer.js"; +export { orders } from "../generated/Order.js"; +``` + +Keep one migration tool per database. To stay on drizzle-kit, point it at this file. To move to +`meta migrate`, adopt the live schema with `meta migrate baseline --from-db`. + +### Zod + +Install the CLI with `npm i -D @metaobjectsdev/cli @metaobjectsdev/codegen-ts typescript`, then run +`npx meta init` once in the project root. + +A Zod project often has no ORM, so generate a contract-only target. An object with no table is an +`object.value`: + +```ts +// metaobjects.config.ts +import { defineConfig } from "@metaobjectsdev/cli"; +import { entityFile } from "@metaobjectsdev/codegen-ts/generators"; + +export default defineConfig({ + outDir: "src/generated", + dialect: "postgres", + targets: { + contract: { outDir: "src/generated", runtime: false }, // Zod schemas and types, no ORM + }, + generators: [entityFile({ target: "contract" })], +}); +``` + +```yaml +metadata: + package: api + children: + - object.value: + name: CreateCustomer + children: + - field.string: { name: email, required: true, maxLength: 254 } + - field.int: { name: seats } + - field.string[]: { name: tags, required: true } +``` + +Run `npx meta gen` to generate. The schema for `CreateCustomer` is `CreateCustomerInsertSchema`. The generated `CreateCustomer` is +the type. Use the same `proof/differs.ts` as above, and compare both `z.input` and `z.output`, +because a default or a transform changes one without the other: + +```ts +import type { z } from "zod"; +import type { CreateCustomer } from "../src/api/schemas.js"; +import type { CreateCustomerInsertSchema } from "../src/generated/CreateCustomer.js"; +import type { Differs, Same } from "./differs.js"; + +export type Proof = [ + Same, z.input>>, + Same, z.output>>, +]; +``` + +`tsc` cannot see `.min()`, `.max()` or `.email()`, because they do not change the type. Zod can +describe them, so add `proof/rules.ts` and run it with your test runner, or with node or bun: + +```ts +import assert from "node:assert/strict"; +import { z } from "zod"; +import { CreateCustomer } from "../src/api/schemas.js"; +import { CreateCustomerInsertSchema } from "../src/generated/CreateCustomer.js"; + +// `required` follows the order of the fields, which does not change what a schema accepts. +const rules = (schema: z.ZodType, io: "input" | "output") => + JSON.parse( + JSON.stringify(z.toJSONSchema(schema, { io, unrepresentable: "any" }), (key, value) => + key === "required" && Array.isArray(value) ? [...value].sort() : value, + ), + ); + +for (const io of ["input", "output"] as const) { + assert.deepEqual(rules(CreateCustomer, io), rules(CreateCustomerInsertSchema, io)); +} +``` + +`z.toJSONSchema` is Zod 4. On Zod 3, sort and compare `zodToJsonSchema(schema)` from the +`zod-to-json-schema` package instead. + +A required string generates `.min(1)`, so a hand-written `z.string()` with no minimum shows up here +as a `minLength` difference. If the existing schema really accepts an empty string, say so in the +model with `validator.length: { min: 0 }` as a child of the field. + +Move a call site by swapping the import and the schema name: + +```ts +import { type CreateCustomer, CreateCustomerInsertSchema } from "./generated/CreateCustomer.js"; + +export function parseSignup(body: unknown): CreateCustomer { + return CreateCustomerInsertSchema.parse(body); +} +``` + +### Pydantic + +Install with `pip install metaobjects "pydantic>=2.8"`. + +The model's field name is the Pydantic attribute, so keep `snake_case` names in the model. A shared +`str` Enum is an abstract `field.enum` that each field `extends`. An enum declared inline on the +field generates a `Literal` instead. + +```yaml +metadata: + package: shop + children: + - field.enum: + name: OrderStatus + abstract: true + values: [pending, paid, shipped] + - object.entity: + name: Order + children: + - source.rdb: { table: orders } + - field.long: { name: id, required: true } + - field.enum: { name: status, extends: OrderStatus, required: true, default: pending } + - field.string[]: { name: tags, required: true, default: "{}" } + - field.date: { name: placed_on, required: true } + - identity.primary: { fields: id, generation: increment } +``` + +Generate beside the existing models: + +```yaml +# metaobjects.config.yaml +metadata: metaobjects +targets: + app: + outDir: generated # beside the existing code; nothing imports it yet + generators: [entity] +``` + +Run `metaobjects gen` from the project root to generate. + +mypy cannot compare two Pydantic models, because a class is its own type, whatever its fields are. +Pydantic can describe a model as JSON Schema, so compare those. Pydantic validates input and writes +output differently, so check both modes. Copy this into `proof/test_equivalence.py` and name each +pair: + +```python +from typing import Any + +import pytest +from pydantic import BaseModel + +import app.models as existing +from generated import Customer, Order, ShippingAddress + + +def shape(model: type[BaseModel], mode: str) -> dict[str, Any]: + """The schema with what does not change behaviour removed: titles and descriptions, the + order of `required`, and `$ref`, which is inlined so a `str` Enum matches a Literal.""" + schema = model.model_json_schema(mode=mode) + defs = schema.pop("$defs", {}) + + def walk(node: Any, seen: tuple[str, ...] = ()) -> Any: + if isinstance(node, list): + return [walk(item, seen) for item in node] + if not isinstance(node, dict): + return node + if "$ref" in node: + name = node["$ref"].rsplit("/", 1)[-1] + if name in seen: # a model that contains itself + return {"$recursive": name} + rest = {key: value for key, value in node.items() if key != "$ref"} + return walk({**defs[name], **rest}, (*seen, name)) + out: dict[str, Any] = {} + for key, value in node.items(): + if key in ("title", "description"): + continue + if key == "properties": + out[key] = {field: walk(prop, seen) for field, prop in value.items()} + elif key == "required": + out[key] = sorted(value) + elif key == "default": + out[key] = value + else: + out[key] = walk(value, seen) + return out + + return walk(schema) + + +@pytest.mark.parametrize("mode", ["validation", "serialization"]) +@pytest.mark.parametrize( + ("hand_written", "generated"), + [ + (existing.Customer, Customer.Customer), + (existing.CustomerCreate, Customer.CustomerCreate), + (existing.ShippingAddress, ShippingAddress.ShippingAddress), + (existing.Order, Order.Order), + ], +) +def test_same_shape(hand_written: type[BaseModel], generated: type[BaseModel], mode: str) -> None: + assert shape(hand_written, mode) == shape(generated, mode) +``` + +Run `python -m pytest proof/ -vv` from the project root. `-vv` prints the whole difference, with +the field name in it. Schemas carry field constraints such as `max_length`, so this also catches a +rule the Python type does not show. + +Move a call site by changing its import: + +```python +from generated.Customer import Customer, CustomerCreate +``` + +Move models that nest together. Pydantic refuses an instance of a different class even when the +shape matches, so a generated `ShippingAddress` passed to the hand-written `Order` raises a +`ValidationError`. If they cannot move together, hand over `address.model_dump()` at the seam. + +The check against a database is `meta verify --db`, which needs the Node CLI, because schema +tooling is TypeScript only. `metaobjects verify` compares the generated code with the model. + +## Fixing the differences + +Fix each one in the model or the config, and leave the existing code alone until it has moved. + +| What you see | Why | Fix | +|---|---|---| +| A timestamp is `Date` on one side and `string` on the other | Generated timestamps read as strings by default | `timestampMode: "date"` in the config (Postgres) | +| A field the database fills is required on insert | The model has no default | Add `default:` to the field | +| A database-set timestamp is accepted on create | The field is writable | `default: now(), mutability: readOnly` | +| An array column's default is reported by `verify --db` | An array `default` must be a string | `default: "{}"`, not `default: []` | +| `verify --db` reports a different foreign key name | The generated name differs | `constraintName` on `identity.reference` | +| `verify --db` reports a different unique index name | The name comes from the identity | Name `identity.secondary` after the existing index | +| A `tsc` failure names a field that is `T \| null` on one side | The existing type allows both null and absent | Model what the code relies on. If the difference is deliberate, keep it visible: `Same, "notes">>` with a comment | + +## What the proof does not cover + +- **Custom logic.** A `.refine()`, a `.transform()` or a Pydantic validator is not in the type or + in the JSON Schema, so neither side of the proof sees it, and the model does not carry it. Keep + it in your code: refine the generated schema, or subclass the generated model. +- **Behaviour.** The proof compares declared shapes. The project's own tests still have to pass + after each move. +- **Native Postgres enums.** A `pgEnum` column cannot be modelled: `field.enum` is text plus a + CHECK. `tsc` passes and `meta verify --db` reports the missing CHECK. Convert the column to text + with a CHECK first, or leave that table out for now. +- **Python enum member names.** JSON Schema sees the values, not the member names. A generated enum + has `UPPER_CASE` members, so `OrderStatus.pending` in a call site must become + `OrderStatus.PENDING`. mypy or pyright finds those call sites. +- **Extra optional fields on a nested TypeScript type.** `Differs` compares a nested field as a + whole, so an optional extra inside it still assigns both ways. Give each nested type its own line. +- **Pairs you did not name.** The proof compares only what it lists, so list every exported type. diff --git a/docs/features/field-types.md b/docs/features/field-types.md index 8995c8d7f..6623784a8 100644 --- a/docs/features/field-types.md +++ b/docs/features/field-types.md @@ -24,7 +24,7 @@ across ports — a `field.currency` is integer minor units everywhere; a | `field.uuid` | `string` | `UUID` | `UUID` | `Guid` | `UUID` | `uuid` | | `field.uri` | `string` | `java.net.URI` | `java.net.URI` | `Uri` | `AnyUrl` | `text` | | `field.inet` | `string` | `InetAddress` | `InetAddress` | `IPAddress` | `IPvAnyAddress` | `inet` | -| `field.enum` | union + `z.enum` | `Enum` | `enum class` | `enum` | `Enum` | `varchar` + `CHECK` | +| `field.enum` | union + `z.enum` | `Enum` | `enum class` | `enum` | `Literal` inline, `Enum` with `UPPER_CASE` members when shared by `extends` | `varchar` + `CHECK` | | `field.object` | nested type | nested class | nested data class | nested class | nested Pydantic model | per `@storage` | | `field.map` | `Record` | `Map` | `Map` | `Dictionary` | `dict[str, V]` | `jsonb` | diff --git a/docs/ports/python.md b/docs/ports/python.md index 8119079ef..e592bd61d 100644 --- a/docs/ports/python.md +++ b/docs/ports/python.md @@ -968,5 +968,6 @@ import lines are stable. - [`server/python/README.md`](../../server/python/README.md) — module-level overview - [`docs/features/`](../features/) — every feature shows the Python output inline +- [Adopting existing models](../features/adopting-existing-models.md) — move Pydantic models into the model and prove they match before you switch a call site - [`docs/superpowers/specs/2026-05-23-python-codegen-engine-entity-generator-design.md`](../superpowers/specs/2026-05-23-python-codegen-engine-entity-generator-design.md) - [`docs/superpowers/specs/2026-05-23-python-codegen-persistence-foundation-roadmap.md`](../superpowers/specs/2026-05-23-python-codegen-persistence-foundation-roadmap.md) diff --git a/docs/ports/typescript.md b/docs/ports/typescript.md index 332291d3b..c274d65c5 100644 --- a/docs/ports/typescript.md +++ b/docs/ports/typescript.md @@ -53,6 +53,10 @@ export default defineConfig({ Already on drizzle-kit and want to keep its migrations? Re-export the generated tables from your own schema file: [drizzle-kit migrations](../recipes/drizzle-kit-migrations.md). +Moving existing Drizzle tables or Zod schemas into the model? [Adopting existing +models](../features/adopting-existing-models.md) generates beside them and proves the types match +before you switch a call site. + A model of only value objects and sourceless projections needs no `dialect`. A sourceless object generates its type and nothing else: no table, no migration, no routes. See [Own your codegen](../features/own-your-codegen.md) for ejecting a generator, and the diff --git a/examples/brownfield/README.md b/examples/brownfield/README.md new file mode 100644 index 000000000..547671f70 --- /dev/null +++ b/examples/brownfield/README.md @@ -0,0 +1,18 @@ +# Moving existing models into MetaObjects + +Three small projects that each start with hand-written models and no MetaObjects dependency in +their source. They are the worked examples for +[Adopting existing models](../../docs/features/adopting-existing-models.md). + +| Project | Existing code | What proves it | +|---|---|---| +| [`drizzle/`](drizzle) | Drizzle tables and a Postgres schema | `tsc` over `proof/`, then `meta verify --db` | +| [`zod/`](zod) | Zod request schemas | `tsc` over `proof/`, then `proof/rules.ts` | +| [`pydantic/`](pydantic) | Pydantic models | `python -m pytest proof/` | + +Each one holds the model, the config and the proof, in the state just after step 3 of the recipe. +Nothing is generated yet, and the source files have not moved. Generated output is not committed. + +These run in CI, including the deliberate mismatches: see +`server/typescript/packages/cli/test/integration/brownfield-recipe.test.ts` and +`server/python/tests/codegen/test_brownfield_recipe.py`. diff --git a/examples/brownfield/drizzle/db/schema.sql b/examples/brownfield/drizzle/db/schema.sql new file mode 100644 index 000000000..aef67e526 --- /dev/null +++ b/examples/brownfield/drizzle/db/schema.sql @@ -0,0 +1,26 @@ +-- The database the application already runs: what `drizzle-kit generate` wrote for src/db/schema.ts. +CREATE TABLE "customers" ( + "id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL, + "email" varchar(254) NOT NULL, + "display_name" varchar(120) NOT NULL, + "loyalty_points" integer DEFAULT 0 NOT NULL, + "marketing_opt_in" boolean DEFAULT false NOT NULL, + "notes" text, + "created_at" timestamp with time zone DEFAULT now() NOT NULL, + CONSTRAINT "customers_email_unique" UNIQUE ("email") +); + +CREATE TABLE "orders" ( + "id" bigserial PRIMARY KEY NOT NULL, + "customer_id" uuid NOT NULL, + "status" text DEFAULT 'pending' NOT NULL, + "total" numeric(12, 2) NOT NULL, + "shipping_address" jsonb, + "tags" text[] DEFAULT '{}'::text[] NOT NULL, + "placed_on" date NOT NULL, + CONSTRAINT "orders_status_chk" CHECK (status IN ('pending', 'paid', 'shipped')) +); + +ALTER TABLE "orders" ADD CONSTRAINT "orders_customer_id_customers_id_fk" + FOREIGN KEY ("customer_id") REFERENCES "public"."customers"("id") + ON DELETE NO ACTION ON UPDATE NO ACTION; diff --git a/examples/brownfield/drizzle/metaobjects.config.ts b/examples/brownfield/drizzle/metaobjects.config.ts new file mode 100644 index 000000000..14ad00714 --- /dev/null +++ b/examples/brownfield/drizzle/metaobjects.config.ts @@ -0,0 +1,8 @@ +import { defineConfig } from "@metaobjectsdev/cli"; + +export default defineConfig({ + outDir: "src/generated", // beside the existing code; nothing imports it yet + dialect: "postgres", + timestampMode: "date", // the existing timestamp columns read as Date + generators: ["entity"], +}); diff --git a/examples/brownfield/drizzle/metaobjects/meta.shop.yaml b/examples/brownfield/drizzle/metaobjects/meta.shop.yaml new file mode 100644 index 000000000..fbf8d74ae --- /dev/null +++ b/examples/brownfield/drizzle/metaobjects/meta.shop.yaml @@ -0,0 +1,39 @@ +metadata: + package: shop + children: + - object.value: + name: ShippingAddress + children: + - field.string: { name: line1, required: true } + - field.string: { name: city, required: true } + - field.string: { name: postcode } + - object.entity: + name: Customer + children: + - source.rdb: { table: customers } + - field.uuid: { name: id, required: true } + - field.string: { name: email, required: true, maxLength: 254 } + - field.string: { name: displayName, required: true, maxLength: 120 } + - field.int: { name: loyaltyPoints, required: true, default: 0 } + - field.boolean: { name: marketingOptIn, required: true, default: false } + - field.string: { name: notes } + - field.timestamp: { name: createdAt, required: true, default: now(), mutability: readOnly } + - identity.primary: { fields: id, generation: uuid } + - identity.secondary: { name: customers_email_unique, fields: email } + - object.entity: + name: Order + children: + - source.rdb: { table: orders } + - field.long: { name: id, required: true } + - field.uuid: { name: customerId, required: true } + - field.enum: { name: status, required: true, values: [pending, paid, shipped], default: pending } + - field.decimal: { name: total, required: true, precision: 12, scale: 2 } + - field.object: { name: shippingAddress, objectRef: ShippingAddress, storage: jsonb } + - field.string[]: { name: tags, required: true, default: "{}" } + - field.date: { name: placedOn, required: true } + - identity.primary: { fields: id, generation: increment } + - identity.reference: + name: fkCustomer + fields: customerId + references: Customer + constraintName: orders_customer_id_customers_id_fk diff --git a/examples/brownfield/drizzle/proof/differs.ts b/examples/brownfield/drizzle/proof/differs.ts new file mode 100644 index 000000000..a79916084 --- /dev/null +++ b/examples/brownfield/drizzle/proof/differs.ts @@ -0,0 +1,21 @@ +// Names every field that differs between two object types, or `never` when they match. +// +// Mutual assignability alone is not enough. A field that is optional and present on only one +// side passes both ways, and `any` is assignable to and from everything, so this compares +// key by key and treats `any` on one side as a difference. Pick<> keeps each field's optional +// modifier. It looks one level down: give a nested type its own line in equivalence.ts. +type Mutual = [A] extends [B] ? ([B] extends [A] ? true : false) : false; +type IsAny = 0 extends 1 & T ? true : false; + +export type Differs = { + [K in keyof A | keyof B]: K extends keyof A & keyof B + ? [IsAny] extends [IsAny] + ? Mutual, Pick> extends true + ? never + : K + : K + : K; +}[keyof A | keyof B]; + +// tsc reports each differing field as: Type '"name"' does not satisfy the constraint 'never'. +export type Same<_NoDifferences extends never> = true; diff --git a/examples/brownfield/drizzle/proof/equivalence.ts b/examples/brownfield/drizzle/proof/equivalence.ts new file mode 100644 index 000000000..7362ad382 --- /dev/null +++ b/examples/brownfield/drizzle/proof/equivalence.ts @@ -0,0 +1,15 @@ +// tsc is the check: it fails this file, naming the field, when a pair stops matching. +// Delete it once the last call site has moved. +import type { Customer, NewCustomer, NewOrder, Order, ShippingAddress } from "../src/db/schema.js"; +import type { Customer as GeneratedCustomer, CustomerInsert } from "../src/generated/Customer.js"; +import type { Order as GeneratedOrder, OrderInsert } from "../src/generated/Order.js"; +import type { ShippingAddress as GeneratedShippingAddress } from "../src/generated/ShippingAddress.js"; +import type { Differs, Same } from "./differs.js"; + +export type Proof = [ + Same>, + Same>, + Same>, + Same>, + Same>, +]; diff --git a/examples/brownfield/drizzle/src/customers.ts b/examples/brownfield/drizzle/src/customers.ts new file mode 100644 index 000000000..14242dc4d --- /dev/null +++ b/examples/brownfield/drizzle/src/customers.ts @@ -0,0 +1,18 @@ +import { eq } from "drizzle-orm"; +import type { PgDatabase, PgQueryResultHKT } from "drizzle-orm/pg-core"; +import { type Customer, customers, type NewCustomer } from "./db/schema.js"; + +type Db = PgDatabase; + +export async function findByEmail(db: Db, email: string): Promise { + const [row] = await db.select().from(customers).where(eq(customers.email, email)); + return row; +} + +export async function register(db: Db, input: NewCustomer): Promise { + await db.insert(customers).values(input); +} + +export function memberSince(customer: Customer): number { + return customer.createdAt.getFullYear(); +} diff --git a/examples/brownfield/drizzle/src/db/schema.ts b/examples/brownfield/drizzle/src/db/schema.ts new file mode 100644 index 000000000..cab34ae40 --- /dev/null +++ b/examples/brownfield/drizzle/src/db/schema.ts @@ -0,0 +1,51 @@ +import { sql } from "drizzle-orm"; +import { + bigserial, + boolean, + check, + date, + integer, + jsonb, + numeric, + pgTable, + text, + timestamp, + uuid, + varchar, +} from "drizzle-orm/pg-core"; + +export const customers = pgTable("customers", { + id: uuid("id").primaryKey().defaultRandom(), + email: varchar("email", { length: 254 }).notNull().unique(), + displayName: varchar("display_name", { length: 120 }).notNull(), + loyaltyPoints: integer("loyalty_points").notNull().default(0), + marketingOptIn: boolean("marketing_opt_in").notNull().default(false), + notes: text("notes"), + createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(), +}); + +export type Customer = typeof customers.$inferSelect; +export type NewCustomer = typeof customers.$inferInsert; + +export type ShippingAddress = { line1: string; city: string; postcode?: string }; + +export const orders = pgTable( + "orders", + { + id: bigserial("id", { mode: "number" }).primaryKey(), + customerId: uuid("customer_id") + .notNull() + .references(() => customers.id), + status: text("status", { enum: ["pending", "paid", "shipped"] }) + .notNull() + .default("pending"), + total: numeric("total", { precision: 12, scale: 2 }).notNull(), + shippingAddress: jsonb("shipping_address").$type(), + tags: text("tags").array().notNull().default(sql`'{}'::text[]`), + placedOn: date("placed_on").notNull(), + }, + () => [check("orders_status_chk", sql`status IN ('pending', 'paid', 'shipped')`)], +); + +export type Order = typeof orders.$inferSelect; +export type NewOrder = typeof orders.$inferInsert; diff --git a/examples/brownfield/drizzle/src/orders.ts b/examples/brownfield/drizzle/src/orders.ts new file mode 100644 index 000000000..6e70e8f87 --- /dev/null +++ b/examples/brownfield/drizzle/src/orders.ts @@ -0,0 +1,17 @@ +import { eq } from "drizzle-orm"; +import type { PgDatabase, PgQueryResultHKT } from "drizzle-orm/pg-core"; +import { type NewOrder, type Order, orders } from "./db/schema.js"; + +type Db = PgDatabase; + +export async function ordersFor(db: Db, customerId: string): Promise { + return db.select().from(orders).where(eq(orders.customerId, customerId)); +} + +export async function place(db: Db, input: NewOrder): Promise { + await db.insert(orders).values(input); +} + +export function shipTo(order: Order): string { + return order.shippingAddress?.city ?? "collection"; +} diff --git a/examples/brownfield/drizzle/tsconfig.json b/examples/brownfield/drizzle/tsconfig.json new file mode 100644 index 000000000..1c42711b7 --- /dev/null +++ b/examples/brownfield/drizzle/tsconfig.json @@ -0,0 +1,12 @@ +{ + "compilerOptions": { + "strict": true, + "target": "esnext", + "module": "nodenext", + "moduleResolution": "nodenext", + "noEmit": true, + "skipLibCheck": true, + "types": ["node"] + }, + "include": ["src", "proof"] +} diff --git a/examples/brownfield/pydantic/app/__init__.py b/examples/brownfield/pydantic/app/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/examples/brownfield/pydantic/app/models.py b/examples/brownfield/pydantic/app/models.py new file mode 100644 index 000000000..0aaec30e7 --- /dev/null +++ b/examples/brownfield/pydantic/app/models.py @@ -0,0 +1,47 @@ +from datetime import date, datetime +from decimal import Decimal +from enum import Enum +from typing import Optional +from uuid import UUID + +from pydantic import BaseModel, Field + + +class OrderStatus(str, Enum): + PENDING = "pending" + PAID = "paid" + SHIPPED = "shipped" + + +class ShippingAddress(BaseModel): + line1: str + city: str + postcode: Optional[str] = None + + +class Customer(BaseModel): + id: UUID + email: str = Field(max_length=254) + display_name: str = Field(max_length=120) + loyalty_points: int = 0 + marketing_opt_in: bool = False + notes: Optional[str] = None + created_at: datetime + + +class CustomerCreate(BaseModel): + email: str = Field(min_length=1, max_length=254) + display_name: str = Field(min_length=1, max_length=120) + loyalty_points: int = 0 + marketing_opt_in: bool = False + notes: Optional[str] = None + + +class Order(BaseModel): + id: int + customer_id: UUID + status: OrderStatus = OrderStatus.PENDING + total: Decimal + shipping_address: Optional[ShippingAddress] = None + tags: list[str] = [] + placed_on: date diff --git a/examples/brownfield/pydantic/app/shipping.py b/examples/brownfield/pydantic/app/shipping.py new file mode 100644 index 000000000..cd0372e4c --- /dev/null +++ b/examples/brownfield/pydantic/app/shipping.py @@ -0,0 +1,5 @@ +from app.models import Order, OrderStatus, ShippingAddress + + +def ship_to(order: Order) -> ShippingAddress | None: + return order.shipping_address if order.status is OrderStatus.PAID else None diff --git a/examples/brownfield/pydantic/app/signup.py b/examples/brownfield/pydantic/app/signup.py new file mode 100644 index 000000000..3248ed738 --- /dev/null +++ b/examples/brownfield/pydantic/app/signup.py @@ -0,0 +1,9 @@ +from app.models import Customer, CustomerCreate + + +def welcome(customer: Customer) -> str: + return f"Welcome, {customer.display_name}" + + +def parse_signup(body: dict[str, object]) -> CustomerCreate: + return CustomerCreate.model_validate(body) diff --git a/examples/brownfield/pydantic/metaobjects.config.yaml b/examples/brownfield/pydantic/metaobjects.config.yaml new file mode 100644 index 000000000..f3c775fd0 --- /dev/null +++ b/examples/brownfield/pydantic/metaobjects.config.yaml @@ -0,0 +1,5 @@ +metadata: metaobjects +targets: + app: + outDir: generated # beside the existing code; nothing imports it yet + generators: [entity] diff --git a/examples/brownfield/pydantic/metaobjects/meta.shop.yaml b/examples/brownfield/pydantic/metaobjects/meta.shop.yaml new file mode 100644 index 000000000..b5af4e45a --- /dev/null +++ b/examples/brownfield/pydantic/metaobjects/meta.shop.yaml @@ -0,0 +1,38 @@ +metadata: + package: shop + children: + - field.enum: + name: OrderStatus + abstract: true + values: [pending, paid, shipped] + - object.value: + name: ShippingAddress + children: + - field.string: { name: line1, required: true } + - field.string: { name: city, required: true } + - field.string: { name: postcode } + - object.entity: + name: Customer + children: + - source.rdb: { table: customers } + - field.uuid: { name: id, required: true } + - field.string: { name: email, required: true, maxLength: 254 } + - field.string: { name: display_name, required: true, maxLength: 120 } + - field.int: { name: loyalty_points, required: true, default: 0 } + - field.boolean: { name: marketing_opt_in, required: true, default: false } + - field.string: { name: notes } + - field.timestamp: { name: created_at, required: true, default: now(), mutability: readOnly } + - identity.primary: { fields: id, generation: uuid } + - object.entity: + name: Order + children: + - source.rdb: { table: orders } + - field.long: { name: id, required: true } + - field.uuid: { name: customer_id, required: true } + - field.enum: { name: status, extends: OrderStatus, required: true, default: pending } + - field.decimal: { name: total, required: true, precision: 12, scale: 2 } + - field.object: { name: shipping_address, objectRef: ShippingAddress, storage: jsonb } + - field.string[]: { name: tags, required: true, default: "{}" } + - field.date: { name: placed_on, required: true } + - identity.primary: { fields: id, generation: increment } + - identity.reference: { name: fkCustomer, fields: customer_id, references: Customer } diff --git a/examples/brownfield/pydantic/proof/test_equivalence.py b/examples/brownfield/pydantic/proof/test_equivalence.py new file mode 100644 index 000000000..e9e022c88 --- /dev/null +++ b/examples/brownfield/pydantic/proof/test_equivalence.py @@ -0,0 +1,59 @@ +"""Pydantic describes each model as JSON Schema, so compare those, in both directions of use. + +Delete this file once the last call site has moved. +""" +from typing import Any + +import pytest +from pydantic import BaseModel + +import app.models as existing +from generated import Customer, Order, ShippingAddress + + +def shape(model: type[BaseModel], mode: str) -> dict[str, Any]: + """The schema with what does not change behaviour removed: titles and descriptions, the + order of `required`, and `$ref`, which is inlined so a `str` Enum matches a Literal.""" + schema = model.model_json_schema(mode=mode) + defs = schema.pop("$defs", {}) + + def walk(node: Any, seen: tuple[str, ...] = ()) -> Any: + if isinstance(node, list): + return [walk(item, seen) for item in node] + if not isinstance(node, dict): + return node + if "$ref" in node: + name = node["$ref"].rsplit("/", 1)[-1] + if name in seen: # a model that contains itself + return {"$recursive": name} + rest = {key: value for key, value in node.items() if key != "$ref"} + return walk({**defs[name], **rest}, (*seen, name)) + out: dict[str, Any] = {} + for key, value in node.items(): + if key in ("title", "description"): + continue + if key == "properties": + out[key] = {field: walk(prop, seen) for field, prop in value.items()} + elif key == "required": + out[key] = sorted(value) + elif key == "default": + out[key] = value + else: + out[key] = walk(value, seen) + return out + + return walk(schema) + + +@pytest.mark.parametrize("mode", ["validation", "serialization"]) +@pytest.mark.parametrize( + ("hand_written", "generated"), + [ + (existing.Customer, Customer.Customer), + (existing.CustomerCreate, Customer.CustomerCreate), + (existing.ShippingAddress, ShippingAddress.ShippingAddress), + (existing.Order, Order.Order), + ], +) +def test_same_shape(hand_written: type[BaseModel], generated: type[BaseModel], mode: str) -> None: + assert shape(hand_written, mode) == shape(generated, mode) diff --git a/examples/brownfield/zod/metaobjects.config.ts b/examples/brownfield/zod/metaobjects.config.ts new file mode 100644 index 000000000..812f4d40f --- /dev/null +++ b/examples/brownfield/zod/metaobjects.config.ts @@ -0,0 +1,11 @@ +import { defineConfig } from "@metaobjectsdev/cli"; +import { entityFile } from "@metaobjectsdev/codegen-ts/generators"; + +export default defineConfig({ + outDir: "src/generated", + dialect: "postgres", + targets: { + contract: { outDir: "src/generated", runtime: false }, // Zod schemas and types, no ORM + }, + generators: [entityFile({ target: "contract" })], +}); diff --git a/examples/brownfield/zod/metaobjects/meta.api.yaml b/examples/brownfield/zod/metaobjects/meta.api.yaml new file mode 100644 index 000000000..995ad1802 --- /dev/null +++ b/examples/brownfield/zod/metaobjects/meta.api.yaml @@ -0,0 +1,18 @@ +metadata: + package: api + children: + - object.value: + name: Address + children: + - field.string: { name: line1, required: true } + - field.string: { name: city, required: true } + - field.string: { name: postcode } + - object.value: + name: CreateCustomer + children: + - field.string: { name: email, required: true, maxLength: 254 } + - field.string: { name: displayName, required: true, maxLength: 120 } + - field.enum: { name: plan, values: [free, team, enterprise] } + - field.int: { name: seats } + - field.object: { name: address, objectRef: Address } + - field.string[]: { name: tags, required: true } diff --git a/examples/brownfield/zod/proof/differs.ts b/examples/brownfield/zod/proof/differs.ts new file mode 100644 index 000000000..a79916084 --- /dev/null +++ b/examples/brownfield/zod/proof/differs.ts @@ -0,0 +1,21 @@ +// Names every field that differs between two object types, or `never` when they match. +// +// Mutual assignability alone is not enough. A field that is optional and present on only one +// side passes both ways, and `any` is assignable to and from everything, so this compares +// key by key and treats `any` on one side as a difference. Pick<> keeps each field's optional +// modifier. It looks one level down: give a nested type its own line in equivalence.ts. +type Mutual = [A] extends [B] ? ([B] extends [A] ? true : false) : false; +type IsAny = 0 extends 1 & T ? true : false; + +export type Differs = { + [K in keyof A | keyof B]: K extends keyof A & keyof B + ? [IsAny] extends [IsAny] + ? Mutual, Pick> extends true + ? never + : K + : K + : K; +}[keyof A | keyof B]; + +// tsc reports each differing field as: Type '"name"' does not satisfy the constraint 'never'. +export type Same<_NoDifferences extends never> = true; diff --git a/examples/brownfield/zod/proof/equivalence.ts b/examples/brownfield/zod/proof/equivalence.ts new file mode 100644 index 000000000..b02c15b02 --- /dev/null +++ b/examples/brownfield/zod/proof/equivalence.ts @@ -0,0 +1,15 @@ +// tsc is the check: it fails this file, naming the field, when a pair stops matching. +// Compare input and output: a default or a transform changes one without the other. +// Delete it once the last call site has moved. +import type { z } from "zod"; +import type { Address, CreateCustomer } from "../src/api/schemas.js"; +import type { AddressInsertSchema } from "../src/generated/Address.js"; +import type { CreateCustomerInsertSchema } from "../src/generated/CreateCustomer.js"; +import type { Differs, Same } from "./differs.js"; + +export type Proof = [ + Same, z.input>>, + Same, z.output>>, + Same, z.input>>, + Same, z.output>>, +]; diff --git a/examples/brownfield/zod/proof/rules.ts b/examples/brownfield/zod/proof/rules.ts new file mode 100644 index 000000000..516b00e21 --- /dev/null +++ b/examples/brownfield/zod/proof/rules.ts @@ -0,0 +1,21 @@ +// tsc cannot see .min(), .max() or .email(): they do not change the type. Zod can: compare the +// JSON Schema each validator describes, on input and on output. Run this with your test runner, +// or with node or bun. +import assert from "node:assert/strict"; +import { z } from "zod"; +import { Address, CreateCustomer } from "../src/api/schemas.js"; +import { AddressInsertSchema } from "../src/generated/Address.js"; +import { CreateCustomerInsertSchema } from "../src/generated/CreateCustomer.js"; + +// `required` follows the order of the fields, which does not change what a schema accepts. +const rules = (schema: z.ZodType, io: "input" | "output") => + JSON.parse( + JSON.stringify(z.toJSONSchema(schema, { io, unrepresentable: "any" }), (key, value) => + key === "required" && Array.isArray(value) ? [...value].sort() : value, + ), + ); + +for (const io of ["input", "output"] as const) { + assert.deepEqual(rules(Address, io), rules(AddressInsertSchema, io)); + assert.deepEqual(rules(CreateCustomer, io), rules(CreateCustomerInsertSchema, io)); +} diff --git a/examples/brownfield/zod/src/api/schemas.ts b/examples/brownfield/zod/src/api/schemas.ts new file mode 100644 index 000000000..f1709c31a --- /dev/null +++ b/examples/brownfield/zod/src/api/schemas.ts @@ -0,0 +1,18 @@ +import { z } from "zod"; + +export const Address = z.object({ + line1: z.string().min(1), + city: z.string().min(1), + postcode: z.string().optional(), +}); +export type Address = z.infer; + +export const CreateCustomer = z.object({ + email: z.string().min(1).max(254), + displayName: z.string().min(1).max(120), + plan: z.enum(["free", "team", "enterprise"]).optional(), + seats: z.number().int().optional(), + address: Address.optional(), + tags: z.array(z.string()), +}); +export type CreateCustomer = z.infer; diff --git a/examples/brownfield/zod/src/shipping.ts b/examples/brownfield/zod/src/shipping.ts new file mode 100644 index 000000000..f561a0b2f --- /dev/null +++ b/examples/brownfield/zod/src/shipping.ts @@ -0,0 +1,9 @@ +import { Address } from "./api/schemas.js"; + +export function parseAddress(body: unknown): Address { + return Address.parse(body); +} + +export function label(address: Address): string { + return `${address.line1}, ${address.city}`; +} diff --git a/examples/brownfield/zod/src/signup.ts b/examples/brownfield/zod/src/signup.ts new file mode 100644 index 000000000..14c90e0dd --- /dev/null +++ b/examples/brownfield/zod/src/signup.ts @@ -0,0 +1,9 @@ +import { CreateCustomer } from "./api/schemas.js"; + +export function parseSignup(body: unknown): CreateCustomer { + return CreateCustomer.parse(body); +} + +export function seatsFor(customer: CreateCustomer): number { + return customer.seats ?? 1; +} diff --git a/examples/brownfield/zod/tsconfig.json b/examples/brownfield/zod/tsconfig.json new file mode 100644 index 000000000..1c42711b7 --- /dev/null +++ b/examples/brownfield/zod/tsconfig.json @@ -0,0 +1,12 @@ +{ + "compilerOptions": { + "strict": true, + "target": "esnext", + "module": "nodenext", + "moduleResolution": "nodenext", + "noEmit": true, + "skipLibCheck": true, + "types": ["node"] + }, + "include": ["src", "proof"] +} diff --git a/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-authoring/SKILL.md b/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-authoring/SKILL.md index 85ac7bc5c..a1e709017 100644 --- a/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-authoring/SKILL.md +++ b/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-authoring/SKILL.md @@ -25,6 +25,7 @@ This file covers what almost every model needs. The topics below live in | File | Open it when | |---|---| | `references/adopting-existing-code.md` | working code or a populated schema already exists for what you are modeling | +| `references/adopting-existing-models.md` | moving existing Zod, Drizzle or Pydantic models into the model: the step-by-step recipe, and how to prove the generated types match before the switch | | `references/choosing-a-shape.md` | no row of "Canonical form for common field needs" fits; you need the full UUID / timestamp / URI / `@autoSet` rule; or you are registering your own vocabulary | | `references/json-columns.md` | a JSON column beyond the ladder below: coverage per port, `field.map` at runtime, version caveats, bag → native array | | `references/identities-and-indexes.md` | declaring a primary, secondary or reference identity, an FK, or a non-unique `index.lookup` | diff --git a/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-code.md b/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-code.md index 5d265812a..b611b9d1d 100644 --- a/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-code.md +++ b/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-code.md @@ -13,6 +13,9 @@ redefining it. (The metadata is still the durable spine *going forward*; only th **The observable predicate:** does working code or a populated schema already exist for what you're modeling? If yes, you are in adoption mode and these rules apply. +For Zod schemas, Drizzle tables or Pydantic models, `adopting-existing-models.md` is the +step-by-step recipe: generate beside them, prove the types match, move one call site at a time. + **Author metadata to match what the code ALREADY IS — not what you'd design fresh.** Read the existing code and schema *first*, then model to reproduce them: - The **native types the code uses** are the spec — model `field.uuid` when the code diff --git a/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-models.md b/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-models.md new file mode 100644 index 000000000..d35d70fe2 --- /dev/null +++ b/fixtures/agent-context-conformance/java-kotlin-react-tanstack/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-models.md @@ -0,0 +1,386 @@ +# Adopting existing models + +> Part of the `metaobjects-authoring` skill. Read it when a project already has Zod schemas, Drizzle tables or Pydantic models and you are moving them into the model. + +MetaObjects never reads the project's source. You read the existing types and write the model, and +the language's own tools compare the two: `tsc` for TypeScript, `model_json_schema()` for Python. + +## Contents +- The recipe +- Drizzle +- Zod +- Pydantic +- Fixing the differences +- What the proof does not cover + +## The recipe + +1. **Write the model from the existing types.** Read the types and model what they are, not what + you would design fresh. `references/adopting-existing-code.md` in the `metaobjects-authoring` + skill has the rules. +2. **Generate beside them.** Send the output to a new directory. Nothing imports it yet. +3. **Write the proof.** It is one test file that names each pair of types. +4. **See it fail once.** Break one field in the model, regenerate, and check that the proof names + that field. Then undo it. +5. **Fix the differences in the model,** not in the existing code, until the proof passes. +6. **Check the database,** if the project has one. +7. **Move one call site at a time.** Run the proof and the project's tests after each move. +8. **Retire the old code,** delete the proof, and keep `meta verify` in CI. + +Step 4 is not optional. A proof that cannot fail looks the same as one that passes. If the proof +stays green with a field broken, its file is outside `include` in `tsconfig.json`, or `strict` is +off, or pytest never collected it. + +### Drizzle + +Install the CLI with `npm i -D @metaobjectsdev/cli @metaobjectsdev/codegen-ts typescript`, then run +`npx meta init` once in the project root. + +Write the model. Field names are the TypeScript property names. `snake_case` is the default for +columns, so `displayName` already means `display_name`. Reuse the names the database has: the name +of a `.unique()` index goes on `identity.secondary`, and an existing foreign key name goes in +`constraintName`. + +```yaml +metadata: + package: shop + children: + - object.entity: + name: Customer + children: + - source.rdb: { table: customers } + - field.uuid: { name: id, required: true } + - field.string: { name: email, required: true, maxLength: 254 } + - field.string: { name: displayName, required: true, maxLength: 120 } + - field.int: { name: loyaltyPoints, required: true, default: 0 } + - field.string: { name: notes } + - field.timestamp: { name: createdAt, required: true, default: now(), mutability: readOnly } + - identity.primary: { fields: id, generation: uuid } + - identity.secondary: { name: customers_email_unique, fields: email } +``` + +Generate beside the existing schema: + +```ts +// metaobjects.config.ts +import { defineConfig } from "@metaobjectsdev/cli"; + +export default defineConfig({ + outDir: "src/generated", // beside the existing code; nothing imports it yet + dialect: "postgres", + timestampMode: "date", // when the existing timestamp columns read as Date + generators: ["entity"], +}); +``` + +Run `npx meta gen` to generate. + +Copy this helper into `proof/differs.ts`: + +```ts +// Names every field that differs between two object types, or `never` when they match. +// +// Mutual assignability alone is not enough. A field that is optional and present on only one +// side passes both ways, and `any` is assignable to and from everything, so this compares +// key by key and treats `any` on one side as a difference. Pick<> keeps each field's optional +// modifier. It looks one level down: give a nested type its own line in equivalence.ts. +type Mutual = [A] extends [B] ? ([B] extends [A] ? true : false) : false; +type IsAny = 0 extends 1 & T ? true : false; + +export type Differs = { + [K in keyof A | keyof B]: K extends keyof A & keyof B + ? [IsAny] extends [IsAny] + ? Mutual, Pick> extends true + ? never + : K + : K + : K; +}[keyof A | keyof B]; + +// tsc reports each differing field as: Type '"name"' does not satisfy the constraint 'never'. +export type Same<_NoDifferences extends never> = true; +``` + +Then name each pair in `proof/equivalence.ts`, inside `include` in `tsconfig.json`, with `strict` +on. Compare what a query reads and what an insert takes: + +```ts +import type { Customer, NewCustomer } from "../src/db/schema.js"; +import type { Customer as GeneratedCustomer, CustomerInsert } from "../src/generated/Customer.js"; +import type { Differs, Same } from "./differs.js"; + +export type Proof = [ + Same>, + Same>, +]; +``` + +`npx tsc --noEmit` is the check. + +Types do not include column names, so `tsc` passes even with a wrong column. `meta verify --db` is +the check for those (it needs `pg` installed): + +```sh +npx meta verify --codegen --db "$DATABASE_URL" +``` + +`--db` runs only the database check unless you name the others, so keep `--codegen` beside it. + +Two Drizzle objects for one table can exist side by side, since each only describes SQL. Move a +call site by changing its import, with an alias where the generated name differs: + +```ts +import { type Customer, customers, type CustomerInsert as NewCustomer } from "./generated/Customer.js"; +``` + +When the last one has moved, replace the old table definitions with re-exports and delete +`proof/`: + +```ts +export { customers } from "../generated/Customer.js"; +export { orders } from "../generated/Order.js"; +``` + +Keep one migration tool per database. To stay on drizzle-kit, point it at this file. To move to +`meta migrate`, adopt the live schema with `meta migrate baseline --from-db`. + +### Zod + +Install the CLI with `npm i -D @metaobjectsdev/cli @metaobjectsdev/codegen-ts typescript`, then run +`npx meta init` once in the project root. + +A Zod project often has no ORM, so generate a contract-only target. An object with no table is an +`object.value`: + +```ts +// metaobjects.config.ts +import { defineConfig } from "@metaobjectsdev/cli"; +import { entityFile } from "@metaobjectsdev/codegen-ts/generators"; + +export default defineConfig({ + outDir: "src/generated", + dialect: "postgres", + targets: { + contract: { outDir: "src/generated", runtime: false }, // Zod schemas and types, no ORM + }, + generators: [entityFile({ target: "contract" })], +}); +``` + +```yaml +metadata: + package: api + children: + - object.value: + name: CreateCustomer + children: + - field.string: { name: email, required: true, maxLength: 254 } + - field.int: { name: seats } + - field.string[]: { name: tags, required: true } +``` + +Run `npx meta gen` to generate. The schema for `CreateCustomer` is `CreateCustomerInsertSchema`. The generated `CreateCustomer` is +the type. Use the same `proof/differs.ts` as above, and compare both `z.input` and `z.output`, +because a default or a transform changes one without the other: + +```ts +import type { z } from "zod"; +import type { CreateCustomer } from "../src/api/schemas.js"; +import type { CreateCustomerInsertSchema } from "../src/generated/CreateCustomer.js"; +import type { Differs, Same } from "./differs.js"; + +export type Proof = [ + Same, z.input>>, + Same, z.output>>, +]; +``` + +`tsc` cannot see `.min()`, `.max()` or `.email()`, because they do not change the type. Zod can +describe them, so add `proof/rules.ts` and run it with your test runner, or with node or bun: + +```ts +import assert from "node:assert/strict"; +import { z } from "zod"; +import { CreateCustomer } from "../src/api/schemas.js"; +import { CreateCustomerInsertSchema } from "../src/generated/CreateCustomer.js"; + +// `required` follows the order of the fields, which does not change what a schema accepts. +const rules = (schema: z.ZodType, io: "input" | "output") => + JSON.parse( + JSON.stringify(z.toJSONSchema(schema, { io, unrepresentable: "any" }), (key, value) => + key === "required" && Array.isArray(value) ? [...value].sort() : value, + ), + ); + +for (const io of ["input", "output"] as const) { + assert.deepEqual(rules(CreateCustomer, io), rules(CreateCustomerInsertSchema, io)); +} +``` + +`z.toJSONSchema` is Zod 4. On Zod 3, sort and compare `zodToJsonSchema(schema)` from the +`zod-to-json-schema` package instead. + +A required string generates `.min(1)`, so a hand-written `z.string()` with no minimum shows up here +as a `minLength` difference. If the existing schema really accepts an empty string, say so in the +model with `validator.length: { min: 0 }` as a child of the field. + +Move a call site by swapping the import and the schema name: + +```ts +import { type CreateCustomer, CreateCustomerInsertSchema } from "./generated/CreateCustomer.js"; + +export function parseSignup(body: unknown): CreateCustomer { + return CreateCustomerInsertSchema.parse(body); +} +``` + +### Pydantic + +Install with `pip install metaobjects "pydantic>=2.8"`. + +The model's field name is the Pydantic attribute, so keep `snake_case` names in the model. A shared +`str` Enum is an abstract `field.enum` that each field `extends`. An enum declared inline on the +field generates a `Literal` instead. + +```yaml +metadata: + package: shop + children: + - field.enum: + name: OrderStatus + abstract: true + values: [pending, paid, shipped] + - object.entity: + name: Order + children: + - source.rdb: { table: orders } + - field.long: { name: id, required: true } + - field.enum: { name: status, extends: OrderStatus, required: true, default: pending } + - field.string[]: { name: tags, required: true, default: "{}" } + - field.date: { name: placed_on, required: true } + - identity.primary: { fields: id, generation: increment } +``` + +Generate beside the existing models: + +```yaml +# metaobjects.config.yaml +metadata: metaobjects +targets: + app: + outDir: generated # beside the existing code; nothing imports it yet + generators: [entity] +``` + +Run `metaobjects gen` from the project root to generate. + +mypy cannot compare two Pydantic models, because a class is its own type, whatever its fields are. +Pydantic can describe a model as JSON Schema, so compare those. Pydantic validates input and writes +output differently, so check both modes. Copy this into `proof/test_equivalence.py` and name each +pair: + +```python +from typing import Any + +import pytest +from pydantic import BaseModel + +import app.models as existing +from generated import Customer, Order, ShippingAddress + + +def shape(model: type[BaseModel], mode: str) -> dict[str, Any]: + """The schema with what does not change behaviour removed: titles and descriptions, the + order of `required`, and `$ref`, which is inlined so a `str` Enum matches a Literal.""" + schema = model.model_json_schema(mode=mode) + defs = schema.pop("$defs", {}) + + def walk(node: Any, seen: tuple[str, ...] = ()) -> Any: + if isinstance(node, list): + return [walk(item, seen) for item in node] + if not isinstance(node, dict): + return node + if "$ref" in node: + name = node["$ref"].rsplit("/", 1)[-1] + if name in seen: # a model that contains itself + return {"$recursive": name} + rest = {key: value for key, value in node.items() if key != "$ref"} + return walk({**defs[name], **rest}, (*seen, name)) + out: dict[str, Any] = {} + for key, value in node.items(): + if key in ("title", "description"): + continue + if key == "properties": + out[key] = {field: walk(prop, seen) for field, prop in value.items()} + elif key == "required": + out[key] = sorted(value) + elif key == "default": + out[key] = value + else: + out[key] = walk(value, seen) + return out + + return walk(schema) + + +@pytest.mark.parametrize("mode", ["validation", "serialization"]) +@pytest.mark.parametrize( + ("hand_written", "generated"), + [ + (existing.Customer, Customer.Customer), + (existing.CustomerCreate, Customer.CustomerCreate), + (existing.ShippingAddress, ShippingAddress.ShippingAddress), + (existing.Order, Order.Order), + ], +) +def test_same_shape(hand_written: type[BaseModel], generated: type[BaseModel], mode: str) -> None: + assert shape(hand_written, mode) == shape(generated, mode) +``` + +Run `python -m pytest proof/ -vv` from the project root. `-vv` prints the whole difference, with +the field name in it. Schemas carry field constraints such as `max_length`, so this also catches a +rule the Python type does not show. + +Move a call site by changing its import: + +```python +from generated.Customer import Customer, CustomerCreate +``` + +Move models that nest together. Pydantic refuses an instance of a different class even when the +shape matches, so a generated `ShippingAddress` passed to the hand-written `Order` raises a +`ValidationError`. If they cannot move together, hand over `address.model_dump()` at the seam. + +The check against a database is `meta verify --db`, which needs the Node CLI, because schema +tooling is TypeScript only. `metaobjects verify` compares the generated code with the model. + +## Fixing the differences + +Fix each one in the model or the config, and leave the existing code alone until it has moved. + +| What you see | Why | Fix | +|---|---|---| +| A timestamp is `Date` on one side and `string` on the other | Generated timestamps read as strings by default | `timestampMode: "date"` in the config (Postgres) | +| A field the database fills is required on insert | The model has no default | Add `default:` to the field | +| A database-set timestamp is accepted on create | The field is writable | `default: now(), mutability: readOnly` | +| An array column's default is reported by `verify --db` | An array `default` must be a string | `default: "{}"`, not `default: []` | +| `verify --db` reports a different foreign key name | The generated name differs | `constraintName` on `identity.reference` | +| `verify --db` reports a different unique index name | The name comes from the identity | Name `identity.secondary` after the existing index | +| A `tsc` failure names a field that is `T \| null` on one side | The existing type allows both null and absent | Model what the code relies on. If the difference is deliberate, keep it visible: `Same, "notes">>` with a comment | + +## What the proof does not cover + +- **Custom logic.** A `.refine()`, a `.transform()` or a Pydantic validator is not in the type or + in the JSON Schema, so neither side of the proof sees it, and the model does not carry it. Keep + it in your code: refine the generated schema, or subclass the generated model. +- **Behaviour.** The proof compares declared shapes. The project's own tests still have to pass + after each move. +- **Native Postgres enums.** A `pgEnum` column cannot be modelled: `field.enum` is text plus a + CHECK. `tsc` passes and `meta verify --db` reports the missing CHECK. Convert the column to text + with a CHECK first, or leave that table out for now. +- **Python enum member names.** JSON Schema sees the values, not the member names. A generated enum + has `UPPER_CASE` members, so `OrderStatus.pending` in a call site must become + `OrderStatus.PENDING`. mypy or pyright finds those call sites. +- **Extra optional fields on a nested TypeScript type.** `Differs` compares a nested field as a + whole, so an optional extra inside it still assigns both ways. Give each nested type its own line. +- **Pairs you did not name.** The proof compares only what it lists, so list every exported type. diff --git a/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-authoring/SKILL.md b/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-authoring/SKILL.md index 85ac7bc5c..a1e709017 100644 --- a/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-authoring/SKILL.md +++ b/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-authoring/SKILL.md @@ -25,6 +25,7 @@ This file covers what almost every model needs. The topics below live in | File | Open it when | |---|---| | `references/adopting-existing-code.md` | working code or a populated schema already exists for what you are modeling | +| `references/adopting-existing-models.md` | moving existing Zod, Drizzle or Pydantic models into the model: the step-by-step recipe, and how to prove the generated types match before the switch | | `references/choosing-a-shape.md` | no row of "Canonical form for common field needs" fits; you need the full UUID / timestamp / URI / `@autoSet` rule; or you are registering your own vocabulary | | `references/json-columns.md` | a JSON column beyond the ladder below: coverage per port, `field.map` at runtime, version caveats, bag → native array | | `references/identities-and-indexes.md` | declaring a primary, secondary or reference identity, an FK, or a non-unique `index.lookup` | diff --git a/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-code.md b/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-code.md index 5d265812a..b611b9d1d 100644 --- a/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-code.md +++ b/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-code.md @@ -13,6 +13,9 @@ redefining it. (The metadata is still the durable spine *going forward*; only th **The observable predicate:** does working code or a populated schema already exist for what you're modeling? If yes, you are in adoption mode and these rules apply. +For Zod schemas, Drizzle tables or Pydantic models, `adopting-existing-models.md` is the +step-by-step recipe: generate beside them, prove the types match, move one call site at a time. + **Author metadata to match what the code ALREADY IS — not what you'd design fresh.** Read the existing code and schema *first*, then model to reproduce them: - The **native types the code uses** are the spec — model `field.uuid` when the code diff --git a/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-models.md b/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-models.md new file mode 100644 index 000000000..d35d70fe2 --- /dev/null +++ b/fixtures/agent-context-conformance/java-react/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-models.md @@ -0,0 +1,386 @@ +# Adopting existing models + +> Part of the `metaobjects-authoring` skill. Read it when a project already has Zod schemas, Drizzle tables or Pydantic models and you are moving them into the model. + +MetaObjects never reads the project's source. You read the existing types and write the model, and +the language's own tools compare the two: `tsc` for TypeScript, `model_json_schema()` for Python. + +## Contents +- The recipe +- Drizzle +- Zod +- Pydantic +- Fixing the differences +- What the proof does not cover + +## The recipe + +1. **Write the model from the existing types.** Read the types and model what they are, not what + you would design fresh. `references/adopting-existing-code.md` in the `metaobjects-authoring` + skill has the rules. +2. **Generate beside them.** Send the output to a new directory. Nothing imports it yet. +3. **Write the proof.** It is one test file that names each pair of types. +4. **See it fail once.** Break one field in the model, regenerate, and check that the proof names + that field. Then undo it. +5. **Fix the differences in the model,** not in the existing code, until the proof passes. +6. **Check the database,** if the project has one. +7. **Move one call site at a time.** Run the proof and the project's tests after each move. +8. **Retire the old code,** delete the proof, and keep `meta verify` in CI. + +Step 4 is not optional. A proof that cannot fail looks the same as one that passes. If the proof +stays green with a field broken, its file is outside `include` in `tsconfig.json`, or `strict` is +off, or pytest never collected it. + +### Drizzle + +Install the CLI with `npm i -D @metaobjectsdev/cli @metaobjectsdev/codegen-ts typescript`, then run +`npx meta init` once in the project root. + +Write the model. Field names are the TypeScript property names. `snake_case` is the default for +columns, so `displayName` already means `display_name`. Reuse the names the database has: the name +of a `.unique()` index goes on `identity.secondary`, and an existing foreign key name goes in +`constraintName`. + +```yaml +metadata: + package: shop + children: + - object.entity: + name: Customer + children: + - source.rdb: { table: customers } + - field.uuid: { name: id, required: true } + - field.string: { name: email, required: true, maxLength: 254 } + - field.string: { name: displayName, required: true, maxLength: 120 } + - field.int: { name: loyaltyPoints, required: true, default: 0 } + - field.string: { name: notes } + - field.timestamp: { name: createdAt, required: true, default: now(), mutability: readOnly } + - identity.primary: { fields: id, generation: uuid } + - identity.secondary: { name: customers_email_unique, fields: email } +``` + +Generate beside the existing schema: + +```ts +// metaobjects.config.ts +import { defineConfig } from "@metaobjectsdev/cli"; + +export default defineConfig({ + outDir: "src/generated", // beside the existing code; nothing imports it yet + dialect: "postgres", + timestampMode: "date", // when the existing timestamp columns read as Date + generators: ["entity"], +}); +``` + +Run `npx meta gen` to generate. + +Copy this helper into `proof/differs.ts`: + +```ts +// Names every field that differs between two object types, or `never` when they match. +// +// Mutual assignability alone is not enough. A field that is optional and present on only one +// side passes both ways, and `any` is assignable to and from everything, so this compares +// key by key and treats `any` on one side as a difference. Pick<> keeps each field's optional +// modifier. It looks one level down: give a nested type its own line in equivalence.ts. +type Mutual = [A] extends [B] ? ([B] extends [A] ? true : false) : false; +type IsAny = 0 extends 1 & T ? true : false; + +export type Differs = { + [K in keyof A | keyof B]: K extends keyof A & keyof B + ? [IsAny] extends [IsAny] + ? Mutual, Pick> extends true + ? never + : K + : K + : K; +}[keyof A | keyof B]; + +// tsc reports each differing field as: Type '"name"' does not satisfy the constraint 'never'. +export type Same<_NoDifferences extends never> = true; +``` + +Then name each pair in `proof/equivalence.ts`, inside `include` in `tsconfig.json`, with `strict` +on. Compare what a query reads and what an insert takes: + +```ts +import type { Customer, NewCustomer } from "../src/db/schema.js"; +import type { Customer as GeneratedCustomer, CustomerInsert } from "../src/generated/Customer.js"; +import type { Differs, Same } from "./differs.js"; + +export type Proof = [ + Same>, + Same>, +]; +``` + +`npx tsc --noEmit` is the check. + +Types do not include column names, so `tsc` passes even with a wrong column. `meta verify --db` is +the check for those (it needs `pg` installed): + +```sh +npx meta verify --codegen --db "$DATABASE_URL" +``` + +`--db` runs only the database check unless you name the others, so keep `--codegen` beside it. + +Two Drizzle objects for one table can exist side by side, since each only describes SQL. Move a +call site by changing its import, with an alias where the generated name differs: + +```ts +import { type Customer, customers, type CustomerInsert as NewCustomer } from "./generated/Customer.js"; +``` + +When the last one has moved, replace the old table definitions with re-exports and delete +`proof/`: + +```ts +export { customers } from "../generated/Customer.js"; +export { orders } from "../generated/Order.js"; +``` + +Keep one migration tool per database. To stay on drizzle-kit, point it at this file. To move to +`meta migrate`, adopt the live schema with `meta migrate baseline --from-db`. + +### Zod + +Install the CLI with `npm i -D @metaobjectsdev/cli @metaobjectsdev/codegen-ts typescript`, then run +`npx meta init` once in the project root. + +A Zod project often has no ORM, so generate a contract-only target. An object with no table is an +`object.value`: + +```ts +// metaobjects.config.ts +import { defineConfig } from "@metaobjectsdev/cli"; +import { entityFile } from "@metaobjectsdev/codegen-ts/generators"; + +export default defineConfig({ + outDir: "src/generated", + dialect: "postgres", + targets: { + contract: { outDir: "src/generated", runtime: false }, // Zod schemas and types, no ORM + }, + generators: [entityFile({ target: "contract" })], +}); +``` + +```yaml +metadata: + package: api + children: + - object.value: + name: CreateCustomer + children: + - field.string: { name: email, required: true, maxLength: 254 } + - field.int: { name: seats } + - field.string[]: { name: tags, required: true } +``` + +Run `npx meta gen` to generate. The schema for `CreateCustomer` is `CreateCustomerInsertSchema`. The generated `CreateCustomer` is +the type. Use the same `proof/differs.ts` as above, and compare both `z.input` and `z.output`, +because a default or a transform changes one without the other: + +```ts +import type { z } from "zod"; +import type { CreateCustomer } from "../src/api/schemas.js"; +import type { CreateCustomerInsertSchema } from "../src/generated/CreateCustomer.js"; +import type { Differs, Same } from "./differs.js"; + +export type Proof = [ + Same, z.input>>, + Same, z.output>>, +]; +``` + +`tsc` cannot see `.min()`, `.max()` or `.email()`, because they do not change the type. Zod can +describe them, so add `proof/rules.ts` and run it with your test runner, or with node or bun: + +```ts +import assert from "node:assert/strict"; +import { z } from "zod"; +import { CreateCustomer } from "../src/api/schemas.js"; +import { CreateCustomerInsertSchema } from "../src/generated/CreateCustomer.js"; + +// `required` follows the order of the fields, which does not change what a schema accepts. +const rules = (schema: z.ZodType, io: "input" | "output") => + JSON.parse( + JSON.stringify(z.toJSONSchema(schema, { io, unrepresentable: "any" }), (key, value) => + key === "required" && Array.isArray(value) ? [...value].sort() : value, + ), + ); + +for (const io of ["input", "output"] as const) { + assert.deepEqual(rules(CreateCustomer, io), rules(CreateCustomerInsertSchema, io)); +} +``` + +`z.toJSONSchema` is Zod 4. On Zod 3, sort and compare `zodToJsonSchema(schema)` from the +`zod-to-json-schema` package instead. + +A required string generates `.min(1)`, so a hand-written `z.string()` with no minimum shows up here +as a `minLength` difference. If the existing schema really accepts an empty string, say so in the +model with `validator.length: { min: 0 }` as a child of the field. + +Move a call site by swapping the import and the schema name: + +```ts +import { type CreateCustomer, CreateCustomerInsertSchema } from "./generated/CreateCustomer.js"; + +export function parseSignup(body: unknown): CreateCustomer { + return CreateCustomerInsertSchema.parse(body); +} +``` + +### Pydantic + +Install with `pip install metaobjects "pydantic>=2.8"`. + +The model's field name is the Pydantic attribute, so keep `snake_case` names in the model. A shared +`str` Enum is an abstract `field.enum` that each field `extends`. An enum declared inline on the +field generates a `Literal` instead. + +```yaml +metadata: + package: shop + children: + - field.enum: + name: OrderStatus + abstract: true + values: [pending, paid, shipped] + - object.entity: + name: Order + children: + - source.rdb: { table: orders } + - field.long: { name: id, required: true } + - field.enum: { name: status, extends: OrderStatus, required: true, default: pending } + - field.string[]: { name: tags, required: true, default: "{}" } + - field.date: { name: placed_on, required: true } + - identity.primary: { fields: id, generation: increment } +``` + +Generate beside the existing models: + +```yaml +# metaobjects.config.yaml +metadata: metaobjects +targets: + app: + outDir: generated # beside the existing code; nothing imports it yet + generators: [entity] +``` + +Run `metaobjects gen` from the project root to generate. + +mypy cannot compare two Pydantic models, because a class is its own type, whatever its fields are. +Pydantic can describe a model as JSON Schema, so compare those. Pydantic validates input and writes +output differently, so check both modes. Copy this into `proof/test_equivalence.py` and name each +pair: + +```python +from typing import Any + +import pytest +from pydantic import BaseModel + +import app.models as existing +from generated import Customer, Order, ShippingAddress + + +def shape(model: type[BaseModel], mode: str) -> dict[str, Any]: + """The schema with what does not change behaviour removed: titles and descriptions, the + order of `required`, and `$ref`, which is inlined so a `str` Enum matches a Literal.""" + schema = model.model_json_schema(mode=mode) + defs = schema.pop("$defs", {}) + + def walk(node: Any, seen: tuple[str, ...] = ()) -> Any: + if isinstance(node, list): + return [walk(item, seen) for item in node] + if not isinstance(node, dict): + return node + if "$ref" in node: + name = node["$ref"].rsplit("/", 1)[-1] + if name in seen: # a model that contains itself + return {"$recursive": name} + rest = {key: value for key, value in node.items() if key != "$ref"} + return walk({**defs[name], **rest}, (*seen, name)) + out: dict[str, Any] = {} + for key, value in node.items(): + if key in ("title", "description"): + continue + if key == "properties": + out[key] = {field: walk(prop, seen) for field, prop in value.items()} + elif key == "required": + out[key] = sorted(value) + elif key == "default": + out[key] = value + else: + out[key] = walk(value, seen) + return out + + return walk(schema) + + +@pytest.mark.parametrize("mode", ["validation", "serialization"]) +@pytest.mark.parametrize( + ("hand_written", "generated"), + [ + (existing.Customer, Customer.Customer), + (existing.CustomerCreate, Customer.CustomerCreate), + (existing.ShippingAddress, ShippingAddress.ShippingAddress), + (existing.Order, Order.Order), + ], +) +def test_same_shape(hand_written: type[BaseModel], generated: type[BaseModel], mode: str) -> None: + assert shape(hand_written, mode) == shape(generated, mode) +``` + +Run `python -m pytest proof/ -vv` from the project root. `-vv` prints the whole difference, with +the field name in it. Schemas carry field constraints such as `max_length`, so this also catches a +rule the Python type does not show. + +Move a call site by changing its import: + +```python +from generated.Customer import Customer, CustomerCreate +``` + +Move models that nest together. Pydantic refuses an instance of a different class even when the +shape matches, so a generated `ShippingAddress` passed to the hand-written `Order` raises a +`ValidationError`. If they cannot move together, hand over `address.model_dump()` at the seam. + +The check against a database is `meta verify --db`, which needs the Node CLI, because schema +tooling is TypeScript only. `metaobjects verify` compares the generated code with the model. + +## Fixing the differences + +Fix each one in the model or the config, and leave the existing code alone until it has moved. + +| What you see | Why | Fix | +|---|---|---| +| A timestamp is `Date` on one side and `string` on the other | Generated timestamps read as strings by default | `timestampMode: "date"` in the config (Postgres) | +| A field the database fills is required on insert | The model has no default | Add `default:` to the field | +| A database-set timestamp is accepted on create | The field is writable | `default: now(), mutability: readOnly` | +| An array column's default is reported by `verify --db` | An array `default` must be a string | `default: "{}"`, not `default: []` | +| `verify --db` reports a different foreign key name | The generated name differs | `constraintName` on `identity.reference` | +| `verify --db` reports a different unique index name | The name comes from the identity | Name `identity.secondary` after the existing index | +| A `tsc` failure names a field that is `T \| null` on one side | The existing type allows both null and absent | Model what the code relies on. If the difference is deliberate, keep it visible: `Same, "notes">>` with a comment | + +## What the proof does not cover + +- **Custom logic.** A `.refine()`, a `.transform()` or a Pydantic validator is not in the type or + in the JSON Schema, so neither side of the proof sees it, and the model does not carry it. Keep + it in your code: refine the generated schema, or subclass the generated model. +- **Behaviour.** The proof compares declared shapes. The project's own tests still have to pass + after each move. +- **Native Postgres enums.** A `pgEnum` column cannot be modelled: `field.enum` is text plus a + CHECK. `tsc` passes and `meta verify --db` reports the missing CHECK. Convert the column to text + with a CHECK first, or leave that table out for now. +- **Python enum member names.** JSON Schema sees the values, not the member names. A generated enum + has `UPPER_CASE` members, so `OrderStatus.pending` in a call site must become + `OrderStatus.PENDING`. mypy or pyright finds those call sites. +- **Extra optional fields on a nested TypeScript type.** `Differs` compares a nested field as a + whole, so an optional extra inside it still assigns both ways. Give each nested type its own line. +- **Pairs you did not name.** The proof compares only what it lists, so list every exported type. diff --git a/fixtures/agent-context-conformance/python/expected/.claude/skills/metaobjects-authoring/SKILL.md b/fixtures/agent-context-conformance/python/expected/.claude/skills/metaobjects-authoring/SKILL.md index 85ac7bc5c..a1e709017 100644 --- a/fixtures/agent-context-conformance/python/expected/.claude/skills/metaobjects-authoring/SKILL.md +++ b/fixtures/agent-context-conformance/python/expected/.claude/skills/metaobjects-authoring/SKILL.md @@ -25,6 +25,7 @@ This file covers what almost every model needs. The topics below live in | File | Open it when | |---|---| | `references/adopting-existing-code.md` | working code or a populated schema already exists for what you are modeling | +| `references/adopting-existing-models.md` | moving existing Zod, Drizzle or Pydantic models into the model: the step-by-step recipe, and how to prove the generated types match before the switch | | `references/choosing-a-shape.md` | no row of "Canonical form for common field needs" fits; you need the full UUID / timestamp / URI / `@autoSet` rule; or you are registering your own vocabulary | | `references/json-columns.md` | a JSON column beyond the ladder below: coverage per port, `field.map` at runtime, version caveats, bag → native array | | `references/identities-and-indexes.md` | declaring a primary, secondary or reference identity, an FK, or a non-unique `index.lookup` | diff --git a/fixtures/agent-context-conformance/python/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-code.md b/fixtures/agent-context-conformance/python/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-code.md index 5d265812a..b611b9d1d 100644 --- a/fixtures/agent-context-conformance/python/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-code.md +++ b/fixtures/agent-context-conformance/python/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-code.md @@ -13,6 +13,9 @@ redefining it. (The metadata is still the durable spine *going forward*; only th **The observable predicate:** does working code or a populated schema already exist for what you're modeling? If yes, you are in adoption mode and these rules apply. +For Zod schemas, Drizzle tables or Pydantic models, `adopting-existing-models.md` is the +step-by-step recipe: generate beside them, prove the types match, move one call site at a time. + **Author metadata to match what the code ALREADY IS — not what you'd design fresh.** Read the existing code and schema *first*, then model to reproduce them: - The **native types the code uses** are the spec — model `field.uuid` when the code diff --git a/fixtures/agent-context-conformance/python/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-models.md b/fixtures/agent-context-conformance/python/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-models.md new file mode 100644 index 000000000..d35d70fe2 --- /dev/null +++ b/fixtures/agent-context-conformance/python/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-models.md @@ -0,0 +1,386 @@ +# Adopting existing models + +> Part of the `metaobjects-authoring` skill. Read it when a project already has Zod schemas, Drizzle tables or Pydantic models and you are moving them into the model. + +MetaObjects never reads the project's source. You read the existing types and write the model, and +the language's own tools compare the two: `tsc` for TypeScript, `model_json_schema()` for Python. + +## Contents +- The recipe +- Drizzle +- Zod +- Pydantic +- Fixing the differences +- What the proof does not cover + +## The recipe + +1. **Write the model from the existing types.** Read the types and model what they are, not what + you would design fresh. `references/adopting-existing-code.md` in the `metaobjects-authoring` + skill has the rules. +2. **Generate beside them.** Send the output to a new directory. Nothing imports it yet. +3. **Write the proof.** It is one test file that names each pair of types. +4. **See it fail once.** Break one field in the model, regenerate, and check that the proof names + that field. Then undo it. +5. **Fix the differences in the model,** not in the existing code, until the proof passes. +6. **Check the database,** if the project has one. +7. **Move one call site at a time.** Run the proof and the project's tests after each move. +8. **Retire the old code,** delete the proof, and keep `meta verify` in CI. + +Step 4 is not optional. A proof that cannot fail looks the same as one that passes. If the proof +stays green with a field broken, its file is outside `include` in `tsconfig.json`, or `strict` is +off, or pytest never collected it. + +### Drizzle + +Install the CLI with `npm i -D @metaobjectsdev/cli @metaobjectsdev/codegen-ts typescript`, then run +`npx meta init` once in the project root. + +Write the model. Field names are the TypeScript property names. `snake_case` is the default for +columns, so `displayName` already means `display_name`. Reuse the names the database has: the name +of a `.unique()` index goes on `identity.secondary`, and an existing foreign key name goes in +`constraintName`. + +```yaml +metadata: + package: shop + children: + - object.entity: + name: Customer + children: + - source.rdb: { table: customers } + - field.uuid: { name: id, required: true } + - field.string: { name: email, required: true, maxLength: 254 } + - field.string: { name: displayName, required: true, maxLength: 120 } + - field.int: { name: loyaltyPoints, required: true, default: 0 } + - field.string: { name: notes } + - field.timestamp: { name: createdAt, required: true, default: now(), mutability: readOnly } + - identity.primary: { fields: id, generation: uuid } + - identity.secondary: { name: customers_email_unique, fields: email } +``` + +Generate beside the existing schema: + +```ts +// metaobjects.config.ts +import { defineConfig } from "@metaobjectsdev/cli"; + +export default defineConfig({ + outDir: "src/generated", // beside the existing code; nothing imports it yet + dialect: "postgres", + timestampMode: "date", // when the existing timestamp columns read as Date + generators: ["entity"], +}); +``` + +Run `npx meta gen` to generate. + +Copy this helper into `proof/differs.ts`: + +```ts +// Names every field that differs between two object types, or `never` when they match. +// +// Mutual assignability alone is not enough. A field that is optional and present on only one +// side passes both ways, and `any` is assignable to and from everything, so this compares +// key by key and treats `any` on one side as a difference. Pick<> keeps each field's optional +// modifier. It looks one level down: give a nested type its own line in equivalence.ts. +type Mutual = [A] extends [B] ? ([B] extends [A] ? true : false) : false; +type IsAny = 0 extends 1 & T ? true : false; + +export type Differs = { + [K in keyof A | keyof B]: K extends keyof A & keyof B + ? [IsAny] extends [IsAny] + ? Mutual, Pick> extends true + ? never + : K + : K + : K; +}[keyof A | keyof B]; + +// tsc reports each differing field as: Type '"name"' does not satisfy the constraint 'never'. +export type Same<_NoDifferences extends never> = true; +``` + +Then name each pair in `proof/equivalence.ts`, inside `include` in `tsconfig.json`, with `strict` +on. Compare what a query reads and what an insert takes: + +```ts +import type { Customer, NewCustomer } from "../src/db/schema.js"; +import type { Customer as GeneratedCustomer, CustomerInsert } from "../src/generated/Customer.js"; +import type { Differs, Same } from "./differs.js"; + +export type Proof = [ + Same>, + Same>, +]; +``` + +`npx tsc --noEmit` is the check. + +Types do not include column names, so `tsc` passes even with a wrong column. `meta verify --db` is +the check for those (it needs `pg` installed): + +```sh +npx meta verify --codegen --db "$DATABASE_URL" +``` + +`--db` runs only the database check unless you name the others, so keep `--codegen` beside it. + +Two Drizzle objects for one table can exist side by side, since each only describes SQL. Move a +call site by changing its import, with an alias where the generated name differs: + +```ts +import { type Customer, customers, type CustomerInsert as NewCustomer } from "./generated/Customer.js"; +``` + +When the last one has moved, replace the old table definitions with re-exports and delete +`proof/`: + +```ts +export { customers } from "../generated/Customer.js"; +export { orders } from "../generated/Order.js"; +``` + +Keep one migration tool per database. To stay on drizzle-kit, point it at this file. To move to +`meta migrate`, adopt the live schema with `meta migrate baseline --from-db`. + +### Zod + +Install the CLI with `npm i -D @metaobjectsdev/cli @metaobjectsdev/codegen-ts typescript`, then run +`npx meta init` once in the project root. + +A Zod project often has no ORM, so generate a contract-only target. An object with no table is an +`object.value`: + +```ts +// metaobjects.config.ts +import { defineConfig } from "@metaobjectsdev/cli"; +import { entityFile } from "@metaobjectsdev/codegen-ts/generators"; + +export default defineConfig({ + outDir: "src/generated", + dialect: "postgres", + targets: { + contract: { outDir: "src/generated", runtime: false }, // Zod schemas and types, no ORM + }, + generators: [entityFile({ target: "contract" })], +}); +``` + +```yaml +metadata: + package: api + children: + - object.value: + name: CreateCustomer + children: + - field.string: { name: email, required: true, maxLength: 254 } + - field.int: { name: seats } + - field.string[]: { name: tags, required: true } +``` + +Run `npx meta gen` to generate. The schema for `CreateCustomer` is `CreateCustomerInsertSchema`. The generated `CreateCustomer` is +the type. Use the same `proof/differs.ts` as above, and compare both `z.input` and `z.output`, +because a default or a transform changes one without the other: + +```ts +import type { z } from "zod"; +import type { CreateCustomer } from "../src/api/schemas.js"; +import type { CreateCustomerInsertSchema } from "../src/generated/CreateCustomer.js"; +import type { Differs, Same } from "./differs.js"; + +export type Proof = [ + Same, z.input>>, + Same, z.output>>, +]; +``` + +`tsc` cannot see `.min()`, `.max()` or `.email()`, because they do not change the type. Zod can +describe them, so add `proof/rules.ts` and run it with your test runner, or with node or bun: + +```ts +import assert from "node:assert/strict"; +import { z } from "zod"; +import { CreateCustomer } from "../src/api/schemas.js"; +import { CreateCustomerInsertSchema } from "../src/generated/CreateCustomer.js"; + +// `required` follows the order of the fields, which does not change what a schema accepts. +const rules = (schema: z.ZodType, io: "input" | "output") => + JSON.parse( + JSON.stringify(z.toJSONSchema(schema, { io, unrepresentable: "any" }), (key, value) => + key === "required" && Array.isArray(value) ? [...value].sort() : value, + ), + ); + +for (const io of ["input", "output"] as const) { + assert.deepEqual(rules(CreateCustomer, io), rules(CreateCustomerInsertSchema, io)); +} +``` + +`z.toJSONSchema` is Zod 4. On Zod 3, sort and compare `zodToJsonSchema(schema)` from the +`zod-to-json-schema` package instead. + +A required string generates `.min(1)`, so a hand-written `z.string()` with no minimum shows up here +as a `minLength` difference. If the existing schema really accepts an empty string, say so in the +model with `validator.length: { min: 0 }` as a child of the field. + +Move a call site by swapping the import and the schema name: + +```ts +import { type CreateCustomer, CreateCustomerInsertSchema } from "./generated/CreateCustomer.js"; + +export function parseSignup(body: unknown): CreateCustomer { + return CreateCustomerInsertSchema.parse(body); +} +``` + +### Pydantic + +Install with `pip install metaobjects "pydantic>=2.8"`. + +The model's field name is the Pydantic attribute, so keep `snake_case` names in the model. A shared +`str` Enum is an abstract `field.enum` that each field `extends`. An enum declared inline on the +field generates a `Literal` instead. + +```yaml +metadata: + package: shop + children: + - field.enum: + name: OrderStatus + abstract: true + values: [pending, paid, shipped] + - object.entity: + name: Order + children: + - source.rdb: { table: orders } + - field.long: { name: id, required: true } + - field.enum: { name: status, extends: OrderStatus, required: true, default: pending } + - field.string[]: { name: tags, required: true, default: "{}" } + - field.date: { name: placed_on, required: true } + - identity.primary: { fields: id, generation: increment } +``` + +Generate beside the existing models: + +```yaml +# metaobjects.config.yaml +metadata: metaobjects +targets: + app: + outDir: generated # beside the existing code; nothing imports it yet + generators: [entity] +``` + +Run `metaobjects gen` from the project root to generate. + +mypy cannot compare two Pydantic models, because a class is its own type, whatever its fields are. +Pydantic can describe a model as JSON Schema, so compare those. Pydantic validates input and writes +output differently, so check both modes. Copy this into `proof/test_equivalence.py` and name each +pair: + +```python +from typing import Any + +import pytest +from pydantic import BaseModel + +import app.models as existing +from generated import Customer, Order, ShippingAddress + + +def shape(model: type[BaseModel], mode: str) -> dict[str, Any]: + """The schema with what does not change behaviour removed: titles and descriptions, the + order of `required`, and `$ref`, which is inlined so a `str` Enum matches a Literal.""" + schema = model.model_json_schema(mode=mode) + defs = schema.pop("$defs", {}) + + def walk(node: Any, seen: tuple[str, ...] = ()) -> Any: + if isinstance(node, list): + return [walk(item, seen) for item in node] + if not isinstance(node, dict): + return node + if "$ref" in node: + name = node["$ref"].rsplit("/", 1)[-1] + if name in seen: # a model that contains itself + return {"$recursive": name} + rest = {key: value for key, value in node.items() if key != "$ref"} + return walk({**defs[name], **rest}, (*seen, name)) + out: dict[str, Any] = {} + for key, value in node.items(): + if key in ("title", "description"): + continue + if key == "properties": + out[key] = {field: walk(prop, seen) for field, prop in value.items()} + elif key == "required": + out[key] = sorted(value) + elif key == "default": + out[key] = value + else: + out[key] = walk(value, seen) + return out + + return walk(schema) + + +@pytest.mark.parametrize("mode", ["validation", "serialization"]) +@pytest.mark.parametrize( + ("hand_written", "generated"), + [ + (existing.Customer, Customer.Customer), + (existing.CustomerCreate, Customer.CustomerCreate), + (existing.ShippingAddress, ShippingAddress.ShippingAddress), + (existing.Order, Order.Order), + ], +) +def test_same_shape(hand_written: type[BaseModel], generated: type[BaseModel], mode: str) -> None: + assert shape(hand_written, mode) == shape(generated, mode) +``` + +Run `python -m pytest proof/ -vv` from the project root. `-vv` prints the whole difference, with +the field name in it. Schemas carry field constraints such as `max_length`, so this also catches a +rule the Python type does not show. + +Move a call site by changing its import: + +```python +from generated.Customer import Customer, CustomerCreate +``` + +Move models that nest together. Pydantic refuses an instance of a different class even when the +shape matches, so a generated `ShippingAddress` passed to the hand-written `Order` raises a +`ValidationError`. If they cannot move together, hand over `address.model_dump()` at the seam. + +The check against a database is `meta verify --db`, which needs the Node CLI, because schema +tooling is TypeScript only. `metaobjects verify` compares the generated code with the model. + +## Fixing the differences + +Fix each one in the model or the config, and leave the existing code alone until it has moved. + +| What you see | Why | Fix | +|---|---|---| +| A timestamp is `Date` on one side and `string` on the other | Generated timestamps read as strings by default | `timestampMode: "date"` in the config (Postgres) | +| A field the database fills is required on insert | The model has no default | Add `default:` to the field | +| A database-set timestamp is accepted on create | The field is writable | `default: now(), mutability: readOnly` | +| An array column's default is reported by `verify --db` | An array `default` must be a string | `default: "{}"`, not `default: []` | +| `verify --db` reports a different foreign key name | The generated name differs | `constraintName` on `identity.reference` | +| `verify --db` reports a different unique index name | The name comes from the identity | Name `identity.secondary` after the existing index | +| A `tsc` failure names a field that is `T \| null` on one side | The existing type allows both null and absent | Model what the code relies on. If the difference is deliberate, keep it visible: `Same, "notes">>` with a comment | + +## What the proof does not cover + +- **Custom logic.** A `.refine()`, a `.transform()` or a Pydantic validator is not in the type or + in the JSON Schema, so neither side of the proof sees it, and the model does not carry it. Keep + it in your code: refine the generated schema, or subclass the generated model. +- **Behaviour.** The proof compares declared shapes. The project's own tests still have to pass + after each move. +- **Native Postgres enums.** A `pgEnum` column cannot be modelled: `field.enum` is text plus a + CHECK. `tsc` passes and `meta verify --db` reports the missing CHECK. Convert the column to text + with a CHECK first, or leave that table out for now. +- **Python enum member names.** JSON Schema sees the values, not the member names. A generated enum + has `UPPER_CASE` members, so `OrderStatus.pending` in a call site must become + `OrderStatus.PENDING`. mypy or pyright finds those call sites. +- **Extra optional fields on a nested TypeScript type.** `Differs` compares a nested field as a + whole, so an optional extra inside it still assigns both ways. Give each nested type its own line. +- **Pairs you did not name.** The proof compares only what it lists, so list every exported type. diff --git a/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-authoring/SKILL.md b/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-authoring/SKILL.md index 85ac7bc5c..a1e709017 100644 --- a/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-authoring/SKILL.md +++ b/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-authoring/SKILL.md @@ -25,6 +25,7 @@ This file covers what almost every model needs. The topics below live in | File | Open it when | |---|---| | `references/adopting-existing-code.md` | working code or a populated schema already exists for what you are modeling | +| `references/adopting-existing-models.md` | moving existing Zod, Drizzle or Pydantic models into the model: the step-by-step recipe, and how to prove the generated types match before the switch | | `references/choosing-a-shape.md` | no row of "Canonical form for common field needs" fits; you need the full UUID / timestamp / URI / `@autoSet` rule; or you are registering your own vocabulary | | `references/json-columns.md` | a JSON column beyond the ladder below: coverage per port, `field.map` at runtime, version caveats, bag → native array | | `references/identities-and-indexes.md` | declaring a primary, secondary or reference identity, an FK, or a non-unique `index.lookup` | diff --git a/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-code.md b/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-code.md index 5d265812a..b611b9d1d 100644 --- a/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-code.md +++ b/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-code.md @@ -13,6 +13,9 @@ redefining it. (The metadata is still the durable spine *going forward*; only th **The observable predicate:** does working code or a populated schema already exist for what you're modeling? If yes, you are in adoption mode and these rules apply. +For Zod schemas, Drizzle tables or Pydantic models, `adopting-existing-models.md` is the +step-by-step recipe: generate beside them, prove the types match, move one call site at a time. + **Author metadata to match what the code ALREADY IS — not what you'd design fresh.** Read the existing code and schema *first*, then model to reproduce them: - The **native types the code uses** are the spec — model `field.uuid` when the code diff --git a/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-models.md b/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-models.md new file mode 100644 index 000000000..d35d70fe2 --- /dev/null +++ b/fixtures/agent-context-conformance/ts-react-tanstack/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-models.md @@ -0,0 +1,386 @@ +# Adopting existing models + +> Part of the `metaobjects-authoring` skill. Read it when a project already has Zod schemas, Drizzle tables or Pydantic models and you are moving them into the model. + +MetaObjects never reads the project's source. You read the existing types and write the model, and +the language's own tools compare the two: `tsc` for TypeScript, `model_json_schema()` for Python. + +## Contents +- The recipe +- Drizzle +- Zod +- Pydantic +- Fixing the differences +- What the proof does not cover + +## The recipe + +1. **Write the model from the existing types.** Read the types and model what they are, not what + you would design fresh. `references/adopting-existing-code.md` in the `metaobjects-authoring` + skill has the rules. +2. **Generate beside them.** Send the output to a new directory. Nothing imports it yet. +3. **Write the proof.** It is one test file that names each pair of types. +4. **See it fail once.** Break one field in the model, regenerate, and check that the proof names + that field. Then undo it. +5. **Fix the differences in the model,** not in the existing code, until the proof passes. +6. **Check the database,** if the project has one. +7. **Move one call site at a time.** Run the proof and the project's tests after each move. +8. **Retire the old code,** delete the proof, and keep `meta verify` in CI. + +Step 4 is not optional. A proof that cannot fail looks the same as one that passes. If the proof +stays green with a field broken, its file is outside `include` in `tsconfig.json`, or `strict` is +off, or pytest never collected it. + +### Drizzle + +Install the CLI with `npm i -D @metaobjectsdev/cli @metaobjectsdev/codegen-ts typescript`, then run +`npx meta init` once in the project root. + +Write the model. Field names are the TypeScript property names. `snake_case` is the default for +columns, so `displayName` already means `display_name`. Reuse the names the database has: the name +of a `.unique()` index goes on `identity.secondary`, and an existing foreign key name goes in +`constraintName`. + +```yaml +metadata: + package: shop + children: + - object.entity: + name: Customer + children: + - source.rdb: { table: customers } + - field.uuid: { name: id, required: true } + - field.string: { name: email, required: true, maxLength: 254 } + - field.string: { name: displayName, required: true, maxLength: 120 } + - field.int: { name: loyaltyPoints, required: true, default: 0 } + - field.string: { name: notes } + - field.timestamp: { name: createdAt, required: true, default: now(), mutability: readOnly } + - identity.primary: { fields: id, generation: uuid } + - identity.secondary: { name: customers_email_unique, fields: email } +``` + +Generate beside the existing schema: + +```ts +// metaobjects.config.ts +import { defineConfig } from "@metaobjectsdev/cli"; + +export default defineConfig({ + outDir: "src/generated", // beside the existing code; nothing imports it yet + dialect: "postgres", + timestampMode: "date", // when the existing timestamp columns read as Date + generators: ["entity"], +}); +``` + +Run `npx meta gen` to generate. + +Copy this helper into `proof/differs.ts`: + +```ts +// Names every field that differs between two object types, or `never` when they match. +// +// Mutual assignability alone is not enough. A field that is optional and present on only one +// side passes both ways, and `any` is assignable to and from everything, so this compares +// key by key and treats `any` on one side as a difference. Pick<> keeps each field's optional +// modifier. It looks one level down: give a nested type its own line in equivalence.ts. +type Mutual = [A] extends [B] ? ([B] extends [A] ? true : false) : false; +type IsAny = 0 extends 1 & T ? true : false; + +export type Differs = { + [K in keyof A | keyof B]: K extends keyof A & keyof B + ? [IsAny] extends [IsAny] + ? Mutual, Pick> extends true + ? never + : K + : K + : K; +}[keyof A | keyof B]; + +// tsc reports each differing field as: Type '"name"' does not satisfy the constraint 'never'. +export type Same<_NoDifferences extends never> = true; +``` + +Then name each pair in `proof/equivalence.ts`, inside `include` in `tsconfig.json`, with `strict` +on. Compare what a query reads and what an insert takes: + +```ts +import type { Customer, NewCustomer } from "../src/db/schema.js"; +import type { Customer as GeneratedCustomer, CustomerInsert } from "../src/generated/Customer.js"; +import type { Differs, Same } from "./differs.js"; + +export type Proof = [ + Same>, + Same>, +]; +``` + +`npx tsc --noEmit` is the check. + +Types do not include column names, so `tsc` passes even with a wrong column. `meta verify --db` is +the check for those (it needs `pg` installed): + +```sh +npx meta verify --codegen --db "$DATABASE_URL" +``` + +`--db` runs only the database check unless you name the others, so keep `--codegen` beside it. + +Two Drizzle objects for one table can exist side by side, since each only describes SQL. Move a +call site by changing its import, with an alias where the generated name differs: + +```ts +import { type Customer, customers, type CustomerInsert as NewCustomer } from "./generated/Customer.js"; +``` + +When the last one has moved, replace the old table definitions with re-exports and delete +`proof/`: + +```ts +export { customers } from "../generated/Customer.js"; +export { orders } from "../generated/Order.js"; +``` + +Keep one migration tool per database. To stay on drizzle-kit, point it at this file. To move to +`meta migrate`, adopt the live schema with `meta migrate baseline --from-db`. + +### Zod + +Install the CLI with `npm i -D @metaobjectsdev/cli @metaobjectsdev/codegen-ts typescript`, then run +`npx meta init` once in the project root. + +A Zod project often has no ORM, so generate a contract-only target. An object with no table is an +`object.value`: + +```ts +// metaobjects.config.ts +import { defineConfig } from "@metaobjectsdev/cli"; +import { entityFile } from "@metaobjectsdev/codegen-ts/generators"; + +export default defineConfig({ + outDir: "src/generated", + dialect: "postgres", + targets: { + contract: { outDir: "src/generated", runtime: false }, // Zod schemas and types, no ORM + }, + generators: [entityFile({ target: "contract" })], +}); +``` + +```yaml +metadata: + package: api + children: + - object.value: + name: CreateCustomer + children: + - field.string: { name: email, required: true, maxLength: 254 } + - field.int: { name: seats } + - field.string[]: { name: tags, required: true } +``` + +Run `npx meta gen` to generate. The schema for `CreateCustomer` is `CreateCustomerInsertSchema`. The generated `CreateCustomer` is +the type. Use the same `proof/differs.ts` as above, and compare both `z.input` and `z.output`, +because a default or a transform changes one without the other: + +```ts +import type { z } from "zod"; +import type { CreateCustomer } from "../src/api/schemas.js"; +import type { CreateCustomerInsertSchema } from "../src/generated/CreateCustomer.js"; +import type { Differs, Same } from "./differs.js"; + +export type Proof = [ + Same, z.input>>, + Same, z.output>>, +]; +``` + +`tsc` cannot see `.min()`, `.max()` or `.email()`, because they do not change the type. Zod can +describe them, so add `proof/rules.ts` and run it with your test runner, or with node or bun: + +```ts +import assert from "node:assert/strict"; +import { z } from "zod"; +import { CreateCustomer } from "../src/api/schemas.js"; +import { CreateCustomerInsertSchema } from "../src/generated/CreateCustomer.js"; + +// `required` follows the order of the fields, which does not change what a schema accepts. +const rules = (schema: z.ZodType, io: "input" | "output") => + JSON.parse( + JSON.stringify(z.toJSONSchema(schema, { io, unrepresentable: "any" }), (key, value) => + key === "required" && Array.isArray(value) ? [...value].sort() : value, + ), + ); + +for (const io of ["input", "output"] as const) { + assert.deepEqual(rules(CreateCustomer, io), rules(CreateCustomerInsertSchema, io)); +} +``` + +`z.toJSONSchema` is Zod 4. On Zod 3, sort and compare `zodToJsonSchema(schema)` from the +`zod-to-json-schema` package instead. + +A required string generates `.min(1)`, so a hand-written `z.string()` with no minimum shows up here +as a `minLength` difference. If the existing schema really accepts an empty string, say so in the +model with `validator.length: { min: 0 }` as a child of the field. + +Move a call site by swapping the import and the schema name: + +```ts +import { type CreateCustomer, CreateCustomerInsertSchema } from "./generated/CreateCustomer.js"; + +export function parseSignup(body: unknown): CreateCustomer { + return CreateCustomerInsertSchema.parse(body); +} +``` + +### Pydantic + +Install with `pip install metaobjects "pydantic>=2.8"`. + +The model's field name is the Pydantic attribute, so keep `snake_case` names in the model. A shared +`str` Enum is an abstract `field.enum` that each field `extends`. An enum declared inline on the +field generates a `Literal` instead. + +```yaml +metadata: + package: shop + children: + - field.enum: + name: OrderStatus + abstract: true + values: [pending, paid, shipped] + - object.entity: + name: Order + children: + - source.rdb: { table: orders } + - field.long: { name: id, required: true } + - field.enum: { name: status, extends: OrderStatus, required: true, default: pending } + - field.string[]: { name: tags, required: true, default: "{}" } + - field.date: { name: placed_on, required: true } + - identity.primary: { fields: id, generation: increment } +``` + +Generate beside the existing models: + +```yaml +# metaobjects.config.yaml +metadata: metaobjects +targets: + app: + outDir: generated # beside the existing code; nothing imports it yet + generators: [entity] +``` + +Run `metaobjects gen` from the project root to generate. + +mypy cannot compare two Pydantic models, because a class is its own type, whatever its fields are. +Pydantic can describe a model as JSON Schema, so compare those. Pydantic validates input and writes +output differently, so check both modes. Copy this into `proof/test_equivalence.py` and name each +pair: + +```python +from typing import Any + +import pytest +from pydantic import BaseModel + +import app.models as existing +from generated import Customer, Order, ShippingAddress + + +def shape(model: type[BaseModel], mode: str) -> dict[str, Any]: + """The schema with what does not change behaviour removed: titles and descriptions, the + order of `required`, and `$ref`, which is inlined so a `str` Enum matches a Literal.""" + schema = model.model_json_schema(mode=mode) + defs = schema.pop("$defs", {}) + + def walk(node: Any, seen: tuple[str, ...] = ()) -> Any: + if isinstance(node, list): + return [walk(item, seen) for item in node] + if not isinstance(node, dict): + return node + if "$ref" in node: + name = node["$ref"].rsplit("/", 1)[-1] + if name in seen: # a model that contains itself + return {"$recursive": name} + rest = {key: value for key, value in node.items() if key != "$ref"} + return walk({**defs[name], **rest}, (*seen, name)) + out: dict[str, Any] = {} + for key, value in node.items(): + if key in ("title", "description"): + continue + if key == "properties": + out[key] = {field: walk(prop, seen) for field, prop in value.items()} + elif key == "required": + out[key] = sorted(value) + elif key == "default": + out[key] = value + else: + out[key] = walk(value, seen) + return out + + return walk(schema) + + +@pytest.mark.parametrize("mode", ["validation", "serialization"]) +@pytest.mark.parametrize( + ("hand_written", "generated"), + [ + (existing.Customer, Customer.Customer), + (existing.CustomerCreate, Customer.CustomerCreate), + (existing.ShippingAddress, ShippingAddress.ShippingAddress), + (existing.Order, Order.Order), + ], +) +def test_same_shape(hand_written: type[BaseModel], generated: type[BaseModel], mode: str) -> None: + assert shape(hand_written, mode) == shape(generated, mode) +``` + +Run `python -m pytest proof/ -vv` from the project root. `-vv` prints the whole difference, with +the field name in it. Schemas carry field constraints such as `max_length`, so this also catches a +rule the Python type does not show. + +Move a call site by changing its import: + +```python +from generated.Customer import Customer, CustomerCreate +``` + +Move models that nest together. Pydantic refuses an instance of a different class even when the +shape matches, so a generated `ShippingAddress` passed to the hand-written `Order` raises a +`ValidationError`. If they cannot move together, hand over `address.model_dump()` at the seam. + +The check against a database is `meta verify --db`, which needs the Node CLI, because schema +tooling is TypeScript only. `metaobjects verify` compares the generated code with the model. + +## Fixing the differences + +Fix each one in the model or the config, and leave the existing code alone until it has moved. + +| What you see | Why | Fix | +|---|---|---| +| A timestamp is `Date` on one side and `string` on the other | Generated timestamps read as strings by default | `timestampMode: "date"` in the config (Postgres) | +| A field the database fills is required on insert | The model has no default | Add `default:` to the field | +| A database-set timestamp is accepted on create | The field is writable | `default: now(), mutability: readOnly` | +| An array column's default is reported by `verify --db` | An array `default` must be a string | `default: "{}"`, not `default: []` | +| `verify --db` reports a different foreign key name | The generated name differs | `constraintName` on `identity.reference` | +| `verify --db` reports a different unique index name | The name comes from the identity | Name `identity.secondary` after the existing index | +| A `tsc` failure names a field that is `T \| null` on one side | The existing type allows both null and absent | Model what the code relies on. If the difference is deliberate, keep it visible: `Same, "notes">>` with a comment | + +## What the proof does not cover + +- **Custom logic.** A `.refine()`, a `.transform()` or a Pydantic validator is not in the type or + in the JSON Schema, so neither side of the proof sees it, and the model does not carry it. Keep + it in your code: refine the generated schema, or subclass the generated model. +- **Behaviour.** The proof compares declared shapes. The project's own tests still have to pass + after each move. +- **Native Postgres enums.** A `pgEnum` column cannot be modelled: `field.enum` is text plus a + CHECK. `tsc` passes and `meta verify --db` reports the missing CHECK. Convert the column to text + with a CHECK first, or leave that table out for now. +- **Python enum member names.** JSON Schema sees the values, not the member names. A generated enum + has `UPPER_CASE` members, so `OrderStatus.pending` in a call site must become + `OrderStatus.PENDING`. mypy or pyright finds those call sites. +- **Extra optional fields on a nested TypeScript type.** `Differs` compares a nested field as a + whole, so an optional extra inside it still assigns both ways. Give each nested type its own line. +- **Pairs you did not name.** The proof compares only what it lists, so list every exported type. diff --git a/fixtures/agent-context-conformance/ts-requirements/expected/.claude/skills/metaobjects-authoring/SKILL.md b/fixtures/agent-context-conformance/ts-requirements/expected/.claude/skills/metaobjects-authoring/SKILL.md index 85ac7bc5c..a1e709017 100644 --- a/fixtures/agent-context-conformance/ts-requirements/expected/.claude/skills/metaobjects-authoring/SKILL.md +++ b/fixtures/agent-context-conformance/ts-requirements/expected/.claude/skills/metaobjects-authoring/SKILL.md @@ -25,6 +25,7 @@ This file covers what almost every model needs. The topics below live in | File | Open it when | |---|---| | `references/adopting-existing-code.md` | working code or a populated schema already exists for what you are modeling | +| `references/adopting-existing-models.md` | moving existing Zod, Drizzle or Pydantic models into the model: the step-by-step recipe, and how to prove the generated types match before the switch | | `references/choosing-a-shape.md` | no row of "Canonical form for common field needs" fits; you need the full UUID / timestamp / URI / `@autoSet` rule; or you are registering your own vocabulary | | `references/json-columns.md` | a JSON column beyond the ladder below: coverage per port, `field.map` at runtime, version caveats, bag → native array | | `references/identities-and-indexes.md` | declaring a primary, secondary or reference identity, an FK, or a non-unique `index.lookup` | diff --git a/fixtures/agent-context-conformance/ts-requirements/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-code.md b/fixtures/agent-context-conformance/ts-requirements/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-code.md index 5d265812a..b611b9d1d 100644 --- a/fixtures/agent-context-conformance/ts-requirements/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-code.md +++ b/fixtures/agent-context-conformance/ts-requirements/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-code.md @@ -13,6 +13,9 @@ redefining it. (The metadata is still the durable spine *going forward*; only th **The observable predicate:** does working code or a populated schema already exist for what you're modeling? If yes, you are in adoption mode and these rules apply. +For Zod schemas, Drizzle tables or Pydantic models, `adopting-existing-models.md` is the +step-by-step recipe: generate beside them, prove the types match, move one call site at a time. + **Author metadata to match what the code ALREADY IS — not what you'd design fresh.** Read the existing code and schema *first*, then model to reproduce them: - The **native types the code uses** are the spec — model `field.uuid` when the code diff --git a/fixtures/agent-context-conformance/ts-requirements/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-models.md b/fixtures/agent-context-conformance/ts-requirements/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-models.md new file mode 100644 index 000000000..d35d70fe2 --- /dev/null +++ b/fixtures/agent-context-conformance/ts-requirements/expected/.claude/skills/metaobjects-authoring/references/adopting-existing-models.md @@ -0,0 +1,386 @@ +# Adopting existing models + +> Part of the `metaobjects-authoring` skill. Read it when a project already has Zod schemas, Drizzle tables or Pydantic models and you are moving them into the model. + +MetaObjects never reads the project's source. You read the existing types and write the model, and +the language's own tools compare the two: `tsc` for TypeScript, `model_json_schema()` for Python. + +## Contents +- The recipe +- Drizzle +- Zod +- Pydantic +- Fixing the differences +- What the proof does not cover + +## The recipe + +1. **Write the model from the existing types.** Read the types and model what they are, not what + you would design fresh. `references/adopting-existing-code.md` in the `metaobjects-authoring` + skill has the rules. +2. **Generate beside them.** Send the output to a new directory. Nothing imports it yet. +3. **Write the proof.** It is one test file that names each pair of types. +4. **See it fail once.** Break one field in the model, regenerate, and check that the proof names + that field. Then undo it. +5. **Fix the differences in the model,** not in the existing code, until the proof passes. +6. **Check the database,** if the project has one. +7. **Move one call site at a time.** Run the proof and the project's tests after each move. +8. **Retire the old code,** delete the proof, and keep `meta verify` in CI. + +Step 4 is not optional. A proof that cannot fail looks the same as one that passes. If the proof +stays green with a field broken, its file is outside `include` in `tsconfig.json`, or `strict` is +off, or pytest never collected it. + +### Drizzle + +Install the CLI with `npm i -D @metaobjectsdev/cli @metaobjectsdev/codegen-ts typescript`, then run +`npx meta init` once in the project root. + +Write the model. Field names are the TypeScript property names. `snake_case` is the default for +columns, so `displayName` already means `display_name`. Reuse the names the database has: the name +of a `.unique()` index goes on `identity.secondary`, and an existing foreign key name goes in +`constraintName`. + +```yaml +metadata: + package: shop + children: + - object.entity: + name: Customer + children: + - source.rdb: { table: customers } + - field.uuid: { name: id, required: true } + - field.string: { name: email, required: true, maxLength: 254 } + - field.string: { name: displayName, required: true, maxLength: 120 } + - field.int: { name: loyaltyPoints, required: true, default: 0 } + - field.string: { name: notes } + - field.timestamp: { name: createdAt, required: true, default: now(), mutability: readOnly } + - identity.primary: { fields: id, generation: uuid } + - identity.secondary: { name: customers_email_unique, fields: email } +``` + +Generate beside the existing schema: + +```ts +// metaobjects.config.ts +import { defineConfig } from "@metaobjectsdev/cli"; + +export default defineConfig({ + outDir: "src/generated", // beside the existing code; nothing imports it yet + dialect: "postgres", + timestampMode: "date", // when the existing timestamp columns read as Date + generators: ["entity"], +}); +``` + +Run `npx meta gen` to generate. + +Copy this helper into `proof/differs.ts`: + +```ts +// Names every field that differs between two object types, or `never` when they match. +// +// Mutual assignability alone is not enough. A field that is optional and present on only one +// side passes both ways, and `any` is assignable to and from everything, so this compares +// key by key and treats `any` on one side as a difference. Pick<> keeps each field's optional +// modifier. It looks one level down: give a nested type its own line in equivalence.ts. +type Mutual = [A] extends [B] ? ([B] extends [A] ? true : false) : false; +type IsAny = 0 extends 1 & T ? true : false; + +export type Differs = { + [K in keyof A | keyof B]: K extends keyof A & keyof B + ? [IsAny] extends [IsAny] + ? Mutual, Pick> extends true + ? never + : K + : K + : K; +}[keyof A | keyof B]; + +// tsc reports each differing field as: Type '"name"' does not satisfy the constraint 'never'. +export type Same<_NoDifferences extends never> = true; +``` + +Then name each pair in `proof/equivalence.ts`, inside `include` in `tsconfig.json`, with `strict` +on. Compare what a query reads and what an insert takes: + +```ts +import type { Customer, NewCustomer } from "../src/db/schema.js"; +import type { Customer as GeneratedCustomer, CustomerInsert } from "../src/generated/Customer.js"; +import type { Differs, Same } from "./differs.js"; + +export type Proof = [ + Same>, + Same>, +]; +``` + +`npx tsc --noEmit` is the check. + +Types do not include column names, so `tsc` passes even with a wrong column. `meta verify --db` is +the check for those (it needs `pg` installed): + +```sh +npx meta verify --codegen --db "$DATABASE_URL" +``` + +`--db` runs only the database check unless you name the others, so keep `--codegen` beside it. + +Two Drizzle objects for one table can exist side by side, since each only describes SQL. Move a +call site by changing its import, with an alias where the generated name differs: + +```ts +import { type Customer, customers, type CustomerInsert as NewCustomer } from "./generated/Customer.js"; +``` + +When the last one has moved, replace the old table definitions with re-exports and delete +`proof/`: + +```ts +export { customers } from "../generated/Customer.js"; +export { orders } from "../generated/Order.js"; +``` + +Keep one migration tool per database. To stay on drizzle-kit, point it at this file. To move to +`meta migrate`, adopt the live schema with `meta migrate baseline --from-db`. + +### Zod + +Install the CLI with `npm i -D @metaobjectsdev/cli @metaobjectsdev/codegen-ts typescript`, then run +`npx meta init` once in the project root. + +A Zod project often has no ORM, so generate a contract-only target. An object with no table is an +`object.value`: + +```ts +// metaobjects.config.ts +import { defineConfig } from "@metaobjectsdev/cli"; +import { entityFile } from "@metaobjectsdev/codegen-ts/generators"; + +export default defineConfig({ + outDir: "src/generated", + dialect: "postgres", + targets: { + contract: { outDir: "src/generated", runtime: false }, // Zod schemas and types, no ORM + }, + generators: [entityFile({ target: "contract" })], +}); +``` + +```yaml +metadata: + package: api + children: + - object.value: + name: CreateCustomer + children: + - field.string: { name: email, required: true, maxLength: 254 } + - field.int: { name: seats } + - field.string[]: { name: tags, required: true } +``` + +Run `npx meta gen` to generate. The schema for `CreateCustomer` is `CreateCustomerInsertSchema`. The generated `CreateCustomer` is +the type. Use the same `proof/differs.ts` as above, and compare both `z.input` and `z.output`, +because a default or a transform changes one without the other: + +```ts +import type { z } from "zod"; +import type { CreateCustomer } from "../src/api/schemas.js"; +import type { CreateCustomerInsertSchema } from "../src/generated/CreateCustomer.js"; +import type { Differs, Same } from "./differs.js"; + +export type Proof = [ + Same, z.input>>, + Same, z.output>>, +]; +``` + +`tsc` cannot see `.min()`, `.max()` or `.email()`, because they do not change the type. Zod can +describe them, so add `proof/rules.ts` and run it with your test runner, or with node or bun: + +```ts +import assert from "node:assert/strict"; +import { z } from "zod"; +import { CreateCustomer } from "../src/api/schemas.js"; +import { CreateCustomerInsertSchema } from "../src/generated/CreateCustomer.js"; + +// `required` follows the order of the fields, which does not change what a schema accepts. +const rules = (schema: z.ZodType, io: "input" | "output") => + JSON.parse( + JSON.stringify(z.toJSONSchema(schema, { io, unrepresentable: "any" }), (key, value) => + key === "required" && Array.isArray(value) ? [...value].sort() : value, + ), + ); + +for (const io of ["input", "output"] as const) { + assert.deepEqual(rules(CreateCustomer, io), rules(CreateCustomerInsertSchema, io)); +} +``` + +`z.toJSONSchema` is Zod 4. On Zod 3, sort and compare `zodToJsonSchema(schema)` from the +`zod-to-json-schema` package instead. + +A required string generates `.min(1)`, so a hand-written `z.string()` with no minimum shows up here +as a `minLength` difference. If the existing schema really accepts an empty string, say so in the +model with `validator.length: { min: 0 }` as a child of the field. + +Move a call site by swapping the import and the schema name: + +```ts +import { type CreateCustomer, CreateCustomerInsertSchema } from "./generated/CreateCustomer.js"; + +export function parseSignup(body: unknown): CreateCustomer { + return CreateCustomerInsertSchema.parse(body); +} +``` + +### Pydantic + +Install with `pip install metaobjects "pydantic>=2.8"`. + +The model's field name is the Pydantic attribute, so keep `snake_case` names in the model. A shared +`str` Enum is an abstract `field.enum` that each field `extends`. An enum declared inline on the +field generates a `Literal` instead. + +```yaml +metadata: + package: shop + children: + - field.enum: + name: OrderStatus + abstract: true + values: [pending, paid, shipped] + - object.entity: + name: Order + children: + - source.rdb: { table: orders } + - field.long: { name: id, required: true } + - field.enum: { name: status, extends: OrderStatus, required: true, default: pending } + - field.string[]: { name: tags, required: true, default: "{}" } + - field.date: { name: placed_on, required: true } + - identity.primary: { fields: id, generation: increment } +``` + +Generate beside the existing models: + +```yaml +# metaobjects.config.yaml +metadata: metaobjects +targets: + app: + outDir: generated # beside the existing code; nothing imports it yet + generators: [entity] +``` + +Run `metaobjects gen` from the project root to generate. + +mypy cannot compare two Pydantic models, because a class is its own type, whatever its fields are. +Pydantic can describe a model as JSON Schema, so compare those. Pydantic validates input and writes +output differently, so check both modes. Copy this into `proof/test_equivalence.py` and name each +pair: + +```python +from typing import Any + +import pytest +from pydantic import BaseModel + +import app.models as existing +from generated import Customer, Order, ShippingAddress + + +def shape(model: type[BaseModel], mode: str) -> dict[str, Any]: + """The schema with what does not change behaviour removed: titles and descriptions, the + order of `required`, and `$ref`, which is inlined so a `str` Enum matches a Literal.""" + schema = model.model_json_schema(mode=mode) + defs = schema.pop("$defs", {}) + + def walk(node: Any, seen: tuple[str, ...] = ()) -> Any: + if isinstance(node, list): + return [walk(item, seen) for item in node] + if not isinstance(node, dict): + return node + if "$ref" in node: + name = node["$ref"].rsplit("/", 1)[-1] + if name in seen: # a model that contains itself + return {"$recursive": name} + rest = {key: value for key, value in node.items() if key != "$ref"} + return walk({**defs[name], **rest}, (*seen, name)) + out: dict[str, Any] = {} + for key, value in node.items(): + if key in ("title", "description"): + continue + if key == "properties": + out[key] = {field: walk(prop, seen) for field, prop in value.items()} + elif key == "required": + out[key] = sorted(value) + elif key == "default": + out[key] = value + else: + out[key] = walk(value, seen) + return out + + return walk(schema) + + +@pytest.mark.parametrize("mode", ["validation", "serialization"]) +@pytest.mark.parametrize( + ("hand_written", "generated"), + [ + (existing.Customer, Customer.Customer), + (existing.CustomerCreate, Customer.CustomerCreate), + (existing.ShippingAddress, ShippingAddress.ShippingAddress), + (existing.Order, Order.Order), + ], +) +def test_same_shape(hand_written: type[BaseModel], generated: type[BaseModel], mode: str) -> None: + assert shape(hand_written, mode) == shape(generated, mode) +``` + +Run `python -m pytest proof/ -vv` from the project root. `-vv` prints the whole difference, with +the field name in it. Schemas carry field constraints such as `max_length`, so this also catches a +rule the Python type does not show. + +Move a call site by changing its import: + +```python +from generated.Customer import Customer, CustomerCreate +``` + +Move models that nest together. Pydantic refuses an instance of a different class even when the +shape matches, so a generated `ShippingAddress` passed to the hand-written `Order` raises a +`ValidationError`. If they cannot move together, hand over `address.model_dump()` at the seam. + +The check against a database is `meta verify --db`, which needs the Node CLI, because schema +tooling is TypeScript only. `metaobjects verify` compares the generated code with the model. + +## Fixing the differences + +Fix each one in the model or the config, and leave the existing code alone until it has moved. + +| What you see | Why | Fix | +|---|---|---| +| A timestamp is `Date` on one side and `string` on the other | Generated timestamps read as strings by default | `timestampMode: "date"` in the config (Postgres) | +| A field the database fills is required on insert | The model has no default | Add `default:` to the field | +| A database-set timestamp is accepted on create | The field is writable | `default: now(), mutability: readOnly` | +| An array column's default is reported by `verify --db` | An array `default` must be a string | `default: "{}"`, not `default: []` | +| `verify --db` reports a different foreign key name | The generated name differs | `constraintName` on `identity.reference` | +| `verify --db` reports a different unique index name | The name comes from the identity | Name `identity.secondary` after the existing index | +| A `tsc` failure names a field that is `T \| null` on one side | The existing type allows both null and absent | Model what the code relies on. If the difference is deliberate, keep it visible: `Same, "notes">>` with a comment | + +## What the proof does not cover + +- **Custom logic.** A `.refine()`, a `.transform()` or a Pydantic validator is not in the type or + in the JSON Schema, so neither side of the proof sees it, and the model does not carry it. Keep + it in your code: refine the generated schema, or subclass the generated model. +- **Behaviour.** The proof compares declared shapes. The project's own tests still have to pass + after each move. +- **Native Postgres enums.** A `pgEnum` column cannot be modelled: `field.enum` is text plus a + CHECK. `tsc` passes and `meta verify --db` reports the missing CHECK. Convert the column to text + with a CHECK first, or leave that table out for now. +- **Python enum member names.** JSON Schema sees the values, not the member names. A generated enum + has `UPPER_CASE` members, so `OrderStatus.pending` in a call site must become + `OrderStatus.PENDING`. mypy or pyright finds those call sites. +- **Extra optional fields on a nested TypeScript type.** `Differs` compares a nested field as a + whole, so an optional extra inside it still assigns both ways. Give each nested type its own line. +- **Pairs you did not name.** The proof compares only what it lists, so list every exported type. diff --git a/scripts/ci-local.sh b/scripts/ci-local.sh index 11b50b23e..6c04a566a 100755 --- a/scripts/ci-local.sh +++ b/scripts/ci-local.sh @@ -600,6 +600,14 @@ gate_prisma_pg() { bun_install && ( cd server/typescript/packages/cli && bun test test/integration/prisma-schema.test.ts ) } +# The brownfield recipe's database check (docs/features/adopting-existing-models.md): the model +# matches a Postgres built from examples/brownfield/drizzle/db/schema.sql, and a wrong column that +# tsc cannot see fails `meta verify --db`. The rest of that file needs no database and runs with +# the cli suite, so this runs only the Postgres half, on the same sidecar as gate_prisma_pg above. +gate_brownfield_pg() { + bun_install && ( cd server/typescript/packages/cli && bun test test/integration/brownfield-recipe.test.ts -t "meta verify --db" ) +} + # ── conformance.yml — per-port conformance corpora (exact CI commands) ──────── gate_conf_ts() { bun_install || return 1 @@ -608,7 +616,7 @@ gate_conf_ts() { ( cd server/typescript/packages/render && bun test test/render-conformance.test.ts test/verify-conformance.test.ts test/extract/extract-conformance.test.ts test/output-prompt-conformance.test.ts ) || return 1 ( cd server/typescript/packages/integration-tests && bun test test/validation-conformance.test.ts test/validation-conformance-runtime.test.ts ) || return 1 ( cd server/typescript/packages/migrate-ts && bun test ) || return 1 - ( cd server/typescript/packages/codegen-ts && bun test ) || return 1 + ( cd server/typescript/packages/codegen-ts && bun test --timeout 30000 ) || return 1 # --timeout 30000: the cli suite invokes full `meta`/run() dispatch, which lazily # imports each command's (sometimes heavy: codegen, migrate-ts) module on first # use. On a cold/contended self-hosted runner that first cold-start import can @@ -1098,6 +1106,7 @@ lane_ts_integ() { step_if bun "migrate-ts real-PG suite" gate_migrate_ts_pg step_if bun "runtime-ts real-PG dialect matrix" gate_runtime_ts_pg step_if bun "prisma real-PG round trip" gate_prisma_pg + step_if bun "brownfield recipe real-PG check" gate_brownfield_pg step "integration-tests (ts)" gate_integration_port ts } lane_cube() { [ "$NO_INTEG" -eq 0 ] && run_cube_lane; return 0; } @@ -1247,6 +1256,7 @@ if [ "$PARALLEL" -eq 0 ]; then in_lane_any ts ts-slow && step_if bun "migrate-ts real-PG suite" gate_migrate_ts_pg in_lane_any ts ts-slow && step_if bun "runtime-ts real-PG dialect matrix" gate_runtime_ts_pg in_lane_any ts ts-slow && step_if bun "prisma real-PG round trip" gate_prisma_pg + in_lane_any ts ts-slow && step_if bun "brownfield recipe real-PG check" gate_brownfield_pg in_lane_any ts ts-slow && run_integration_for ts ts in_lane_any java java-slow && run_integration_for java java kotlin in_lane python && run_integration_for python python diff --git a/server/python/src/metaobjects/codegen/generators/entity_model.py b/server/python/src/metaobjects/codegen/generators/entity_model.py index 885c1ae8b..e695ff2e2 100644 --- a/server/python/src/metaobjects/codegen/generators/entity_model.py +++ b/server/python/src/metaobjects/codegen/generators/entity_model.py @@ -53,7 +53,7 @@ from ..runtime import fr019_shared_enum as fr019 from ..runtime.m2m_codegen import build_object_index, resolve_m2m_descriptors -from ..runtime.shared_predicates import has_literal_default, is_sql_expr_default +from ..runtime.shared_predicates import array_default, has_literal_default def _is_int(value: object) -> bool: @@ -348,11 +348,21 @@ def _with_type_checking(imports: list[str]) -> list[str]: return sorted([*imports, "from typing import TYPE_CHECKING"]) +def _literal_default_expr(field: MetaField, enum_type_name: str | None) -> str: + """The Python source of a literal @default: the list an array default spells, an enum + member (Type.MEMBER, UPPER_CASE), or the value's own literal.""" + default_raw = field.attrs().get(fc.FIELD_ATTR_DEFAULT) + if field_is_array(field): + return repr(array_default(field)) + if enum_type_name is not None: + return f"{enum_type_name}.{str(default_raw).upper()}" + return repr(default_raw) + + def _field_line(field: MetaField, imports: set[str], config: GenConfig) -> tuple[str, bool]: """Return (source line, uses_field). Collects required imports into *imports*.""" type_expr, enum_type_name = _type_expr_for_field(field, imports, config, wire=False) required = field.attrs().get(fc.FIELD_ATTR_REQUIRED) is True - default_raw = field.attrs().get(fc.FIELD_ATTR_DEFAULT) # A server-side expression default (now(), gen_random_uuid(), CURRENT_TIMESTAMP) # is filled by the DB — the Python model carries no default for it (the field keeps # its required/optional shape). Only literal defaults become Pydantic defaults. @@ -361,14 +371,7 @@ def _field_line(field: MetaField, imports: set[str], config: GenConfig) -> tuple parts = _constraint_parts(_validator_constraints(field, wire=False)) uses_field = bool(parts) - # @default — render an enum member (Type.MEMBER, UPPER_CASE) or a Python literal. - if has_default: - if enum_type_name is not None and not field_is_array(field): - default_expr = f"{enum_type_name}.{str(default_raw).upper()}" - else: - default_expr = repr(default_raw) - else: - default_expr = None + default_expr = _literal_default_expr(field, enum_type_name) if has_default else None # #195 — a field derived by an origin.aggregate any|all|collect always COALESCEs # to a value (any→false, all→true, collect→[]), so its read type is mandatory @@ -472,7 +475,7 @@ def _create_field_line( type_expr, enum_type_name = _type_expr_for_field(field, imports, config, wire=True) required = field.attrs().get(fc.FIELD_ATTR_REQUIRED) is True default_raw = field.attrs().get(fc.FIELD_ATTR_DEFAULT) - literal_default = default_raw is not None and not is_sql_expr_default(default_raw) + literal_default = has_literal_default(field) auto_set = field.attrs().get(fc.FIELD_ATTR_AUTO_SET) is_auto_set = auto_set in (fc.AUTO_SET_ON_CREATE, fc.AUTO_SET_ON_UPDATE) # A field is required in the CREATE shape only when @required AND not server-filled: @@ -489,12 +492,9 @@ def _create_field_line( py_name = field.name # FR-036 #3 — validation models key by the WIRE name (field.name) if literal_default: - # Preserve the literal @default (enum member or Python literal) exactly as the - # read model does — the caller may omit the field and the model fills the default. - if enum_type_name is not None and not field_is_array(field): - default_expr = f"{enum_type_name}.{str(default_raw).upper()}" - else: - default_expr = repr(default_raw) + # Preserve the literal @default exactly as the read model does — the caller may + # omit the field and the model fills the default. + default_expr = _literal_default_expr(field, enum_type_name) annotation = type_expr assignment = ( f" = Field(default={default_expr}, {', '.join(parts)})" diff --git a/server/python/src/metaobjects/codegen/runtime/api_docs_builder.py b/server/python/src/metaobjects/codegen/runtime/api_docs_builder.py index e5e2476c2..16e2a438b 100644 --- a/server/python/src/metaobjects/codegen/runtime/api_docs_builder.py +++ b/server/python/src/metaobjects/codegen/runtime/api_docs_builder.py @@ -45,7 +45,7 @@ resolve_m2m_descriptors, ) from metaobjects.codegen.generators.find_inbound import is_xml, response_shape -from metaobjects.codegen.value_objects import bound_payload_vo, is_field_required, pkg_of +from metaobjects.codegen.value_objects import bound_payload_vo, is_field_required from metaobjects.codegen.instance_artifacts import ( has_item_route, is_abstract, diff --git a/server/python/src/metaobjects/codegen/runtime/shared_predicates.py b/server/python/src/metaobjects/codegen/runtime/shared_predicates.py index 0a0cdded2..9318226f4 100644 --- a/server/python/src/metaobjects/codegen/runtime/shared_predicates.py +++ b/server/python/src/metaobjects/codegen/runtime/shared_predicates.py @@ -18,10 +18,14 @@ """ from __future__ import annotations +import json +import math import re +from collections.abc import Callable from metaobjects.codegen.generators.tph_plan import is_tph_subtype from metaobjects.codegen.instance_artifacts import emits_instance_artifacts +from metaobjects.codegen.type_map import field_is_array from metaobjects.meta.core.field import field_constants as fc from metaobjects.meta.core.field.meta_field import MetaField from metaobjects.meta.core.object.meta_object import MetaObject @@ -54,13 +58,95 @@ def is_sql_expr_default(value: object) -> bool: return isinstance(value, str) and any(p.search(value) for p in SQL_EXPR_DEFAULT_PATTERNS) +def _parse_bool_element(raw: object) -> bool: + text = str(raw).lower() + if text not in ("true", "false"): + raise ValueError(raw) + return text == "true" + + +def _parse_text_element(raw: object) -> str: + """A string, or a JSON number or boolean read as its text. null and a nested value are + refused.""" + if isinstance(raw, str): + return raw + if isinstance(raw, bool | int | float): + return json.dumps(raw) + raise ValueError(raw) + + +def _parse_int_element(raw: object) -> int: + return int(str(raw)) + + +def _parse_float_element(raw: object) -> float: + """float() also reads ``nan`` and ``inf``, whose repr is not Python source.""" + value = float(str(raw)) + if not math.isfinite(value): + raise ValueError(raw) + return value + + +# How an array @default's non-empty elements read in Python, per element subtype. A subtype +# with no entry has no Python spelling here, so its array default stays with the database. +_ARRAY_ELEMENT_PARSERS: dict[str, Callable[[object], object]] = { + fc.FIELD_SUBTYPE_STRING: _parse_text_element, + fc.FIELD_SUBTYPE_INT: _parse_int_element, + fc.FIELD_SUBTYPE_LONG: _parse_int_element, + fc.FIELD_SUBTYPE_DOUBLE: _parse_float_element, + fc.FIELD_SUBTYPE_FLOAT: _parse_float_element, + fc.FIELD_SUBTYPE_BOOLEAN: _parse_bool_element, +} + + +def array_default(field: MetaField) -> list[object] | None: + """The list an array field's ``@default`` spells, or None when it is not one this can + read. The metadata value is a string — ``{}``, ``[]``, ``{a,b}`` or ``["a","b"]`` — or a + native list (``default: [a, b]`` in YAML, which the loader keeps as written). The four + string shapes the TS column-mapper's parseArrayDefault accepts are read here, with the + same refusals: a quote or nested brace in the Postgres form is an escaping shape neither + port models. A native list goes through the same element parsers.""" + raw = field.attrs().get(fc.FIELD_ATTR_DEFAULT) + elements: list[object] + if isinstance(raw, list): + elements = raw + elif not isinstance(raw, str): + return None + elif (text := raw.strip()).startswith("[") and text.endswith("]"): + try: + elements = json.loads(text) + except ValueError: + return None + if not isinstance(elements, list): + return None + elif text.startswith("{") and text.endswith("}"): + inner = text[1:-1].strip() + if re.search(r'["{}]', inner): + return None + elements = [p.strip() for p in inner.split(",")] if inner else [] + else: + return None + if not elements: + return [] + parse = _ARRAY_ELEMENT_PARSERS.get(field.sub_type) + if parse is None: + return None + try: + return [parse(e) for e in elements] + except ValueError: + return None + + def has_literal_default(field: MetaField) -> bool: """True iff the read model gives *field* a Python default: a ``@default`` that is not a - server-side SQL expression. Such a field is annotated ``T`` (not ``T | None``) with that - default, so code that builds the model without a value for it must OMIT the keyword - rather than pass ``None`` — the extractor's mapper relies on exactly this.""" + server-side SQL expression (nor an array default Python cannot spell). Such a field is + annotated ``T`` (not ``T | None``) with that default, so code that builds the model + without a value for it must OMIT the keyword rather than pass ``None`` — the extractor's + mapper relies on exactly this.""" default_raw = field.attrs().get(fc.FIELD_ATTR_DEFAULT) - return default_raw is not None and not is_sql_expr_default(default_raw) + if default_raw is None or is_sql_expr_default(default_raw): + return False + return not field_is_array(field) or array_default(field) is not None # F22 — the read-only source kinds that still get a REST surface. A view (and a diff --git a/server/python/tests/codegen/test_brownfield_recipe.py b/server/python/tests/codegen/test_brownfield_recipe.py new file mode 100644 index 000000000..bc9859fb8 --- /dev/null +++ b/server/python/tests/codegen/test_brownfield_recipe.py @@ -0,0 +1,128 @@ +"""docs/features/adopting-existing-models.md, end to end, on examples/brownfield/pydantic. + +Each test does what the recipe tells an agent to do: generate beside the existing models, prove +the two match with `model_json_schema()`, see the proof fail on a deliberate mismatch, move one +call site, then retire the old models. No MetaObjects command reads the project's source. +""" +from __future__ import annotations + +import shutil +import subprocess +import sys +from pathlib import Path + +import pytest + +EXAMPLE = Path(__file__).resolve().parents[4] / "examples" / "brownfield" / "pydantic" + + +def run(root: Path, *args: str) -> subprocess.CompletedProcess[str]: + return subprocess.run( + [sys.executable, *args], + cwd=root, + capture_output=True, + text=True, + check=False, + ) + + +def metaobjects(root: Path, *args: str) -> subprocess.CompletedProcess[str]: + return run(root, "-m", "metaobjects.cli", *args) + + +def proof(root: Path) -> subprocess.CompletedProcess[str]: + return run(root, "-m", "pytest", "proof", "-vv", "-p", "no:cacheprovider") + + +def edit(root: Path, name: str, old: str, new: str) -> None: + path = root / name + text = path.read_text() + assert old in text + path.write_text(text.replace(old, new)) + + +@pytest.fixture +def project(tmp_path: Path) -> Path: + root = tmp_path / "project" + shutil.copytree(EXAMPLE, root, ignore=shutil.ignore_patterns("__pycache__", "generated", ".metaobjects")) + generated = metaobjects(root, "gen") + assert generated.returncode == 0, generated.stderr + return root + + +def test_generated_models_match_and_the_recipe_runs_to_retiring_the_old_ones(project: Path) -> None: + proved = proof(project) + assert proved.returncode == 0, proved.stdout + + # One call site moves. The other still imports the hand-written models, and both load. + edit( + project, + "app/signup.py", + "from app.models import Customer, CustomerCreate", + "from generated.Customer import Customer, CustomerCreate", + ) + assert run(project, "-c", "import app.signup, app.shipping").returncode == 0 + + # A generated model's instance is not the hand-written class, so nested models move together. + refused = run( + project, + "-c", + "import app.models, generated.ShippingAddress as g, uuid, datetime;" + "app.models.Order(id=1, customer_id=uuid.uuid4(), total=1, placed_on=datetime.date.today()," + " shipping_address=g.ShippingAddress(line1='a', city='b'))", + ) + assert refused.returncode != 0 + assert "valid dictionary or instance of ShippingAddress" in refused.stderr + + edit( + project, + "app/shipping.py", + "from app.models import Order, OrderStatus, ShippingAddress", + "from generated.enums import OrderStatus\n" + "from generated.Order import Order\n" + "from generated.ShippingAddress import ShippingAddress", + ) + (project / "app" / "models.py").unlink() + shutil.rmtree(project / "proof") + assert run(project, "-c", "import app.signup, app.shipping").returncode == 0 + verified = metaobjects(project, "verify") + assert verified.returncode == 0, verified.stderr + + +def test_proof_fails_naming_the_field_when_the_existing_model_differs(project: Path) -> None: + edit(project, "app/models.py", " loyalty_points: int = 0\n marketing_opt_in: bool = False\n notes: Optional[str] = None\n created_at", " loyalty_points: str = '0'\n marketing_opt_in: bool = False\n notes: Optional[str] = None\n created_at") + failed = proof(project) + assert failed.returncode == 1 + assert "loyalty_points" in failed.stdout + assert "test_same_shape" in failed.stdout + + +def test_proof_catches_a_rule_the_python_type_does_not_carry(project: Path) -> None: + edit(project, "app/models.py", "email: str = Field(max_length=254)\n display_name: str = Field(max_length=120)\n loyalty", "email: str = Field(max_length=100)\n display_name: str = Field(max_length=120)\n loyalty") + failed = proof(project) + assert failed.returncode == 1 + assert "maxLength" in failed.stdout + + +def test_proof_catches_a_model_that_lost_a_field_on_a_nested_type(project: Path) -> None: + edit(project, "metaobjects/meta.shop.yaml", " - field.string: { name: postcode }\n", "") + regenerated = metaobjects(project, "gen") + assert regenerated.returncode == 0, regenerated.stderr + failed = proof(project) + assert failed.returncode == 1 + assert "postcode" in failed.stdout + + +def test_a_str_enum_and_a_literal_compare_equal(project: Path) -> None: + """The model may declare the enum inline, which generates a Literal. The proof must not + report that as a difference: both accept and write the same strings.""" + edit( + project, + "metaobjects/meta.shop.yaml", + "- field.enum: { name: status, extends: OrderStatus, required: true, default: pending }", + "- field.enum: { name: status, values: [pending, paid, shipped], required: true, default: pending }", + ) + regenerated = metaobjects(project, "gen") + assert regenerated.returncode == 0, regenerated.stderr + proved = proof(project) + assert proved.returncode == 0, proved.stdout diff --git a/server/python/tests/codegen/test_entity_model.py b/server/python/tests/codegen/test_entity_model.py index 02dc80f42..2e2fbd750 100644 --- a/server/python/tests/codegen/test_entity_model.py +++ b/server/python/tests/codegen/test_entity_model.py @@ -247,6 +247,69 @@ def test_field_default_is_emitted_as_literal() -> None: assert "s: str = 'approx'" in out +@pytest.mark.parametrize( + ("subtype", "raw", "expected"), + [ + (fc.FIELD_SUBTYPE_STRING, "{}", "[]"), + (fc.FIELD_SUBTYPE_STRING, "[]", "[]"), + (fc.FIELD_SUBTYPE_STRING, "{a,b}", "['a', 'b']"), + (fc.FIELD_SUBTYPE_STRING, '["a","b"]', "['a', 'b']"), + (fc.FIELD_SUBTYPE_LONG, "{1,2}", "[1, 2]"), + (fc.FIELD_SUBTYPE_DOUBLE, "[1.5]", "[1.5]"), + (fc.FIELD_SUBTYPE_BOOLEAN, "{true,false}", "[True, False]"), + # JSON numbers and booleans in a text array read as their text, as TypeScript reads them. + (fc.FIELD_SUBTYPE_STRING, "[true, 1]", "['true', '1']"), + # An empty array needs no element parsing, so every element subtype has it. + (fc.FIELD_SUBTYPE_UUID, "{}", "[]"), + (fc.FIELD_SUBTYPE_DECIMAL, "[]", "[]"), + # YAML `default: [a, b]` reaches the generator as a native list, not a string. + (fc.FIELD_SUBTYPE_STRING, ["a", "b"], "['a', 'b']"), + (fc.FIELD_SUBTYPE_STRING, [], "[]"), + (fc.FIELD_SUBTYPE_LONG, [1, 2], "[1, 2]"), + (fc.FIELD_SUBTYPE_BOOLEAN, [True, False], "[True, False]"), + (fc.FIELD_SUBTYPE_UUID, [], "[]"), + ], +) +def test_array_default_is_a_python_list_not_the_metadata_string( + subtype: str, raw: object, expected: str +) -> None: + """An array @default is authored as a string or a native list; both models must get the + list it spells, because Pydantic does not validate a default and `tags: list[str] = '{}'` + hands every instance a str.""" + tags = _f("tags", subtype, is_array=True) + tags.set_attr(fc.FIELD_ATTR_DEFAULT, raw) + out = render_entity_model(_entity("M", [tags])) + assert f"] = {expected}\n" in out.split("class MCreate")[0] + assert f"] = {expected}\n" in out.split("class MCreate")[1] + + +@pytest.mark.parametrize( + ("subtype", "raw"), + [ + (fc.FIELD_SUBTYPE_STRING, '{"a,b",c}'), + (fc.FIELD_SUBTYPE_INT, "{1,x}"), + (fc.FIELD_SUBTYPE_UUID, "{a}"), + # float() reads these, but `[nan]` is not Python source and the module would not import. + (fc.FIELD_SUBTYPE_DOUBLE, "{nan}"), + (fc.FIELD_SUBTYPE_DOUBLE, "{inf}"), + (fc.FIELD_SUBTYPE_STRING, "[null]"), + (fc.FIELD_SUBTYPE_STRING, '[["a"]]'), + (fc.FIELD_SUBTYPE_INT, ["x"]), + (fc.FIELD_SUBTYPE_UUID, ["a"]), + ], +) +def test_array_default_python_cannot_spell_is_left_to_the_database( + subtype: str, raw: str | list[str] +) -> None: + """A shape this parser does not model carries no Python default, as a SQL expression + does: the column has the default, and the model must not invent one.""" + tags = _f("tags", subtype, is_array=True) + tags.set_attr(fc.FIELD_ATTR_DEFAULT, raw) + out = render_entity_model(_entity("M", [tags])) + assert str(raw) not in out + assert "] | None = None" in out.split("class MCreate")[0] + + def test_dbcolumntype_jsonb_field_is_any_with_import() -> None: """Issue #98 — a field.string @dbColumnType:jsonb stores genuinely-open JSON, which pg8000 decodes to a native Python object at read time. The generated diff --git a/server/typescript/packages/cli/test/integration/brownfield-recipe.test.ts b/server/typescript/packages/cli/test/integration/brownfield-recipe.test.ts new file mode 100644 index 000000000..191859aeb --- /dev/null +++ b/server/typescript/packages/cli/test/integration/brownfield-recipe.test.ts @@ -0,0 +1,276 @@ +/** + * docs/features/adopting-existing-models.md, end to end, on the two TypeScript projects under + * examples/brownfield/. Each test does what the recipe tells an agent to do: generate beside the + * existing code, prove the two match, see the proof fail on a deliberate mismatch, move one call + * site, then retire the old code. + * + * The proof is `tsc` over a test file the recipe writes. No MetaObjects command reads the + * project's source. That is the point of the recipe, so the mismatches below are caught by + * `tsc`, by Zod and by `meta verify --db`, each on the thing it can see. + * + * The drizzle `verify --db` half needs a server: MIGRATE_TS_PG_URL (the ts-slow lane's sidecar). + * Each run creates a database of its own and drops it. + */ +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { cpSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { join, resolve } from "node:path"; +import pg from "pg"; +import { run } from "../../src/index.js"; + +const CLI_ROOT = resolve(import.meta.dirname, "../.."); +const TSC_BIN = join(CLI_ROOT, "node_modules", ".bin", "tsc"); +const EXAMPLES = resolve(CLI_ROOT, "../../../../examples/brownfield"); +// Under the cli package so the example resolves zod, drizzle-orm and @types/node through the +// workspace (gitignored). The examples carry no package.json of their own. +const WORKSPACE_TMP = resolve(import.meta.dirname, "../fixtures/__tmp__"); +const PG_URL = process.env.MIGRATE_TS_PG_URL; + +function project(name: "drizzle" | "zod"): string { + mkdirSync(WORKSPACE_TMP, { recursive: true }); + const root = mkdtempSync(join(WORKSPACE_TMP, `brownfield-${name}-`)); + cpSync(join(EXAMPLES, name), root, { recursive: true }); + return root; +} + +function spawn(cmd: string, args: string[], root: string): { code: number; output: string } { + const p = Bun.spawnSync([cmd, ...args], { cwd: root }); + return { code: p.exitCode, output: `${p.stdout.toString()}${p.stderr.toString()}` }; +} + +function tsc(root: string): { code: number; output: string } { + return spawn(TSC_BIN, ["-p", join(root, "tsconfig.json")], root); +} + +function bun(root: string, script: string): { code: number; output: string } { + return spawn(process.execPath, [script], root); +} + +function edit(root: string, file: string, from: string, to: string): void { + const path = join(root, file); + const text = readFileSync(path, "utf8"); + expect(text).toContain(from); + writeFileSync(path, text.replace(from, to)); +} + +let roots: string[] = []; +let quiet: { log: typeof console.log; error: typeof console.error }; +beforeEach(() => { + roots = []; + quiet = { log: console.log, error: console.error }; + console.log = () => {}; + console.error = () => {}; +}); +afterEach(() => { + console.log = quiet.log; + console.error = quiet.error; + for (const root of roots) rmSync(root, { recursive: true, force: true }); +}); +function fresh(name: "drizzle" | "zod"): string { + const root = project(name); + roots.push(root); + return root; +} + +describe("brownfield recipe — Drizzle", () => { + test("the generated tables match the hand-written ones, and the recipe runs to retiring them", async () => { + const root = fresh("drizzle"); + expect(await run(["gen", "--cwd", root])).toBe(0); + expect(tsc(root)).toEqual({ code: 0, output: "" }); + + // One call site moves. The other still reads the old schema, and both compile. + edit( + root, + "src/customers.ts", + 'import { type Customer, customers, type NewCustomer } from "./db/schema.js";', + 'import { type Customer, customers, type CustomerInsert as NewCustomer } from "./generated/Customer.js";', + ); + expect(tsc(root)).toEqual({ code: 0, output: "" }); + + // The rest moves, the old schema becomes re-exports, the proof goes, and verify still passes. + edit( + root, + "src/orders.ts", + 'import { type NewOrder, type Order, orders } from "./db/schema.js";', + 'import { type OrderInsert as NewOrder, type Order, orders } from "./generated/Order.js";', + ); + writeFileSync( + join(root, "src/db/schema.ts"), + 'export { customers } from "../generated/Customer.js";\nexport { orders } from "../generated/Order.js";\n', + ); + rmSync(join(root, "proof"), { recursive: true }); + expect(tsc(root)).toEqual({ code: 0, output: "" }); + expect(await run(["verify", "--cwd", root])).toBe(0); + }, 120_000); + + test("tsc names the field when the model stops matching the existing types", async () => { + const root = fresh("drizzle"); + edit(root, "metaobjects/meta.shop.yaml", '{ name: tags, required: true, default: "{}" }', "{ name: tags, required: true }"); + edit(root, "metaobjects/meta.shop.yaml", "{ name: notes }", "{ name: notes, required: true }"); + expect(await run(["gen", "--cwd", root])).toBe(0); + const { code, output } = tsc(root); + expect(code).not.toBe(0); + expect(output).toContain(`Type '"tags"' does not satisfy the constraint 'never'`); + expect(output).toContain(`Type '"notes"' does not satisfy the constraint 'never'`); + }, 120_000); + + test("without timestampMode the proof catches a Date column generated as a string", async () => { + const root = fresh("drizzle"); + edit(root, "metaobjects.config.ts", ' timestampMode: "date", // the existing timestamp columns read as Date\n', ""); + expect(await run(["gen", "--cwd", root])).toBe(0); + const { code, output } = tsc(root); + expect(code).not.toBe(0); + expect(output).toContain(`Type '"createdAt"' does not satisfy the constraint 'never'`); + }, 120_000); + + describe("meta verify --db (postgres, env-gated)", () => { + if (PG_URL === undefined || PG_URL.length === 0) { + // The ts-slow lane arms this: there the URL is the sidecar's, and a missing one is a broken + // lane, not a skip. + if (process.env.MIGRATE_TS_PG_EXPECT === "1") { + test("MIGRATE_TS_PG_URL is set where MIGRATE_TS_PG_EXPECT=1 says a Postgres is", () => { + expect(PG_URL).toBeDefined(); + }); + } else { + test.skip("skipped — MIGRATE_TS_PG_URL not set", () => {}); + } + return; + } + + test("the model matches the database the app runs; a wrong column tsc cannot see fails", async () => { + const dbName = `mo_brownfield_${process.pid}_${Date.now()}`; + const admin = new pg.Client({ connectionString: PG_URL }); + await admin.connect(); + await admin.query(`CREATE DATABASE ${dbName}`); + const url = new URL(PG_URL); + url.pathname = `/${dbName}`; + const root = fresh("drizzle"); + try { + const db = new pg.Client({ connectionString: url.toString() }); + await db.connect(); + await db.query(readFileSync(join(root, "db/schema.sql"), "utf8")); + await db.end(); + + expect(await run(["gen", "--cwd", root])).toBe(0); + expect(await run(["verify", "--cwd", root, "--db", url.toString()])).toBe(0); + + // The physical column is not in the type, so the proof passes and only the database + // comparison can fail. + edit(root, "metaobjects/meta.shop.yaml", "{ name: displayName, required: true,", "{ name: displayName, column: display_nam, required: true,"); + expect(await run(["gen", "--cwd", root])).toBe(0); + expect(tsc(root)).toEqual({ code: 0, output: "" }); + expect(await run(["verify", "--cwd", root, "--db", url.toString()])).toBe(1); + } finally { + await admin.query(`DROP DATABASE IF EXISTS ${dbName} WITH (FORCE)`); + await admin.end(); + } + }, 120_000); + }); +}); + +describe("brownfield recipe — Zod", () => { + test("the generated schemas match the hand-written ones, and the recipe runs to retiring them", async () => { + const root = fresh("zod"); + expect(await run(["gen", "--cwd", root])).toBe(0); + expect(tsc(root)).toEqual({ code: 0, output: "" }); + expect(bun(root, "proof/rules.ts").code).toBe(0); + + // One call site moves; the other still parses with the old schema. + edit( + root, + "src/shipping.ts", + 'import { Address } from "./api/schemas.js";', + 'import { type Address, AddressInsertSchema } from "./generated/Address.js";', + ); + edit(root, "src/shipping.ts", "return Address.parse(body);", "return AddressInsertSchema.parse(body);"); + expect(tsc(root)).toEqual({ code: 0, output: "" }); + + edit( + root, + "src/signup.ts", + 'import { CreateCustomer } from "./api/schemas.js";', + 'import { type CreateCustomer, CreateCustomerInsertSchema } from "./generated/CreateCustomer.js";', + ); + edit(root, "src/signup.ts", "return CreateCustomer.parse(body);", "return CreateCustomerInsertSchema.parse(body);"); + rmSync(join(root, "src/api"), { recursive: true }); + rmSync(join(root, "proof"), { recursive: true }); + expect(tsc(root)).toEqual({ code: 0, output: "" }); + expect(await run(["verify", "--cwd", root])).toBe(0); + }, 120_000); + + test("tsc names a type mismatch, on input and on output", async () => { + const root = fresh("zod"); + expect(await run(["gen", "--cwd", root])).toBe(0); + edit(root, "src/api/schemas.ts", "seats: z.number().int().optional()", "seats: z.string().optional()"); + const { code, output } = tsc(root); + expect(code).not.toBe(0); + expect(output.match(/Type '"seats"' does not satisfy the constraint 'never'/g)?.length).toBe(2); + }, 120_000); + + test("tsc names an optional field present on one side, which mutual assignability would pass", async () => { + const root = fresh("zod"); + expect(await run(["gen", "--cwd", root])).toBe(0); + edit(root, "src/api/schemas.ts", "tags: z.array(z.string()),", "tags: z.array(z.string()),\n referrer: z.string().optional(),"); + const { code, output } = tsc(root); + expect(code).not.toBe(0); + expect(output).toContain(`Type '"referrer"' does not satisfy the constraint 'never'`); + + // The naive check this replaces: both directions assign, so it would have called these equal. + writeFileSync( + join(root, "proof/naive.ts"), + `import type { z } from "zod"; +import type { CreateCustomer } from "../src/api/schemas.js"; +import type { CreateCustomerInsertSchema } from "../src/generated/CreateCustomer.js"; +type A = z.input; +type B = z.input; +export const forward: B = {} as A; +export const backward: A = {} as B; +`, + ); + expect(tsc(root).output).not.toContain("proof/naive.ts"); + }, 120_000); + + test("an existing string that accepts empty differs by rule, and the model can say so", async () => { + const root = fresh("zod"); + edit(root, "src/api/schemas.ts", "displayName: z.string().min(1).max(120)", "displayName: z.string().max(120)"); + expect(await run(["gen", "--cwd", root])).toBe(0); + const rules = bun(root, "proof/rules.ts"); + expect(rules.code).not.toBe(0); + expect(rules.output).toContain("minLength"); + + edit( + root, + "metaobjects/meta.api.yaml", + "- field.string: { name: displayName, required: true, maxLength: 120 }", + "- field.string:\n name: displayName\n required: true\n maxLength: 120\n children:\n - validator.length: { min: 0 }", + ); + expect(await run(["gen", "--cwd", root])).toBe(0); + expect(bun(root, "proof/rules.ts").code).toBe(0); + }, 120_000); + + test("fields in a different order are not a difference", async () => { + const root = fresh("zod"); + expect(await run(["gen", "--cwd", root])).toBe(0); + edit(root, "src/api/schemas.ts", " email: z.string().min(1).max(254),\n displayName: z.string().min(1).max(120),\n", " displayName: z.string().min(1).max(120),\n email: z.string().min(1).max(254),\n"); + expect(tsc(root)).toEqual({ code: 0, output: "" }); + expect(bun(root, "proof/rules.ts").code).toBe(0); + }, 120_000); + + test("tsc names a field that became any, which assigns both ways", async () => { + const root = fresh("zod"); + expect(await run(["gen", "--cwd", root])).toBe(0); + edit(root, "src/api/schemas.ts", "tags: z.array(z.string()),", "tags: z.any() as z.ZodType,"); + const { code, output } = tsc(root); + expect(code).not.toBe(0); + expect(output).toContain(`Type '"tags"' does not satisfy the constraint 'never'`); + }, 120_000); + + test("a rule the type cannot see passes tsc and fails the JSON Schema comparison", async () => { + const root = fresh("zod"); + expect(await run(["gen", "--cwd", root])).toBe(0); + edit(root, "src/api/schemas.ts", "email: z.string().min(1).max(254)", "email: z.string().min(1).max(100)"); + expect(tsc(root)).toEqual({ code: 0, output: "" }); + const rules = bun(root, "proof/rules.ts"); + expect(rules.code).not.toBe(0); + expect(rules.output).toContain("maxLength"); + }, 120_000); +}); diff --git a/server/typescript/packages/sdk/test/agent-context/recipe-mirrors.test.ts b/server/typescript/packages/sdk/test/agent-context/recipe-mirrors.test.ts index 7c2cd9d1f..1071a222b 100644 --- a/server/typescript/packages/sdk/test/agent-context/recipe-mirrors.test.ts +++ b/server/typescript/packages/sdk/test/agent-context/recipe-mirrors.test.ts @@ -11,6 +11,8 @@ import { join } from "node:path"; const REPO = join(import.meta.dir, "../../../../../.."); const REFS = join(REPO, "agent-context/skills/metaobjects-codegen/references"); +const AUTHORING_REFS = join(REPO, "agent-context/skills/metaobjects-authoring/references"); +const BROWNFIELD = join(REPO, "examples/brownfield"); test("the shipped MongoDB repository generator is the tested recipe, byte for byte", () => { const shipped = readFileSync(join(REFS, "typescript-document-store.md"), "utf8"); @@ -35,3 +37,23 @@ test("the shipped Prisma guide carries the recipe's setup, workflow, mapping and expect(shipped.includes(MARK)).toBe(true); expect(shipped.slice(shipped.indexOf(MARK))).toBe(original.slice(original.indexOf(MARK))); }); + +test("the shipped adoption guide carries the recipe unchanged", () => { + const MARK = "## The recipe"; + const shipped = readFileSync(join(AUTHORING_REFS, "adopting-existing-models.md"), "utf8"); + const original = readFileSync(join(REPO, "docs/features/adopting-existing-models.md"), "utf8"); + expect(original.includes(MARK)).toBe(true); + expect(shipped.slice(shipped.indexOf(`\n${MARK}\n`) + 1)).toBe(original.slice(original.indexOf(`\n${MARK}\n`) + 1)); +}); + +// The recipe prints these helpers for an agent to copy. brownfield-recipe.test.ts and +// test_brownfield_recipe.py run the example copies, so the printed text must be those. +test("the helpers the adoption recipe prints are the tested ones, byte for byte", () => { + const recipe = readFileSync(join(REPO, "docs/features/adopting-existing-models.md"), "utf8"); + const differs = readFileSync(join(BROWNFIELD, "drizzle/proof/differs.ts"), "utf8"); + expect(readFileSync(join(BROWNFIELD, "zod/proof/differs.ts"), "utf8")).toBe(differs); + expect(recipe).toContain(`\`\`\`ts\n${differs}\`\`\``); + const python = readFileSync(join(BROWNFIELD, "pydantic/proof/test_equivalence.py"), "utf8"); + const body = python.slice(python.indexOf("from typing import Any")); + expect(recipe).toContain(`\`\`\`python\n${body}\`\`\``); +});