Skip to content

feat(init): scaffold from a five-template catalog, with templates as the single source of truth - #3

Merged
coccor merged 13 commits into
mainfrom
worktree-init-templates
Aug 20, 2026
Merged

feat(init): scaffold from a five-template catalog, with templates as the single source of truth#3
coccor merged 13 commits into
mainfrom
worktree-init-templates

Conversation

@coccor

@coccor coccor commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

What

pz init now scaffolds from a named catalog of five built-in templates instead of a hardcoded two-way switch, and the templates themselves move to a top-level templates/ directory that is simultaneously the browsable examples, the trees the test suite compiles in place, and pz init's only source.

pz init <name> [-t|--template <id>] [--list-templates]
id shape runnability
minimal (default) project.yml + connections.yml, commented, ready to author against nothing to run
sample four-pipeline demo over local CSVs offline
incremental watermark-bounded reads; run twice, the second run lands nothing offline
http GitHub REST API to a parquet delta log needs internet, no credentials
sqlserver incremental merge, all six check kinds, optional remote state needs a live database

Why

src/Pz.Cli/Templates/init/ was a hand-maintained copy of samples/hello-pz, and the two had already drifted — every file differed except two CSVs (connections.yml by 24 lines, README.md by 69). The csproj comment documented the decision to embed rather than generate precisely so they could not drift silently; they drifted anyway, by hand, because nothing compared them.

One directory now serves all three purposes, so drift is impossible rather than merely discouraged.

Notable changes

  • {{PROJECT_NAME}}pz_new_project. The moustache was invalid YAML ({{ opens a flow mapping), which is exactly what stopped a template directory from being a loadable project. A YAML-valid sentinel means every template loads, compiles, and (where its dependencies allow) runs from its own directory — so the in-place compile guards verify the real scaffold source rather than a copy of it.
  • --sample is removed, not deprecated. v0.2.0 predates it, so it has never appeared in a release.
  • New error codes: PZ0131 (unknown template id) and PZ0132 (invalid invocation — no name, or a name alongside --list-templates).
  • samples/hello-pz and samples/http-api are deleted, promoted into templates/. samples/ keeps sqlserver-tour, mysql-native, and mssql-mart — references rather than starting points.
  • Three set-equality tests bind TemplateCatalog.All ↔ on-disk templates/ ↔ embedded resources in both directions, so a template added wrongly fails the build.
  • MCP: pz_init_project takes template (default "minimal") and names all five ids with their runnability, so an agent picks in one shot. Unknown ids return PZ0131 before touching the filesystem.

Verification

  • Build: 0 warnings (TreatWarningsAsErrors).
  • Full suite: 3,103 tests across 21 projects, 0 failures (~50 docker-gated skips), run unpiped.
  • scripts/verify-tool-install.sh: PASS end to end — pack → local-feed-only tool install → offline pz init → offline --template sample--list-templatespz run --all writing non-empty outputs. Now also asserts the scaffolded README.md and .gitignore landed, the one place a dotfile lost in NuGet packaging would surface.

Follow-up outside this repo

pipelinez.dev (the pz-site repo) documents pz init. Its CLI reference and quickstart need --template/--list-templates, the five ids and what each needs to run, the removal of --sample, and PZ0131/PZ0132.

coccor added 13 commits August 20, 2026 19:17
…ption, PZ0131 validation

- scripts/verify-tool-install.sh: --sample -> --template sample (matches the deleted flag)
- PzMcpServer's pz_init_project description: name the real 'template' parameter and its
  two current ids instead of the removed 'minimal' boolean
- McpCommand.InitProject: validate templateId against TemplateCatalog before the
  directory-empty check, returning PZ0131 instead of misreporting PZ0603
samples/hello-pz and templates/sample had drifted into two hand-maintained
copies of the same project; templates/sample already held the reconciled
result of every differing hunk, so promoting it means deleting the sample,
not merging content. Verified hunk-by-hunk (old sample -> kept template):

- connections.yml: connection renamed crm -> raw (it now holds customers,
  orders, AND products; crm mischaracterized a product catalog as CRM data)
- connections.yml: customers moved from a call-site source() (no columns:)
  to a YAML entities: block with a full columns: contract -- matches the
  README's own claim that customers/orders are the YAML-declared pair
- connections.yml: added a top-of-file comment on the connection/entity/
  direction vocabulary, for a reader who just ran `pz init`
- project.yml: kept the pz_new_project sentinel name (mandatory -- every
  template's project.yml must keep it for scaffold-time string replacement)
  and the retention: keep_last: 10 block
- README.md: kept templates/sample's version wholesale, written for someone
  who just scaffolded the project, not someone browsing the repo
- pipelines/*.sql: source('crm', ...) -> source('raw', ...) throughout;
  orders_enriched.sql's customers read dropped its now-redundant path/
  format kwargs (redeclaring them would be PZ0341, declared both places);
  sink() kwarg order (strategy before format) unified across all three
  pipelines
- pipelines/configs/orders_enriched.yml: dropped the stale `crm` tag
- kept templates/sample's third flow (product_catalog / data/products.csv)
  and its .gitignore, per the stated one-of-each-surface pedagogy

Promoting the sample to two independent flows and giving every entity a
contract had real test consequences, not just a path rename:

- every bare `pz run --project <copy>` against the promoted sample now
  trips PZ0215 (two independent flows) and needs --all
- two ValidateCommandTests relied on customers being contract-less
  (PZ0330 tier-5 gap) and call-site-declared (PZ0331 drift); both now
  mutate a copy of the shipped tree to reconstruct those cases --
  products (call-site-declared already) replaces the customers/
  orders_enriched.sql hand-edit for the drift test
- SqlDryCompilerTests loaded samples/hello-pz straight off disk outside
  the .csproj content-link, so deleting the directory broke it; retargeted
  to a mutated copy of templates/sample the same way
- src_crm__* staging node names -> src_raw__*
- RunCommandTests.Retention_off_sweeps_nothing_and_prints_nothing used to
  append a second `retention:` key to project.yml, now duplicate since the
  template ships its own; changed to replace the existing block

Also fixes a doc comment in EntityPipelineAuthoringTests.cs whose "delete
six files first" count predated this branch, reworded to avoid a count.
…e doc cites

tests/Pz.Cli.Tests/Pz.Cli.Tests.csproj's content link for templates/sample
was missing the .pz/out Exclude that src/Pz.Cli/Pz.Cli.csproj's
EmbeddedResource over the same tree already carries: the sample is
runnable in place, so running it locally leaves .pz/ and out/ behind,
gitignored and invisible to git status. Without the exclude, MSBuild
would copy those leftovers into the test output's TemplatesSample
directory, and CopyTree would seed every temp project from them --
corrupting run-directory counts in retention tests on that machine only,
never in CI. Verified with a throwaway .pz/runs/probe and out/ dir: after
the fix, neither shows up under the rebuilt test output.

docs/reference/authoring-for-agents.md is embedded into the Pz.Mcp binary
and written to a stranger's disk with no source tree by `pz mcp init`, so
its three `(from samples/hello-pz/...)` provenance citations would 404
for every reader once that directory is gone. Removed the citations
rather than repointing them at templates/sample: the snippets' content no
longer matches that tree verbatim after the reconciliation, so repointing
would just be a more specific stale claim -- the syntax they teach is
correct either way. Also updated the minimal project.yml example's
`name: hello_pz` to `name: pz_new_project` to match the sentinel every
template now ships.
… install gate

Names every built-in template id and what it needs to run in the pz_init_project
tool description, adds packaging and --list-templates smoke checks to
verify-tool-install.sh, and updates README/CLAUDE.md/CONTRIBUTING.md references
that still pointed at the retired --sample flag and src/Pz.Cli/Templates/ path.
- templates/incremental: stop claiming extraction is bounded when only
  the pipeline is; point at the sqlserver template for real pushdown
- templates/sqlserver: add a freshness check so "all six check kinds"
  is true, matching templates/sample
- templates/http: drop dead samples/mssql-mart and
  extract-from-http-api.md citations a scaffolded project can't reach
- templates/minimal, InitCommandTests, verify-tool-install.sh: stop
  counting the minimal template's files so the claim can't go stale
- templates/sample: a URL cannot live in a repository
- InitCommandTests, Pz.Mcp.csproj: fix stale Templates/init/** and
  {{PROJECT_NAME}} citations from the pre-catalog scaffold
- TemplateCatalog: escape <name> in an XML doc comment; reorder All to
  minimal, sample, incremental, http, sqlserver
- InitCommand: fail with PZ0131 instead of reporting success when a
  catalog template's resource prefix matches zero embedded resources
- TemplateCatalogTests: note the tree-vs-embed check compares on-disk
  files, not git-tracked ones
@coccor
coccor marked this pull request as ready for review August 20, 2026 19:17
@coccor
coccor merged commit f8dad53 into main Aug 20, 2026
3 checks passed
@coccor
coccor deleted the worktree-init-templates branch August 20, 2026 19:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant