diff --git a/CHANGELOG.md b/CHANGELOG.md index f797d4563..69192d6a7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,118 @@ Unreleased entries live in [`changelog.d/`](changelog.d/), one file per issue — lanes do not edit this file. A cut assembles them (`scripts/assemble_changelog.py`). +## 0.114.3 (2026-08-13) — **`resume_from`, the fifth reserved model input** — the load half of resume, plus the mint supervision cut (th#1834 Phase 3) and the `.yaml` workflow rename that the PyPI publisher entry must follow + +> **CUT FROM A PINNED COMMIT: `d9174a18`.** 31 commits since `v0.114.2`, assembled from 7 +> fragments. Cut under an enqueue freeze (pgw#1226), with the queue verified EMPTY and all seven +> open PRs pinned and `--auto`-disarmed at freeze time. +> +> **pgw#1243's finalize-wedge fix does NOT ride this cut.** It is PR #791 and was still open — and +> `UNMERGEABLE` in the queue — when this was pinned. The wedge is the P0 that stands between the +> platform and its first published compiled graph, so read this wheel as *"everything except the +> wedge"*: it exists because a funded H100 booking needs `resume_from` published, not because the +> mint is fixed. **The wedge fix gets 0.114.4.** +> +> ⚠️ **This is the FIRST tag to run through the renamed `publish.yaml`** (th#1918, PR #784, merged +> 02:47Z). 0.114.2 published at 01:44Z on the old `publish.yml`. PyPI trusted publishing binds to +> the workflow filename, so if the human-side publisher edit has not landed, the upload 403s after +> CI, build and provenance pass. A 403 uploads nothing and invalidates nothing: the tag stands and +> publish re-runs on the same tag once the entry is fixed. +> +> The sweep against `origin/master` found **th#1918 itself owed a fragment and had none** — written +> here rather than shipped as a silent release note, which matters more than usual because it is the +> entry that explains the 403 a reader may be staring at. + +- **pgw#1206 C3: the residency teardown leaves `executor.py`.** `shutdown_instances` + and `_vacate_record` — the two mutators over the instance record book, parked in + the executor since C2 because both open with `await self.abandon_background_mint(...)` + — move to `gen_worker.models.records` as `shutdown_instances` / `vacate_record`, + against th#1834's ruled seam: `abandon_background_mint(rec, *, reason, + code="unspecified", free_targets=False)` stays with the mint supervisor and is + injected into the record book as a typed callable, so residency CALLS supervision + instead of owning it. A pure move — no behaviour change, no shim, every caller + (`lifecycle`'s drain, five executor sites, three test modules) repointed at the + defining module. `executor.py` 11,248 → 11,143. + +- **pgw#1215 step 2b (th#1834 Phase 3 keystone): the process that TRACES a graph class is the process that COMPILES and PACKS it, so the `ExportedProgram` never crosses a process boundary.** The mint parent used to `torch.export.save` each program and a per-class compile child used to `torch.export.load` it back, at a **36.04 s median** (pgw#1216, P0-E §5c) — ~22 min of a 36-class sdxl mint spent deserializing what another process in the same pod had just serialized. `aot_compile_child` now receives a SHARE (`rows[i::K]`) plus the four facts `boot_key.TraceJob` already carries (`function`/`modules`/`slots`/`cfg`), builds its own weight-free pipeline, runs `aot_mint.trace_for_key(compile_now=True)` over its rows, and packs each one with `aot_mint.pack_graph_classes` (lifted out in step 2a for exactly this). The save/load pair is DELETED, not relocated. +- **Three pieces of round-trip repair machinery go with it, because the round trip is gone.** `EntryJob.program`, and `EntryJob.symbol_values`/`symbol_labels` with the whole `aot_shape_hints` module (pgw#998 — `torch.export`'s save/load rebuilds the ShapeEnv keyed by size EXPRESSIONS, so a derived symbol left every non-key extent unlowerable); and `structure_only.to_meta_for_save`/`as_meta_for_save`/`has_meta_params`/`revirtualize_from_meta` (pgw#1111 — a FAKE tensor has no storage to serialize, so a weight-free program had to cross as META). Each existed only to repair the crossing. +- **The span partition follows the code in the same commit.** `child_program_load_s` leaves `aot_compile_spans.PARTITIONS`; `child_setup_s` (composing the weight-free pipeline), `child_trace_s` (`torch.export` itself) and `child_pack_s` (the packaging tail the child now owns) take its place, and the child seeds every declared member to zero so a share that refuses before it traces records a zero instead of leaving the residual to absorb it. Both halves of the closure are red-proven: dropping the term without editing `PARTITIONS` reports *"partition member(s) `['child_program_load_s']` were never recorded"*, and emitting it after the edit reports *"child_wall_s=200.000 but its members sum to 163.960"*. +- **`EntryCompilePool` dispatches shares instead of staging programs.** K children are spawned once from one recipe; there is no producer iterator, no staged-program slack and no multi-GB write, because the parent produces nothing the children consume. `EntryReport` hands back packed graph classes (key + artifact + envelope + that class's own `export_s`/`compile_s`), and a new `_assert_shares_whole` proves the shares reconstruct the WHOLE declared class set from the children's own declared counts — the parent holds no pipeline and therefore never enumerated it. +- **The child packs INSIDE its trace loop, one graph class at a time.** pgw#1183's ordering, held at the class: the artifact is on disk before the next class is traced, and each row is released straight after, so the child holds one compiled program rather than its whole share. A crash costs the ONE class in flight, and a refusal at class *k* REPORTS the *k−1* artifacts that already exist instead of discarding them (`EntryReport.classes` rides the refusal, `EntryCompilePool.refused_classes` banks it, and the failure sentence says how many survived). Batch-packing a share would have reinstated exactly the all-or-nothing loss th#1825 spent 1 h 37 m on. +- **`aot_mint._mint_cell` is the SERIAL path and nothing else**, and `aot_mint.mint_graph_classes` is the K-wide one — **wired as the production path**: `mint_child._mint_aot` now hands the children the recipe (`function`/`modules`/`slots`/`cfg`/`execution_lane`/`out_dir`, every one of them already on `MintRequest`) and enumerates the class count for the width, instead of exporting in-process. The old hybrid — export in the parent, compile in children — is what the keystone abolishes, so pgw#1052's overlapped producer, `_drive_pool`'s per-entry fold and `_compile_entries_parallel` are deleted with it. `release_residents` survives as `aot_mint.release_mint_residents`, called at the same moment on the new path: after the last read of the pipeline, before the first child spawns. +- **A compile child stamps NO coverage label.** `manifest_digest` is declaration-wide and a child holds one share, so `pack_graph_classes(manifest="")` records absence rather than a share-local digest that would read like a whole one; `mint_graph_classes` folds the real label across every share through the one shared `class_manifest`. The publish path already treats an empty label as honest (`fleet_cells._identity_axes`), and the label reaches no key. +- **A family with NO structure-only path still mints — the compile child PREFERS the weight-free composition and falls back to real weights, exactly as `mint_child._load` has since pgw#1123.** The child composes its own target now, so a composition that only ever asked for structure-only would have made every stranded family (a quantized artifact lane, a class the tree's `model_index.json` does not name) refuse `StructureOnlyUnsupported` before anything else ran — a family that minted last week minting nothing this week, reported as a perfectly typed terminal refusal. `StructureNotHonored` stays FATAL and is caught first, because it is a subclass and means the opposite: the target WAS built weight-free and the pipeline discarded it, so falling back there would export ~weight-scale REAL tensors while the child reports weightless (ie#638's silent 40 GiB OOM). The fallback keeps `place=False` — this process never runs a forward, so the serving placement ladder must not move real components onto the parent's card (pgw#1124). Red-proven: severing the fallback reds the stranded-family composition and the cardless-decline parity row; severing the fail-closed handler reds the discarded-target row. +- ⚠️ **The parent-side resume bank (pgw#848 item 5) loses its driver**: admission re-derived its graph hash from an ExportedProgram the parent no longer holds. `aot_resume` keeps the mint-scoped inductor cache dir and its ledger row — so a killed mint's next attempt still hits torch's own FX graph cache — and re-homing file-level resume at the packed graph-class artifact is owed by step 3/4. +- **One artifact per KEY, enforced where it is the only enforceable place.** A compile child holds one share and cannot see a peer's keys, so two classes carrying one key survive the pool's name-keyed collection intact and the collision is first discovered by the HUB — a duplicate-key 409 on the second publish, on the pod, after both compiles are paid for. `mint_graph_classes` now groups the returned rows by key, keeps one artifact per key, and records the absorbed names as that entry's `aliases` (pgw#917's shape, and not a `class_hash` fact, so an alias cannot re-key its survivor). The coverage label folds over the SURVIVORS, so it does not depend on how the declaration was sharded. Both halves red-proven: severing the grouping returns two artifacts under one key; folding the label over the absorbed row as well moves the label. +- ⚠️ **pgw#917's INGRESS merge still does not run on the sharded path, and the earlier note here had its mechanism wrong.** It claimed a mergeable pair "keys IDENTICALLY" because `class_hash` folds no name. Measured: `class_hash` folds `class_dims`, which is the one axis a mergeable pair differs on (`_class_identity` requires every other axis to match), so the pair keys **APART** — `6decad0789e30a3a` vs `a185615c3fd880e4` for the same block at two coordinates. The consequence is therefore not a 409 but the original pgw#917 defect returning: both rows compile, both publish, both arm, and `EntryDispatch.select` answers `entry_ambiguous` on every call they carry — a SILENT 100 %-eager degrade rather than a loud rejection. The REFUSAL half is lost too: a cluster that collides at ingress but is *not* one class used to be a terminal `MintRefused` naming the differing axis. Both need the whole declaration BEFORE sharding, so they are owed at step 4; a family whose declaration carries a mergeable pair (sdxl regional: 36 of 72 compiles) is affected until then. +- **pgw#1215 step 4 (th#1834 Phase 3): th#1299 becomes a fence, the ungated router is deleted, and the peak-RSS contract takes the compiled-graph name.** `scripts/lint_serving_process_compiles.py` is new and gating: `torch.export` and the AOTI compile entry points may only be called by a module that runs inside a compile/trace child (three, each declared with the child that reaches it), so the serving process cannot acquire an export the day the middle mint-child tier goes away. `torch.compile` is deliberately not banned — th#1834 keeps the dynamo JIT-intake arm — and its half of th#1299 is now structural instead: `hot_swap.Router` refuses an ungated router typed (`RouterNotGated`) at `enable()`, at a warm-job enqueue and at a pgw#680 guard-miss heal, and `route()`'s degrade-to-inline-compile fallback is gone with the "ungated legacy" mode it belonged to. That mode survived on kill-switch parity with `GEN_WORKER_BG_YIELD`, which pgw#995 deleted and which `test_no_env_restores_the_pre_pgw677_tree` asserts cannot exist. `Executor.declares_compile` — caller-less, pinned only by a `hasattr` — is deleted with its pin retargeted onto the deletion. `mint_workers.entry_peak_rss` / `record_entry_peak_rss` / the `entry_peak_rss_bytes` wire field are renamed `compiled_graph_peak_rss*` across all six participants in one change. +- **pgw#1215 step 4 (th#1834 Phase 3, the reroute's final act): the SERVING PARENT supervises its compile children directly, and the middle mint-child tier is deleted.** `mint_delegate.py` is gone. `Executor._background_mint_run` / `_delegated_mint_run` / `_advertise_minted_cells` are replaced by `_supervise_mint` + `_advertise_compiled_graphs` over one new module, `mint_supervisor`, which drives `aot_mint.mint_graph_classes` on a worker thread and adopts what comes back through the ordinary delivered-cell path. Three process tiers become two: the tier that died existed only to hold a weight-free pipeline the parent already holds a real, resident, SERVING copy of. th#1299's property is kept by the lint fence that landed ahead of it, not by the process. +- **The parent-side gates the middle tier used to run now run where the pipeline is.** The declared-blocker refusal is `mint_supervisor.DeclaredBlockerRefusal` — deliberately NOT `MintRefused`, which is live in `aot_contract` for *"every declared class refused"*; sharing the name would make the two indistinguishable at the terminus with nothing going red and tensorhub's blame ladder attributing them identically. The class ENUMERATION (`aot_mint.declared_class_rows`) runs on the live pipeline, because the adapter fork is a property of the composed pipeline and no child can be told how many classes exist — only which share of them is its own. +- **pgw#947 re-based: a serving parent STATES its lane instead of benchmarking one.** The mint child A/B'd the kernel lanes by loading one full pipeline per candidate onto an empty card. A serving parent has neither an empty card nor the right to take one, and it does not need the probe — the lane this pod serves on is a fact it holds first-hand and is the lane the artifact must be stamped with. Recorded as a typed `kernel_path.unmeasured` verdict with its reason, never as an absence. +- **A RETRY CONSUMES WHAT THE LAST ATTEMPT PACKED.** `mint_delegate.build_cell`'s retry loop gave every attempt a fresh `child-N` directory, so attempt 2 of a 36-class mint re-traced and re-compiled 35 finished classes to retry one, at a measured 156-509 s each. That loop is deleted, not ported: the supervisor accretes into an attempt-STABLE directory, reads the packed artifacts back off disk (each names its own `cg-key-v1` key and carries its envelope), and hands the children `EntryJob.have_classes` so a held row is dropped from the share BEFORE it is exported. The held classes JOIN the result and the manifest fold, so a retry's honest report is "36 of 36" rather than "1 of 36", and `EntryCompilePool._assert_shares_whole` counts coverage rather than this attempt's work. Red-proven at both ends and in the middle: severing the skip re-exports the held class; severing the hand-off re-compiles it; severing the whole-set accounting refuses every successful retry. +- **pgw#917's INGRESS MERGE AND ITS REFUSAL HALF ARE BACK on the sharded path — `aot_mint.canonicalize_packed_classes`.** Restated because the earlier note had the mechanism wrong twice: a mergeable pair keys APART (`class_hash` folds `class_dims`, the one axis such a pair differs on), so the by-key dedupe never sees it and the loss is SILENT — both rows compile, both publish, both arm, and `EntryDispatch.select` answers `entry_ambiguous` on every call they carry, which is the 4,200-refusal defect pgw#917 was filed to fix. The gate now runs at the only seam a supervisor can reach: the packed envelope, which carries EVERY axis `_class_identity` folds (`graph_witness` for the body, `range_digest` for the ingress, `graph.pytree`/`graph.literals`/`graph.specialization`/`graph.lifted_inputs`, plus `precision`/`lora_bucket`/`strict_export`/`source_digest` off the metadata). Asked through `aot_serve.contract_from_meta` and `aot_serve.assert_ingress` themselves, so the gate cannot drift from what dispatch will do. Both halves red-proven. +- ⚠️ **What it does NOT recover, stated rather than implied: the duplicate COMPILE.** Both members of a mergeable cluster are already built by the time an envelope exists, so pgw#847's "36 of sdxl regional's 72 compiles bought nothing" still costs what it costs on this path. Recovering that needs the decision BEFORE the trace, and the merge predicate is a property of the traced program — which the serving parent may not produce, by the fence it also owns. Correctness first: a merged entry serves those coordinates COMPILED today, where two published rows served them eager. +- **The fence learns the status change it was written for.** `lint_serving_process_compiles.py` gains a `SUPERVISED` list — the opposite of an exemption: naming a module there ASSERTS the default serving-process rule over it, and a stale row reds exactly as a stale `CHILD_ONLY` row does. `aot_compile_pool` and `aot_compile_child` were exempt until the keystone; the supervisor now calls the pool from the serving loop, so their status moved from "exempt" to "must be clean" and the fence says so. +- **`mint_process` / `mint_child` survive as the OPERATOR one-shot path** (`scripts/micro_mint_rig.py`, the pre-publish proof harness th#1834's census marks RE-POINT, never delete) and inherit `MintTask` / `build_request` / `cfg_spec` from the deleted module. They have no production caller any more, which is the point of the change and is recorded as a debt with an owner and an expiry in `scripts/unreached_surface_baseline.txt` rather than as a silent exemption. +- ⚠️ **`aot_resume.py` is NOT deleted here, and the plan's "its only importer" is wrong.** Measured: `aot_compile_pool:73/1147` and `mint_child:453/465` both import and call it, and the first is a module Phase 3 keeps. Only `aot_resume.discard` lost its caller — the supervisor keeps no resume bank at all, because coverage accretes in the artifact directory and a retry consumes the artifacts. Its deletion stays sequenced where th#1834 put it: the change that proves boot resolves already-compiled graphs locally. +- **Built extraction-aware.** The compiled-graph interior — the accretion loop, held-artifact consumption, key/identity reads, the packed-envelope merge — sits in `mint_supervisor` and `aot_mint`; residency (which pipe holds what, and what the wire is told) stays in `executor.py`. That seam is the `torch-compiled-graphs` migration surface: the interior lifts whole. + +- **pgw#1237: Cozy runtime inputs are one pinned contract.** The worker now + binds the five shared runtime environment names, path semantics and optional + secret classification to a public corpus that cozy-local fences byte-for-byte. +- **pgw#1237: worker values have a public carrier.** Activity decisions, + cell-resolve and trust refusals, compile-cache ranks, execution lanes, layout + identity, fabric admission, the pickle ban and all seven typed callout + refusals are pinned as exact values or explicitly bounded relations for + Tensorhub's peer fence. The carrier also binds the byte-compatible compile + cache flavor label, model-source and serving vocabularies, request fallback + and eager-posture reasons, hardware-unsuitable reason classes, the LoRA + weight bound, boot phases/outcomes/sources, and the intentionally + one-way `FnDegraded.ran` relation: the worker's current producer set is a + subset of the hub's accepted set because the hub retains the retired + `emergency_quant` value for compatibility. Unknown future boot phases remain + storable by Tensorhub; the fenced set is what this worker currently produces + and the hub currently interprets. + +### Fixed + +- Pin the Hub-to-worker launch ABI in a public, source-bound contract corpus so + Tensorhub can fail closed when launch environment names, endpoint-owned + secret names, forbidden C2PA key material, topology delivery, managed fill, + or build-input classification drift between repositories. + +- **pgw#1242 (te#185): `resume_from`, the fifth reserved model input — a training endpoint can finally be handed back a checkpoint it published.** `ctx.save_checkpoint` has always published to the job workspace, and there was no door in the other direction: a model artifact reaches tenant code only as a reserved payload field, and `_reserved_repo_info` hardcoded four names — `source`/`destination`/`text_encoder`/`candidate` — none of which means "the adapter to continue from", with `source` already spoken for by the base model. There is no runtime model-ref materialization either (`resolve_dataset` has no model equivalent, pgw#684), and `kind="training"` declares no slots. So the SAVE half of resume worked, the LOAD half did not exist, and a multi-hour training run restarted from zero on pod loss — `image_lora_finetuner`'s *"Resume v1 = clean restart"* is that gap rather than a preference. te#185's H3 trainer is 4xH100 at $13.16/hr for hours, where "restart from zero" is the whole booking. +- **Rides the seam pgw#594 generalized and pgw#684 last extended**: `_materialize_source` already takes a `field_name` plus a `set_path` callback, so this is the same ~15-line shape as `candidate` — read the struct, pass it to the producer ctx, materialize it after `text_encoder`/`candidate`, expose `ctx.resume_from` / `ctx.resume_from_path`. **pgw#684's ruling is followed deliberately, not merely echoed: "fourth hardcoded name NOW, declarative NEXT."** pgw#690 remains the declarative successor and is explicitly not this change — coupling a reserved-field-contract refactor to an unblock a consumer is waiting on is what that ruling declined, and the same reasoning applies here. +- **Absent field is a no-op, which is the whole safety argument for a hardcoded name.** Every existing payload struct lacks `resume_from`, so it stays `{}`, no `ensure_local` fires and `resume_from_path` stays `None`. Producer-only, like its four siblings: an inference endpoint that happens to embed a `SourceRepo` never triggers materialization. A blank ref is a typed `payload.resume_from.ref` validation error naming the field the tenant actually wrote, and a fetch failure classifies RETRYABLE exactly as `source` does — restarting a paid continuation from zero because a checkpoint 404'd is the failure this field exists to prevent, so it must never degrade to a silent fresh start. Red-verified: 7 of the 8 new tests fail against the unpatched tree, and the one that passes is the inference no-op, correctly. + +- **th#1887: two behaviour-switch environment variables are deleted.** Paul: + *"get rid of these envs please, unless you really need them. I hate envs like + this."* `GEN_WORKER_VIDEO_ENCODER` is gone — the NVENC probe was always the + real decision, since `x264` only SKIPPED the probe and `nvenc` already fell + back to x264 when it failed, so the switch's only power was to make a pod + encode on CPU while its NVENC ASIC sat idle and never report the gap. + `GEN_WORKER_SVDQ_ENGINE` is gone — a PROCESS-wide pin over a PER-ARTIFACT + decision, so one incident's pin outlived the incident and silently served + every later artifact on the wrong engine; the typed per-call `override=` + argument survives and cannot leak into the next load. Both deletions are + proved by tests that assert the env is INERT: exporting the value that used + to win now changes nothing, checked against a forced-positive NVENC probe and + against int4 (whose only candidate is nunchaku) so neither can pass for the + wrong reason. `GEN_WORKER_NATIVE_KERNELS` is deliberately NOT deleted here — + it gates an unshipped rollout rather than switching a finished feature, and + removing it forces an activate-or-delete decision on the native-kernel + subsystem itself; reported for an owner instead of resolved blind. + +- **th#1918: the workflows are `.yaml`, and the PyPI trusted publisher must be renamed to match.** + Every workflow file moves `.yml` → `.yaml` per Paul's directive. This is not cosmetic for + releases: **PyPI trusted publishing binds to the workflow FILENAME**, so the OIDC token now + presents `publish.yaml` and PyPI refuses a publisher entry that still names `publish.yml`. The + failure is late and clean — CI, build and provenance all pass, then the upload step 403s, and no + artifact is uploaded. 0.114.2 published on the old name roughly an hour before this landed, so + **0.114.3 is the first tag to run through `publish.yaml`.** The fix is a one-field human edit + (PyPI → gen-worker → Manage → Publishing → Workflow name); a 403 costs nothing but a re-run of + publish on the same tag. + ## 0.114.2 (2026-08-13) — **the compiled-graph format ships as v1 under a qualified name** — DESIGN-RULINGS §1.38b, the last identity contract to settle before the family train mints > **CUT FROM A PINNED COMMIT: `a98f232f`.** 28 commits since `v0.114.1`, assembled from 5 fragments. diff --git a/changelog.d/pgw1206.md b/changelog.d/pgw1206.md deleted file mode 100644 index 85c922fe2..000000000 --- a/changelog.d/pgw1206.md +++ /dev/null @@ -1,10 +0,0 @@ -- **pgw#1206 C3: the residency teardown leaves `executor.py`.** `shutdown_instances` - and `_vacate_record` — the two mutators over the instance record book, parked in - the executor since C2 because both open with `await self.abandon_background_mint(...)` - — move to `gen_worker.models.records` as `shutdown_instances` / `vacate_record`, - against th#1834's ruled seam: `abandon_background_mint(rec, *, reason, - code="unspecified", free_targets=False)` stays with the mint supervisor and is - injected into the record book as a typed callable, so residency CALLS supervision - instead of owning it. A pure move — no behaviour change, no shim, every caller - (`lifecycle`'s drain, five executor sites, three test modules) repointed at the - defining module. `executor.py` 11,248 → 11,143. diff --git a/changelog.d/pgw1215.md b/changelog.d/pgw1215.md deleted file mode 100644 index b6f2db4f2..000000000 --- a/changelog.d/pgw1215.md +++ /dev/null @@ -1,22 +0,0 @@ -- **pgw#1215 step 2b (th#1834 Phase 3 keystone): the process that TRACES a graph class is the process that COMPILES and PACKS it, so the `ExportedProgram` never crosses a process boundary.** The mint parent used to `torch.export.save` each program and a per-class compile child used to `torch.export.load` it back, at a **36.04 s median** (pgw#1216, P0-E §5c) — ~22 min of a 36-class sdxl mint spent deserializing what another process in the same pod had just serialized. `aot_compile_child` now receives a SHARE (`rows[i::K]`) plus the four facts `boot_key.TraceJob` already carries (`function`/`modules`/`slots`/`cfg`), builds its own weight-free pipeline, runs `aot_mint.trace_for_key(compile_now=True)` over its rows, and packs each one with `aot_mint.pack_graph_classes` (lifted out in step 2a for exactly this). The save/load pair is DELETED, not relocated. -- **Three pieces of round-trip repair machinery go with it, because the round trip is gone.** `EntryJob.program`, and `EntryJob.symbol_values`/`symbol_labels` with the whole `aot_shape_hints` module (pgw#998 — `torch.export`'s save/load rebuilds the ShapeEnv keyed by size EXPRESSIONS, so a derived symbol left every non-key extent unlowerable); and `structure_only.to_meta_for_save`/`as_meta_for_save`/`has_meta_params`/`revirtualize_from_meta` (pgw#1111 — a FAKE tensor has no storage to serialize, so a weight-free program had to cross as META). Each existed only to repair the crossing. -- **The span partition follows the code in the same commit.** `child_program_load_s` leaves `aot_compile_spans.PARTITIONS`; `child_setup_s` (composing the weight-free pipeline), `child_trace_s` (`torch.export` itself) and `child_pack_s` (the packaging tail the child now owns) take its place, and the child seeds every declared member to zero so a share that refuses before it traces records a zero instead of leaving the residual to absorb it. Both halves of the closure are red-proven: dropping the term without editing `PARTITIONS` reports *"partition member(s) `['child_program_load_s']` were never recorded"*, and emitting it after the edit reports *"child_wall_s=200.000 but its members sum to 163.960"*. -- **`EntryCompilePool` dispatches shares instead of staging programs.** K children are spawned once from one recipe; there is no producer iterator, no staged-program slack and no multi-GB write, because the parent produces nothing the children consume. `EntryReport` hands back packed graph classes (key + artifact + envelope + that class's own `export_s`/`compile_s`), and a new `_assert_shares_whole` proves the shares reconstruct the WHOLE declared class set from the children's own declared counts — the parent holds no pipeline and therefore never enumerated it. -- **The child packs INSIDE its trace loop, one graph class at a time.** pgw#1183's ordering, held at the class: the artifact is on disk before the next class is traced, and each row is released straight after, so the child holds one compiled program rather than its whole share. A crash costs the ONE class in flight, and a refusal at class *k* REPORTS the *k−1* artifacts that already exist instead of discarding them (`EntryReport.classes` rides the refusal, `EntryCompilePool.refused_classes` banks it, and the failure sentence says how many survived). Batch-packing a share would have reinstated exactly the all-or-nothing loss th#1825 spent 1 h 37 m on. -- **`aot_mint._mint_cell` is the SERIAL path and nothing else**, and `aot_mint.mint_graph_classes` is the K-wide one — **wired as the production path**: `mint_child._mint_aot` now hands the children the recipe (`function`/`modules`/`slots`/`cfg`/`execution_lane`/`out_dir`, every one of them already on `MintRequest`) and enumerates the class count for the width, instead of exporting in-process. The old hybrid — export in the parent, compile in children — is what the keystone abolishes, so pgw#1052's overlapped producer, `_drive_pool`'s per-entry fold and `_compile_entries_parallel` are deleted with it. `release_residents` survives as `aot_mint.release_mint_residents`, called at the same moment on the new path: after the last read of the pipeline, before the first child spawns. -- **A compile child stamps NO coverage label.** `manifest_digest` is declaration-wide and a child holds one share, so `pack_graph_classes(manifest="")` records absence rather than a share-local digest that would read like a whole one; `mint_graph_classes` folds the real label across every share through the one shared `class_manifest`. The publish path already treats an empty label as honest (`fleet_cells._identity_axes`), and the label reaches no key. -- **A family with NO structure-only path still mints — the compile child PREFERS the weight-free composition and falls back to real weights, exactly as `mint_child._load` has since pgw#1123.** The child composes its own target now, so a composition that only ever asked for structure-only would have made every stranded family (a quantized artifact lane, a class the tree's `model_index.json` does not name) refuse `StructureOnlyUnsupported` before anything else ran — a family that minted last week minting nothing this week, reported as a perfectly typed terminal refusal. `StructureNotHonored` stays FATAL and is caught first, because it is a subclass and means the opposite: the target WAS built weight-free and the pipeline discarded it, so falling back there would export ~weight-scale REAL tensors while the child reports weightless (ie#638's silent 40 GiB OOM). The fallback keeps `place=False` — this process never runs a forward, so the serving placement ladder must not move real components onto the parent's card (pgw#1124). Red-proven: severing the fallback reds the stranded-family composition and the cardless-decline parity row; severing the fail-closed handler reds the discarded-target row. -- ⚠️ **The parent-side resume bank (pgw#848 item 5) loses its driver**: admission re-derived its graph hash from an ExportedProgram the parent no longer holds. `aot_resume` keeps the mint-scoped inductor cache dir and its ledger row — so a killed mint's next attempt still hits torch's own FX graph cache — and re-homing file-level resume at the packed graph-class artifact is owed by step 3/4. -- **One artifact per KEY, enforced where it is the only enforceable place.** A compile child holds one share and cannot see a peer's keys, so two classes carrying one key survive the pool's name-keyed collection intact and the collision is first discovered by the HUB — a duplicate-key 409 on the second publish, on the pod, after both compiles are paid for. `mint_graph_classes` now groups the returned rows by key, keeps one artifact per key, and records the absorbed names as that entry's `aliases` (pgw#917's shape, and not a `class_hash` fact, so an alias cannot re-key its survivor). The coverage label folds over the SURVIVORS, so it does not depend on how the declaration was sharded. Both halves red-proven: severing the grouping returns two artifacts under one key; folding the label over the absorbed row as well moves the label. -- ⚠️ **pgw#917's INGRESS merge still does not run on the sharded path, and the earlier note here had its mechanism wrong.** It claimed a mergeable pair "keys IDENTICALLY" because `class_hash` folds no name. Measured: `class_hash` folds `class_dims`, which is the one axis a mergeable pair differs on (`_class_identity` requires every other axis to match), so the pair keys **APART** — `6decad0789e30a3a` vs `a185615c3fd880e4` for the same block at two coordinates. The consequence is therefore not a 409 but the original pgw#917 defect returning: both rows compile, both publish, both arm, and `EntryDispatch.select` answers `entry_ambiguous` on every call they carry — a SILENT 100 %-eager degrade rather than a loud rejection. The REFUSAL half is lost too: a cluster that collides at ingress but is *not* one class used to be a terminal `MintRefused` naming the differing axis. Both need the whole declaration BEFORE sharding, so they are owed at step 4; a family whose declaration carries a mergeable pair (sdxl regional: 36 of 72 compiles) is affected until then. -- **pgw#1215 step 4 (th#1834 Phase 3): th#1299 becomes a fence, the ungated router is deleted, and the peak-RSS contract takes the compiled-graph name.** `scripts/lint_serving_process_compiles.py` is new and gating: `torch.export` and the AOTI compile entry points may only be called by a module that runs inside a compile/trace child (three, each declared with the child that reaches it), so the serving process cannot acquire an export the day the middle mint-child tier goes away. `torch.compile` is deliberately not banned — th#1834 keeps the dynamo JIT-intake arm — and its half of th#1299 is now structural instead: `hot_swap.Router` refuses an ungated router typed (`RouterNotGated`) at `enable()`, at a warm-job enqueue and at a pgw#680 guard-miss heal, and `route()`'s degrade-to-inline-compile fallback is gone with the "ungated legacy" mode it belonged to. That mode survived on kill-switch parity with `GEN_WORKER_BG_YIELD`, which pgw#995 deleted and which `test_no_env_restores_the_pre_pgw677_tree` asserts cannot exist. `Executor.declares_compile` — caller-less, pinned only by a `hasattr` — is deleted with its pin retargeted onto the deletion. `mint_workers.entry_peak_rss` / `record_entry_peak_rss` / the `entry_peak_rss_bytes` wire field are renamed `compiled_graph_peak_rss*` across all six participants in one change. -- **pgw#1215 step 4 (th#1834 Phase 3, the reroute's final act): the SERVING PARENT supervises its compile children directly, and the middle mint-child tier is deleted.** `mint_delegate.py` is gone. `Executor._background_mint_run` / `_delegated_mint_run` / `_advertise_minted_cells` are replaced by `_supervise_mint` + `_advertise_compiled_graphs` over one new module, `mint_supervisor`, which drives `aot_mint.mint_graph_classes` on a worker thread and adopts what comes back through the ordinary delivered-cell path. Three process tiers become two: the tier that died existed only to hold a weight-free pipeline the parent already holds a real, resident, SERVING copy of. th#1299's property is kept by the lint fence that landed ahead of it, not by the process. -- **The parent-side gates the middle tier used to run now run where the pipeline is.** The declared-blocker refusal is `mint_supervisor.DeclaredBlockerRefusal` — deliberately NOT `MintRefused`, which is live in `aot_contract` for *"every declared class refused"*; sharing the name would make the two indistinguishable at the terminus with nothing going red and tensorhub's blame ladder attributing them identically. The class ENUMERATION (`aot_mint.declared_class_rows`) runs on the live pipeline, because the adapter fork is a property of the composed pipeline and no child can be told how many classes exist — only which share of them is its own. -- **pgw#947 re-based: a serving parent STATES its lane instead of benchmarking one.** The mint child A/B'd the kernel lanes by loading one full pipeline per candidate onto an empty card. A serving parent has neither an empty card nor the right to take one, and it does not need the probe — the lane this pod serves on is a fact it holds first-hand and is the lane the artifact must be stamped with. Recorded as a typed `kernel_path.unmeasured` verdict with its reason, never as an absence. -- **A RETRY CONSUMES WHAT THE LAST ATTEMPT PACKED.** `mint_delegate.build_cell`'s retry loop gave every attempt a fresh `child-N` directory, so attempt 2 of a 36-class mint re-traced and re-compiled 35 finished classes to retry one, at a measured 156-509 s each. That loop is deleted, not ported: the supervisor accretes into an attempt-STABLE directory, reads the packed artifacts back off disk (each names its own `cg-key-v1` key and carries its envelope), and hands the children `EntryJob.have_classes` so a held row is dropped from the share BEFORE it is exported. The held classes JOIN the result and the manifest fold, so a retry's honest report is "36 of 36" rather than "1 of 36", and `EntryCompilePool._assert_shares_whole` counts coverage rather than this attempt's work. Red-proven at both ends and in the middle: severing the skip re-exports the held class; severing the hand-off re-compiles it; severing the whole-set accounting refuses every successful retry. -- **pgw#917's INGRESS MERGE AND ITS REFUSAL HALF ARE BACK on the sharded path — `aot_mint.canonicalize_packed_classes`.** Restated because the earlier note had the mechanism wrong twice: a mergeable pair keys APART (`class_hash` folds `class_dims`, the one axis such a pair differs on), so the by-key dedupe never sees it and the loss is SILENT — both rows compile, both publish, both arm, and `EntryDispatch.select` answers `entry_ambiguous` on every call they carry, which is the 4,200-refusal defect pgw#917 was filed to fix. The gate now runs at the only seam a supervisor can reach: the packed envelope, which carries EVERY axis `_class_identity` folds (`graph_witness` for the body, `range_digest` for the ingress, `graph.pytree`/`graph.literals`/`graph.specialization`/`graph.lifted_inputs`, plus `precision`/`lora_bucket`/`strict_export`/`source_digest` off the metadata). Asked through `aot_serve.contract_from_meta` and `aot_serve.assert_ingress` themselves, so the gate cannot drift from what dispatch will do. Both halves red-proven. -- ⚠️ **What it does NOT recover, stated rather than implied: the duplicate COMPILE.** Both members of a mergeable cluster are already built by the time an envelope exists, so pgw#847's "36 of sdxl regional's 72 compiles bought nothing" still costs what it costs on this path. Recovering that needs the decision BEFORE the trace, and the merge predicate is a property of the traced program — which the serving parent may not produce, by the fence it also owns. Correctness first: a merged entry serves those coordinates COMPILED today, where two published rows served them eager. -- **The fence learns the status change it was written for.** `lint_serving_process_compiles.py` gains a `SUPERVISED` list — the opposite of an exemption: naming a module there ASSERTS the default serving-process rule over it, and a stale row reds exactly as a stale `CHILD_ONLY` row does. `aot_compile_pool` and `aot_compile_child` were exempt until the keystone; the supervisor now calls the pool from the serving loop, so their status moved from "exempt" to "must be clean" and the fence says so. -- **`mint_process` / `mint_child` survive as the OPERATOR one-shot path** (`scripts/micro_mint_rig.py`, the pre-publish proof harness th#1834's census marks RE-POINT, never delete) and inherit `MintTask` / `build_request` / `cfg_spec` from the deleted module. They have no production caller any more, which is the point of the change and is recorded as a debt with an owner and an expiry in `scripts/unreached_surface_baseline.txt` rather than as a silent exemption. -- ⚠️ **`aot_resume.py` is NOT deleted here, and the plan's "its only importer" is wrong.** Measured: `aot_compile_pool:73/1147` and `mint_child:453/465` both import and call it, and the first is a module Phase 3 keeps. Only `aot_resume.discard` lost its caller — the supervisor keeps no resume bank at all, because coverage accretes in the artifact directory and a retry consumes the artifacts. Its deletion stays sequenced where th#1834 put it: the change that proves boot resolves already-compiled graphs locally. -- **Built extraction-aware.** The compiled-graph interior — the accretion loop, held-artifact consumption, key/identity reads, the packed-envelope merge — sits in `mint_supervisor` and `aot_mint`; residency (which pipe holds what, and what the wire is told) stays in `executor.py`. That seam is the `torch-compiled-graphs` migration surface: the interior lifts whole. diff --git a/changelog.d/pgw1237.md b/changelog.d/pgw1237.md deleted file mode 100644 index 0cad1280c..000000000 --- a/changelog.d/pgw1237.md +++ /dev/null @@ -1,16 +0,0 @@ -- **pgw#1237: Cozy runtime inputs are one pinned contract.** The worker now - binds the five shared runtime environment names, path semantics and optional - secret classification to a public corpus that cozy-local fences byte-for-byte. -- **pgw#1237: worker values have a public carrier.** Activity decisions, - cell-resolve and trust refusals, compile-cache ranks, execution lanes, layout - identity, fabric admission, the pickle ban and all seven typed callout - refusals are pinned as exact values or explicitly bounded relations for - Tensorhub's peer fence. The carrier also binds the byte-compatible compile - cache flavor label, model-source and serving vocabularies, request fallback - and eager-posture reasons, hardware-unsuitable reason classes, the LoRA - weight bound, boot phases/outcomes/sources, and the intentionally - one-way `FnDegraded.ran` relation: the worker's current producer set is a - subset of the hub's accepted set because the hub retains the retired - `emergency_quant` value for compatibility. Unknown future boot phases remain - storable by Tensorhub; the fenced set is what this worker currently produces - and the hub currently interprets. diff --git a/changelog.d/pgw1239.md b/changelog.d/pgw1239.md deleted file mode 100644 index 7e89ae0aa..000000000 --- a/changelog.d/pgw1239.md +++ /dev/null @@ -1,6 +0,0 @@ -### Fixed - -- Pin the Hub-to-worker launch ABI in a public, source-bound contract corpus so - Tensorhub can fail closed when launch environment names, endpoint-owned - secret names, forbidden C2PA key material, topology delivery, managed fill, - or build-input classification drift between repositories. diff --git a/changelog.d/pgw1242.md b/changelog.d/pgw1242.md deleted file mode 100644 index b7ce3a19c..000000000 --- a/changelog.d/pgw1242.md +++ /dev/null @@ -1,3 +0,0 @@ -- **pgw#1242 (te#185): `resume_from`, the fifth reserved model input — a training endpoint can finally be handed back a checkpoint it published.** `ctx.save_checkpoint` has always published to the job workspace, and there was no door in the other direction: a model artifact reaches tenant code only as a reserved payload field, and `_reserved_repo_info` hardcoded four names — `source`/`destination`/`text_encoder`/`candidate` — none of which means "the adapter to continue from", with `source` already spoken for by the base model. There is no runtime model-ref materialization either (`resolve_dataset` has no model equivalent, pgw#684), and `kind="training"` declares no slots. So the SAVE half of resume worked, the LOAD half did not exist, and a multi-hour training run restarted from zero on pod loss — `image_lora_finetuner`'s *"Resume v1 = clean restart"* is that gap rather than a preference. te#185's H3 trainer is 4xH100 at $13.16/hr for hours, where "restart from zero" is the whole booking. -- **Rides the seam pgw#594 generalized and pgw#684 last extended**: `_materialize_source` already takes a `field_name` plus a `set_path` callback, so this is the same ~15-line shape as `candidate` — read the struct, pass it to the producer ctx, materialize it after `text_encoder`/`candidate`, expose `ctx.resume_from` / `ctx.resume_from_path`. **pgw#684's ruling is followed deliberately, not merely echoed: "fourth hardcoded name NOW, declarative NEXT."** pgw#690 remains the declarative successor and is explicitly not this change — coupling a reserved-field-contract refactor to an unblock a consumer is waiting on is what that ruling declined, and the same reasoning applies here. -- **Absent field is a no-op, which is the whole safety argument for a hardcoded name.** Every existing payload struct lacks `resume_from`, so it stays `{}`, no `ensure_local` fires and `resume_from_path` stays `None`. Producer-only, like its four siblings: an inference endpoint that happens to embed a `SourceRepo` never triggers materialization. A blank ref is a typed `payload.resume_from.ref` validation error naming the field the tenant actually wrote, and a fetch failure classifies RETRYABLE exactly as `source` does — restarting a paid continuation from zero because a checkpoint 404'd is the failure this field exists to prevent, so it must never degrade to a silent fresh start. Red-verified: 7 of the 8 new tests fail against the unpatched tree, and the one that passes is the inference no-op, correctly. diff --git a/changelog.d/th1887.md b/changelog.d/th1887.md deleted file mode 100644 index b6891dce5..000000000 --- a/changelog.d/th1887.md +++ /dev/null @@ -1,17 +0,0 @@ -- **th#1887: two behaviour-switch environment variables are deleted.** Paul: - *"get rid of these envs please, unless you really need them. I hate envs like - this."* `GEN_WORKER_VIDEO_ENCODER` is gone — the NVENC probe was always the - real decision, since `x264` only SKIPPED the probe and `nvenc` already fell - back to x264 when it failed, so the switch's only power was to make a pod - encode on CPU while its NVENC ASIC sat idle and never report the gap. - `GEN_WORKER_SVDQ_ENGINE` is gone — a PROCESS-wide pin over a PER-ARTIFACT - decision, so one incident's pin outlived the incident and silently served - every later artifact on the wrong engine; the typed per-call `override=` - argument survives and cannot leak into the next load. Both deletions are - proved by tests that assert the env is INERT: exporting the value that used - to win now changes nothing, checked against a forced-positive NVENC probe and - against int4 (whose only candidate is nunchaku) so neither can pass for the - wrong reason. `GEN_WORKER_NATIVE_KERNELS` is deliberately NOT deleted here — - it gates an unshipped rollout rather than switching a finished feature, and - removing it forces an activate-or-delete decision on the native-kernel - subsystem itself; reported for an owner instead of resolved blind. diff --git a/pyproject.toml b/pyproject.toml index b36e014ea..bf191a654 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "gen-worker" -version = "0.114.2" +version = "0.114.3" description = "A library used to build custom functions in Cozy Creator's serverless function platform." readme = "README.md" license = "MIT" diff --git a/uv.lock b/uv.lock index 19611fae3..df0bf511d 100644 --- a/uv.lock +++ b/uv.lock @@ -564,7 +564,7 @@ wheels = [ [[package]] name = "gen-worker" -version = "0.114.2" +version = "0.114.3" source = { editable = "." } dependencies = [ { name = "blake3" },