Skip to content

pgw#1648: hardcut streaming fills to tensorfs - #1191

Draft
PaulFidika wants to merge 3 commits into
masterfrom
1648-tensorfs-fill-client
Draft

pgw#1648: hardcut streaming fills to tensorfs#1191
PaulFidika wants to merge 3 commits into
masterfrom
1648-tensorfs-fill-client

Conversation

@PaulFidika

@PaulFidika PaulFidika commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Summary

  • replace PGW streaming reads, pinned ring, copy walk, checkpoint image pread, warm torch copy, and cold varena refill submission with primitive destination/source maps handed to tensorfs
  • require the native tensorfs store; delete the bridge fallback and streaming staging.py in the same cut
  • preserve the construction census predicate/fence and admit the binding-carried expected census through the exact hook coordinated with th#2290
  • keep the seam torch-free: strings, integer pointers/capacities/offsets, shapes, element width, layout token, and immutable file-range/hole records only

Current proof at 26b2e80

  • real CPU native-fill pipeline is byte-equal across more than 100 tensors
  • repeated fresh load after interpreter/config warmup: /proc/self/io write_bytes == 0
  • 91 combined checkpoint, seam, streaming, lane-dtype, quantized, tied-weight, and census tests green
  • warm B-to-A-to-B swap arm uses identical reservation addresses and records zero compile calls
  • cold swap arm uses the same tensorfs client with complete file/hole records and honest destination byte accounting
  • ruff and strict mypy green on touched source
  • deletion census: no BridgeWeightStore, StagingPool, cudaMemcpyAsync, legacy _Placement/_walk, checkpoint _pread, torch warm copy, or varena refill submit remains in the cut paths; streaming/staging.py is deleted

Executed red/green evidence

  • tensorfs planted mutation: forcing every requested arrangement to torch.contiguous@1 made the independent channels-last oracle fail at byte index 1; restoring the requested arrangement returned it green
  • existing null/short destination, wrong-rank, unratified-layout, unknown-name, invalid-ledger, and partial-switch refusal arms remain green

Explicit blockers before ready/merge

  • tensorfs PR changelog: gw#456 entry for 0.13.5 #153 merged as 1828c90 and v0.1.0 is tagged there; release run 32538259817 proved all gates, six wheel builds/smokes and two sdist installs green, then PyPI refused with invalid-publisher because no trusted publisher matches repo cozy-creator/tensorfs, workflow .github/workflows/publish.yaml, environment pypi. Both project and exact-version endpoints remain 404. Configure the PyPI publisher, then rerun failed jobs unchanged; do not retag/refire.
  • tcg#63: torchcg's unversioned tensorfs Git source conflicts with PGW's exact 1828c90 draft source pin. Once 0.1.0 is public, torchcg must depend on exact tensorfs 0.1.0 and delete its source override; PGW will not weaken its pin.
  • one real endpoint same-card/same-tree parity measurement remains required; no endpoint will be spent/refired blindly

Fence

Tracker: pgw#1648

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant