Speed repeated local learning preparation with verified caches

.github/workflows/ci.yml

@@ -78,6 +78,8 @@ jobs:
           dvidia-piper-recovery --help
           dvidia-piper-reliability --help
           dvidia-training-doctor --json
+      - name: Verify pinned learning caches
+        run: python -m unittest tests.test_camera_action_cache tests.test_camera_cache_cli tests.test_camera_action_export tests.test_camera_action_model -v
       - name: Test footage, visual training and loopback studio
         run: python -m unittest tests.test_footage_data tests.test_footage_train tests.test_footage_pipeline tests.test_footage_benchmark tests.test_training_studio tests.test_doctor tests.test_coverage tests.test_episode_tape tests.test_reaction tests.test_reaction_metrics tests.test_reaction_convergence tests.test_reaction_benchmark tests.test_motor_memory tests.test_motor_metrics tests.test_motor_benchmark tests.test_observations tests.test_observation_models tests.test_motion tests.test_motion_benchmark tests.test_transition_model tests.test_transition_pilot tests.test_transition_cli tests.test_embodiment_calibration tests.test_embodiment_cli tests.test_reference_sim -v
 

README.md

@@ -6,6 +6,14 @@
 
 [Recorded demo](https://huggingface.co/spaces/Dvidia/dvidia-training) · [Synthetic datasets](https://huggingface.co/datasets/Dvidia/dvidia-training-examples) · [Pilot models](https://huggingface.co/Dvidia/dvidia-training-pilot)
 
+**[Verified local learning cache](docs/camera-action-export-cache-v01.md).**
+The **0.1.3.dev5 preview** adds `dvidia-train cache` to reuse verified camera/action
+features with caller-pinned data, split and source identities. On one 1,000-sample
+training episode, validated loads took a median **2.952 ms**, versus **1.636 s**
+for strict raw export; numeric values and model dataset identity were unchanged.
+Cache hits verify derived bytes, not raw recordings. This speeds repeated local
+data preparation; it does not improve the frozen **5/64** placement result.
+
 **[Camera-to-action skill v0.1](docs/camera-action-v01.md).**
 The **0.1.3.dev4 preview** adds a local demonstration-to-action learning loop:
 25 Hz RGB-D, 50 Hz measured joints and demonstrated targets, a compact CPU

benchmarks/camera-action-export-cache-v01/README.md

@@ -0,0 +1,35 @@
+# Camera-action export cache: bounded local benchmark
+
+`results.json` is the byte-identical result from one first-protocol-ordered training episode (`train-00-goal0-teacher`). No fitting, native simulation or confirmation-episode data was used.
+
+| Operation | Wall time |
+|---|---:|
+| Unchanged strict export | 1.636 s |
+| Strict export + cache preparation | 1.053 s |
+| Three validated cache loads | 2.952 / 2.831 / 3.214 ms |
+
+All 1,000 feature/target samples, groups, provenance and model dataset identity match exactly for every load. This episode's strict-export features are float32, as stored in the cache. Multi-episode strict export promotes float32 values to float64; caching those values as float32 preserves numeric values and model identity but does not preserve the combiner's array dtype or NPY bytes.
+
+The median ratio was 554× for repeated preparation of this one warm shared-host episode. The 1,137,058-byte cache replaces repeated validation of an episode containing 6,011 files and 197,398,789 bytes for this use; original recordings remain intact. A cache hit does **not** reverify those raw files. No inference-speed, grip-success, task-success or hardware claim follows from these timings.
+
+Host scope: arm64 macOS, Python 3.12.9, NumPy 2.5.3. CPU and memory sysctl metadata was unavailable in the sandbox. Peak process RSS was 163,463,168 bytes including strict export and preparation. Validation is timed; Python imports and explicit post-load equality comparison are excluded. There was no OS file-cache flush or exclusive CPU reservation. The shorter cache-build time is an order/cache effect, not a faster strict exporter.
+
+Three data-preparation measurements are retained. The first reporter attempt completed data operations but failed on restricted optional host metadata; its local cache/pins directory remains intact. The corrected reporter persisted progress and produced the prior-source measurement in `results-prior-source.json` (SHA-256 `5e28976319740bc6da85c833084c87b0df4a48aea62f0c249a1007c9cddbc922`, cache module `ad79c142b51798281c3f2ce0645662e24da8a21e778ae7785ce9c2c44102c185`).
+
+Review then identified a Windows portability issue: setting the temporary hard link read-only before unlinking its temporary name can prevent cleanup. The writer now unlinks the still-writable temporary name before sealing the complete target read-only. The current measurement reran the same episode in a fresh directory, bound to cache module `9191f2eb9f96bd09d4f711c31966a583bd938853a874e6018f8ddebfb0bc00d8`. Times across these shared-host runs are not a controlled before/after comparison or evidence that this cleanup fix accelerated export. There were no native/model trials or benchmark-scene replacements. Original source bytes and local directories are preserved.
+
+The focused CLI/cache/export suite passes 25 tests, including cleanup order, no-overwrite behavior, disk-space refusal and fixtures that skip unavailable platform link privileges. The cache tests are included in the existing Ubuntu/macOS/Windows CI matrix; local execution here was macOS only. A fresh offline `0.1.3.dev5` wheel install outside the checkout loaded the new cache using independently retained benchmark pins and matched all 19 frozen source hashes plus the cache source. It also passed wrong-split refusal, original robot stages, and footage help checks without installing native simulation dependencies.
+
+- Result SHA-256: `a4b4b1675e8d157ee9808e556e37c6c697134f529a084488b930299028084353`.
+- Dataset SHA-256: `856205dec60ad1c452373fabfd1a75e72a875ff76075971ac2ed26ada8a8bdf3`.
+- Protocol SHA-256: `576851e321034a0f6d3cb4d0818b3a5ca97975fcc6f2c7fa30b07ffd203a2d76`.
+- `cli-proof.json` records fresh installed-wheel source, inspection/load, wrong-split refusal and existing CLI checks, outside the three timed loads. `cli-proof-prior-source.json` retains the earlier source-bound checks.
+- [Usage, trust boundary and reproduction](../../docs/camera-action-export-cache-v01.md).
+
+## Complete training dataset follow-up
+
+[full-training-results.json](full-training-results.json) covers the complete registered training split: 64 teacher and 32 mixed-control episodes, 96,000 samples, 32 start groups. Both caches were built once with strict source validation in **160.671 s**. The three verified load-and-combine measurements were **0.347 / 0.280 / 0.279 s**; the four original model-reference comparisons and blind projection took a separate **0.596 / 0.491 / 0.490 s**. All four sealed model dataset/provenance identities matched on every load (12/12 checks).
+
+Derived caches total 108,804,852 bytes, from a source inventory of 18,971,932,592 bytes / 577,056 files. Whole-process peak RSS was 1,631,633,408 bytes and total validation time was 163.738 s. The same shared-host caveats apply. There was no redundant strict-export baseline, no new fit, native simulation or held-out recording export, and no task-success improvement or qualification change. The repeated-load measurement does not include training. Raw recordings are verified at cache construction, not again on a hit.
+
+[validate_full_training.py](validate_full_training.py) is a portable version of the exact executed script, with required absolute CLI paths replacing host-specific paths. [full-training-script-provenance.json](full-training-script-provenance.json) pins both original and portable hashes; the portable variant was not run for a new measurement. All original source, protocol, stage, model and complete training-roster checks remain intact. [Reproduction and scope](../../docs/camera-action-export-cache-v01.md#complete-training-data-validation) explain the fixed dev5 environment and fresh-output requirements. This benchmark-and-documentation follow-up does not replace the released wheel or tag; the candidate remains unqualified.

benchmarks/camera-action-export-cache-v01/benchmark.py

@@ -0,0 +1,104 @@
+#!/usr/bin/env python3
+"""One registered training episode: strict export, cache build, three verified hits.
+
+No fitting, teacher execution, simulator or confirmation-scene reads. Output is
+fresh and outside the source study. Wall times include validation, not imports.
+"""
+import argparse
+from hashlib import sha256
+import json
+from pathlib import Path
+import platform
+import resource
+import subprocess
+import time
+
+import numpy as np
+
+from dvidia_training import camera_action_cache as cache
+from dvidia_training import camera_action_export as export
+from dvidia_training import camera_action_model as model
+
+
+def array_hash(value):
+    return sha256(np.ascontiguousarray(value).tobytes()).hexdigest()
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('study'); parser.add_argument('output'); parser.add_argument('--protocol-sha256', required=True)
+    args = parser.parse_args()
+    root = Path(args.study).resolve(); out = Path(args.output).absolute()
+    if root == out or root in out.parents:
+        raise ValueError('Proof output must be outside source study.')
+    out.mkdir()  # fresh output, no rerun/overwrite
+    def progress(event):
+        with (out/'progress.jsonl').open('a') as stream:
+            stream.write(json.dumps(event, sort_keys=True) + '\n')
+    protocol = json.loads((root/'protocol.json').read_text())
+    if sha256((root/'protocol.json').read_bytes()).hexdigest() != args.protocol_sha256:
+        raise ValueError('Unexpected protocol identity.')
+    entry = next(r for r in protocol['rosters']['demonstrations'] if r['case']['split'] == 'train')
+    relative = 'episodes/demonstrations/' + entry['id']
+    directory = root/relative
+    metadata_before = {name: sha256((directory/name).read_bytes()).hexdigest()
+                       for name in ('recording.json','integrity.json','rollout.json')}
+    start = time.perf_counter(); strict = export.export_episode(directory); strict_s = time.perf_counter()-start
+    progress(dict(stage='strict_export', seconds=strict_s))
+    start = time.perf_counter()
+    pins = cache.build_cache(root, out/'cache', expected_protocol_sha256=args.protocol_sha256,
+                             split='train', stage='demonstrations', attempt_ids=[entry['id']])
+    build_s = time.perf_counter()-start
+    progress(dict(stage='strict_cache_build', seconds=build_s))
+    (out/'pins.json').write_text(json.dumps(pins, sort_keys=True, indent=2)+'\n')
+    (out/'pins.json').chmod(0o444)
+    loads, checks = [], []
+    for _ in range(3):
+        start = time.perf_counter()
+        loaded = cache.load_cache(out/'cache', expected_manifest_sha256=pins['manifest_sha256'],
+             expected_protocol_sha256=args.protocol_sha256, expected_split='train', expected_source_sha256=pins['source_sha256'])
+        loads.append(time.perf_counter()-start)
+        progress(dict(stage='verified_hit', index=len(loads), seconds=loads[-1]))
+        # Equality comparisons are outside the timed hit; byte/hash/semantic
+        # validation performed by load_cache itself is inside every timed hit.
+        checks.append(dict(features=np.array_equal(strict['features'], loaded['features']),
+            targets=np.array_equal(strict['targets'], loaded['targets']), groups=strict['groups']==loaded['groups'],
+            provenance=strict['provenance']==loaded['provenance'],
+            dataset_sha256=model._dataset(strict, blind=False)[-1]==model._dataset(loaded, blind=False)[-1]))
+    if not all(all(c.values()) for c in checks):
+        raise ValueError('Cache output differs from strict export.')
+    metadata_after = {name: sha256((directory/name).read_bytes()).hexdigest() for name in metadata_before}
+    if metadata_before != metadata_after:
+        raise ValueError('Source episode metadata changed during benchmark.')
+    hardware = dict(platform=platform.platform(), machine=platform.machine(), python=platform.python_version(), numpy=np.__version__)
+    if platform.system() == 'Darwin':
+        for name in ('machdep.cpu.brand_string','hw.physicalcpu','hw.memsize'):
+            query = subprocess.run(['sysctl','-n',name], check=False, text=True, capture_output=True)
+            hardware[name] = query.stdout.strip() if query.returncode == 0 else {'status':'unavailable', 'reason':'host metadata query refused'}
+    files = [p for p in directory.rglob('*') if p.is_file()]
+    result = dict(schema='dvidia.camera-action-export-cache-benchmark/v1', scope=__doc__.strip(),
+        attempt_id=entry['id'], split='train', protocol_sha256=args.protocol_sha256,
+        source_identity=cache.source_identity(), pins=pins, episode_metadata_sha256=metadata_after,
+        samples=len(strict['groups']), features_sha256=array_hash(strict['features']), targets_sha256=array_hash(strict['targets']),
+        groups_sha256=model.digest(strict['groups']), provenance_sha256=model.digest(strict['provenance']),
+        equality_checks=checks, strict_export_seconds=strict_s, strict_cache_build_seconds=build_s,
+        verified_hit_seconds=loads, verified_hit_median_seconds=float(np.median(loads)),
+        strict_to_hit_median_ratio=strict_s/float(np.median(loads)),
+        source_episode_files=len(files), source_episode_bytes=sum(p.stat().st_size for p in files),
+        cache_bytes=sum(p.stat().st_size for p in (out/'cache').iterdir()),
+        peak_process_rss=resource.getrusage(resource.RUSAGE_SELF).ru_maxrss,
+        peak_process_rss_units='bytes' if platform.system()=='Darwin' else 'KiB',
+        hardware=hardware, raw_source_reverified_on_hit=False, native_runs=0, fitting_runs=0,
+        qualification_changed=False,
+        timing_caveats=['One fixed training episode, one strict export and one strict cache build, then three sequential cache hits.',
+                       'Existing OS file cache was not flushed; shared laptop, no exclusive CPU reservation.',
+                       'Imports are excluded; validation and source-hash checks are included. Equality comparison is outside timed hit.',
+                       'RSS is cumulative process high water including strict export and build, not isolated cache memory.',
+                       'No evidence of faster inference, improved control, whole-dataset throughput or hardware readiness.'])
+    (out/'benchmark.json').write_text(json.dumps(result, sort_keys=True, indent=2)+'\n')
+    print(json.dumps({k:result[k] for k in ('attempt_id','samples','strict_export_seconds','strict_cache_build_seconds',
+                      'verified_hit_seconds','strict_to_hit_median_ratio','cache_bytes')}, indent=2))
+
+
+if __name__ == '__main__':
+    main()

benchmarks/camera-action-export-cache-v01/cli-proof-prior-source.json

@@ -0,0 +1,13 @@
+{
+  "scope": "Fresh Python CLI processes, cached artifact only, no training or raw-data verification.",
+  "valid_inspect_exit": 0,
+  "valid_inspection": {
+    "dataset_sha256": "856205dec60ad1c452373fabfd1a75e72a875ff76075971ac2ed26ada8a8bdf3",
+    "raw_source_reverified_on_hit": false,
+    "samples": 1000,
+    "scope": "Cached derived learning arrays, not raw recording revalidation, source authentication, training, native simulation or hardware qualification.",
+    "status": "verified"
+  },
+  "wrong_split_exit": 1,
+  "wrong_split_refused": true
+}