Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions .github/workflows/prepared-export-browser.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
name: Prepared export browser verification

on:
push:
branches: [fix/large-media-exports]
pull_request:
workflow_dispatch:

permissions:
contents: read

jobs:
browser:
name: Prepared jobs and native playback — attempt ${{ matrix.attempt }}
runs-on: ubuntu-24.04
timeout-minutes: 25
strategy:
fail-fast: false
matrix:
attempt: [1, 2]
steps:
- uses: actions/checkout@v4
- run: bash .github/scripts/ci-media-apt.sh update
- run: bash .github/scripts/ci-media-apt.sh install ffmpeg python3-venv openssl
- name: Build the exact checkout runtime image
run: docker build -t simplenvr-export-test .
- name: Install pinned Chrome tooling
run: |
python3 -m venv "$RUNNER_TEMP/export-browser-venv"
. "$RUNNER_TEMP/export-browser-venv/bin/activate"
python -m pip install 'playwright==1.56.0'
python -m playwright install --with-deps chrome
- name: Protocol regressions and real HTTPS/browser acceptance
run: |
. "$RUNNER_TEMP/export-browser-venv/bin/activate"
python -m unittest discover -s tests -p 'test_prepared_export_protocol.py' -v
bash tests/runtime-browser-acceptance.sh simplenvr-export-test "$RUNNER_TEMP/prepared-export-browser"
- name: Retain media, network, browser and container evidence
if: always()
uses: actions/upload-artifact@v4
with:
name: prepared-export-browser-${{ github.sha }}-${{ github.run_attempt }}-${{ matrix.attempt }}
path: ${{ runner.temp }}/prepared-export-browser
if-no-files-found: warn
retention-days: 14
51 changes: 51 additions & 0 deletions .github/workflows/prepared-export-go.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
name: Prepared export Go verification

on:
push:
branches: [fix/large-media-exports]
pull_request:
workflow_dispatch:

permissions:
contents: read

jobs:
go:
name: Full Go build, race suite and vet
runs-on: ubuntu-24.04
timeout-minutes: 25
env:
GOFLAGS: -mod=readonly
SIMPLENVR_REQUIRE_FFMPEG: '1'
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version-file: go.mod
cache: false
- uses: actions/setup-node@v4
with:
node-version: '22'
- run: bash .github/scripts/ci-media-apt.sh update
- run: bash .github/scripts/ci-media-apt.sh install ffmpeg
- name: Prepared protocol regression tests
run: python3 -m unittest discover -s tests -p 'test_prepared_export_protocol.py' -v
- name: Verify dependencies and build the exact checkout
run: |
go mod download
go mod verify
git diff --exit-code -- go.mod go.sum
go build ./...
- name: Full race-enabled Go suite including Node and native media tests
run: bash tests/core-ci-evidence.sh "$RUNNER_TEMP/prepared-export-go"
- name: Static analysis
if: always()
run: go vet ./...
- name: Retain Go execution evidence even on failure
if: always()
uses: actions/upload-artifact@v4
with:
name: prepared-export-go-${{ github.sha }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/prepared-export-go
if-no-files-found: warn
retention-days: 14
13 changes: 13 additions & 0 deletions docs/EXPORT_API_MODES.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Export modes: integration and verification status

The HTTP planner and browser are wired to prepared copy/accurate exports. Copy is the default and never silently switches to re-encoding. Whole single-source copies preserve original bytes in prepared storage; compatible multiple sources preserve compressed A/V payloads when remuxed. This is not zero-copy original-file HTTP delivery.

Subsection copy supports non-reordered video with usable decoded keyframe timestamps. The server proposes video boundaries at keyframe/file boundaries, reinspects expanded coverage and returns copy_range_approval_required before admitting a job. The UI shows the requested/proposed intervals and AAC packet allowance and waits for explicit approval. Canceling a proposal creates no job. Approval retains the original selection and sends accepted_start/end for fresh server checks. Changed proposals require another click. Unsupported copy must be changed to Accurate explicitly.

AAC packet edges may extend outside the video-aligned interval by the disclosed allowance. Timing is metadata/frame/packet granular, not calibrated camera-clock time. Candidate copy outputs are duration-probed and externally decoded before readiness; this is not a proof of every source frame/sample or colour value.

Accurate explicitly re-encodes video with libx264 CRF 18 and audio with AAC. Both tracks share the video/cut origin; supported audio offsets are preserved. Full-range input becomes limited-range yuv420p, so accurate exports are not lossless or byte-identical. Leading/trailing silence can maintain the selected window without stretching audio. Mixed audio configurations, extra tracks, rotation and higher bit depths remain unsupported.

The live-container browser acceptance harness has been migrated to prepared jobs, real downloads and server release. Its container now mounts private export scratch. Branch-triggered Go and two-attempt Chrome workflows test the same pushed commit. See PREPARED_EXPORT_VERIFICATION.md for assertions and the distinction between local helper checks and actual CI results. The older routed-fixture browser script remains separate and unmigrated.

Release readiness is NOT established until both workflows pass on the final commit and failures are resolved. Nothing in this change merges, deploys or certifies real-camera/full-resolution/multi-GiB operation. Scratch settings, quotas, session isolation and leases remain described in LARGE_MEDIA_EXPORTS.md.
23 changes: 23 additions & 0 deletions docs/EXPORT_RENDERING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Prepared export rendering backend: implementation status

This is a staged backend implementation, not a completed integration. export_media.go adds copy/accurate rendering primitives and export_media_test.go adds media integration tests. The HTTP planner still calls the legacy renderer and the browser does not yet send a mode choice. These paths are NOT yet selectable through the app. Do not release the branch as complete.

## Copy

A single complete source is copied byte for byte into bounded prepared-output storage. It is not yet the planned zero-copy original-file HTTP delivery. Several complete sources are remuxed with explicit audio/video mapping, stream copy, and matching codec configuration/extradata checks. No nominal/average frame-rate equality test is used. Packet conversion is disabled for compatible concatenation, preserving compressed payloads. Additional tracks or incompatible configurations are refused, never silently selected/dropped or transcoded.

Subsections in copy mode currently return a boundary error. Keyframe probing, actual-range preview and explicit consent are still required before adjusted-boundary copy is enabled. No intervals are widened automatically.

## Accurate

Audio/video windows use the same video timestamp origin and cut reference, preserving relative audio offset rather than independently resetting tracks. AAC is retained by re-encoding in accurate mode; its sample rate/channel count are validated. Leading/trailing missing audio in an accepted input window is represented as silence to maintain the window duration, without stretching audio. Mixed audio presence or mismatched audio configurations across parts are currently refused.

8-bit yuv420p and full-range yuvj420p are supported within existing dimension bounds. Accurate full-range input is explicitly converted to limited-range yuv420p output. Video is re-encoded with libx264 CRF 18; accurate exports are not byte-identical or lossless. Copy does not perform this conversion. Rotation and higher-bit-depth profiles remain unsupported. Frame-rate strings are not used as a proxy for valid timestamps. Video is frame-granular, not exact camera wall-clock timing.

Fragmented MP4 accurate output uses delayed moov/edit-list handling with negative timestamps enabled for AAC encoder priming; this avoids treating encoder delay as source timing. Output must be a fresh empty regular file. Sources are pinned and revalidated, output writes are bounded, output tracks/durations are probed, and all subprocesses are joined before return. Checks do not by themselves certify every frame, colour value or audio sample.

## Evidence and remaining work

Local FFmpeg command experiments (not execution of the Go implementation) produced decodable H.264/AAC single/multi-part accurate output, preserved a deliberately delayed audio onset at approximately 250 ms, and preserved compressed video/AAC packet hashes when copying complete identical-profile clips. Go integration tests cover the camera metadata profile, original-byte copying, copied-packet hashes, multi-part accurate output, external A/V decode, delayed audio onset, source digests, cancellation and output limits. Go is unavailable in the assistant environment, so these Go tests have NOT been executed or compilation-verified.

Still required: HTTP job renderer wiring; API and UI explicit mode selection; keyframe-adjusted copy planning/consent; original descriptor delivery; real-camera and fractional/B-frame boundary timing validation; colour-range validation; large-file and cancellation-under-load tests; migration and execution of the real Chrome acceptance suite.
38 changes: 38 additions & 0 deletions docs/LARGE_MEDIA_EXPORTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# Large-media exports: work in progress

Do not release this branch as the completed export redesign.

## Implemented

- Bounded disk-backed job store, random IDs, owner checks, independent seekable readers, cancellation, expiry and shutdown joining.
- Authenticated POST /api/exports/create and /api/exports/cancel, plus GET/HEAD /api/exports/status?id=ID and /api/exports/download?id=ID. Event tokens cannot access these routes; jobs are bound to session and credential version.
- Startup and shutdown integration. Download responses use ServeContent with Range support.
- Browser UI creates jobs, polls state, and offers a normal same-origin download. Page JavaScript does not fetch/buffer video or impose the old 64 MiB cap. Control responses are bounded to 8 KiB.
- Range controls are retained. Cameras are processed sequentially; the next job starts after release or expiry of the prepared file.
- Cancel/release requires server acknowledgement; failures remain visible with a retry control. Lost creation responses are not retried automatically. Navigation leaves jobs to server expiry rather than canceling a download during handoff.

## Configuration

Mount trusted disk-backed scratch storage outside /recordings and camera directories (for example /exports) and set NVR_EXPORT_SCRATCH_DIR=/exports. It must exist and must not resolve through symlinks. Without it the new endpoints return 503; the job UI reports that configuration is required and does not fall back to the old Blob path. Invalid configured limits/storage fail startup. Do not put GiB exports in a tmpfs or container writable layer.

| Setting | Default | Meaning |
| --- | --- | --- |
| NVR_EXPORT_MAX_BYTES | 8589934592 | Per-job prepared output (8 GiB) |
| NVR_EXPORT_TOTAL_BYTES | 17179869184 | Aggregate job-output reservation (16 GiB) |
| NVR_EXPORT_MAX_JOBS | 8 | Retained records |
| NVR_EXPORT_PARALLEL | 1 | Concurrent preparations |
| NVR_EXPORT_TTL | 30m | Processing expiry; retention resets on completion |
| NVR_EXPORT_PREPARE_TIMEOUT | 30m | Preparation deadline including queue wait |

Each admitted job reserves its maximum output; defaults allow at most two reservations. Active expired downloads remain charged until their leases end. Legacy trim intermediates retain separate existing bounds. Reservations are not physical disk reservations or a bound on all filesystem allocations. Monitor free space. Jobs do not survive restart.

POST create JSON contains camera_id, start_time and end_time (ordered RFC3339 timestamps). A 202 response has id, state, message, bytes, expires_at and a Location status URL. Ready status adds download_url. POST cancel uses {"id":"JOB_ID"} and confirms with 204. Subsequent requests require the creating session. Cancellation/expiry prevent new leases; already-started responses may continue. Logout and credential changes block subsequent requests, not necessarily existing responses.

## Still required before release

- Original-file delivery, compatible audio/video stream-copy assembly, and consent for adjusted keyframe ranges. The current renderer and its audio/pixel-format/frame-rate restrictions are unchanged.
- Optional accurate audio-preserving trimming and representative media validation.
- Migrate the real Chrome and routed browser acceptance harnesses from Blob URLs to job create/status/download/release semantics; enable scratch mounts in runtime fixtures. Existing browser harnesses still contain old assumptions and are not expected to pass this intermediate branch. Do not delete their media, hash, duration, playback or cancellation checks.
- Real large-file, slow-client, media, Range, authenticated-session and shutdown runtime checks.

JavaScript syntax and the updated Node export UI/range regressions passed locally in UTC and America/Chicago. The 5 GiB test uses reported size metadata, not an actual 5 GiB transfer. Go and real-browser tests have not been run here.
11 changes: 11 additions & 0 deletions docs/PREPARED_EXPORT_VERIFICATION.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Prepared export verification

Both prepared-export workflows trigger on pushes to fix/large-media-exports, pull requests and manual dispatch. A push runs the full Go build/race/vet workflow and two independent Chrome acceptance attempts against the same event commit. Require both workflows to pass on that exact commit before completion is claimed. No tests were replaced by an export-only Go subset.

The live HTTPS browser harness now uses create/status/download/cancel endpoints. It checks anonymous access refusal, whole-copy compressed video packet hashes, external decode/duration, HEAD and Range, missing-coverage refusal, accurate AAC output and duration, copy proposal cancellation and explicit approval, active accurate-render cancellation and child reaping, browser-saved SHA-256/duration/decode, and HTTP 404 after server release. It no longer expects Blob URL revocation or fetches video into page JavaScript for hashing. Test-directed range/seek state is retained; application functions and response bodies are not replaced.

The container has a private scratch bind mount at /exports, outside /recordings, with explicit small test quotas. Non-reordered fixtures exercise supported subsection copy; reordered inputs remain covered by Go refusal tests. Native playback, real login/navigation diagnostics and the expanded playback-edge suite are retained. Reports include the commit, image identity, browser version, media and network evidence. Passing synthetic acceptance is not production-camera or multi-GiB qualification.

Local checks: Python/shell/YAML syntax; all 30 embedded browser-JavaScript expressions parsed by Node; media helper check copied 160 video packets without hash changes and externally decoded a 16-second fixture. The earlier six protocol regressions passed. These are not a live-backend or browser pass. Go and Docker are unavailable locally; actual Go/Chrome/container results must come from CI after this commit is pushed.

The older routed-response tests/browser-export-acceptance.py is a separate legacy harness, not the real-runtime acceptance entrypoint. It has not been migrated or reported as passing here. Do not confuse that script with the migrated tests/runtime-browser-acceptance.py executed by both browser workflows.
Loading
Loading