Skip to content

feat(export): run the public data export unattended and publish to object storage #852

Description

@bencap

Problem

The export assumes an operator: an SSH tunnel to production, DB_* environment variables exported by hand, the script run from a workstation, a ZIP written to the working directory, and verification by reading unzip | grep output. There is no machine-checkable success condition, so a scheduled run would have no basis for deciding whether its output is publishable.

Proposed behavior

The export runs on a schedule on the existing worker, against production, with no operator present, and publishes to the dump bucket.

  • Destination bucket and key prefix come from configuration. Local filesystem output stays available for development.
  • The archive is written under a dated key.
  • The run emits a result record: asOf, experiment set count, score set count, archive byte size, archive SHA-256, and object key.
  • The run maintains a published-release registry — the set of archives eligible to be served, and which one is current.
  • The generated archive is verified after generation, see feat(export): verify a generated archive before it enters the registry #853

Acceptance criteria

  • The export completes with no interactive input and no tunnel from a workstation, given only configuration and credentials.
  • The archive lands in the configured bucket under a dated key, byte-identical to what the manual procedure produces for the same database state.
  • The result record is emitted and includes the archive SHA-256 and object key.
  • A run that fails partway leaves nothing in the registry and no object reachable as current.
  • Failures reach the alerting path in Slack.
  • The registry names exactly one current archive at any time.
  • The registry is not uploaded if feat(export): verify a generated archive before it enters the registry #853 does not pass.

Metadata

Metadata

Assignees

No one assigned

    Labels

    app: backendTask implementation touches the backend

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions