Problem
Two gaps exist between what MaveDB publishes and what a platform integrator needs:
- No predictable URL. The archive is published to Zenodo under concept DOI
10.5281/zenodo.11201736, which resolves to the latest version but is not a fetchable file URL. An integrator has no address to point a loader at. Download throughput via Zenodo is also poor.
- No cadence. The release is a manual procedure: SSH tunnel to production, run script, browser upload to Zenodo and performed roughly quarterly. There is currently no mechanism to run it unattended.
We are also limited in that serving ourselves moves egress costs onto our AWS account, which Zenodo currently absorbs. There is no AWS mechanism that caps this spend, so our exposure has to be bounded by design.
Architecture
One artifact. The published archive keeps its current shape and contents. This work changes how it is produced, where it is served, and how often it is uploaded- not what it contains.
Production. The export runs on a schedule on the existing worker, writes a compressed archive to the dump bucket under a dated key. A partial run must never become current.
Serving. A dedicated CloudFront distribution over the dump bucket, separate from the UI's. Dated paths are immutable and retained; /latest resolves to the newest published archive.
Bounding egress. Two controls doing two different jobs:
- Spikes are bounded by a WAF rate-based rule with
CONSTANT aggregation, which counts all requests to the archive path regardless of source. This caps total burst throughput rather than handling it on a per-client basis. This is enforced inline and acts in real time.
- Sustained volume is bounded by CloudWatch alarms. A rate alarm on bytes-per-interval, and a cumulative ladder against the monthly budget. CloudFront metrics arrive minutes late, so alarms cannot stop a burst. However, they can tell us when we are close to our monthly limit.
We can then implement a kill switch which flips the WAF rule to Block, which takes effect in about a minute.
At roughly 2 GB per compressed archive, the account's 1 TB CloudFront allowance is about 500 downloads per month, and that allowance is shared with the UI distribution. Overage runs about $0.085/GB, so a thousand downloads in a month costs on the order of $80. CDN caching does not reduce this: S3-to-CloudFront origin transfer is already free and delivery is charged identically on cache hits.
Spike protection protects us from reaching the limit quickly before we can be alerted via our CloudWatch alarms.
Problem
Two gaps exist between what MaveDB publishes and what a platform integrator needs:
10.5281/zenodo.11201736, which resolves to the latest version but is not a fetchable file URL. An integrator has no address to point a loader at. Download throughput via Zenodo is also poor.We are also limited in that serving ourselves moves egress costs onto our AWS account, which Zenodo currently absorbs. There is no AWS mechanism that caps this spend, so our exposure has to be bounded by design.
Architecture
One artifact. The published archive keeps its current shape and contents. This work changes how it is produced, where it is served, and how often it is uploaded- not what it contains.
Production. The export runs on a schedule on the existing worker, writes a compressed archive to the dump bucket under a dated key. A partial run must never become current.
Serving. A dedicated CloudFront distribution over the dump bucket, separate from the UI's. Dated paths are immutable and retained;
/latestresolves to the newest published archive.Bounding egress. Two controls doing two different jobs:
CONSTANTaggregation, which counts all requests to the archive path regardless of source. This caps total burst throughput rather than handling it on a per-client basis. This is enforced inline and acts in real time.We can then implement a kill switch which flips the WAF rule to
Block, which takes effect in about a minute.At roughly 2 GB per compressed archive, the account's 1 TB CloudFront allowance is about 500 downloads per month, and that allowance is shared with the UI distribution. Overage runs about $0.085/GB, so a thousand downloads in a month costs on the order of $80. CDN caching does not reduce this: S3-to-CloudFront origin transfer is already free and delivery is charged identically on cache hits.
Spike protection protects us from reaching the limit quickly before we can be alerted via our CloudWatch alarms.