Skip to content

fix: sign use_http_get reads so private S3/MinIO buckets work MAPCO-11391 - #95

Open
razbroc wants to merge 1 commit into
masterfrom
feat/sign-s3-http-get-reads-mapco-11391
Open

fix: sign use_http_get reads so private S3/MinIO buckets work MAPCO-11391#95
razbroc wants to merge 1 commit into
masterfrom
feat/sign-s3-http-get-reads-mapco-11391

Conversation

@razbroc

@razbroc razbroc commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Mirrors the upstream MapProxy fork's signed-read fix (mapproxy/mapproxy#1477) into this repo's vendored config/patch/s3.py, applied at Docker build time when PATCH_FILES=true.

is_cached/load_tile previously issued completely unauthenticated urllib3 HEAD/GET requests when use_http_get is enabled. A private bucket returns 403 to any unsigned request, and the old code treated 403 identically to 404 ("not cached"). Writes remain authenticated via boto3 and always succeed, so tiles get written to a private bucket but could never be read back through this path — any deployment using an S3-compatible cache (e.g. MinIO) with use_http_get: true against a private bucket effectively never served tiles from cache.

Changes

  • is_cached/load_tile now sign their HEAD/GET requests using a boto3-generated presigned URL, built locally from the same session already used for writes (no extra network round-trip).
  • Bucket addressing uses Bucket=self.bucket_name, matching the write path — the {endpoint}/{username}:{bucket}/{key} URL convention is retired from this path.
  • A genuine 403 on a signed request now raises S3ConnectionError instead of being treated as a cache miss; 404 remains a normal cache miss, unchanged.
  • Debug logs print the object key instead of the full request URL, which would otherwise carry the presigned signature.
  • Only the pre-existing, documented config-surface differences (env vars, ImageSource) remain between this file and the upstream fork.

No changes to mapproxy-api/mapproxinator or this repo's Helm chart.

Test plan

  • Docker build with PATCH_FILES=true completes and passes the existing patch-apply/import sanity check.
  • Verified end-to-end against a real MinIO instance (see MAPCO-11392): a tile written to a private bucket is served from cache (unchanged ETag/timestamp) on a subsequent request; a credential with GetObject denied surfaces S3ConnectionError with a clear traceback rather than a silent, permanent cache miss; existing public-bucket + use_http_get behavior is unchanged.

MAPCO-11391

…1391

Mirrors the upstream fork's feat/sign-s3-http-get-reads fix into the
vendored s3.py patch. is_cached/load_tile now sign HTTP-GET reads with
a boto3 presigned URL addressed via Bucket=self.bucket_name (matching
the write path), instead of issuing unsigned requests that a private
bucket rejects with 403 and the old code treated as a cache miss. 403
now raises S3ConnectionError; 404 remains a normal miss. Debug logs
print the object key instead of the full (now signature-bearing) URL.

Only the pre-existing, documented config-surface differences (env
vars, ImageSource) remain between this file and the upstream fork.
Verified the Docker build's patch-apply/import sanity check still
passes with PATCH_FILES=true.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant