Tile-Ripper downloads a small area and time range from supported scientific raster datasets into Zarr, keeping the original stored values, native grid and masks, and recording the exact source files and versions it read.
Documentation · Quickstart · Supported data · Installation · Apache-2.0
Status: source alpha (0.1.0-alpha.1). There is no packaged release yet;
build from source. The API and request format may still change. Tile-Ripper is
not a general Zarr reader, reprojection engine, mosaic or temporal compositor.
This request asks for Landsat 8/9 surface reflectance (red, near-infrared and
their quality layers) over a ~1 km box in North Carolina on one day
(examples/product-landsat.json):
{
"product": "landsat-c2-l2-sr",
"aoi": {"kind": "bounds", "crs": "EPSG:4326", "bbox": [-78.95,35.95,-78.94,35.96]},
"start": "2024-06-29T00:00:00Z",
"end": "2024-06-29T23:59:59Z",
"bands": ["SR_B4", "SR_B5", "QA_PIXEL", "QA_RADSAT", "SR_QA_AEROSOL"],
"quality_policies": ["landsat_pixel_clear"],
"max_observations": 8,
"buffer_bytes": 33554432,
"chunk_edge": 256
}Build without GDAL, inspect what will be read, then download:
cargo build --release --locked --no-default-features --features native-cog
./target/release/tile-ripper plan examples/product-landsat.json --gdal-free --human
./target/release/tile-ripper fetch examples/product-landsat.json --gdal-free --output ./landsat.zarrplan lists the scenes, bands, native grid windows and reads before any
download. fetch writes a new Zarr store (it never overwrites an existing
path) with stored values, native coordinates and separate masks, and records
the exact source files and versions in its provenance. Landsat is read from
Microsoft Planetary Computer and needs no account. The
product workflow covers discovery, streaming and
authenticated sources.
| Data | What works | Access | Details |
|---|---|---|---|
| Landsat | Collection 2 surface reflectance and QA for Landsat 4/5 TM, 7 ETM+ and 8/9 | No account | Landsat |
| HLS | L30 and S30 v2 reflectance with Fmask | No account; NASA Earthdata optional | HLS |
| OPERA | DSWx-HLS, DSWx-S1, DIST-ALERT (HLS and S1), annual DIST-ANN and RTC-S1; DSWx-NI as a prototype | NASA Earthdata for archive access | OPERA |
| GOES | GOES-19 ABI cloud and moisture imagery: CONUS C01/C13 and mesoscale-1 C02 | No account | GOES |
| SMAP | SPL3SMP_E v6 soil moisture, global and northern polar grids, AM and PM | NASA Earthdata | SMAP |
| NISAR | GCOV frequency-A HHHH backscatter on 10 m and 20 m UTM 16N grids, selected by lat/lon box | NASA Earthdata | NISAR |
| Icechunk GFS | Dynamical GFS analysis, materialized or GRIB-backed, pinned to one repository snapshot | Separate icechunk build |
Icechunk GFS |
Support covers the specific product versions and grids checked in each linked page; a mission name alone does not mean every product from it works. Sentinel-2, SWOT Raster, public Zarr stores (ERA5, IMERG, GFS) and an AlphaEarth embeddings prototype are also available; see all sources and limits and more workflows.
Retrieve native scientific raster subsets with one Rust library and one
plan / fetch / stream CLI. Inspect the selected observations, native grids and physical
reads before downloading data. Outputs include a completion/provenance manifest.
Source alpha (Apache-2.0; no package/binary release yet): support is limited to the profiles below. This is not a general
Zarr reader, reprojection engine, mosaic or temporal compositor. No Python is
required to acquire data. The default build uses GDAL; opt-in builds support
GDAL-free Sentinel-2 polygon acquisition and Zarr acquisition.
Use native CLI selection and public Rust examples for
--backend native and the explicit --gdal-free build guard.
Install Rust/Cargo and system GDAL. On macOS with Homebrew:
brew install gdal pkg-config
cargo build --release --locked --bin tile-ripper
./target/release/tile-ripper --helpFor Sentinel-2 polygon/date/band acquisition without GDAL:
cargo build --release --locked --no-default-features --features native-cog
./target/release/tile-ripper fetch docs/native-sentinel/request.json \
--gdal-free --workers 2 --output ./sentinel.zarrThe complete workflow accepts ordinary WGS84
polygons, discovers Earth Search Sentinel-2 bands, maps the AOI to supported UTM
grids and streams masked UInt16 chunks with pinned provenance. The
scientific output workflow saves actual
values to portable Zarr v2, with separate observation groups for different dates,
bands and grids. Open a group with ordinary Python Zarr/xarray; Rust code is not
needed. The durable-output measurements and safety gate
cover transactional synchronization, lossless encoding candidates and bounded
streaming; ordinary fetch retains uncompressed Zarr v2. Direct COG assets
and native bounds remain supported for the admitted TIFF profiles.
GeoTIFF writing requires the optional gdal feature. Zarr-only builds can omit
both features. C/C++ codec build tools remain required; see
installation. GDAL remains the default build and reader.
Completed COG observation outputs can also feed the checkout's
same-grid stack and direct Chronozarr bridge.
Use stack when you need a persistent scientific cube; the Python playback
bridge joins a selected quality mask to its science scene and feeds Chronozarr's
existing encoder without writing that intermediate cube.
The matched HLS benchmark with sixteen-way acquisition
records full workflow timing, actual request concurrency and independent value,
QA and source-provenance checks.
Run from a checkout of this branch. Installation and platform status cover native dependencies, local installation and release preparation.
This small public ERA5 example retrieves air temperature at two times without credentials. The source chunk is roughly 0.6 MB even though the subset is tiny:
./target/release/tile-ripper plan \
--source https://storage.googleapis.com/weatherbench2/datasets/era5/1959-2023_01_10-6h-64x32_equiangular_conservative.zarr/ \
--format zarr_v2 --bbox 10,30,25,50 \
--start 1959-01-01T00:00:00Z --end 1959-01-01T06:00:00Z \
--variables 2m_temperature --human
./target/release/tile-ripper fetch examples/weatherbench2.json --output ./era5-subset --humanThe second command uses the equivalent complete request file. Choose a new
destination; existing directories are never overwritten. The result is
era5-subset/era5.zarr plus manifest.json. For direct flags, use the same
selection with fetch --output ./new-subset; its asset/group name is subset.zarr.
--buffer-mib 32 is the direct interface's default payload budget, not a process
memory limit. STAC defaults cap discovery at 8 items/8 assets and output at one
million pixels. Use JSON for advanced budgets and catalog limits.
Open the result with familiar tools (optional Python with xarray and Zarr v2):
import xarray as xr
subset = xr.open_zarr("era5-subset/era5.zarr", consolidated=False, chunks=None)
print(subset)
print(subset["2m_temperature"].values) # native Kelvin; NaNs remain invalidThe development branch adds bounded native chunk streaming and shared acquisition scheduling. API, CLI demonstrations and resource limits include cancellation, completion checks and manual concurrency overrides. The native HTTP efficiency decision records current request, byte, latency and resource comparisons for the supported native profile. Controlled benchmark protocol describes the performance gate. Measured results and the interoperability recommendation retain wins, ties and losses; no public-provider speed advantage is promised.
The standalone native statistics consumer uses the public API with provisional chunks and a validated completion receipt. The real Sentinel multi-scene gate compares version-pinned native acquisitions with a tuned GDAL reader.
| Source | Supported selection and output |
|---|---|
| Landsat 8/9 C2 Level-2 SR | Native UInt16 SR, UInt16 pixel/saturation QA and UInt8 aerosol QA; bounded Planetary Computer discovery/runtime signing; product workflow |
| Landsat 4/5 TM and Landsat 7 ETM+ C2 SR | Separate sensor profiles; six SR bands, pixel/saturation/LEDAPS QA and opacity; raw SLC-off gaps retained; Wave 2 evidence |
| OPERA DSWx-S1 V1.0/SAS1.2 | Four UInt8 layers and separate radar water policies; exact current PO.DAAC route; source and output evidence |
| OPERA DIST-ALERT-HLS V1 | Nine native status/confidence/onset/last-date/current-mask layers; separate accumulated detection and observation quality; qualification |
| OPERA DSWx-NI v0.1 engineering prototype | Four public, exact-ETag UInt8 WTR/BWTR/CONF/DIAG COGs; pinned-only and not final NI v1.0 or a production route; bounded qualification |
| OPERA DIST-ALERT-S1 V1.0 | Ten qualified native layers over the exact ASF Cumulus route; independent raw bits, masks, coordinates, stream and reopened output; scope |
| OPERA DIST-ANN-HLS V1 | Twenty-one qualified annual layers on northern T17SNU and southern T20LRP; original source CRS/northings, provider coverage and CF time bounds retained; southern qualification |
| HLS L30 and S30 v2 | Distinct Int16 reflectance and UInt8 Fmask profiles; public GSFC references and separately qualified explicit Earthdata COG access; 30 m UTM groups |
| OPERA DSWx-HLS v1 | Exact UInt8 classified water and UInt16 diagnostics, separate named class masks; public reference and current PO.DAAC v1.1 authenticated access |
| OPERA RTC-S1 v1 | Exact Float32 linear gamma0 VV/VH and UInt8 mask on public references and a current authenticated ASF Cumulus Sentinel-1C burst |
| Earth Search Sentinel-2 C1 L2A | STAC dates, geographic bounds/Polygon, named native UInt16 bands; GDAL-free per-observation Zarr v2 or masked native stream, optional GDAL GeoTIFF output |
| WeatherBench2 ERA5 64×32 product above | Anonymous HTTP Zarr v2, Float32/NaN, (time,longitude,latitude); native Zarr subset |
| NASA POWER IMERG daily spatial | Anonymous HTTP Zarr v2, Float32/NaN, (time,lat,lon), verified daily time and storage filter; native Zarr subset |
| Archived GOES-19 RRQPE and cloud-top-temperature NetCDF-4 samples | Opt-in native-hdf5; exact UInt16/Float32 rank-two array slices, stored one-dimensional axes, conditional compressed-chunk ranges and provisional streaming; profile and decision gate |
| Inspected SWOT Raster Version C 100 m UTM19T scene | Native Float32 /wse with matching compressed Float64 /x,/y; exact index slices qualified against h5py/NetCDF-C on an identity-pinned bounded mirror. The Version C live archive profile is not qualified; product, access and science gate |
| Inspected SWOT Raster Version D PGD0 100 m UTM19T scene | Aligned native WSE, UInt8 summary quality, UInt32 detailed flags/counts and Float32 uncertainty, separate explicit quality policies, exact stored coordinates and bounded streaming; qualified against h5py/NetCDF-C on a pinned mirror and directly from NASA through opt-in authenticated conditional ranges; live gate, authentication workflow and science |
| Held-out SWOT Raster Version D PGD0 100 m UTM01W selection | Same five-layer CLI/reader and Earthdata policy, 1,310 good/suspect accepted cells, both independent references and live source-range resume; bounded selection, resource diagnostics and platform evidence. This qualifies the documented selection and policy. |
| Public GFS v3 archive | Native Float32 subset, ordinary/sharded codecs and conditional selective ranges; exact v3 profile |
| Local custom-grid Zarr / direct GeoTIFF | Narrow UInt16 profiles; advanced lifecycle and local Zarr |
HDF5 v1 index pruning is an explicit experiment: set
TILE_RIPPER_HDF5_INDEX_POLICY=selective. Full traversal remains the default
because pruning cannot validate unread subtrees. Bounded metadata pages are
available independently; see the large-file traversal results.
Public requests: Sentinel-2, ERA5, IMERG. For IMERG, follow the paired precipitation guide. Other stores must fit the exact admitted geographic/codec profile; V3 has a separate admitted profile and measured limitations. This is not generic CF support. Unsupported metadata, dtypes, dimensions and codecs fail explicitly. Zarr supports rectangular selection, not polygon masking. No remote authentication providers, arbitrary calendars, curvilinear grids, target grids, resampling, unit conversion, mosaicking or temporal aggregation.
Bounds are west,south,east,north in geographic x/y order with no antimeridian
crossing. Use the source longitude convention (ERA5 uses 0–360); no automatic
longitude wrapping. Zarr selects native cells whose area intersects the rectangle,
so outer centers may lie outside it. Both interval endpoints are included.
COG Polygon extraction instead masks by pixel center. Scientific coordinates
are preserved; EPSG:4326 on these Zarr profiles is an interpretation of geographic
axes, not a provider-declared precise datum. Original codec decoding is applied
once; outputs do not repeat IMERG's lossy storage filter.
Opt-in Icechunk acquisition adds snapshot-pinned
plan / stream / fetch for two inspected Dynamical GFS analysis profiles:
materialized Float32 shards and virtual Float64 GRIB messages. Native values,
coordinates, source CRS and physical references are retained. Build with
--no-default-features --features icechunk; default features remain unchanged.
The products have different numerical representations and remain explicit
source choices. Measured access patterns and limits
explain when each layout helps and why a small AOI can still read a whole GRIB grid.
The standalone AlphaEarth GeoZarr prototype streams bounded rank-four native Int8 embeddings using selective shard ranges and sampled coordinate validation. It preserves the live source’s original band dimension and positional channel order. Durable embedding output remains outside this prototype.
Bounded SWOT Raster D discovery now finds candidates by WGS84 bounding box and
date through NASA CMR. discover reports catalog relevance and exclusions;
discover-job accepts explicit native windows or WGS84 bounding boxes and
polygons, resolves geographic requests against inspected source CRS and stored
pixel centers, and writes a version-pinned job after admission. Geographic
outputs retain a distinct aoi_mask beside source validity and quality masks.
See the geographic workflow and contract,
runnable AOI selection and
independent scientific gate. The original
native-index workflow remains available.
Geographic job generation now reuses one immutable source-grid inspection per
granule. See the inspection API and checkpoint boundaries
and matched three-product measurements.
The source-validation lifecycle and measurements
document the subsequent single-binding HEAD reduction, source-failure latches,
retained authorization/completion checks, and COG/Zarr request controls.
Explicit one-to-four-observation SWOT Raster D PGD0 jobs now use normal
plan / fetch commands to write one scientific Zarr v2 dataset. Each granule
keeps its native grid, raw variable types, source masks and quality policy.
See the runnable workflow and Rust API and
scientific, recovery, resource and measurement evidence.
Opt-in authenticated source-range resume for the qualified SWOT Raster D
workflow is available through the normal plan/stream CLI and Rust APIs.
See the tested workflow and
the local decision gate. A resumed scientific
stream starts at its beginning; results remain provisional until finish().
The held-out UTM01W qualification
resolves the sparse-reference failure and verifies a nonempty second PGD0
selection, live recovery and Linux ARM64 checkpoint behavior.
./target/release/tile-ripper plan examples/imerg.json --human
./target/release/tile-ripper plan examples/imerg.json --json > inspection.json
./target/release/tile-ripper fetch examples/imerg.json --dry-run --humanplan and --dry-run read bounded metadata/coordinate objects and inspect selected
object identities; they do not GET scientific Zarr chunks or write outputs.
GDAL may read TIFF headers/directories. Planning is not an offline operation.
Human output summarizes variables, dates, grids, windows, reads and estimates.
JSON exposes physical chunk keys/tile selections, output records, transformations,
validation details, limitations and measured planning traffic. Logical output byte
estimates exclude encoding, coordinates and filesystem overhead. Estimated reads
are library operations, not measured HTTP requests for GDAL.
Saved inspections cannot be replayed: fetch accepts the original request and
prepares it again. Metadata and data identities are checked before/after execution;
HTTP Zarr uses conditional ETags per object, not an atomic dataset snapshot.
Diagnostics redact remote locators; runtime request files may contain sensitive
information and should not be published. Anonymous Zarr rejects URL credentials,
query tokens and redirects. Use explicit supported endpoints.
./target/release/tile-ripper fetch examples/imerg.json --output ./precipitation --human
./target/release/tile-ripper fetch examples/unified-stac.json --output ./sentinel --humanJSON is the default stdout format for scripts. --human adds concise stderr
progress and a completion path; the manifest records observations separately from
estimates, source references, native metadata, processing decisions and timings.
Zarr has measured HTTP totals; GDAL HTTP totals may be unavailable. A successful
manifest has status: "complete"; failures return nonzero and remove newly created
outputs. An interrupted process can leave an incomplete directory: inspect the
manifest and choose a new path before retrying. Never treat partial files as success.
Direct STAC syntax uses --source ENDPOINT --format stac --collection NAME,
--variables B04,B08, dates and --bbox … or --aoi polygon.geojson (a GeoJSON
Polygon geometry, not a Feature). Do not mix JSON selection and direct flags;
--output, --human, --cache and --dry-run still work with request files.
The CLI translates flags into the public Request; it calls the same library as
applications. There is no CLI dependency in the planner:
use tile_ripper::{acquisition, cog::LayoutCache, Request, Result};
fn fetch(request: &Request) -> Result<()> {
let prepared = acquisition::prepare(request, &mut LayoutCache::default())?;
println!("{}", prepared.human());
let manifest = prepared.fetch()?;
assert_eq!(manifest["status"], "complete");
Ok(())
}The runnable example accepts any shared request; a typed request example constructs the ERA5 query directly:
cargo run --release --locked --example acquire -- examples/weatherbench2.json --fetchPrepared::execute also delivers owned native Pixels::UInt8, Pixels::UInt16, Pixels::Int16, Pixels::UInt32, or Pixels::Float32
buffers synchronously, with backpressure, coordinates, validity and provenance.
Use .pixels.len(), .as_u16() or .as_f32(); callback consumers must not retain
unbounded buffers. Results are provisional until execution returns success.
The planner models assets, variables, grids, windows and reads; runtime adapters
own codecs and source bindings. cog::LayoutCache is a historical naming/coupling
rough edge; its default works for both formats. API is not yet stable.
Native values, coordinates, NaNs/nodata and metadata are checked independently against GDAL and xarray. Current private-alpha gate records the fresh-context agent trial, installation evidence and remaining alpha blockers. Alpha contract, small scientific examples, and licensing audit define the candidate scope. Previous scientific gate and raw repeated benchmarks preserve methods and limits. There is no demonstrated consistent end-to-end speed advantage. xarray is preferable for broad CF processing and analysis; GDAL/rasterio for general raster operations. See ordinary alternatives and matching-work limits. Tile-Ripper's present advantage is a shared inspectable acquisition lifecycle with bounded native selection and provenance, not a new decoder or universal speedup.
cargo test --locked
cargo fmt --check
cargo clippy --locked --all-targets -- -D warningsSee architecture,
implementation map and
historical study. Detailed milestone reports in docs/ retain
historical scope and results; use this README and the current gate for current support.
No ChronoZarr dependency, plugin system, Python binding or paid service is required.
The current COG efficiency decision gate uses a two-worker default, source-local GDAL reader reuse and bounded parallel layout inspection. The COG efficiency report records measured traffic, matched GDAL comparisons and the tested 192-MiB resource profile. Working-buffer reservations are not a process RSS ceiling. This branch remains unreleased.
The opt-in native-hdf5 adapter additionally retains stored Pixels::UInt8 and
Pixels::UInt32. Integer quality flags are never cast to Float32. Existing
UInt16/Float32 accessors and consumers are preserved; exhaustive matches need
handlers for the two additional variants. HDF5 remains callback/stream output.
For the inspected Version D schema, select the five swot::PATHS datasets in
one ordinary native request, then apply an explicit separate policy:
let prepared = tile_ripper::swot::prepare(
&request,
tile_ripper::swot::Policy::cookbook(), // good (0) or suspect (1)
&mut cache,
)?;
let mut stream = prepared.into_stream()?;
for tile in stream.by_ref() {
let tile = tile?; // five aligned raw layers and separate derived validity
consume(tile)?;
}
let report = stream.finish()?; // final source validation authorizes completionRaw source masks and named required/excluded bitwise conditions are available
without overwriting science values. Preparation checks the source definitions;
there is no universal good-pixel policy. The real PGD0 file, physical layout,
C-versus-D differences, independent science/fault/resource checks, benchmark
are documented in the Version D workflow.
The separate authenticated live gate qualifies
this exact D product directly from NASA with all five layers. Build with
native-hdf5,authenticated-http, construct a caller-owned
authenticated_http::AuthenticatedHttp::earthdata session from the canonical
URL and a private netrc file path, then pass it to
swot::prepare_authenticated. Usage and scope.
Ordinary prepare remains anonymous; credentials and signed URLs never enter
request JSON or provenance.
- ALPHA.md: candidate contract, private alpha installation and first scientist trial
- Consolidation checklist: local checks and remaining owner decisions
- Product coverage summary, scientific band and quality definitions and the coverage roadmap
- Provider routing, signing and access limits