This repository demonstrates ingestion and rudimentary analysis of RTMA (Real-Time Mesoscale Analysis) weather data from NOAA using Arraylake from Earthmover.
- Source: NOAA RTMA via AWS Open Data
- Product: rtma2p5 (2.5km resolution grid) - others are available but not required at the moment
- Temporal coverage: Hourly data
- Variables: Wind data at 10m (u and v components, direction, speed, and gustiness) - more variables are available
-
Missing Days: Some days have incomplete data (fewer than 24 hourly files). The ingestion leaves gaps in the time series.
-
Changing Domain: Different file extensions indicate different spatial domains:
.grb2- base domain.grb2_ext- extended to the north.grb2_wexp- extended towards the west
Smaller domains are padded to the largest common grid size during ingestion. Note that the changing domains make virtualizing this data (to save on storage costs) complicated.
The AWS Open Data (source GRIB files) is located in us-east-1, and the ingested data bucket is also stored in us-east-1.
- Install Pixi
- Run
pixi install - Run
pixi shellto activate the environment - Authenticate with Arraylake:
arraylake auth login
We use coiled to run these ingestion/analysis examples.
pixi run coiled-notebookThis starts a Jupyter notebook on Coiled with 32GB memory in us-east-1 region.
You can explore the data repository in the app, use Flux to e.g. generate Web Map Tiles or load it via the python client
import arraylake as al
import xarray as xr
client = al.Client()
#client.login() # Only needed for private datasets
repo = client.get_repo('earthmover/rtma')
session = repo.readonly_session('main')
ds = xr.open_zarr(session.store, group='test-ingestion-cooperative-write')