cloud-scatterplots-cli-ta fails on hybrid-coordinate models: CMOR check rejects duplicate air_pressure standard_name

[lewisjared]

esmvaltool/cloud-scatterplots-cli-ta fails for roughly three quarters of the models it is solved for, and the split is systematic by model rather than by member or by data gap.

Observed

From a full CMIP6 historical solve (11,170 datasets):

outcome	models
failed (16 executions)	ACCESS-ESM1-5, CMCC-ESM2, CMCC-CM2-SR5, CanESM5, CanESM5-1
succeeded (6 executions)	AWI-ESM-1-1-LR, BCC-ESM1, CESM2-FV2, CESM2-WACCM-FV2

Every member of a failing model fails and every member of a succeeding model succeeds, which points at a model property rather than at missing files.

Error

ValueError: Can not concatenate cubes: failed to concatenate into a single cube.
  Auxiliary coordinates are unequal for phenomenon `mass_fraction_of_cloud_ice_in_air`:
a: _CoordAndDims(coord=<AuxCoord: air_pressure / (Pa)  <lazy>+bounds  shape(38, 145, 192)>, dims=(1, 2, 3))
b: _CoordAndDims(coord=<AuxCoord: air_pressure / (Pa)  <lazy>+bounds  shape(38, 145, 192)>, dims=(1, 2, 3))

The resulting cube in the case I inspected is on atmosphere_hybrid_height_coordinate. The two air_pressure aux coords have identical shape but unequal values, which is expected: on a hybrid vertical coordinate the derived pressure is time-varying, so two time chunks genuinely differ and iris refuses to concatenate them.

The failure propagates out as subprocess.CalledProcessError from the micromamba run, so the execution is marked failed with no metrics.

I have only confirmed the vertical coordinate for one of the failing models, so treat the common mechanism across all five as likely rather than established.

Why it matters

At AFT scale this silently drops a cloud diagnostic for a large share of the ensemble, and the models affected include several widely used ones. Dropping the derived air_pressure aux coord before concatenation, or concatenating on the model levels and deriving pressure afterwards, are the usual remedies.

Seen on climate-ref v0.16.2, climate-ref-esmvaltool 0.16.2.

[lewisjared]

Sharper numbers as the run has worked further through this diagnostic. It is now 51 failed against 6 succeeded, and the successes have not moved at all since the first report, so every new execution is failing.

outcome	models (executions)
failed	CanESM5-1 (42), CAS-ESM2-0 (4), ACCESS-ESM1-5 (1), CMCC-CM2-HR4 (1), CMCC-CM2-SR5 (1), CMCC-ESM2 (1), CanESM5 (1)
succeeded	BCC-ESM1 (3), AWI-ESM-1-1-LR (1), CESM2-FV2 (1), CESM2-WACCM-FV2 (1)

The per-model split still holds exactly: every member of a failing model fails, every member of a succeeding model succeeds, now across 11 models rather than 9.

Worth noting the ensemble-size amplification. CanESM5-1 alone accounts for 42 of the 51 failures because each of its members is solved separately, so a single incompatible model dominates the failure count for the whole diagnostic. On a full AFT run with all members this diagnostic will produce failures in the hundreds while succeeding only for the handful of models with a compatible vertical coordinate.

[lewisjared]

Further data that sharpens this considerably: cloud-scatterplots-cli-ta is not just failing, it is also OOM-killing its workers, and it is the only member of the family doing either.

diagnostic	ok	failed	pending
cloud-scatterplots-cli-ta	25	65	20
cloud-scatterplots-clivi-lwcre	122	5	0
cloud-scatterplots-clt-swcre	127	1	0
cloud-scatterplots-clwvi-pr	120	5	0

Every OOM kill in the deployment traces to this diagnostic. All six esmvaltool pods, 45 restarts between them, and the last Executing line before each kill is esmvaltool/cloud-scatterplots-cli-ta in every case. pmp, ilamb and the orchestrator have zero restarts.

The likely common factor is that cli and ta are the only 4D fields in the family, on model levels. The siblings all work on single-level fields. That would explain both symptoms from one cause: being on model levels is what produces the duplicate air_pressure standard_name that the CMOR check rejects on hybrid-coordinate models, and the full 4D volume is what exhausts memory when the check passes.

There is also a scheduling consequence. An OOM-killed worker never acks, so with a 6 hour CELERY_VISIBILITY_TIMEOUT the task is not redelivered for six hours.

Chart defaults in use for esmvaltool: max_parallel_tasks: 2, Dask local_distributed with 2 workers and memory_limit: 4GiB, Celery concurrency 1.

[k-a-webb]

A fix for CanESM5 and CanESM5-1 was submitted to ESMValCore recently, ESMValGroup/ESMValCore#3165.
I've tested this fix in my install of REF v0.16.2 and it appears to work.

My understanding was that for CMIP6 the CV convention for the ps coordinate was incorrectly specified standard_name to be air_pressure (or at least, several modelling centers made this mistake). ESMValCore cmor "fix" files are used to update the standard_name to surface_air_pressure.