From 3b14b7776545ea2f9060bad472b0daaf602aedb5 Mon Sep 17 00:00:00 2001 From: Jared Lewis Date: Thu, 13 Aug 2026 00:25:01 +0000 Subject: [PATCH 1/2] chore(datasets): drop 17 unreachable obs4REF reference files No data requirement can select these files. Four are superseded by a newer version of the same dataset that a diagnostic does request: LORA-1-1, the pre-2023 RAPID record, and the two unversioned Hoffman files. The other thirteen are variables of otherwise-used sources that no requirement names: WECANN-1-0 hfls/hfss, TropFlux-1-0 hfns/tas/tauv, Hoffman-1-0 fgco2, and WOA-23 no3/o2/po4/si/ohc/ohcJm2. Reachability was checked against every reference-typed DataRequirement in the esmvaltool, pmp and ilamb providers, matching on the facets the registry key carries, plus the sources ILAMB resolves by literal registry key from configure/ilamb.yaml. GPCP-2-3 pr and FLUXNET2015-1-0 gpp are only reachable that second way and are kept. Verified with two isolated configs holding the same CMIP6 (CanESM5 + ACCESS-ESM1-5, 1855 datasets) and pmp-climatology catalogs, differing only in whether the 17 files are present: `ref solve --dry-run` plans 2819 executions in both, with identical per-diagnostic counts and an identical set of diagnostics that plan nothing. Constraint: obs4REF entries can also be referenced by literal registry key from ilamb.yaml, which requirement matching alone does not see Rejected: Also drop the ESACCI-CLOUD, CALIPSO-ICECLOUD and SAGE-CCI-OMPS entries | unreachable today, but they look staged for migrating the ESMValTool cloud/ozone diagnostics off the esmvaltool-datasets registry Rejected: Prune the same 8 files from sample_data.txt | that manifest tracks an upstream ref-sample-data tag, so it needs a release there first Confidence: high Scope-risk: narrow Directive: Before removing an obs4REF entry, check both the providers' data requirements AND literal `obs4REF/...` keys in provider configure files Not-tested: Diagnostics needing model toz/o3, burntFractionAll, or experiments outside the CMIP activity plan nothing with these two models, so their reference use is unexercised --- changelog/863.trivial.md | 7 +++++ .../dataset_registry/obs4ref_reference.txt | 26 ------------------- 2 files changed, 7 insertions(+), 26 deletions(-) create mode 100644 changelog/863.trivial.md diff --git a/changelog/863.trivial.md b/changelog/863.trivial.md new file mode 100644 index 000000000..4c967ba98 --- /dev/null +++ b/changelog/863.trivial.md @@ -0,0 +1,7 @@ +Dropped 17 obs4REF reference files that no diagnostic can select: +four superseded by a newer version of the same dataset +(LORA-1-1, the pre-2023 RAPID record and the two unversioned Hoffman files), +and thirteen variables that no data requirement asks for +(WECANN-1-0 `hfls`/`hfss`, TropFlux-1-0 `hfns`/`tas`/`tauv`, Hoffman-1-0 `fgco2`, +and WOA-23 `no3`/`o2`/`po4`/`si`/`ohc`/`ohcJm2`). +A solve over the same catalog plans exactly the same executions as before the removal. diff --git a/packages/climate-ref/src/climate_ref/dataset_registry/obs4ref_reference.txt b/packages/climate-ref/src/climate_ref/dataset_registry/obs4ref_reference.txt index ba11f3964..d600e0fa9 100644 --- a/packages/climate-ref/src/climate_ref/dataset_registry/obs4ref_reference.txt +++ b/packages/climate-ref/src/climate_ref/dataset_registry/obs4ref_reference.txt @@ -1,13 +1,8 @@ # source: https://dap.ceda.ac.uk/neodc/obs_for_ref_v2/data/ARCCSS/LORA-1-0/mon/mrro/50km/v20250902/mrro_mon_LORA-1-0_REF_gn_198001-201212.nc obs4REF/ARCCSS/LORA-1-0/mon/mrro/gn/v20250902/mrro_mon_LORA-1-0_REF_gn_198001-201212.nc md5:c1d290a26b4c0a094724b3dc19230f73 -obs4REF/ARCCSS/LORA-1-1/mon/mrro/gn/20250516/mrro_mon_LORA-1-1_REF_gn_198001-201212.nc md5:4cfbbfa3be9632b14de99b18066fbffe obs4REF/CNES/AVISO-1-0/mon/zos/gn/v20210727/zos_mon_AVISO-1-0_PCMDI_gn_199301-201912.nc md5:91252303cb65548fee5ff42dd3024825 # source: https://dap.ceda.ac.uk/neodc/obs_for_ref_v2/data/ColumbiaU/WECANN-1-0/mon/gpp/100km/v20250902/gpp_mon_WECANN-1-0_REF_gn_200701-201512.nc obs4REF/ColumbiaU/WECANN-1-0/mon/gpp/gn/v20250902/gpp_mon_WECANN-1-0_REF_gn_200701-201512.nc md5:eb4fcde64bcdf605b4e72c95e9bbdd54 -# source: https://dap.ceda.ac.uk/neodc/obs_for_ref_v2/data/ColumbiaU/WECANN-1-0/mon/hfls/100km/v20250902/hfls_mon_WECANN-1-0_REF_gn_200701-201512.nc -obs4REF/ColumbiaU/WECANN-1-0/mon/hfls/gn/v20250902/hfls_mon_WECANN-1-0_REF_gn_200701-201512.nc md5:dd3bd01233eb17cb6cb566c9eed7349f -# source: https://dap.ceda.ac.uk/neodc/obs_for_ref_v2/data/ColumbiaU/WECANN-1-0/mon/hfss/100km/v20250902/hfss_mon_WECANN-1-0_REF_gn_200701-201512.nc -obs4REF/ColumbiaU/WECANN-1-0/mon/hfss/gn/v20250902/hfss_mon_WECANN-1-0_REF_gn_200701-201512.nc md5:1605e0d962c3794206e979e51c04f2f9 # source: https://dap.ceda.ac.uk/neodc/obs_for_ref_v2/data/DWD/ESACCI-CLOUD-AVHRR-AMPM-3-0/mon/clivi/50km/v20250723/clivi_mon_ESACCI-CLOUD-AVHRR-AMPM-3-0_ESMValTool-REF_gn_198201-201612.nc obs4REF/DWD/ESACCI-CLOUD-AVHRR-AMPM-3-0/mon/clivi/gn/v20250723/clivi_mon_ESACCI-CLOUD-AVHRR-AMPM-3-0_ESMValTool-REF_gn_198201-201612.nc md5:1eaf07b5d5c7d70c2c40ae9eea4bfd4a # source: https://dap.ceda.ac.uk/neodc/obs_for_ref_v2/data/DWD/ESACCI-CLOUD-AVHRR-AMPM-3-0/mon/clt/50km/v20250723/clt_mon_ESACCI-CLOUD-AVHRR-AMPM-3-0_ESMValTool-REF_gn_198201-201612.nc @@ -36,14 +31,10 @@ obs4REF/DWD/ESACCI-CLOUD-AVHRR-AMPM-3-0/mon/rsutcs/gn/v20250723/rsutcs_mon_ESACC obs4REF/DoESS-UCI/GFED-5-0/mon/burntFractionAll/gr/v20260128/burntFractionAll_mon_GFED-5-0_REF_gr_199701-202012.nc md5:7cddfd3b1d5e50e22d2f2621c9b0393e obs4REF/ESSO/TropFlux-1-0/mon/hfls/gn/v20210727/hfls_mon_TropFlux-1-0_PCMDI_gn_197901-201707.nc md5:2f05191d6727068e1500d8d4ed90098a obs4REF/ESSO/TropFlux-1-0/mon/hfls/gn/v20250415/hfls_mon_TropFlux-1-0_PCMDI_gn_197901-201707.nc md5:e607167a08a2521b65e55eb186182003 -obs4REF/ESSO/TropFlux-1-0/mon/hfns/gn/v20210727/hfns_mon_TropFlux-1-0_PCMDI_gn_197901-201707.nc md5:7a9019e51a41d9e4ab1fcfb072d8ca8d obs4REF/ESSO/TropFlux-1-0/mon/hfss/gn/v20210727/hfss_mon_TropFlux-1-0_PCMDI_gn_197901-201707.nc md5:1da9d8fe862c61bc49c36c18b6527213 obs4REF/ESSO/TropFlux-1-0/mon/hfss/gn/v20250415/hfss_mon_TropFlux-1-0_PCMDI_gn_197901-201707.nc md5:6f766ca0332a8e566c408d237571a924 -obs4REF/ESSO/TropFlux-1-0/mon/tas/gn/v20210727/tas_mon_TropFlux-1-0_PCMDI_gn_197901-201707.nc md5:a6057931b5f6bc000a44514a1a8c891f -obs4REF/ESSO/TropFlux-1-0/mon/tas/gn/v20250415/tas_mon_TropFlux-1-0_PCMDI_gn_197901-201707.nc md5:527ab1b9becf2a793df558532eccfe69 obs4REF/ESSO/TropFlux-1-0/mon/tauu/gn/v20210727/tauu_mon_TropFlux-1-0_PCMDI_gn_197901-201707.nc md5:7c73a3deed3403fa9d21caef3a4d988d obs4REF/ESSO/TropFlux-1-0/mon/tauu/gn/v20250415/tauu_mon_TropFlux-1-0_PCMDI_gn_197901-201707.nc md5:0822e2002e61472277116d38e5e19498 -obs4REF/ESSO/TropFlux-1-0/mon/tauv/gn/v20210727/tauv_mon_TropFlux-1-0_PCMDI_gn_197901-201707.nc md5:8abc7a724a7a297826e2f783a4ea14f9 obs4REF/ESSO/TropFlux-1-0/mon/ts/gn/v20210727/ts_mon_TropFlux-1-0_PCMDI_gn_197901-201707.nc md5:8697d3d7862f6e3b72bb5a161aa75ee8 # source: https://dap.ceda.ac.uk/neodc/obs_for_ref_v2/data/FMI/SAGE-CCI-OMPS-v00008/mon/o3/1000km/v20250718/o3_mon_SAGE-CCI-OMPS-v0008_SAGE-CCI-OMPS_REF_gnz_198410-202212.nc obs4REF/FMI/SAGE-CCI-OMPS-v00008/mon/o3/gnz/v20250718/o3_mon_SAGE-CCI-OMPS-v0008_SAGE-CCI-OMPS_REF_gnz_198410-202212.nc md5:e8f320054b6fc9a92a6e6ed94bb54563 @@ -65,30 +56,13 @@ obs4REF/NOAA-ESRL-PSD/20CR/mon/ts/gn/v20210727/ts_mon_20CR_PCMDI_gn_187101-20121 obs4REF/NOAA-NCEI/CMAP-V1902/mon/pr/gn/v20210727/pr_mon_CMAP-V1902_PCMDI_gn_197901-201901.nc md5:9d943d2dd0645850b616820f246aedf3 obs4REF/NOAA-NCEI/GPCP-2-3/mon/pr/gn/v20210727/pr_mon_GPCP-2-3_PCMDI_gn_197901-201907.nc md5:0877f014868b83547448f96c3e7c83e9 obs4REF/NOAA-NCEI/GPCP-2-3/mon/pr/gn/v20231205/pr_mon_GPCP-Monthly-3-2_RSS_gn_198301-202303.nc md5:6970c22443e2097c45de5db8947318eb -# source: https://dap.ceda.ac.uk/neodc/obs_for_ref_v2/data/NOAA-NCEO-OCL/WOA-23/monC/no3/1x1degree/v20250902/no3_monC_WOA-23_REF_gn_200501-202212.nc -obs4REF/NOAA-NCEO-OCL/WOA-23/monC/no3/gn/v20250902/no3_monC_WOA-23_REF_gn_200501-202212.nc md5:fd8c3af50d5dc6fedec730d38879d2b4 -# source: https://dap.ceda.ac.uk/neodc/obs_for_ref_v2/data/NOAA-NCEO-OCL/WOA-23/monC/o2/1x1degree/v20250902/o2_monC_WOA-23_REF_gn_200501-202212.nc -obs4REF/NOAA-NCEO-OCL/WOA-23/monC/o2/gn/v20250902/o2_monC_WOA-23_REF_gn_200501-202212.nc md5:8bc541d7fe602b156054863867ad6cc7 -# source: https://dap.ceda.ac.uk/neodc/obs_for_ref_v2/data/NOAA-NCEO-OCL/WOA-23/monC/po4/1x1degree/v20250902/po4_monC_WOA-23_REF_gn_200501-202212.nc -obs4REF/NOAA-NCEO-OCL/WOA-23/monC/po4/gn/v20250902/po4_monC_WOA-23_REF_gn_200501-202212.nc md5:abc02f6a8606f276b27fe1ceb2557fe4 -# source: https://dap.ceda.ac.uk/neodc/obs_for_ref_v2/data/NOAA-NCEO-OCL/WOA-23/monC/si/1x1degree/v20250902/si_monC_WOA-23_REF_gn_200501-202212.nc -obs4REF/NOAA-NCEO-OCL/WOA-23/monC/si/gn/v20250902/si_monC_WOA-23_REF_gn_200501-202212.nc md5:8a42fb90cc49bfa83aa285d9d0edf31b # source: https://dap.ceda.ac.uk/neodc/obs_for_ref_v2/data/NOAA-NCEO-OCL/WOA-23/monC/so/1x1degree/v20251024/so_monC_WOA-23_REF_gn_200501-202212.nc obs4REF/NOAA-NCEO-OCL/WOA-23/monC/so/gn/v20251024/so_monC_WOA-23_REF_gn_200501-202212.nc md5:28a6049a9ef0435bce0a9e13c83fd77c # source: https://dap.ceda.ac.uk/neodc/obs_for_ref_v2/data/NOAA-NCEO-OCL/WOA-23/monC/thetao/1x1degree/v20251024/thetao_monC_WOA-23_REF_gn_200501-202212.nc obs4REF/NOAA-NCEO-OCL/WOA-23/monC/thetao/gn/v20251024/thetao_monC_WOA-23_REF_gn_200501-202212.nc md5:a76187904de6d66281af2267d3dddfa3 -# source: https://dap.ceda.ac.uk/neodc/obs_for_ref_v2/data/NOAA-NCEO-OCL/WOA-23/yr/ohc/site/v20250903/ohc_yr_WOA-23_REF_gm_200501-202212.nc -obs4REF/NOAA-NCEO-OCL/WOA-23/yr/ohc/gm/v20250903/ohc_yr_WOA-23_REF_gm_200501-202212.nc md5:98f20f959d9d297d50ac81a4e697b1fb -# source: https://dap.ceda.ac.uk/neodc/obs_for_ref_v2/data/NOAA-NCEO-OCL/WOA-23/yr/ohcJm2/1x1degree/v20250903/ohcjm2_yr_WOA-23_REF_gn_200501-202212.nc -obs4REF/NOAA-NCEO-OCL/WOA-23/yr/ohcJm2/gn/v20250903/ohcjm2_yr_WOA-23_REF_gn_200501-202212.nc md5:1e4e7a86227d98d15baad7a06130cb7e # source: https://dap.ceda.ac.uk/neodc/obs_for_ref_v2/data/NOC/RAPID-2023-1a/mon/msftmz/site/v20250902/msftmz_mon_RAPID-2023-1a_REF_gm_200404-202302.nc obs4REF/NOC/RAPID-2023-1a/mon/msftmz/gm/v20250902/msftmz_mon_RAPID-2023-1a_REF_gm_200404-202302.nc md5:f5ff38176be71f717e3c4d7460938571 -obs4REF/NOC/RAPID/mon/msftmz/NA/20250516/msftmz_mon_RAPID_REF_NA_200404-202302.nc md5:e9ee555d923d39112dcbf591440000b3 # source: https://dap.ceda.ac.uk/neodc/obs_for_ref_v2/data/NR/CCI-CryoClim-FSC-1/mon/snc/0.5x0.5degree/v20250519/snc_mon_CCI-CryoClim-FSC-1_BE_gn_198201-201906.nc obs4REF/NR/CCI-CryoClim-FSC-1/mon/snc/gn/v20250519/snc_mon_CCI-CryoClim-FSC-1_BE_gn_198201-201906.nc md5:d53cddd61a90776f353ee8627dc992a5 -# source: https://dap.ceda.ac.uk/neodc/obs_for_ref_v2/data/UCI-ORNL/Hoffman-1-0/yr/fgco2/site/v20251117/fgco2_yr_Hoffman-1-0_REF_gm_1850-2010.nc -obs4REF/UCI-ORNL/Hoffman-1-0/yr/fgco2/gm/v20251117/fgco2_yr_Hoffman-1-0_REF_gm_1850-2010.nc md5:1808ca5fd11be62934ba062df63d1be1 # source: https://dap.ceda.ac.uk/neodc/obs_for_ref_v2/data/UCI-ORNL/Hoffman-1-0/yr/nbp/site/v20251117/nbp_yr_Hoffman-1-0_REF_gm_1850-2010.nc obs4REF/UCI-ORNL/Hoffman-1-0/yr/nbp/gm/v20251117/nbp_yr_Hoffman-1-0_REF_gm_1850-2010.nc md5:ea86edee4cb869699cdb9b15ed957d2b -obs4REF/UCI-ORNL/Hoffman/yr/fgco2/gm/20250516/fgco2_yr_Hoffman_REF_gm_185007-201007.nc md5:93419dd752e0ae20e69b7db3f8ea9f5b -obs4REF/UCI-ORNL/Hoffman/yr/nbp/gm/20250516/nbp_yr_Hoffman_REF_gm_185007-201007.nc md5:1c0b93c81e2608c668ec39c45ca828de From 85d4a4f63e29bab5088eb63f93f9dbbfe3e856c6 Mon Sep 17 00:00:00 2001 From: Jared Lewis Date: Thu, 13 Aug 2026 00:25:01 +0000 Subject: [PATCH 2/2] docs: correct the claim that obs4REF datasets participate in solving The 0.16.2 changelog entry for #814 says obs4REF datasets participate in solving. They do not: no diagnostic declares an obs4REF data requirement, and the solver matches a requirement against its own source type only, so a collection ingested as obs4ref is never selected. Ingesting the obs4REF collection that way silently drops eight ILAMB diagnostics to zero executions. Leave the released entry as shipped and correct the record in the current changelog, and say so at the point of use in the ingest guide, where the obs4mips source type is prescribed without explaining why the obs4ref one is not used. Rejected: Edit the 0.16.2 entry in place | rewriting a released changelog section hides that the behaviour was mis-described Rejected: Make obs4ref ingest warn or fail | the source type is a deliberate step toward obs4REF solving, so the fix belongs with that work, not with a doc correction Confidence: high Scope-risk: narrow --- changelog/863.docs.md | 6 ++++++ docs/getting-started/03-ingest.md | 7 +++++++ 2 files changed, 13 insertions(+) create mode 100644 changelog/863.docs.md diff --git a/changelog/863.docs.md b/changelog/863.docs.md new file mode 100644 index 000000000..7ce800f89 --- /dev/null +++ b/changelog/863.docs.md @@ -0,0 +1,6 @@ +Corrected the record on the `obs4ref` source type. +The 0.16.2 entry for [#814](https://github.com/Climate-REF/climate-ref/pull/814) stated that obs4REF datasets participate in solving. +They do not yet: no diagnostic declares an `obs4REF` data requirement, +and the solver matches a requirement against its own source type only, +so a collection ingested as `obs4ref` is never selected. +Ingest the obs4REF collection with `--source-type obs4mips`, as the ingest guide describes. diff --git a/docs/getting-started/03-ingest.md b/docs/getting-started/03-ingest.md index 74520a90e..1b412ec4f 100644 --- a/docs/getting-started/03-ingest.md +++ b/docs/getting-started/03-ingest.md @@ -18,6 +18,13 @@ ref datasets ingest --source-type obs4mips $REF_CONFIGURATION/datasets/obs4ref Replace `$REF_CONFIGURATION/datasets/obs4ref` with the directory used when [fetched the obs4REF data](02-download-datasets.md#fetch-obs4ref-datasets). +!!! note "Why not the `obs4ref` source type?" + + An `obs4ref` source type also exists, but ingesting this collection with it will leave the data unused: + no diagnostic declares an `obs4REF` data requirement yet, + and the solver matches a requirement against its own source type only. + Use `obs4mips` until that changes. + ## 2. Ingest CMIP6 data To ingest CMIP6 files, point the CLI at a directory of netCDF files and set `cmip6` as the source type: