Migrated to obs4REF, ready for esgvoc integration

[dwest77a]

No description provided.

[dwest77a]

@ltroussellier could you take a look at this PR for adding an esgvoc manifest to obs4REF and see if this can be added to esgvoc now?

[dwest77a]

The guidance I have includes adding all the CV values here to the WCRP universe as well. I'm attempting to do this, but have noticed the following:

• Obs4REF nominal resolutions have no space in the value (i.e 5km) versus the value in the WCRP-universe that includes a dash (i.e 5-km). Should this be changed in Obs4REF to match the universe?
• Obs4REF folders (e.g activity_id, variable_id) do not match their universe counter parts (e.g activity, variable) in the naming convention. Is this acceptable for the schema or should this be changed to match the universe?
• Other Obs4REF folders have similar naming differences (grid_label -> grid, references -> reference)
• The values for source_type in Obs4REF do not appear to be similar to the equivalent in the universe - is this the correct set of values for source type or is this a different facet's values?

[dwest77a]

These also apply to the obs4MIPs repo

[ltroussellier]

Hum,

• the _CVs directory is useless for esgvoc, it is meant as a archive (for tracability) .. So i dont mind, you can do whatever on it. But esgvoc will NOT read it (not the good structure)

• the manifest, good try, but the manifest (at least in CMIP7 and other projects) is dealt by CI/CD when change are accepted from esgvoc_dev into main/esgvoc (the prod branch).

Obs4REF nominal resolutions have no space in the value (i.e 5km) versus the value in the WCRP-universe that includes a dash (i.e 5-km). Should this be changed in Obs4REF to match the universe?

this one i have to think more about it

Obs4REF folders (e.g activity_id, variable_id) do not match their universe counter parts (e.g activity, variable) in the naming convention. Is this acceptable for the schema or should this be changed to match the universe?

Other Obs4REF folders have similar naming differences (grid_label -> grid, references -> reference)

The values for source_type in Obs4REF do not appear to be similar to the equivalent in the universe - is this the correct set of values for source type or is this a different facet's values?

The mismatch between Obs4REF folder names (e.g., activity_id, variable_id) and their corresponding Universe counterparts (e.g., activity, variable) is handled through the 000_context.json mapping.
For instance : https://github.com/Climate-REF/Obs4REF_CVs/blob/main/source_type/000_context.jsonld
this context file defines the relationship between an Obs4REF directory and its corresponding Universe directory. For example, in the source_type directory, the 000_context.jsonld file specifies the @base mapping, indicating that this directory corresponds to obs_type in the Universe.
Therefore, the directory names do not need to be identical. This approach allows different projects to use their own naming conventions while still referencing the same concepts in the Universe.

[dwest77a]

Great, thanks for the comments. Some of these will be addressed in a meeting tomorrow, specifically about the nominal resolutions, plus the values for source_type that may be incorrect as well.

[dwest77a]

@ltroussellier Are you able to review this PR again to make sure everything looks OK and we can try to get a first version of Obs4REF schema into ESGF-NG? Thanks

[ltroussellier]

@dwest77a
Thanks for this migration work!

i have just released the latest snapshot for obs4REF CV.
and i have looked at this PR / first by triggering the test action :D https://github.com/Climate-REF/Obs4REF_CVs/actions/runs/31014089263/job/98203487423?pr=7
I tested the build locally with esgvoc admin build (esgvoc 5.0.1) against the esgvoc_dev branch of WCRP-universe, and the build completes but 48 terms fail ingestion — all because the corresponding terms don't exist yet in the WCRP-universe. These need to be added there first (or simultaneously via a coordinated PR).

Here's the breakdown by collection:

conventions (1 missing)

• ods-2.6

grid (2 missing)

• site, site-us-arm-sgp

institution (14 missing)

• cmip-ipo, cru-uea, doess-uci, emoryu, enveo, jma, knmi-wur, met-norway, mohc-cru-ncas, nasa-gmao, nerc-nsf-noaa, nr, ucsb-chc, wur

nominal_resolution (6 missing)

• 0.5-km, 1-km, 2.5-km, 5-km, 1x1degree, site

obs_type (1 missing)

• ai_upscaling

source (24 missing)

• 20cr-v3, ai-upscaling-csoil-1-0, cci-swe-v4-0, ceres-ebaf-4-2-1, chirps-v3, cruts-4-0-9, esacci-prec-l3c-no2, esacci-scfg-modis-v4-0, gpcp-3-3,
  gsmap-v4, gsmap-v5, jasmes-snowcover-1-0, jra-3q, lora-1-0, merra-2, mgohcta-woa09, noaa-ncei-cdr-olr-v02r00, noaa-ncei-ersstv6,
  noaa-ncei-lai-avhrr-5-0, noaa-ncei-lai-viirs-1-0, noaa-ncei-oisst-v2-1, rss-prw-v07r00, wang2024-csoil-v2, woa-23

What needs to happen before this can be merged:

1. All 48 terms above must be added to WCRP-universe (on esgvoc_dev branch) first — each project term must have a matching universe term.
2. The old _CVs/ files are deleted in this PR — that's fine since esgvoc doesn't use them, but make sure nothing else depends on them.
3. A note on nominal_resolution IDs: the term ID (= the filename) must match what exists in the universe — you can't invent new IDs on the project side. However, if obs4REF needs a different display string (e.g. for DRS or global attributes), you can override drs_name in the project term file. For example, if the universe has 10-km but you want the DRS to use 10 km (with a space):

// nominal_resolution/10-km.json (in obs4REF_CVs)

{
    "@context": "000_context.jsonld",
    "id": "10-km",
    "type": "resolution",
    "drs_name": "10 km"
}

3. This keeps the ID aligned with the universe while letting the project control how it appears in filenames/attributes. The terms that genuinely don't exist yet (0.5-km, 1-km, 2.5-km, 5-km, 1x1degree, site) still need to be added to WCRP-universe first.

Once the universe terms are in place, the build should be clean. You can test locally with:

  esgvoc admin build --project-repo dwest77a/Obs4REF_CVs --project-ref esgvoc \
    --universe-repo WCRP-CMIP/WCRP-universe --universe-ref esgvoc_dev \
    --output obs4ref.db --project-id obs4ref --cv-version dev --validate

I presume it is more or less for the new obs4MIP repo (althrough i ll have to update the esgvoc library to add this new "project"

[dwest77a]

@ltroussellier I have the PR open for adding these terms to the universe here: WCRP-CMIP/WCRP-universe#224 if you can take a look that would be great, thanks.