diff --git a/data-index/README.adoc b/data-index/README.adoc index c2d1029..43790df 100644 --- a/data-index/README.adoc +++ b/data-index/README.adoc @@ -13,7 +13,8 @@ directory's `configs.yml` for the page's `title`, `favicon` and `description`. This directory once also generated a per-repo Jekyll `_config.yml` (`bin/gen-data-index-config` into `generated/`). support#58 (relaton/relaton#83) replaced the Jekyll build with `relaton index`, and a sweep -of all 30 repos found that none had ever carried a committed `_config.yml`, so +of every `relaton-data-*` repo found that none had ever carried a committed +`_config.yml`, so the generator, its snapshot and the `paginate` / `pubid_class` / `pubid_require` keys that fed it were removed. Git history holds them if they are ever wanted. ==== @@ -152,7 +153,7 @@ for r in $(gh repo list relaton --limit 200 --json name -q '.[].name' | grep '^r done ---- -PR-time validation is `check-index.yml`, mapped into all 30 repos: a PR touching +PR-time validation is `check-index.yml`, mapped into all 31 repos: a PR touching `data/**` builds the whole index and fails if it does not parse. `check_data.yml` keeps its own `push`/`pull_request` triggers on top of that in 3 repos (bipm, iho, iso) as a cheaper schema check. @@ -171,7 +172,7 @@ a run carrying fresh data. [NOTE] ==== -Cimas replaces `deploy.yml` and `check-index.yml` wholesale in all 30 mapped +Cimas replaces `deploy.yml` and `check-index.yml` wholesale in all 31 mapped repos, so nothing per-repo can live in a caller's `with:` block and survive. That is also why a run's *kind* is derived centrally from its ref rather than from a `validate-only` input: the input would have to arrive in exactly that block. @@ -308,10 +309,16 @@ Small flavors (bipm, iana) finish in under two minutes. == Status (verified 2026-08-03) -`configs.yml` covers *30 repos*: the 26 rollout targets plus the 3 already-live +`configs.yml` covers *31 repos*: the 26 rollout targets plus the 3 already-live (*ids, oasis, w3c*), folded in so this file is the single source for every site, -plus *iala* (added 2026-08-17; `main`-default, `index-v2.yaml`). The 3 live ones -carry per-repo `favicon`/`description` overrides. +plus *iala* (added 2026-08-17; `main`-default, `index-v2.yaml`) and *ietf* (added +2026-08-24). The 3 live ones carry per-repo `favicon`/`description` overrides. + +`ietf` closes the last gap between this file and `cimas.yml`: Cimas had always +synced `deploy.yml` into `relaton-data-ietf`, but the repo published no index, so +it carried no row and its site fell back to a bare `IETF Index`. It now crawls +with `Relaton::Ietf::DataFetcher`, which commits `index-v1.yaml` (177,230 +records) on `main`. Every repo Cimas syncs `deploy.yml` into now has a row. [WARNING] ==== @@ -357,7 +364,7 @@ Re-check current state any time with: [source,sh] ---- -bin/check-data-pages # site + raw-index for all 30 +bin/check-data-pages # site + raw-index for all 31 bin/check-data-pages --site-only # just the Pages sites (the 404s to fix) ---- @@ -387,7 +394,7 @@ cimas open-prs -f $CIMAS_CONFIG -r $CIMAS_WD -b data-pages -m 'Pages deploy' -g ---- + NOTE: `check-index.yml` triggers on `pull_request`, so `open-prs` starts a -`Check index` run in every repo it opens a PR in — ~30 full builds at once, +`Check index` run in every repo it opens a PR in — ~31 full builds at once, 2–3 min each on the largest flavors. That is a free fleet-wide smoke test of the build, not a loop. Inspect one repo's synced tree in `$CIMAS_WD` before pushing all of them: for a target that has never had this file, the sync is an *add*. diff --git a/data-index/configs.yml b/data-index/configs.yml index be1d9ae..9a2587e 100644 --- a/data-index/configs.yml +++ b/data-index/configs.yml @@ -14,7 +14,7 @@ # (`baseurl` + `source`) both return 200. # # Branding lives here, and not in each caller's `with:` block, because -# cimas.yml maps `.github/workflows/deploy.yml` as a whole-file copy for 30 +# cimas.yml maps `.github/workflows/deploy.yml` as a whole-file copy for 31 # repos: a `with:` block is wiped on the next `cimas sync` and the site silently # loses its favicon and description. # @@ -99,6 +99,30 @@ repos: description: >- Combined bibliographic index of ITU-R and ITU-T publications in Relaton format. + # Cimas has always synced deploy.yml into relaton-data-ietf. Until that repo + # migrated to `Relaton::Ietf::DataFetcher` it published no index, so it + # deliberately had no row here and its site fell back to a bare "IETF Index" + # with no favicon and no description. The crawl now commits index-v1.yaml + # (177,230 records) on main, so the site has a real index and this row gives + # it an identity. + # + # `source` FOLLOWS the publish here, unlike the iana row above. relaton-ietf's + # DataFetcher is moving index-v1 -> index-v2 as a straight replacement, never + # both; lib/relaton_ietf/data_fetcher.rb still declares index-v1. Flip this + # key only after a crawl has committed index-v2.yaml, or bin/check-data-pages + # reports a 404. + # + # The rfcs/rfcsubseries/ids rows below stay: every released relaton reads + # index-v1 from those three repos. Retire them the staged way the itu-r -> itu + # comment above describes, once relaton's Ietf::Scraper reads this index. + - repo: ietf + display: IETF + source: index-v1.yaml + branch: main + favicon: 'https://www.ietf.org/static/img/ietf-logo.dc9646b3c716.svg' + description: >- + Bibliographic data for IETF RFCs, Internet-Drafts and RFC sub-series in + Relaton format - { repo: jcgm, display: JCGM, source: index-v1.yaml, branch: main } - { repo: jis, display: JIS, source: index-v2.yaml, branch: v2 } - { repo: nist, display: NIST, source: index-v2.yaml, branch: v2 } @@ -110,7 +134,7 @@ repos: - { repo: xsf, display: XSF, source: index-v1.yaml, branch: v2 } # Already-live repos (their sites returned 200 before this file existed). - # Folded in so this file covers all 30; they carry per-repo favicon/ + # Folded in so this file covers all 31; they carry per-repo favicon/ # description overrides. branch: v2 is # each repo's real default (index-v1.yaml is served from v2). - repo: ids diff --git a/lib/data_index_config.rb b/lib/data_index_config.rb index 10145c5..8a49bb2 100644 --- a/lib/data_index_config.rb +++ b/lib/data_index_config.rb @@ -53,16 +53,19 @@ def entry(repo) # The branding `relaton index` renders into the Pages site — title, favicon and # `` — resolved centrally rather than passed by each # caller. cimas.yml maps `.github/workflows/deploy.yml` as a whole-file copy for - # 30 repos, so a `with:` block carrying these values is wiped on the next + # 31 repos, so a `with:` block carrying these values is wiped on the next # `cimas sync` and the site silently loses them. # # Precedence: an explicit non-blank argument (a caller's workflow input) beats # this repo's configs.yml entry, which beats the shared default. # - # Deliberately never raises, unlike #entry: Cimas syncs deploy.yml into - # relaton-data-ietf, which publishes no document index and so has no configs.yml - # row. An unknown repo falls back to what the workflow's own shell derivation - # produced before this method existed — " Index" and no branding. + # Deliberately never raises, unlike #entry. Every repo Cimas syncs deploy.yml + # into now carries a configs.yml row, so nothing exercises the fallback today — + # it is a safety net, not a live path. It stays because a repo added to + # cimas.yml before its row lands would otherwise fail its own Pages build on a + # missing row. Such a repo falls back to what the workflow's own shell + # derivation produced before this method existed — " Index" and no + # branding. spec/cimas_data_pages_spec.rb is what makes the gap loud. # # => { "title" => String, "favicon" => String, "description" => String } def branding(repo, title: nil, favicon: nil, description: nil) @@ -122,7 +125,8 @@ def entry_description(entry) end # What the workflow's retired shell step produced for a repo configs.yml does - # not cover: the slug, upcased. Keeps relaton-data-ietf building unchanged. + # not cover: the slug, upcased. Keeps such a repo building unchanged rather + # than failing its deploy on a missing row. def derived_title(repo) "#{self.class.flavor(repo).upcase} Index" end diff --git a/spec/cimas_data_pages_spec.rb b/spec/cimas_data_pages_spec.rb index fd18eca..507f032 100644 --- a/spec/cimas_data_pages_spec.rb +++ b/spec/cimas_data_pages_spec.rb @@ -64,7 +64,7 @@ # The inverse of the guard this example replaces. `relaton index` needs # no Gemfile: `source: gem` installs relaton-cli, `source: git` writes # $RUNNER_TEMP/Gemfile.index and points BUNDLE_GEMFILE at it. A restored - # mapping would re-create the retired Jekyll bundle in 30 repos and + # mapping would re-create the retired Jekyll bundle in 31 repos and # re-establish the "this repo builds with Jekyll" signal the migration # removes — harmless to the build, which is exactly why it would stick. expect(repositories.fetch(cimas_key).fetch("files")).not_to have_key("Gemfile.deploy") @@ -74,13 +74,14 @@ it "every relaton-data-* repo in the `data` group with a deploy.yml is covered by configs.yml" do # The reverse direction: a data repo that Cimas pushes deploy.yml into but that - # has no configs.yml row would deploy the theme with no per-repo _config.yml. - # Excludes are the deliberate non-index repos (see data-pages-rollout hand-off). + # has no configs.yml row deploys a site with no title, favicon or description. known = configs.repos.map { |e| "relaton-data-#{e['repo']}" } - # ietf gets deploy.yml but is deliberately not a Pages index (no document - # index published); every other data-group repo is covered by configs.yml, - # including the already-live ids/oasis/w3c now folded in. - excluded = %w[relaton-data-ietf] + # Empty, and that is the point: configs.yml now covers every data-group repo + # that gets deploy.yml. relaton-data-ietf was the last exclusion — it had no + # document index until it migrated to Relaton::Ietf::DataFetcher, which + # commits index-v1.yaml as it crawls. The list stays so a future deliberate + # non-index repo has somewhere to go, and so removing one is a visible edit. + excluded = %w[] deploys_pages = data_group.select do |name| files = repositories.fetch(name, {}).fetch("files", nil) || {} @@ -93,11 +94,11 @@ end it "syncs deploy.yml and check-index.yml together, or neither" do - # Not folded into the per-repo examples above: those iterate configs.yml, - # which deliberately has no relaton-data-ietf row (it gets deploy.yml but - # publishes no document index). A repo that merges data with no PR-time - # build is exactly the gap this file exists to make loud, whether or not it - # has a page — so drive this one off cimas.yml instead. + # Not folded into the per-repo examples above: those iterate configs.yml, so + # they are blind to a repo cimas.yml already syncs but configs.yml has not + # caught up with. A repo that merges data with no PR-time build is exactly + # the gap this file exists to make loud, whether or not it has a row yet — + # so drive this one off cimas.yml instead. # # Both directions, because both halves fail silently and the per-repo # examples above can only see repos configs.yml already knows about. A repo @@ -120,7 +121,7 @@ end # The failure mode that made this whole change necessary is entirely silent: a - # per-repo value in a Cimas-synced template is copied verbatim into 30 repos, + # per-repo value in a Cimas-synced template is copied verbatim into 31 repos, # and a per-repo value hand-added to a *synced destination* is reverted on the # next sync with nothing red in CI. These two guards make either a test # failure, for every template cimas.yml syncs — not just deploy.yml. diff --git a/spec/data_deploy_caller_template_spec.rb b/spec/data_deploy_caller_template_spec.rb index 24914c9..ca31133 100644 --- a/spec/data_deploy_caller_template_spec.rb +++ b/spec/data_deploy_caller_template_spec.rb @@ -63,8 +63,9 @@ it "accepts every default branch name in use across the fleet" do # `workflow_run`'s branch filter matches the *triggering* run's branch, so # this list has to cover every default branch the synced repos actually - # use (23 of the 30 are on v2, six on main, one on master) for the file to - # stay repo-agnostic. It is only a pre-filter: a repo carrying more than + # use (23 of the 31 are on v2, eight on main; none is on master today, and + # the list keeps that name because over-covering costs nothing) for the file + # to stay repo-agnostic. It is only a pre-filter: a repo carrying more than # one of those names lets a crawl on a dormant branch through, so # data-deploy.yml gates on `workflow_run.head_branch` being the repo's # real default branch, and publication on the run's own ref. @@ -117,7 +118,7 @@ it "leaves no push or pull_request trigger in Deploy or Crawler" do # Stated separately from the two `contain_exactly` examples above because # this is the invariant that fails silently: re-adding either trigger just - # burns Actions minutes fleet-wide, in 30 repos at once, with nothing red. + # burns Actions minutes fleet-wide, in 31 repos at once, with nothing red. # # Note what re-adding `pull_request` *here* would not be: a restored # pre-merge check. That lives in check-index.yml, which reaches the same @@ -168,7 +169,7 @@ # nothing user-visible to notice either. expect(File.exist?(check_index_path)).to be(true), "the pre-merge check caller is missing; without it a PR " \ - "touching data/ runs no build at all in 30 repos" + "touching data/ runs no build at all in 31 repos" end it "triggers on pull requests and nothing else" do @@ -235,7 +236,7 @@ it "passes no inputs at all" do # THE reason branding moved into data-index/configs.yml. cimas.yml maps - # this file into 30 repos as a whole-file copy, so anything in a `with:` + # this file into 31 repos as a whole-file copy, so anything in a `with:` # here is either wrong for the other 29 or — once a repo hand-edits it — # silently reverted by the next `cimas sync`. Branding failed silently (the # page just loses its favicon), which is why it moved to configs.yml. @@ -261,7 +262,7 @@ # # Narrowing this to `contents: read` — the reflexive "minimal permissions" # edit — does not fail at parse time. It fails minutes into the build, at the - # deploy step, in all 30 repos at once, with: + # deploy step, in all 31 repos at once, with: # # The workflow is requesting 'pages: write', but is only allowed 'pages: none'. # diff --git a/spec/data_deploy_workflow_spec.rb b/spec/data_deploy_workflow_spec.rb index 6babf84..80679ef 100644 --- a/spec/data_deploy_workflow_spec.rb +++ b/spec/data_deploy_workflow_spec.rb @@ -165,7 +165,7 @@ describe "resolving branding from relaton/support" do it "checks out relaton/support, the branding source of truth" do # Branding cannot live in the caller: cimas.yml maps - # .github/workflows/deploy.yml as a whole-file copy for 30 repos, so a + # .github/workflows/deploy.yml as a whole-file copy for 31 repos, so a # `with:` block is wiped on the next sync and the page silently loses its # favicon and description. expect(support_checkout).not_to be_nil diff --git a/spec/data_index_config_spec.rb b/spec/data_index_config_spec.rb index ba0380c..f88c96e 100644 --- a/spec/data_index_config_spec.rb +++ b/spec/data_index_config_spec.rb @@ -20,7 +20,7 @@ def branding(repo) it "applies the default favicon to repos without an override" do # The convention is the SDO's own icon where there is a stable URL for one, # so this roster grows. Adding a `favicon:` to a row means adding it here. - overridden = %w[iana ids itu oasis w3c] + overridden = %w[iana ids ietf itu oasis w3c] default_favicon = config.defaults.fetch("favicon") config.repos.reject { |e| overridden.include?(e["repo"]) }.each do |e| @@ -84,17 +84,21 @@ def branding(repo) end describe "configs.yml data" do - it "covers exactly the 30 repos with no duplicates" do + it "covers exactly the 31 repos with no duplicates" do repos = config.repos.map { |e| e["repo"] } - expect(repos.size).to eq(30) - expect(repos.uniq.size).to eq(30) + expect(repos.size).to eq(31) + expect(repos.uniq.size).to eq(31) expect(repos).to include("iso", "ieee", "jis", "adobe", "easc", "gost", "jcgm", "oiml", "iala") expect(repos).to include("ids", "oasis", "w3c") # already-live, folded in # Both ITU rows: `itu` is the combined ITU-R + ITU-T corpus, `itu-r` the # ITU-R-only repo the relaton gem still consumes. Neither is a typo for # the other, and dropping either silently 404s a site. expect(repos).to include("itu", "itu-r") - expect(repos).not_to include("sdo", "ietf", "misc") + # ietf was the one repo Cimas synced deploy.yml into with no row here. It + # publishes an index since migrating to Relaton::Ietf::DataFetcher, so the + # exclusion is lifted and this file now covers the whole synced fleet. + expect(repos).to include("ietf") + expect(repos).not_to include("sdo", "misc") end it "gives every entry the required fields" do diff --git a/spec/index_branding_spec.rb b/spec/index_branding_spec.rb index 26cbb88..e8f664f 100644 --- a/spec/index_branding_spec.rb +++ b/spec/index_branding_spec.rb @@ -7,8 +7,8 @@ # of truth for these values, and the caller template carries no `with:` at all. # # The load-bearing invariants here are the fallbacks: the workflow passes all -# three flags unconditionally, and Cimas syncs deploy.yml into one repo -# (relaton-data-ietf) that configs.yml deliberately does not cover. +# three flags unconditionally, and a repo added to cimas.yml before its +# configs.yml row lands must still resolve rather than fail its own deploy. require "English" # $CHILD_STATUS require "shellwords" @@ -101,11 +101,14 @@ end it "falls back to the derived title and no branding for a repo configs.yml omits" do - # relaton-data-ietf gets deploy.yml from Cimas but publishes no document - # index, so it has no configs.yml row (see cimas_data_pages_spec.rb). + # No live repo exercises this any more: configs.yml now covers every repo + # Cimas syncs deploy.yml into, relaton-data-ietf included. relaton-data-sdo + # is a deliberate stand-in — it appears in neither file, so it stays a + # stable witness. The fallback stays guarded because a repo added to + # cimas.yml before its row lands would otherwise fail its own Pages build. # Its result must match what the retired shell derivation produced. - expect(config.branding("relaton/relaton-data-ietf")) - .to eq("title" => "IETF Index", "favicon" => "", "description" => "") + expect(config.branding("relaton/relaton-data-sdo")) + .to eq("title" => "SDO Index", "favicon" => "", "description" => "") end it "does not raise for an unknown repo, unlike #entry" do @@ -136,7 +139,7 @@ # The unit examples above all bypass the executable the workflow actually runs. # Without these, renaming a method on GithubOutput or DataIndexConfig would - # leave the whole suite green and break the resolve step in all 30 repos. + # leave the whole suite green and break the resolve step in all 31 repos. describe "bin/index-branding" do bin = File.join(repo_root, "bin/index-branding") @@ -155,11 +158,14 @@ end it "resolves a repo configs.yml does not cover" do - out = `#{bin.shellescape} relaton/relaton-data-ietf 2>/dev/null` + # Same stand-in as the unit example above: relaton-data-sdo is in neither + # configs.yml nor cimas.yml, so it exercises the fallback the executable + # must not fail on. + out = `#{bin.shellescape} relaton/relaton-data-sdo 2>/dev/null` expect($CHILD_STATUS).to be_success expect(parse.call(out)) - .to eq("title" => "IETF Index", "favicon" => "", "description" => "") + .to eq("title" => "SDO Index", "favicon" => "", "description" => "") end it "applies the flags the workflow always passes, blanks included" do