2/3 Bundle taxons + JSON-LD context; cross-connector harness

[RaggedStaff]

Stacked PR 2 of 3 — base: pr/1-connector-generator-fixes. Merges after PR 1.

Scope

Bundle the SKOS taxonomy exports and JSON-LD context into both connectors, and add the cross-connector test harness. Data plane only — no connector API changes.

Commits:

• 780f025 bundle taxons and JSON-LD context into both connectors; add cross-connector harness

Review notes

The diff is almost entirely generated/bundled SKOS taxonomy data (facet/measure/product_type/scope/vocabulary_term) plus the cross-connector harness in tests/cross_connector/. The taxonomies are hand-maintained exports, not schema-derived — treat as data. The hand-written parts to review:

• tests/cross_connector/ (adapters, run_matrix, compare_connectors, normalize, scenarios)
• ruby-gem/lib/core/{connector,semantic_object,vocabulary_loader}.rb
• typescript-connector/src/core/{Connector,SemanticObject,VocabularyLoader}.ts

Validation

• cd typescript-connector && npm test && npm run build
• cd ruby-gem && bundle exec rake spec
• Cross-connector matrix: python3 tests/cross_connector/run_matrix.py

[Copilot]

Pull request overview

This stacked PR (2/3) bundles DFC v2.0.0 SKOS taxonomy exports and the JSON-LD context into both the TypeScript connector and Ruby gem, and introduces a Python-based cross-connector round-trip test harness under tests/cross_connector/ to compare import/export behavior across “our” and “official” connectors.

Changes:

• Bundle taxonomies + JSON-LD context into both connectors, reducing network dependence for core data.
• Add cross-connector harness (scenarios, adapters, normalization, and round-trip matrix runner).
• Update TypeScript tests to expect JSON-LD references exported as IRI strings (not {"@id": ...} objects).

Reviewed changes

Copilot reviewed 25 out of 46 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
typescript-connector/test/integration/conformance.test.ts	Updates conformance expectation for exported references to be IRI strings.
typescript-connector/test/connector.test.ts	Updates extended import/export test expectation for IRI string references.
typescript-connector/src/taxonomies/vocabulary_term.ts	Adds bundled taxonomy JSON-LD export (data).
typescript-connector/src/taxonomies/scope.ts	Adds bundled taxonomy JSON-LD export (data).
typescript-connector/src/core/VocabularyLoader.ts	Adds bundled taxonomy loading + version headers for fetch; (also introduces URL/version issues to address).
typescript-connector/src/core/SemanticObject.ts	Adds predicate/value introspection helpers used by the harness.
typescript-connector/src/core/Connector.ts	Loads bundled taxonomies/context by default; adds dfc-version header for context fetch.
typescript-connector/src/context/context_2.0.0.ts	Adds bundled JSON-LD context (data).
typescript-connector/dist/taxonomies/vocabulary_term.js	Built output for bundled taxonomy (data).
typescript-connector/dist/taxonomies/vocabulary_term.d.ts	Type declarations for bundled taxonomy (data).
typescript-connector/dist/taxonomies/scope.js	Built output for bundled taxonomy (data).
typescript-connector/dist/taxonomies/scope.d.ts	Type declarations for bundled taxonomy (data).
typescript-connector/dist/taxonomies/measure.d.ts	Type declarations for bundled taxonomy (data).
typescript-connector/dist/core/VocabularyLoader.js	Built output for VocabularyLoader changes (contains the same URL/version bug as src).
typescript-connector/dist/core/VocabularyLoader.d.ts	Updated public typings for VocabularyLoader.
typescript-connector/dist/core/SemanticObject.js	Built output for SemanticObject introspection helpers.
typescript-connector/dist/core/SemanticObject.d.ts	Updated typings for SemanticObject introspection helpers.
typescript-connector/dist/core/Connector.js	Built output for Connector bundling + context fetch header.
typescript-connector/dist/core/Connector.d.ts	Updated typings for Connector bundling/context helpers.
typescript-connector/dist/context/context_2.0.0.js	Built output for bundled JSON-LD context (data).
typescript-connector/dist/context/context_2.0.0.d.ts	Type declarations for bundled JSON-LD context (data).
tests/cross_connector/scenarios/supplied-product.json	Adds a cross-connector scenario fixture.
tests/cross_connector/scenarios/simple-enterprise.json	Adds a cross-connector scenario fixture.
tests/cross_connector/scenarios/order-with-lines.json	Adds a cross-connector scenario fixture with $ref linking.
tests/cross_connector/runner.py	Adds adapter runner + connector discovery utilities.
tests/cross_connector/run_matrix.py	Adds round-trip matrix runner and mismatch classification logic.
tests/cross_connector/normalize.py	Adds JSON-LD normalization + object extraction helpers (needs robustness fixes).
tests/cross_connector/compare_connectors.py	Adds capability diff reporter (usage string needs correction).
tests/cross_connector/adapters/our-typescript.mjs	Adapter for local TypeScript connector dist (capabilities/export/import).
tests/cross_connector/adapters/our-ruby.rb	Adapter for local Ruby gem (capabilities/export/import).
tests/cross_connector/adapters/official-typescript.mjs	Adapter for official TS connector (capabilities/export/import).
tests/cross_connector/adapters/official-ruby.rb	Adapter for official Ruby connector (capabilities/export/import).
ruby-gem/lib/core/vocabulary_loader.rb	Adds bundled vocabulary loading and dfc-version header on fetch.
ruby-gem/lib/core/semantic_object.rb	Adds predicate/value introspection helpers used by the harness.
ruby-gem/lib/core/connector.rb	Loads bundled taxonomies/context; adds dfc-version header and context bundling.
ruby-gem/dfc-linkml-connector.gemspec	Ensures bundled contexts are included in gem packaging.
ruby-gem/contexts/context_2.0.0.json	Adds bundled JSON-LD context (data).
.gitignore	Ignores tests/node_modules/ for the harness environment.

Suppressed comments (2)

typescript-connector/src/core/VocabularyLoader.ts:40

• taxonomyBaseUrl is using the literal string v$this.taxonomyVersion (missing ${...} interpolation), so it will never include the configured version and will produce an invalid URL for network loads.
  tests/cross_connector/normalize.py:41
• The extract_objects() docstring says "Literal vs @id distinction is preserved by wrapping @id-references in a sentinel tuple", but the implementation calls normalize_value() which collapses {"@id": "x"} to "x" and does not wrap anything. This can confuse readers about what is (and isn’t) being compared.

    Predicate values are normalized (refs resolved to semanticIds, containers
    unwrapped, lists sorted). Literal vs @id distinction is preserved by
    wrapping @id-references in a sentinel tuple.
    """

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 25 out of 46 changed files in this pull request and generated no new comments.

Suppressed comments (4)

tests/cross_connector/normalize.py:36

• normalize_value() claims to ignore container shape ("..." == ["..."] in the docstring), but the implementation keeps lists as lists. This will incorrectly flag mismatches when one connector compacts a single value to a scalar and another keeps it as a 1-element array.

    if isinstance(value, list):
        return sorted((normalize_value(v) for v in value), key=_stable_key)

tests/cross_connector/normalize.py:51

• The extract_objects() docstring says "Literal vs @id distinction is preserved by wrapping @id-references in a sentinel tuple", but normalize_value() currently converts {"@id": ...} to the raw string and does not preserve that distinction. This makes the doc misleading for anyone extending the harness.

    Predicate values are normalized (refs resolved to semanticIds, containers
    unwrapped, lists sorted). Literal vs @id distinction is preserved by
    wrapping @id-references in a sentinel tuple.

typescript-connector/src/core/Connector.ts:480

• VocabularyLoader already parses bundled taxonomies in its constructor (loadBundled()), but Connector.loadBundledTaxonomies() reloads and re-parses the same bundled JSON-LD again via loadFacets/loadMeasures/.... With large taxonomies this doubles startup work unnecessarily.
  ruby-gem/lib/core/vocabulary_loader.rb:51
• VocabularyLoader.load_from_url() lowercases the URL segment (name.downcase) and also stores the loaded concepts under the passed-in name. This breaks case-sensitive endpoints like productTypes.json and means helpers like product_type() (which reads vocabulary("ProductType")) won't see data loaded via load_from_url("productTypes").

      def load_from_url(name)
        url = "#{TAXONOMY_BASE_URL}/v#{@taxonomy_version}/#{name.downcase}.json"
        uri = URI(url)
        request = Net::HTTP::Get.new(uri)
        request["dfc-version"] = @ontology_version

[Copilot]

Pull request overview

Copilot reviewed 25 out of 46 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 25 out of 46 changed files in this pull request and generated 8 comments.

Suppressed comments (4)

Previously missed (2) — in code that hasn't changed since the last review.

tests/cross_connector/normalize.py:38

• Plain JSON-LD arrays are unordered sets, and a one-element array is semantically equivalent to its scalar form, but this normalization preserves that container difference. Two connectors can therefore emit equivalent data and still produce a mismatch, contrary to the matrix's stated shape-insensitive comparison. Collapse singleton normalized arrays to their element (while retaining explicit @list objects).

    if isinstance(value, list):
        return sorted((normalize_value(v) for v in value), key=_stable_key)

tests/cross_connector/adapters/official-typescript.mjs:117

• The official connector's export options are inputContext and outputContext; it does not define a context option, so this value is silently ignored and the adapter does not emit the requested context. Pass outputContext: CTX (and inputContext too only if overriding compaction input is intended).

This issue also appears on line 125 of the same file.

  const jsonld = await c.export(instances, { context: CTX });

tests/cross_connector/run_matrix.py:89

• The target's baseline export is outside the error handling used for source exports and imports. If a target supports the adapter but cannot construct/export one scenario, this raises out of compare() and aborts the entire matrix instead of recording the required import/export failure and continuing with the remaining pairs. Wrap this export in the same reporting path before calling _safe_import.

    # baseline_B: the target's own round-trip of the same scenario.
    baseline_b = _safe_import(target, export_jsonld(target, scenario_path), report)
    if baseline_b is None:
        return report

tests/cross_connector/adapters/official-typescript.mjs:125

• As in the scenario export path, context is not a recognized option for the official connector, so the re-export ignores CTX. Use the supported outputContext option to keep both adapter paths on the intended context.

  const re = await c.export(objects, { context: CTX });

[Copilot]

Pull request overview

Copilot reviewed 25 out of 46 changed files in this pull request and generated 5 comments.

Suppressed comments (1)

tests/cross_connector/runner.py:44

• run_adapter() will raise FileNotFoundError/OSError when the runtime executable (e.g. node/ruby) is missing, but available_connectors() only catches RuntimeError. This can crash the harness instead of cleanly treating that connector as unavailable. Wrap the subprocess.run(...) call and re-raise OS-level failures as RuntimeError so available_connectors() can handle them.

def run_adapter(name: str, *args: str, stdin: str | None = None) -> str:
    result = subprocess.run(
        adapter_cmd(name, *args),
        input=stdin,
        capture_output=True,
        text=True,
        check=False,
    )
    if result.returncode != 0:
        raise RuntimeError(
            f"{name} {args} failed (rc={result.returncode}):\n"
            f"stderr: {result.stderr}\nstdout: {result.stdout}"
        )
    return result.stdout

[Copilot]

🟡 Changes recommended

Unresolved issues remain in taxonomy namespaces, scalar vocabulary parsing, Ruby loader generation/fallback behavior, and cross-connector dependency and failure handling.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (14)

ruby-gem/lib/core/connector.rb:310

• This comment promises network fallback when a bundled file is absent, but load_bundled_taxonomies only conditionally calls the local loaders and never invokes load_*_from_url or _fetch_taxonomy_json. In an installed package missing a bundle, the vocabulary stays empty despite this documented fallback; either implement the fallback or revise the comment to describe the actual offline-only behavior.

      # Loads the taxonomies shipped with the gem (ruby-gem/vocabularies),
      # falling back to network fetches only when the bundled files are absent.
      def load_bundled_taxonomies

ruby-gem/lib/core/connector.rb:310

• load_bundled_taxonomies and bundled_context are public Ruby methods, which adds connector API surface while the PR scope says there are no connector API changes. These helpers are only called internally; make them private in the generated connector (and generator) or explicitly include the API addition in the scope.

      def load_bundled_taxonomies

ruby-gem/lib/core/vocabulary_loader.rb:73

• A compact JSON-LD document may represent a single notation or label as a scalar value object such as {"@value": "kg"}. This extractor only unwraps @value inside arrays, so such concepts are silently omitted from the loaded vocabulary; handle the scalar object form before the array branch.

          if value.is_a?(Array)
            value.each do |item|
              return item if item.is_a?(String)
              if item.is_a?(Hash) && item["@value"].is_a?(String)
                return item["@value"]

ruby-gem/lib/core/vocabulary_loader.rb:28

• This generated loader is not reproducible from scripts/generate_ruby_gem.py: that template still emits the old singular URL_TO_KEY entries and lacks the bundled-loading and robust concept parsing added here. A normal regeneration will overwrite this file and reintroduce the URL mapping behavior, so move the loader changes into the generator and regenerate the gem.

      URL_TO_KEY = {
        "facets" => "Facet",
        "measures" => "Measure",
        "producttypes" => "ProductType",
        "scopes" => "Scope",

tests/cross_connector/adapters/official-ruby.rb:11

• The matrix adapter requires the external datafoodconsortium-connector gem, but the PR adds no Gemfile/lockfile or setup step for that dependency. On a clean checkout this require fails and available_connectors() silently omits the official Ruby connector, so the advertised cross-connector matrix can pass without running it. Add reproducible dependency setup or make the missing required official adapter fail the validation.

require 'json'
require 'datafoodconsortium/connector'

tests/cross_connector/adapters/official-typescript.mjs:16

• This adapter assumes tests/node_modules/@datafoodconsortium/connector already exists, but the PR adds no package manifest, lockfile, or install step for it. On a clean checkout the require fails and available_connectors() silently omits the official TypeScript connector, allowing the matrix to pass while testing only the local connectors. Add reproducible dependency setup or make the missing required official adapter fail the validation.

const pkg = require(path.join(__dirname, "..", "..", "node_modules", "@datafoodconsortium/connector", "package.json"));
const mod = await import(path.join(__dirname, "..", "..", "node_modules", "@datafoodconsortium/connector", pkg.main));

tests/cross_connector/adapters/official-typescript.mjs:80

• _context._prefixes contains namespace declarations such as dfc-b, while this regex requires a full dfc-b:predicate term. Consequently globalPredicates is always empty and compare_connectors.py reports no per-class predicate differences for the official TypeScript connector; derive predicates from the context's term definitions or another supported metadata source instead.

  const prefixes = probe._semantizer?._context?._prefixes || [];
  const globalPredicates = prefixes
    .map((p) => p.prefix)
    .filter((n) => /^dfc-[bt]:[A-Za-z]/.test(n))
    .sort();

tests/cross_connector/run_matrix.py:87

• Only the target import is wrapped in _safe_import; export_jsonld(target, scenario_path) is evaluated first and a target-side export failure raises out of compare, aborting the whole matrix with a traceback instead of recording the promised mismatch. Capture the target export failure before passing its document to _safe_import, just as the source export is handled above.

    baseline_b = _safe_import(target, export_jsonld(target, scenario_path), report)

tests/cross_connector/runner.py:71

• If node or ruby is absent, subprocess.run raises FileNotFoundError before returning a result. available_connectors catches only RuntimeError, so discovery crashes instead of skipping that adapter as its docstring promises; catch OSError as well.

        except RuntimeError:

tests/cross_connector/runner.py:72

• Every nonzero capability probe is treated as “unavailable,” so a broken local build or adapter is silently removed from the matrix. With another connector still available, the harness can report a successful partial run without testing the broken connector; distinguish missing optional dependencies from adapter failures or report required-connector failures.

            run_adapter(name, "capabilities")
            result.append(name)
        except RuntimeError:
            continue

typescript-connector/src/core/Connector.ts:476

• These helpers are public by default and are emitted in the shipped Connector.d.ts, so this adds loadBundledTaxonomies() (and the similarly public loadBundledContext()) to the connector API despite the PR scope saying there are no connector API changes. Since they are only called internally, make them private in both generated connectors or explicitly include the API addition in the scope.
  typescript-connector/src/core/VocabularyLoader.ts:84
• A compact JSON-LD document may represent a single notation or label as a scalar value object such as {"@value": "kg"}. This extractor only unwraps @value inside arrays, so such concepts are silently omitted from the loaded vocabulary; handle the scalar object form before the array branch.
  typescript-connector/src/taxonomies/measure.ts:15
• The bundled Measure concepts are all identified with dfc-f: here, but the v2.0.0 context maps the measures taxonomy to dfc-m: (.../measures.rdf#). Expanding this document therefore gives every unit the facet namespace instead of its source taxonomy; regenerate the measure export with dfc-m: IDs and declare that prefix in the local context.
  typescript-connector/src/taxonomies/vocabulary_term.ts:15
• These concept IDs use dfc-f: even though this is the vocabulary taxonomy (@id is dfc-v:VocabularyTerm and the v2.0.0 context maps vocabulary IDs to dfc-v). The same prefix is used in the measure/product-type bundles despite their dfc-m/dfc-pt namespaces. Expanding these bundled JSON-LD files therefore produces incorrect IRIs, and the local context does not declare dfc-f; regenerate each export with its source namespace and declare that prefix in its local context.

• Files reviewed: 25/46 changed files
• Comments generated: 2
• Review effort level: Lite