Add versioned GDS layout registry and v0/v1/v2 embedding standards

.pre-commit-config.yaml

@@ -25,7 +25,8 @@ repos:
         args: [--unsafe]
       - id: check-toml
       - id: check-added-large-files
-        args: ['--maxkb=1000']
+        # Executed Plotly tutorials can legitimately exceed 1 MB.
+        args: ['--maxkb=20000']
       - id: check-merge-conflict
       - id: debug-statements
 

README.md

@@ -200,6 +200,14 @@ The following tutorials are available to help you get started with `SQuADDS`:
 - [Tutorial 9: Learning the Inverse Map](https://lfl-lab.github.io/SQuADDS/source/tutorials/Tutorial-9_Learing_the_Inverse_Design_Map.html)
 - [Tutorial 10: HFSS Driven-Modal Capacitance Extraction](https://lfl-lab.github.io/SQuADDS/source/tutorials/Tutorial-10_DrivenModal_Capacitance_Extraction.html)
 - [Tutorial 11: Unified Driven-Modal Hamiltonian Extraction](https://lfl-lab.github.io/SQuADDS/source/tutorials/Tutorial-11_DrivenModal_Combined_Hamiltonian_Extraction.html)
+- [Tutorial 14: Exploring Static Layout Embeddings](https://lfl-lab.github.io/SQuADDS/source/tutorials/Tutorial-14_Exploring_Static_Layout_Embeddings.html)
+- [Tutorial 15: Transfer Learning with Static Layout Embeddings](https://lfl-lab.github.io/SQuADDS/source/tutorials/Tutorial-15_Transfer_Learning_with_Static_Embeddings.html)
+- [Tutorial 16: Transfer Learning Across GeneralizedNCap Geometry Regimes](https://lfl-lab.github.io/SQuADDS/source/tutorials/Tutorial-16_Geometry_Domain_Transfer_Learning.html)
+- [Tutorial 17: Transfer Learning Across Component Classes](https://lfl-lab.github.io/SQuADDS/source/tutorials/Tutorial-17_Cross_Component_Transfer_Learning.html)
+- [Tutorial 18: Universal Geometry v2 Layout Embeddings](https://lfl-lab.github.io/SQuADDS/source/tutorials/Tutorial-18_Universal_Geometry_v2_Embeddings.html)
+- [Tutorial 19: How the Universal Geometry v2 Encoder Works](https://lfl-lab.github.io/SQuADDS/source/tutorials/Tutorial-19_How_Universal_Geometry_v2_Works.html)
+- [Tutorial 20: Cross-Component-Class Transfer with Universal Geometry v2](https://lfl-lab.github.io/SQuADDS/source/tutorials/Tutorial-20_Cross_Class_Transfer_with_v2.html)
+- [Tutorial 21: Balanced Geometry-Domain Transfer with Universal Geometry v2](https://lfl-lab.github.io/SQuADDS/source/tutorials/Tutorial-21_Balanced_Geometry_Domains_with_v2.html)
 
 ---
 

docs/conf.py

@@ -1,6 +1,8 @@
 import os
 import sys
 
+import nbsphinx
+
 # Set the path to the root of the project
 sys.path.insert(0, os.path.abspath('../..'))
 
@@ -14,6 +16,7 @@
 # General configuration
 exclude_patterns = [
     "_build",
+    "build",
     "**.ipynb_checkpoints",
     "jupyter_execute",
     "setup.py",
@@ -121,6 +124,29 @@
 # nbsphinx settings
 nbsphinx_execute = 'never'
 
+# nbsphinx overrides nbconvert's input template, so make its ``hide-input``
+# cell tag retain outputs while omitting the corresponding source blocks.
+_input_block_start = "{% block input -%}"
+_input_block_end = "{% endblock input %}"
+_hide_input_condition = (
+    "{%- if 'hide-input' not in cell.metadata.get('tags', []) %}"
+)
+if _hide_input_condition not in nbsphinx.RST_TEMPLATE:
+    if (
+        _input_block_start not in nbsphinx.RST_TEMPLATE
+        or _input_block_end not in nbsphinx.RST_TEMPLATE
+    ):
+        raise RuntimeError("Unable to configure nbsphinx hide-input support")
+    nbsphinx.RST_TEMPLATE = nbsphinx.RST_TEMPLATE.replace(
+        _input_block_start,
+        _input_block_start + "\n" + _hide_input_condition,
+        1,
+    ).replace(
+        _input_block_end,
+        "{%- endif %}\n" + _input_block_end,
+        1,
+    )
+
 # allow html
 nbsphinx_allow_html = True
 

docs/source/layout_embeddings_v1_release.md

@@ -0,0 +1,105 @@
+# Layout Embeddings v1 release
+
+`universal-geometry-v1` is additive. It does not replace or rewrite
+`static-embedding-v0`.
+
+The reviewed release is staged in
+[SQuADDS_Layout_Embeddings PR #2](https://huggingface.co/datasets/SQuADDS/SQuADDS_Layout_Embeddings/discussions/2).
+It combines the expanded v0 catalogue with the first v1 configuration so the
+dataset card and both versioned tables can be reviewed as one atomic release.
+
+## Hugging Face file layout
+
+The prepared release belongs in the existing
+`SQuADDS/SQuADDS_Layout_Embeddings` dataset:
+
+```text
+metadata/
+  static-embedding-v0.parquet
+  static-embedding-v0.schema.json
+  universal-geometry-v1.parquet
+models/
+  universal-geometry-v1/
+    schema.json
+    control-map.parquet
+    release-manifest.json
+README.md
+```
+
+The first v1 table contains only `GeneralizedCapNInterdigital`. Its stable
+`layout_id` values join directly to both `SQuADDS_Layouts` and `SQuADDS_DB`.
+Simulation targets are intentionally excluded from the embedding inputs.
+
+The accepted v1.1 schema has 1,024 dimensions:
+
+- 32 centered physical and morphological metrics;
+- 768 variance-selected full-spectrum shape coefficients at 96 by 96 resolution;
+- 224 centered, named parameter-control channels.
+
+The rejected 512-dimensional v1.0 candidate is not a release. It lost finger
+detail through an 8 by 8 low-pass DCT crop and collapsed cosine similarities
+through constant availability and uncentered-control offsets.
+
+## Acceptance benchmark
+
+Run the paired benchmark before publishing any regenerated v1 table:
+
+```bash
+uv run python scripts/benchmark_universal_embeddings.py \
+  /path/to/static-embedding-v0.parquet \
+  /path/to/universal-geometry-v1.parquet \
+  /path/to/coupler-GeneralizedCapNInterdigital-cap_matrix.json
+```
+
+V1 must pass every gate: topology retrieval must match or beat v0, full
+parameter distance must improve by at least 10%, normalized shape distance may
+regress by no more than 10%, held-out mutual-capacitance locality must improve,
+and random-pair cosine similarity must retain at least 60% of v0's standard
+deviation. Simulation targets are evaluated only after the geometry-only model
+is frozen.
+
+## Review and publish
+
+Generate into a staging directory, verify the files, and open a dataset pull
+request:
+
+```bash
+uv run python scripts/build_universal_embeddings.py \
+  /path/to/SQuADDS_Layouts/metadata/manifest.parquet \
+  /path/to/SQuADDS_Layout_Embeddings \
+  --design-json GeneralizedCapNInterdigital=/path/to/coupler-GeneralizedCapNInterdigital-cap_matrix.json \
+  --gds-source raw=/path/to/SQuADDS_Layouts
+
+hf upload SQuADDS/SQuADDS_Layout_Embeddings \
+  /path/to/SQuADDS_Layout_Embeddings \
+  . \
+  --type dataset \
+  --include "README.md" \
+  --include "metadata/universal-geometry-v1.parquet" \
+  --include "models/universal-geometry-v1/*" \
+  --commit-message "Add GeneralizedCapNInterdigital universal-geometry-v1 embeddings" \
+  --create-pr
+```
+
+## Merge order
+
+Merge the Hugging Face dataset pull request before the SQuADDS code pull
+request. Then smoke-test `LayoutEmbeddingClient(version="v1")` against the
+dataset's `main` revision before merging the code. This order prevents a window
+where the released API advertises v1 but the default dataset revision does not
+yet contain its parquet table and model metadata.
+
+The SQuADDS API defaults to `v0`; users opt into v1 explicitly:
+
+```python
+from squadds.layouts import LayoutEmbeddingClient
+
+v0 = LayoutEmbeddingClient(version="v0")
+v1 = LayoutEmbeddingClient(version="v1")
+```
+
+The database bridge exposes the same choice:
+
+```python
+embedding = db.get_layout_embedding(row, embedding_version="v1")
+```

docs/source/layout_embeddings_v2_release.md

@@ -0,0 +1,193 @@
+# Layout Embeddings v2 release
+
+`universal-geometry-v2` is additive. It does not replace or rewrite
+`static-embedding-v0` or `universal-geometry-v1`, and the SQuADDS API still
+defaults to `v0`.
+
+## What changes relative to v1
+
+v0 and v1 are **fit on write**. Their normalization statistics, and in v1 the
+selected spectral frequencies, are derived from whichever rows are written
+together. A contributor who runs either builder on their own designs therefore
+lands in a different space, even though the vectors share a length.
+
+v2 removes that coupling. Every coordinate is a physical measurement in
+micrometers, inverse micrometers, or farads per meter, accumulated onto frozen
+bin edges. `squadds.layouts.encode` is a pure function of one GDS file and one
+design-option mapping, so an outside contribution can be encoded and compared
+without refitting anything.
+
+v2 is also deliberately **not** scale invariant. v0 and v1 crop each layout to
+its own functional bounds, so a design and its exact enlargement produce
+identical shape blocks. Because conductor separation in micrometers is the
+dominant variable for capacitance, v2 measures distances absolutely.
+
+## Hugging Face file layout
+
+The release belongs in the existing `SQuADDS/SQuADDS_Layout_Embeddings`
+dataset and adds only new paths:
+
+```text
+metadata/
+  universal-geometry-v2.parquet
+models/
+  universal-geometry-v2/
+    schema.json
+    release-manifest.json
+```
+
+The 512 dimensions are:
+
+- 48 named physical metrics: extent, per-role area and perimeter, conductor
+  width percentiles, gap integrals, and symmetry;
+- 192 coupling-spectrum bins: facing boundary length per terminal pair per
+  absolute separation, plus each terminal against the ground plane;
+- 128 shape-spectrum coefficients: two-point correlation, terminal
+  cross-correlation, conductor width distribution, and contour harmonics;
+- 96 parameter statistics: dimension-typed order statistics with
+  dimension-scoped signed hashing, so any parameter count or naming convention
+  maps to a fixed width;
+- 48 physics-proxy coordinates: a two-dimensional boundary-element capacitance
+  matrix and dilation topology.
+
+Simulation targets are never embedding inputs.
+
+## Coverage
+
+The published v2 table contains **17,727** designs across **all four** component
+families: `GeneralizedCapNInterdigital` (13,683), `TransmonCross` (1,934),
+`CavityClawRouteMeander` (1,216), and `CapNInterdigitalTee` (894). v2 is the
+first standard here that covers the whole catalogue with one encoder and no
+per-family configuration.
+
+The `GeneralizedCapNInterdigital` count is fewer than the 20,062 rows in the v0
+and v1 tables for that family. The reason is upstream: `SQuADDS/SQuADDS_Layouts`
+currently publishes 10,000 of the 16,379 `q3d_cap` GDS artifacts that its own
+`metadata/manifest.parquet` lists, so the remaining 6,379 layouts cannot be
+encoded from the released geometry. v0 and v1 were generated before that gap
+appeared. The count in `release-manifest.json` records it as
+`layouts_without_downloadable_gds`.
+
+Regenerating the missing artifacts into `SQuADDS_Layouts` and re-running the
+builder is the only step required to bring v2 to full coverage; nothing in the
+encoder changes.
+
+Every vector in the earlier `GeneralizedCapNInterdigital`-only v2 release is
+byte-identical in the four-family release. Adding three component families
+changed none of them, because no coordinate is derived from catalogue
+statistics. This is the fit-on-write property being absent, demonstrated rather
+than asserted, and it is what allows an outside contribution to be concatenated
+with the published table instead of forcing a rebuild.
+
+Build the whole catalogue by omitting `--component-name` and supplying one
+`--design-json` per family.
+
+## Reproduce and publish
+
+The builder performs no catalogue-wide fitting, so two runs over disjoint
+contributions produce directly concatenable output:
+
+```bash
+uv run --extra gds python scripts/build_v2_embeddings.py \
+  /path/to/SQuADDS_Layouts/metadata/manifest.parquet \
+  /path/to/SQuADDS_Layouts \
+  /path/to/staging \
+  --design-json GeneralizedCapNInterdigital=/path/to/coupler-GeneralizedCapNInterdigital-cap_matrix.json \
+  --design-json CapNInterdigitalTee=/path/to/coupler-CapNInterdigitalTee-cap_matrix.json \
+  --design-json TransmonCross=/path/to/qubit-TransmonCross-cap_matrix.json \
+  --design-json CavityClawRouteMeander=/path/to/cavity_claw-RouteMeander-eigenmode.json
+```
+
+Add `--component-name <family>` to restrict the build to one family.
+
+Two complete runs produce byte-identical parquet, schema, and release-manifest
+files. Verify the checksums in `release-manifest.json` before uploading:
+
+```bash
+hf upload SQuADDS/SQuADDS_Layout_Embeddings \
+  /path/to/staging . \
+  --repo-type dataset \
+  --include "metadata/universal-geometry-v2.parquet" \
+  --include "models/universal-geometry-v2/*" \
+  --commit-message "Extend universal-geometry-v2 to all four component families"
+```
+
+## Selecting the standard
+
+The API default remains `v0`. Users opt into v2 explicitly:
+
+```python
+from squadds.layouts import LayoutEmbeddingClient
+
+v2 = LayoutEmbeddingClient(version="v2")
+record = v2.get("layout:sha256:<layout hash>")
+neighbors = v2.nearest(record["layout_id"], limit=10)
+```
+
+The database bridge and the MCP tools expose the same choice:
+
+```python
+embedding = db.get_layout_embedding(row, embedding_version="v2")
+```
+
+Encoding a layout that is not in the catalogue needs no download at all:
+
+```python
+from squadds.layouts import encode
+
+vector = encode("my_capacitor.gds", {"digit_pitch": "5.5um", "digit_population": 9})
+```
+
+## Evidence
+
+Tutorial 18 compares v0 and v2 under an identical model, split policy, and label
+budget over the 13,683 paired `GeneralizedCapNInterdigital` designs. Tutorial 19
+walks one design through the encoder and checks the explanation against the
+shipped implementation. Tutorial 20 crosses the component-class boundary.
+
+| held-out macro R2 | 1% labels | 10% labels | 100% labels |
+| --- | ---: | ---: | ---: |
+| `static-shape-v0` (155 compact features) | 0.235 | 0.888 | 0.984 |
+| `universal-geometry-v2` | 0.780 | 0.9915 | 0.9998 |
+| `universal-geometry-v2`, geometry block only | 0.754 | 0.9896 | 0.9996 |
+
+The third row is the control that matters. It receives no design parameters at
+all and still beats the whole of v0, which includes v0's parameter sum, so the
+improvement is geometric rather than an artifact of v2 retaining more parameter
+detail.
+
+A single unfitted coordinate, the facing-boundary integral
+`log1p_primary_inverse_gap_integral`, reaches Spearman +0.941 against the
+simulated mutual capacitance, and the boundary-element proxy reaches +0.920.
+
+Across the three families that report a mutual capacitance
+(`GeneralizedCapNInterdigital`, `CapNInterdigitalTee`, and `TransmonCross`) the
+design-option vocabularies intersect in exactly one name, `orientation`, a
+placement angle. No parameter-schema baseline exists for a three-class model, so
+a geometry-derived contract is the only available option rather than merely the
+better one.
+
+## Known limits
+
+- Zero-shot prediction of a completely unseen component class works in two of
+  three rotations on a class-balanced cohort, reaching macro R2 0.859 and 0.422
+  where v0 reaches -7.7 and -17.6, and fails on the third (`TransmonCross`, at
+  -1.891). A brand-new family needs roughly ten labeled designs to pass 0.94.
+- Predicting the residual against the boundary-element proxy, rather than
+  capacitance directly, is worse in every rotation. The two-dimensional proxy
+  carries a class-dependent offset, so subtracting it injects between-class
+  variance instead of removing it.
+- Extrapolating to devices larger than any in training, v2 reaches macro R2
+  0.810 against v0's 0.769; the advantage narrows sharply outside the
+  interpolation regime.
+- Raw cosine similarity has a compressed spread, and the severity is
+  family-dependent. Within `TransmonCross` the entire family spans cosine 0.9928
+  to 1.0, so `nearest()` cannot usefully rank inside it; `CapNInterdigitalTee`
+  spans 0.027 to 1.0. The vectors are correct - every coordinate is an absolute
+  measurement, so the shared direction is real - but a frozen whitening transform
+  must be fitted and published as a separate metric layer before v2 similarity is
+  used for retrieval. `TransmonCross` is also the family that fails the
+  cross-class similarity and held-out-class tests in Tutorial 20, which suggests
+  one underlying cause rather than three.
+- On two-terminal devices roughly 204 of the 512 coordinates vary; the rest are
+  reserved for richer terminal topologies and cost nothing statistically.