Add a compas_pb migrate command for older serialized data

.github/workflows/build.yml

@@ -18,9 +18,10 @@ jobs:
         python: ["3.9", "3.10", "3.11", "3.12", "3.13"]
 
     steps:
-      - uses: compas-dev/compas-actions.build@v4
+      - uses: compas-dev/compas-actions.build@v5
         with:
           python: ${{ matrix.python }}
+          management_tool: uv
           invoke_lint: true
           invoke_test: true
   create-assets:

.github/workflows/release.yml

@@ -14,9 +14,10 @@ jobs:
         python: ["3.11", "3.12", "3.13"]
 
     steps:
-      - uses: compas-dev/compas-actions.build@v4
+      - uses: compas-dev/compas-actions.build@v5
         with:
           python: ${{ matrix.python }}
+          management_tool: uv
           invoke_lint: true
           invoke_test: true
 

CHANGELOG.md

@@ -9,8 +9,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+* Added a `compas_pb migrate` command that re-encodes data written by an older, wire-incompatible version into the current format. It decodes the blob in an ephemeral `uv` environment holding the version that wrote it, bridges through COMPAS JSON, and re-encodes with the current build. Reads from a file or stdin, writes to a file or stdout, and `--inspect` reports which version wrote a blob without migrating it.
+* Added `compas_pb.cli.migrate_bytes` and `compas_pb.cli.detect_wire_version` for the same migration from Python.
+* Added a migration guide at `docs/migration.md`.
+
 ### Changed
 
+* Changed the incompatible-wire-format error to point at `compas_pb migrate` instead of only telling the user to re-serialize the source.
+
 ### Removed
 
 

docs/migration.md

@@ -0,0 +1,91 @@
+# Migrating older data
+
+`compas_pb` refuses to read data written by an incompatible version. If you have blobs written
+by an older release, `compas_pb migrate` re-encodes them into the current wire format.
+
+```bash
+compas_pb migrate old.pb -o new.pb
+```
+
+## Why old data is refused rather than best-effort parsed
+
+`compas_pb` reuses protobuf field numbers across format revisions. Version 1.0.0 changed what
+those fields hold: coordinates went from `float` to `double`, mesh points went from one message
+per point to a packed `repeated double`, faces moved to CSR form, and per-element attributes
+became columnar. Protobuf does not reject a blob whose field numbers still line up, so an old
+blob read by a new build can *silently misparse* into plausible-looking but wrong geometry.
+
+Deserialization therefore checks the version tag and raises rather than guessing. Compatibility
+follows SemVer: under `0.x` every minor release is a break, and from `1.0` on only major bumps
+are. So `1.0` and `1.2` interoperate, while `0.5` and `1.0` do not.
+
+## How migration works
+
+There is no way to read both wire formats in one process, the two builds generate protobuf
+descriptors from the same file path, and only one can be registered at a time. So `migrate`
+decodes the data in a throwaway environment holding the version that *wrote* it, bridges through
+COMPAS JSON, and re-encodes with the current version:
+
+```
+old.pb ──> [ephemeral env, compas_pb 0.5.0] ──> COMPAS JSON ──> [this build] ──> new.pb
+```
+
+This needs [`uv`](https://docs.astral.sh/uv/) on `PATH`, and network access the first time a
+given old version is fetched. Nothing is installed into your own environment.
+
+## Usage
+
+```bash
+# Check what wrote a blob, without migrating it
+compas_pb migrate old.pb --inspect
+
+# File in, file out
+compas_pb migrate old.pb -o new.pb
+
+# Pipes work too
+cat old.pb | compas_pb migrate - > new.pb
+
+# Blobs written before v0.4.1 carry no version tag, so name the version yourself
+compas_pb migrate ancient.pb --from-version 0.3.1 -o new.pb
+
+# Pin the interpreter for the ephemeral environment
+compas_pb migrate old.pb --python 3.12 -o new.pb
+```
+
+Migrating a directory is a shell loop:
+
+```bash
+for f in data/*.pb; do compas_pb migrate "$f" -o "migrated/$(basename "$f")"; done
+```
+
+The same thing is available from Python:
+
+```python
+from compas_pb.cli import migrate_bytes
+
+with open("old.pb", "rb") as f:
+    migrated = migrate_bytes(f.read())
+```
+
+## What survives, and what does not
+
+Geometry, datastructures, attributes and explicitly set guids all come across. Two caveats are
+inherent to the old data rather than to migration:
+
+- **Precision is not recovered.** Pre-1.0 coordinates were `float32` on the wire. Migration
+  preserves exactly what was stored, so a point written as `0.1` comes back as
+  `0.10000000149011612`. It will now round-trip losslessly from here on, but the precision the
+  old format discarded is gone.
+- **Integral floats may arrive as ints.** Pre-1.0 routed values through `google.protobuf.Value`,
+  which returned whole numbers as `int`. A vertex attribute written as `3.0` migrates as `3`.
+  This is the old reader's behaviour, faithfully carried forward.
+
+Auto-generated guids are a non-issue in practice: a guid that was on the old wire is treated as
+explicitly set when read back, so it survives the re-encode even though 1.0 no longer serializes
+session-local ones.
+
+!!! note
+
+    `compas_pb` output is not byte-for-byte reproducible: protobuf serializes map fields in an
+    unspecified order, so migrating the same blob twice yields equal data in different bytes.
+    Compare deserialized objects, not hashes.

mkdocs.yml

@@ -139,6 +139,7 @@ nav:
       - Installation: installation.md
       - Examples: examples.md
       - Missing COMPAS Type?: missing_compas_type.md
+      - Migrating Older Data: migration.md
   - API Reference:
       - compas_pb: reference/compas_pb.md
       - compas_pb.core: reference/compas_pb.core.md

pyproject.toml

@@ -67,6 +67,9 @@ docs = [
     "tomli >=2.0; python_version < '3.11'",
 ]
 
+[project.scripts]
+compas_pb = "compas_pb.cli:main"
+
 [project.urls]
 Homepage = "https://gramaziokohler.github.io/compas_pb"
 Documentation = "https://gramaziokohler.github.io/compas_pb"
@@ -110,6 +113,9 @@ addopts = [
     "--tb=short",
     "--import-mode=importlib",
 ]
+markers = [
+    "network: needs network access to fetch an older compas_pb into an ephemeral environment",
+]
 doctest_optionflags = [
     "NORMALIZE_WHITESPACE",
     "IGNORE_EXCEPTION_DETAIL",

src/compas_pb/cli.py

@@ -0,0 +1,203 @@
+"""Command line interface for compas_pb."""
+
+import argparse
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+from importlib.metadata import version
+
+import compas
+
+from compas_pb.api import pb_dump_bts
+from compas_pb.api import pb_load_bts
+from compas_pb.core import _wire_compat_key
+
+_CURRENT_VERSION: str = version("compas_pb")
+
+# Runs inside the ephemeral environment of the *writing* version, so it may only use API that
+# version already had. ``pb_load`` and COMPAS JSON are the two things stable across all of them.
+_DECODE_SCRIPT = """
+import sys
+import compas
+from compas_pb import pb_load
+
+compas.json_dump(pb_load(sys.argv[1]), sys.argv[2])
+"""
+
+
+def _read_varint(buf: bytes, pos: int):
+    """Read a base-128 varint from ``buf`` at ``pos``, returning ``(value, new_pos)``."""
+    result = 0
+    shift = 0
+    while True:
+        if pos >= len(buf):
+            raise ValueError("truncated varint")
+        byte = buf[pos]
+        pos += 1
+        result |= (byte & 0x7F) << shift
+        if not byte & 0x80:
+            return result, pos
+        shift += 7
+        if shift > 63:
+            raise ValueError("varint overflows 64 bits")
+
+
+def detect_wire_version(blob: bytes):
+    """Read ``MessageData.version`` out of a blob without parsing its payload.
+
+    The payload of an incompatible blob cannot be parsed by this build -- that is the whole
+    reason migration is needed -- so this walks only the top-level fields, skipping field 1
+    (``data``) by its length prefix. ``version`` has been field 2 of ``MessageData`` since the
+    tag was introduced, and top-level framing has not changed since.
+
+    Returns ``None`` for a blob written before the version tag existed.
+    """
+    pos = 0
+    end = len(blob)
+    while pos < end:
+        tag, pos = _read_varint(blob, pos)
+        field_no, wire_type = tag >> 3, tag & 0x07
+        if wire_type == 0:
+            _, pos = _read_varint(blob, pos)
+        elif wire_type == 1:
+            pos += 8
+        elif wire_type == 5:
+            pos += 4
+        elif wire_type == 2:
+            length, pos = _read_varint(blob, pos)
+            if field_no == 2:
+                return blob[pos : pos + length].decode("utf-8")
+            pos += length
+        else:
+            raise ValueError("unsupported protobuf wire type {}; not a compas_pb message".format(wire_type))
+    return None
+
+
+def _decode_with(source_version: str, blob: bytes, python: str = None) -> str:
+    """Decode ``blob`` in an ephemeral environment holding ``source_version``, returning COMPAS JSON."""
+    if shutil.which("uv") is None:
+        raise RuntimeError("`uv` is required to migrate older data but was not found on PATH. See https://docs.astral.sh/uv/getting-started/installation/")
+
+    workdir = tempfile.mkdtemp(prefix="compas_pb_migrate_")
+    blob_path = os.path.join(workdir, "source.pb")
+    json_path = os.path.join(workdir, "bridge.json")
+    script_path = os.path.join(workdir, "decode.py")
+    try:
+        with open(blob_path, "wb") as f:
+            f.write(blob)
+        with open(script_path, "w") as f:
+            f.write(_DECODE_SCRIPT)
+
+        cmd = ["uv", "run", "--no-project", "--quiet"]
+        if python:
+            cmd += ["--python", python]
+        cmd += ["--with", "compas_pb=={}".format(source_version), "python", script_path, blob_path, json_path]
+
+        result = subprocess.run(cmd, capture_output=True, text=True, cwd=workdir)
+        if result.returncode != 0:
+            raise RuntimeError("failed to read the data with compas_pb {}:\n{}".format(source_version, (result.stderr or result.stdout).strip()))
+
+        with open(json_path, "r") as f:
+            return f.read()
+    finally:
+        shutil.rmtree(workdir, ignore_errors=True)
+
+
+def migrate_bytes(blob: bytes, source_version: str = None, python: str = None) -> bytes:
+    """Re-encode a blob written by an older compas_pb into the current wire format.
+
+    Parameters
+    ----------
+    blob : bytes
+        The serialized data to migrate.
+    source_version : str, optional
+        Version that wrote the blob. Detected from the blob when omitted, which is only
+        possible if it carries a version tag.
+    python : str, optional
+        Python version for the ephemeral environment, e.g. ``"3.12"``.
+
+    Returns
+    -------
+    bytes
+        The same data in the current wire format.
+
+    """
+    if source_version is None:
+        source_version = detect_wire_version(blob)
+    if source_version is None:
+        raise ValueError("this blob carries no version tag, so the version that wrote it cannot be detected; pass --from-version explicitly")
+
+    if _wire_compat_key(source_version) == _wire_compat_key(_CURRENT_VERSION):
+        raise ValueError("data written by {} is already readable by this build ({}); no migration needed".format(source_version, _CURRENT_VERSION))
+
+    bridge = _decode_with(source_version, blob, python=python)
+    migrated = pb_dump_bts(compas.json_loads(bridge))
+
+    # Cheap proof the result is readable before it reaches the user's disk.
+    pb_load_bts(migrated)
+    return migrated
+
+
+def _cmd_migrate(args) -> int:
+    blob = sys.stdin.buffer.read() if args.input == "-" else open(args.input, "rb").read()
+    if not blob:
+        print("error: no input data", file=sys.stderr)
+        return 1
+
+    if args.inspect:
+        detected = detect_wire_version(blob)
+        print("written by:  {}".format(detected or "<no version tag>"))
+        print("this build:  {}".format(_CURRENT_VERSION))
+        if detected and _wire_compat_key(detected) == _wire_compat_key(_CURRENT_VERSION):
+            print("status:      readable as-is")
+        else:
+            print("status:      needs migration")
+        return 0
+
+    if args.output == "-" and sys.stdout.isatty():
+        print("error: refusing to write binary data to a terminal; pass -o OUTPUT or redirect stdout", file=sys.stderr)
+        return 1
+
+    try:
+        migrated = migrate_bytes(blob, source_version=args.from_version, python=args.python)
+    except (ValueError, RuntimeError) as e:
+        print("error: {}".format(e), file=sys.stderr)
+        return 1
+
+    if args.output == "-":
+        sys.stdout.buffer.write(migrated)
+    else:
+        with open(args.output, "wb") as f:
+            f.write(migrated)
+        print("migrated {} -> {} ({} bytes)".format(args.input, args.output, len(migrated)), file=sys.stderr)
+    return 0
+
+
+def main(argv=None) -> int:
+    parser = argparse.ArgumentParser(prog="compas_pb", description="Utilities for compas_pb serialized data.")
+    subparsers = parser.add_subparsers(dest="command", required=True)
+
+    migrate = subparsers.add_parser(
+        "migrate",
+        help="re-encode data written by an older compas_pb into the current wire format",
+        description=(
+            "Reads a blob written by an older compas_pb, decodes it in an ephemeral environment "
+            "holding that version, and re-encodes it with this one. Requires `uv` and network "
+            "access the first time a given version is fetched."
+        ),
+    )
+    migrate.add_argument("input", help="path to the data to migrate, or - for stdin")
+    migrate.add_argument("-o", "--output", default="-", help="where to write the migrated data, or - for stdout (default)")
+    migrate.add_argument("--from-version", default=None, help="version that wrote the data; detected from the blob when omitted")
+    migrate.add_argument("--python", default=None, help="python version for the ephemeral environment, e.g. 3.12")
+    migrate.add_argument("--inspect", action="store_true", help="report the version that wrote the data and exit")
+    migrate.set_defaults(func=_cmd_migrate)
+
+    args = parser.parse_args(argv)
+    return args.func(args)
+
+
+if __name__ == "__main__":
+    sys.exit(main())

src/compas_pb/core.py

@@ -411,6 +411,7 @@ def _check_version_compatibility(any_data: message_pb2.MessageData) -> None:
     if _wire_compat_key(incoming) != _wire_compat_key(_CURRENT_VERSION):
         raise ValueError(
             "Incompatible compas_pb wire format: message was written by version {} but this "
-            "reader is {}. The binary schema differs between these versions; re-serialize the "
-            "source or read it with a matching compas_pb version.".format(incoming, _CURRENT_VERSION)
+            "reader is {}. The binary schema differs between these versions; migrate the data "
+            "with `compas_pb migrate <file> -o <file>` or read it with a matching compas_pb "
+            "version.".format(incoming, _CURRENT_VERSION)
         )