From 9db33fdf26c7d847b8b8c90106828bf4a15504e8 Mon Sep 17 00:00:00 2001 From: Gonzalo Casas Date: Wed, 5 Aug 2026 15:23:32 +0200 Subject: [PATCH 1/2] Add a `compas_pb migrate` command for older serialized data The wire-version check is a hard gate, so data written by an incompatible version is refused outright. That was the right call -- field numbers are reused across format revisions, so an old blob read by a new build can silently misparse into plausible but wrong geometry -- but it left users with archived blobs no way forward beyond hand-rolling a two-environment dance. Both formats cannot be read in one process: the two builds generate protobuf descriptors from the same file path, and only one can register. So `migrate` decodes the blob in an ephemeral `uv` environment holding the version that wrote it, bridges through COMPAS JSON, and re-encodes with the current build. Nothing is installed into the caller's environment. The source version is read by scanning the top-level framing rather than by parsing the message. `ParseFromString` does recover the tag, but it eagerly misparses the whole payload to get one field, which is the exact hazard the gate exists to prevent. Scanning field 2 and skipping field 1 by its length prefix cannot fail on payload contents. Blobs written before v0.4.1 carry no version tag at all, hence `--from-version`. Verified against blobs written by real 0.5.0 and 0.4.10 installs: geometry, datastructures, attributes and explicitly set guids all survive. Two caveats are inherent to the old data and are documented rather than fixed -- float32 precision is not recovered, and integral floats come back as ints because that is how the old reader unwrapped protobuf `Value`. The test that spawns `uv` is marked `network` so it can be deselected in CI. Co-Authored-By: Claude Opus 5 --- CHANGELOG.md | 6 + docs/migration.md | 91 ++++++++++++++ mkdocs.yml | 1 + pyproject.toml | 6 + src/compas_pb/cli.py | 203 +++++++++++++++++++++++++++++++ src/compas_pb/core.py | 5 +- tests/test_cli.py | 78 ++++++++++++ tests/test_data/mesh_v0.5.0.data | Bin 0 -> 1082 bytes 8 files changed, 388 insertions(+), 2 deletions(-) create mode 100644 docs/migration.md create mode 100644 src/compas_pb/cli.py create mode 100644 tests/test_cli.py create mode 100644 tests/test_data/mesh_v0.5.0.data diff --git a/CHANGELOG.md b/CHANGELOG.md index 3cb9ade..585325b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,8 +9,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +* Added a `compas_pb migrate` command that re-encodes data written by an older, wire-incompatible version into the current format. It decodes the blob in an ephemeral `uv` environment holding the version that wrote it, bridges through COMPAS JSON, and re-encodes with the current build. Reads from a file or stdin, writes to a file or stdout, and `--inspect` reports which version wrote a blob without migrating it. +* Added `compas_pb.cli.migrate_bytes` and `compas_pb.cli.detect_wire_version` for the same migration from Python. +* Added a migration guide at `docs/migration.md`. + ### Changed +* Changed the incompatible-wire-format error to point at `compas_pb migrate` instead of only telling the user to re-serialize the source. + ### Removed diff --git a/docs/migration.md b/docs/migration.md new file mode 100644 index 0000000..a31f89d --- /dev/null +++ b/docs/migration.md @@ -0,0 +1,91 @@ +# Migrating older data + +`compas_pb` refuses to read data written by an incompatible version. If you have blobs written +by an older release, `compas_pb migrate` re-encodes them into the current wire format. + +```bash +compas_pb migrate old.pb -o new.pb +``` + +## Why old data is refused rather than best-effort parsed + +`compas_pb` reuses protobuf field numbers across format revisions. Version 1.0.0 changed what +those fields hold: coordinates went from `float` to `double`, mesh points went from one message +per point to a packed `repeated double`, faces moved to CSR form, and per-element attributes +became columnar. Protobuf does not reject a blob whose field numbers still line up, so an old +blob read by a new build can *silently misparse* into plausible-looking but wrong geometry. + +Deserialization therefore checks the version tag and raises rather than guessing. Compatibility +follows SemVer: under `0.x` every minor release is a break, and from `1.0` on only major bumps +are. So `1.0` and `1.2` interoperate, while `0.5` and `1.0` do not. + +## How migration works + +There is no way to read both wire formats in one process, the two builds generate protobuf +descriptors from the same file path, and only one can be registered at a time. So `migrate` +decodes the data in a throwaway environment holding the version that *wrote* it, bridges through +COMPAS JSON, and re-encodes with the current version: + +``` +old.pb ──> [ephemeral env, compas_pb 0.5.0] ──> COMPAS JSON ──> [this build] ──> new.pb +``` + +This needs [`uv`](https://docs.astral.sh/uv/) on `PATH`, and network access the first time a +given old version is fetched. Nothing is installed into your own environment. + +## Usage + +```bash +# Check what wrote a blob, without migrating it +compas_pb migrate old.pb --inspect + +# File in, file out +compas_pb migrate old.pb -o new.pb + +# Pipes work too +cat old.pb | compas_pb migrate - > new.pb + +# Blobs written before v0.4.1 carry no version tag, so name the version yourself +compas_pb migrate ancient.pb --from-version 0.3.1 -o new.pb + +# Pin the interpreter for the ephemeral environment +compas_pb migrate old.pb --python 3.12 -o new.pb +``` + +Migrating a directory is a shell loop: + +```bash +for f in data/*.pb; do compas_pb migrate "$f" -o "migrated/$(basename "$f")"; done +``` + +The same thing is available from Python: + +```python +from compas_pb.cli import migrate_bytes + +with open("old.pb", "rb") as f: + migrated = migrate_bytes(f.read()) +``` + +## What survives, and what does not + +Geometry, datastructures, attributes and explicitly set guids all come across. Two caveats are +inherent to the old data rather than to migration: + +- **Precision is not recovered.** Pre-1.0 coordinates were `float32` on the wire. Migration + preserves exactly what was stored, so a point written as `0.1` comes back as + `0.10000000149011612`. It will now round-trip losslessly from here on, but the precision the + old format discarded is gone. +- **Integral floats may arrive as ints.** Pre-1.0 routed values through `google.protobuf.Value`, + which returned whole numbers as `int`. A vertex attribute written as `3.0` migrates as `3`. + This is the old reader's behaviour, faithfully carried forward. + +Auto-generated guids are a non-issue in practice: a guid that was on the old wire is treated as +explicitly set when read back, so it survives the re-encode even though 1.0 no longer serializes +session-local ones. + +!!! note + + `compas_pb` output is not byte-for-byte reproducible: protobuf serializes map fields in an + unspecified order, so migrating the same blob twice yields equal data in different bytes. + Compare deserialized objects, not hashes. diff --git a/mkdocs.yml b/mkdocs.yml index b2fe5cb..7dd7b27 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -139,6 +139,7 @@ nav: - Installation: installation.md - Examples: examples.md - Missing COMPAS Type?: missing_compas_type.md + - Migrating Older Data: migration.md - API Reference: - compas_pb: reference/compas_pb.md - compas_pb.core: reference/compas_pb.core.md diff --git a/pyproject.toml b/pyproject.toml index a268ca0..914f319 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -67,6 +67,9 @@ docs = [ "tomli >=2.0; python_version < '3.11'", ] +[project.scripts] +compas_pb = "compas_pb.cli:main" + [project.urls] Homepage = "https://gramaziokohler.github.io/compas_pb" Documentation = "https://gramaziokohler.github.io/compas_pb" @@ -110,6 +113,9 @@ addopts = [ "--tb=short", "--import-mode=importlib", ] +markers = [ + "network: needs network access to fetch an older compas_pb into an ephemeral environment", +] doctest_optionflags = [ "NORMALIZE_WHITESPACE", "IGNORE_EXCEPTION_DETAIL", diff --git a/src/compas_pb/cli.py b/src/compas_pb/cli.py new file mode 100644 index 0000000..ca65ace --- /dev/null +++ b/src/compas_pb/cli.py @@ -0,0 +1,203 @@ +"""Command line interface for compas_pb.""" + +import argparse +import os +import shutil +import subprocess +import sys +import tempfile +from importlib.metadata import version + +import compas + +from compas_pb.api import pb_dump_bts +from compas_pb.api import pb_load_bts +from compas_pb.core import _wire_compat_key + +_CURRENT_VERSION: str = version("compas_pb") + +# Runs inside the ephemeral environment of the *writing* version, so it may only use API that +# version already had. ``pb_load`` and COMPAS JSON are the two things stable across all of them. +_DECODE_SCRIPT = """ +import sys +import compas +from compas_pb import pb_load + +compas.json_dump(pb_load(sys.argv[1]), sys.argv[2]) +""" + + +def _read_varint(buf: bytes, pos: int): + """Read a base-128 varint from ``buf`` at ``pos``, returning ``(value, new_pos)``.""" + result = 0 + shift = 0 + while True: + if pos >= len(buf): + raise ValueError("truncated varint") + byte = buf[pos] + pos += 1 + result |= (byte & 0x7F) << shift + if not byte & 0x80: + return result, pos + shift += 7 + if shift > 63: + raise ValueError("varint overflows 64 bits") + + +def detect_wire_version(blob: bytes): + """Read ``MessageData.version`` out of a blob without parsing its payload. + + The payload of an incompatible blob cannot be parsed by this build -- that is the whole + reason migration is needed -- so this walks only the top-level fields, skipping field 1 + (``data``) by its length prefix. ``version`` has been field 2 of ``MessageData`` since the + tag was introduced, and top-level framing has not changed since. + + Returns ``None`` for a blob written before the version tag existed. + """ + pos = 0 + end = len(blob) + while pos < end: + tag, pos = _read_varint(blob, pos) + field_no, wire_type = tag >> 3, tag & 0x07 + if wire_type == 0: + _, pos = _read_varint(blob, pos) + elif wire_type == 1: + pos += 8 + elif wire_type == 5: + pos += 4 + elif wire_type == 2: + length, pos = _read_varint(blob, pos) + if field_no == 2: + return blob[pos : pos + length].decode("utf-8") + pos += length + else: + raise ValueError("unsupported protobuf wire type {}; not a compas_pb message".format(wire_type)) + return None + + +def _decode_with(source_version: str, blob: bytes, python: str = None) -> str: + """Decode ``blob`` in an ephemeral environment holding ``source_version``, returning COMPAS JSON.""" + if shutil.which("uv") is None: + raise RuntimeError("`uv` is required to migrate older data but was not found on PATH. See https://docs.astral.sh/uv/getting-started/installation/") + + workdir = tempfile.mkdtemp(prefix="compas_pb_migrate_") + blob_path = os.path.join(workdir, "source.pb") + json_path = os.path.join(workdir, "bridge.json") + script_path = os.path.join(workdir, "decode.py") + try: + with open(blob_path, "wb") as f: + f.write(blob) + with open(script_path, "w") as f: + f.write(_DECODE_SCRIPT) + + cmd = ["uv", "run", "--no-project", "--quiet"] + if python: + cmd += ["--python", python] + cmd += ["--with", "compas_pb=={}".format(source_version), "python", script_path, blob_path, json_path] + + result = subprocess.run(cmd, capture_output=True, text=True, cwd=workdir) + if result.returncode != 0: + raise RuntimeError("failed to read the data with compas_pb {}:\n{}".format(source_version, (result.stderr or result.stdout).strip())) + + with open(json_path, "r") as f: + return f.read() + finally: + shutil.rmtree(workdir, ignore_errors=True) + + +def migrate_bytes(blob: bytes, source_version: str = None, python: str = None) -> bytes: + """Re-encode a blob written by an older compas_pb into the current wire format. + + Parameters + ---------- + blob : bytes + The serialized data to migrate. + source_version : str, optional + Version that wrote the blob. Detected from the blob when omitted, which is only + possible if it carries a version tag. + python : str, optional + Python version for the ephemeral environment, e.g. ``"3.12"``. + + Returns + ------- + bytes + The same data in the current wire format. + + """ + if source_version is None: + source_version = detect_wire_version(blob) + if source_version is None: + raise ValueError("this blob carries no version tag, so the version that wrote it cannot be detected; pass --from-version explicitly") + + if _wire_compat_key(source_version) == _wire_compat_key(_CURRENT_VERSION): + raise ValueError("data written by {} is already readable by this build ({}); no migration needed".format(source_version, _CURRENT_VERSION)) + + bridge = _decode_with(source_version, blob, python=python) + migrated = pb_dump_bts(compas.json_loads(bridge)) + + # Cheap proof the result is readable before it reaches the user's disk. + pb_load_bts(migrated) + return migrated + + +def _cmd_migrate(args) -> int: + blob = sys.stdin.buffer.read() if args.input == "-" else open(args.input, "rb").read() + if not blob: + print("error: no input data", file=sys.stderr) + return 1 + + if args.inspect: + detected = detect_wire_version(blob) + print("written by: {}".format(detected or "")) + print("this build: {}".format(_CURRENT_VERSION)) + if detected and _wire_compat_key(detected) == _wire_compat_key(_CURRENT_VERSION): + print("status: readable as-is") + else: + print("status: needs migration") + return 0 + + if args.output == "-" and sys.stdout.isatty(): + print("error: refusing to write binary data to a terminal; pass -o OUTPUT or redirect stdout", file=sys.stderr) + return 1 + + try: + migrated = migrate_bytes(blob, source_version=args.from_version, python=args.python) + except (ValueError, RuntimeError) as e: + print("error: {}".format(e), file=sys.stderr) + return 1 + + if args.output == "-": + sys.stdout.buffer.write(migrated) + else: + with open(args.output, "wb") as f: + f.write(migrated) + print("migrated {} -> {} ({} bytes)".format(args.input, args.output, len(migrated)), file=sys.stderr) + return 0 + + +def main(argv=None) -> int: + parser = argparse.ArgumentParser(prog="compas_pb", description="Utilities for compas_pb serialized data.") + subparsers = parser.add_subparsers(dest="command", required=True) + + migrate = subparsers.add_parser( + "migrate", + help="re-encode data written by an older compas_pb into the current wire format", + description=( + "Reads a blob written by an older compas_pb, decodes it in an ephemeral environment " + "holding that version, and re-encodes it with this one. Requires `uv` and network " + "access the first time a given version is fetched." + ), + ) + migrate.add_argument("input", help="path to the data to migrate, or - for stdin") + migrate.add_argument("-o", "--output", default="-", help="where to write the migrated data, or - for stdout (default)") + migrate.add_argument("--from-version", default=None, help="version that wrote the data; detected from the blob when omitted") + migrate.add_argument("--python", default=None, help="python version for the ephemeral environment, e.g. 3.12") + migrate.add_argument("--inspect", action="store_true", help="report the version that wrote the data and exit") + migrate.set_defaults(func=_cmd_migrate) + + args = parser.parse_args(argv) + return args.func(args) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/src/compas_pb/core.py b/src/compas_pb/core.py index 4d903a9..a82a278 100644 --- a/src/compas_pb/core.py +++ b/src/compas_pb/core.py @@ -411,6 +411,7 @@ def _check_version_compatibility(any_data: message_pb2.MessageData) -> None: if _wire_compat_key(incoming) != _wire_compat_key(_CURRENT_VERSION): raise ValueError( "Incompatible compas_pb wire format: message was written by version {} but this " - "reader is {}. The binary schema differs between these versions; re-serialize the " - "source or read it with a matching compas_pb version.".format(incoming, _CURRENT_VERSION) + "reader is {}. The binary schema differs between these versions; migrate the data " + "with `compas_pb migrate -o ` or read it with a matching compas_pb " + "version.".format(incoming, _CURRENT_VERSION) ) diff --git a/tests/test_cli.py b/tests/test_cli.py new file mode 100644 index 0000000..8675586 --- /dev/null +++ b/tests/test_cli.py @@ -0,0 +1,78 @@ +import pytest + +from compas_pb import pb_load_bts +from compas_pb.cli import detect_wire_version +from compas_pb.cli import main +from compas_pb.cli import migrate_bytes + + +@pytest.fixture +def legacy_blob(): + # Genuinely written by compas_pb 0.5.0, not synthesized: a mesh with mesh-level and + # vertex-level attributes plus a pointcloud, so it uses the pre-1.0 per-point message and + # map-based attribute layouts that this build can no longer parse. Regenerate with: + # uv run --no-project --with compas_pb==0.5.0 python -c "..." + with open("tests/test_data/mesh_v0.5.0.data", "rb") as f: + return f.read() + + +@pytest.fixture +def current_blob(): + with open("tests/test_data/frame.data", "rb") as f: + return f.read() + + +def test_detect_wire_version(legacy_blob, current_blob): + assert detect_wire_version(legacy_blob) == "0.5.0" + assert detect_wire_version(current_blob) == "1.0.0" + + +def test_detect_wire_version_without_tag(legacy_blob): + # Blobs written before v0.4.1 carry no version field at all. + stripped = legacy_blob[: legacy_blob.rfind(b"\x12\x050.5.0")] + assert detect_wire_version(stripped) is None + + +def test_detect_wire_version_rejects_garbage(): + with pytest.raises(ValueError): + detect_wire_version(b"\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff") + + +def test_migrate_refuses_current_data(current_blob): + with pytest.raises(ValueError, match="already readable"): + migrate_bytes(current_blob) + + +def test_migrate_needs_explicit_version_when_untagged(legacy_blob): + stripped = legacy_blob[: legacy_blob.rfind(b"\x12\x050.5.0")] + with pytest.raises(ValueError, match="no version tag"): + migrate_bytes(stripped) + + +def test_legacy_blob_is_refused_by_the_gate(legacy_blob): + with pytest.raises(ValueError, match="Incompatible compas_pb wire format"): + pb_load_bts(legacy_blob) + + +def test_inspect_reports_versions(legacy_blob, capsys): + assert main(["migrate", "tests/test_data/mesh_v0.5.0.data", "--inspect"]) == 0 + out = capsys.readouterr().out + assert "0.5.0" in out + assert "needs migration" in out + + +@pytest.mark.network +def test_migrate_legacy_blob(legacy_blob): + """Full migration through an ephemeral environment holding compas_pb 0.5.0.""" + migrated = migrate_bytes(legacy_blob) + + data = pb_load_bts(migrated) + mesh, cloud = data["mesh"], data["cloud"] + + assert mesh.attributes["label"] == "old-blob" + assert mesh.face_vertices(0) == [0, 1, 2, 3] + assert [mesh.vertex_attribute(v, "load") for v in mesh.vertices()] == [0, 1.5, 3, 4.5] + assert len(cloud.points) == 2 + # Guids that were on the old wire come back as explicitly set, so they survive the + # re-encode even though 1.0 no longer serializes auto-generated ones. + assert mesh.guid is not None diff --git a/tests/test_data/mesh_v0.5.0.data b/tests/test_data/mesh_v0.5.0.data new file mode 100644 index 0000000000000000000000000000000000000000..31c7e0890b5a1ee0f8a8182de34a0a9ce1e0db80 GIT binary patch literal 1082 zcmb`F&1w`u6oso}TF0vkZ3)7F8<}jBJE_0!suoQK2`>Brap^|tHx6{{ff>QL%F1_$ zD_=ly;RCpql?x0+xk*(=4E5u|(}_kD{(*$(5zv+ajN9 zt{v>}-&sA_f4Gis_x9F1wN@rbDw8mn5@n?=YFTmQPB;^VkfK(<)r9PhRtb4GLP9=| z;LTZB-VW2}#bFLV^M#N-O4sLQzvL;na7qRrQ7K-d6hUKMRVp?1sjE@ODpo9al0L#9 zYVT-NF*Vcgt%ZN3vIZ{?fDIBi^JKk%f zuV7FbW)ez^!U~aCc!}Cdk5=gz66BgzCp3x0?nd`mB3ua@IEENgKr0;@TWgGh>nNGf zp3G$QxrrQ(P>d-)o!nTGLwoX$c-cCW?Eoxe(Yte{lt_z;e?X zUtB28UncWaj5;%RI%YpMIx~7YW`uNR{B+EEqgxfoF54R5e^I+j?pJULvYlhlQnWOA T{mQ literal 0 HcmV?d00001 From 2e55acc932833d95a1300c859df5a7fa55ca78ed Mon Sep 17 00:00:00 2001 From: Gonzalo Casas Date: Wed, 5 Aug 2026 15:40:39 +0200 Subject: [PATCH 2/2] Build with uv so the migration tests can run `compas_pb migrate` shells out to uv to read data written by older versions, so the migration test fails on a runner without it. compas-actions.build v5 takes a `management_tool` input, which installs uv before running `invoke test`. Verified the full suite, including the migration test, on Python 3.9 -- the oldest cell in the matrix -- so the ephemeral environment resolves compas_pb 0.5.0 there too. Note that v5 skips the build entirely on macOS with Python 3.9, so that cell no longer runs. Co-Authored-By: Claude Opus 5 --- .github/workflows/build.yml | 3 ++- .github/workflows/release.yml | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index ffa79eb..c2f36b0 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -18,9 +18,10 @@ jobs: python: ["3.9", "3.10", "3.11", "3.12", "3.13"] steps: - - uses: compas-dev/compas-actions.build@v4 + - uses: compas-dev/compas-actions.build@v5 with: python: ${{ matrix.python }} + management_tool: uv invoke_lint: true invoke_test: true create-assets: diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 387b1df..0be553e 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -14,9 +14,10 @@ jobs: python: ["3.11", "3.12", "3.13"] steps: - - uses: compas-dev/compas-actions.build@v4 + - uses: compas-dev/compas-actions.build@v5 with: python: ${{ matrix.python }} + management_tool: uv invoke_lint: true invoke_test: true