Skip to content

feat(display-format): decode Protobuf schemas and Schema Registry values - #3979

Merged
datlechin merged 30 commits into
TableProApp:mainfrom
J2TeamNNL:feat/3607-protobuf-schema-registry
Nov 1, 2026
Merged

datlechin merged 30 commits into
TableProApp:mainfrom
J2TeamNNL:feat/3607-protobuf-schema-registry

Conversation

@J2TeamNNL

@J2TeamNNL J2TeamNNL commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Display As > Protobuf Schema… on binary columns takes a .proto file or a descriptor set (protoc --descriptor_set_out), then asks which message when the file has several. A --include_imports set opens on the file that imports the others. The tree names each field and badges its declared type.
  • Text matches protoc --decode (libprotoc 36.2): map entries sorted by key, the last oneof member only, proto3 and editions fields at their default left out, a closed enum's undeclared number with the unknown fields, a singular message sent twice merged. Fields the schema does not name, or names with another wire type, show by number. A declared message or packed field that does not parse, or a proto3 string that is not UTF-8, fails the value as it fails protoc, so the cell stays hex and the viewer says why.
  • Display As > Schema Registry, offered when the connection names a registry. Each value's prefix gives the schema ID, fetched from GET /schemas/ids/{id} with its references from /subjects/{subject}/versions/{version} (Avro and Protobuf). Protobuf picks the message by the frame's message index; JSON Schema values open in the JSON tree.
  • The grid owns the fetches: it collects the column's schema IDs when rows load, rows arrive or the format is set, fetches each once, and redraws only that column when a batch lands. At most 4 fetches run at once and 256 wait, so a page of bytes that happen to start with zero cannot flood the registry. Formatting a cell never touches the network.
  • Schemas are kept for the session per registry URL and credentials, so an edited registry reads from the new one after the next connect. A failed fetch is kept 30 seconds; Refresh or opening the value asks again at once. The viewer waits for the fetch, never shows the previous value meanwhile, and redraws when a schema lands. A value filter on the column, in every tab and hidden or not, follows the text from stored bytes to decoded and across a registry switch.
  • Kafka connection settings gain Schema Registry URL, User and Password (Keychain) under Authentication, in the plugin and the curated snapshot. The app reads them by field id, so any driver can declare them, and an installed plugin from before them keeps the curated fields. Only http and https; responses are capped at 4 MB and 60 seconds.
  • Kafka: a value that starts with a zero byte makes the column binary. With an older plugin a framed value can land in a text column; the cell, inspector and double-click all decode it from its UTF-8 bytes.
  • SameOriginRedirectGuard replaces the MCP transport's private guard and is used by the registry client. It now treats http://h and http://h:80 as one origin; nothing else it allows changed.
  • Schema files: SchemaFileCache holds the shared file handling for AvroSchemaStore and ProtobufSchemaStore (ProtobufSchemas/<hash>.pbschema), remembers an unreadable file instead of reading it per cell, and choosing a lost schema again redraws the grid columns that name it (Avro too).
  • A .proto file is read alone (imports are not opened); a descriptor set built with --include_imports covers imported types. The registry is reached directly, not through the connection's SSH tunnel.

Test plan

  • ProtobufSchemaDecoderTests: fixtures from protoc --encode, expected text from protoc --decode (libprotoc 36.2), for a .proto and the descriptor set built from it: every scalar, enums, packed and unpacked repeats, maps with repeated keys, oneof in both orders, proto3 and proto2 presence, editions features.field_presence, closed enums, groups, merged messages, --include_imports primary file, a truncated descriptor set, optional as a type name, budgets inside unknown fields and nested messages.
  • SchemaRegistryTests: responses recorded from confluentinc/cp-schema-registry 7.7.1, values from confluent-kafka 2.6.1. Avro, Protobuf with a reference and message index [0] and [1], basic auth, the 404 reason, the pure cache read, one fetch per schema, the fetch bounds, the failure lifetime and reload retry, an endpoint change, no URL.
  • DataGridSchemaRegistryTests: prefetch on format change, on inserted rows and from a text column; a registry change or a restored schema file clears only the affected column.
  • SameOriginRedirectGuardTests, CellInteractionResolverBinaryTests, FieldEditorResolverTests, AvroSchemaStoreTests, ValueDisplayFormatTests, BinaryDisplayFormatTests, ValueDisplayFormatStorageTests, DataGridDisplayCacheTests, DataGridValueFilterTests.

Docs: TableProApp/docs#155 (draft, merges after this one).

Part of #3607

J2TeamNNL and others added 13 commits November 1, 2026 04:28
igbinary decodes into the PHP serialized tree, numbered as serialize() numbers it. Avro decodes with a schema file chosen per column, stored by a hash of its text.
…formats readable by 0.83

Fixes the review of TableProApp#3962: igbinary r:/R: numbering, copies, caps and float text; Avro framing, decimal width, named logical types and zero-byte caps; a second storage key for formats 0.83 lacks; schema files instead of defaults.
…hemas from a Schema Registry

Display As > Protobuf Schema… reads a .proto file or descriptor set and shows field names and declared types. Display As > Schema Registry fetches the Avro, Protobuf or JSON schema each Kafka value names by its wire-format prefix, from the registry set on the connection.
datlechin and others added 4 commits November 1, 2026 16:45
…hema-registry

# Conflicts:
#	apple/TablePro/Resources/Localizable.xcstrings
…schema-registry

# Conflicts:
#	apple/TablePro/Core/Services/Formatting/BinaryValues/AvroDecoder.swift
#	apple/TablePro/Core/Services/Formatting/BinaryValues/AvroSchema.swift
#	apple/TablePro/Core/Services/Formatting/BinaryValues/BinaryValueDecoding.swift
#	apple/TablePro/Core/Services/Formatting/BinaryValues/BinaryValueReader.swift
#	apple/TablePro/Core/Services/Formatting/ValueDisplayFormat.swift
#	apple/TablePro/Core/Services/Formatting/ValueDisplayFormatter.swift
#	apple/TablePro/Core/Storage/AvroSchemaStore.swift
#	apple/TablePro/Core/Storage/ValueDisplayFormatStorage.swift
#	apple/TablePro/Resources/Localizable.xcstrings
#	apple/TablePro/Views/DataGrid/Commands/DataGridViewController+HeaderMenu.swift
#	apple/TablePro/Views/Results/BinaryValueViewerView.swift
#	apple/TablePro/Views/Results/CellInteractionResolver.swift
#	apple/TablePro/Views/Shared/FieldEditors/FieldEditorResolver.swift
#	apple/TableProTests/Core/Services/Formatting/BinaryValues/BinaryDisplayFormatTests.swift
#	apple/TableProTests/Core/Services/Formatting/ValueDisplayFormatTests.swift
#	apple/TableProTests/Core/Storage/ValueDisplayFormatStorageTests.swift
auto-merge was automatically disabled November 1, 2026 14:28

Head branch was pushed to by a user without write access

… announce loads to every reader, parse dropped members in order
…on so the password reads back from the Keychain
…hema-registry

# Conflicts:
#	apple/TablePro/Resources/Localizable.xcstrings
@datlechin
datlechin merged commit e824f8f into TableProApp:main Nov 1, 2026
8 checks passed
datlechin added a commit that referenced this pull request Nov 2, 2026
* fix(security): tier SQL with one classifier for macOS and iOS

Discovered by Anthropic's OSS Scanner, as vulnerability ANT-2026-7WQZ7IBV.
Discovered by Anthropic's OSS Scanner, as vulnerability ANT-2026-85FM8CNF.
Discovered by Anthropic's OSS Scanner, as vulnerability ANT-2026-BO7086MJ.
Discovered by Anthropic's OSS Scanner, as vulnerability ANT-2026-DXB5GXHF.
Discovered by Anthropic's OSS Scanner, as vulnerability ANT-2026-RSL7CZQC.
Discovered by Anthropic's OSS Scanner, as vulnerability ANT-2026-TUTSEW23.
Discovered by Anthropic's OSS Scanner, as vulnerability ANT-2026-VEO0AP8A.
Advisory: GHSA-3x6q-h487-r38w

* fix(linux-grid): stop column moves at the row numbers, keep the focus in view and announce them (#3985)

* feat(linux-sidebar): list functions, procedures, sequences and triggers (#3969)

Co-authored-by: Ngo Quoc Dat <datlechin@gmail.com>

* feat(plugin-influxdb): add an InfluxDB 2 and 3 driver over HTTP (#3976)

Co-authored-by: Ngo Quoc Dat <datlechin@gmail.com>

* feat(plugin-qdrant): add a Qdrant driver over the REST API (#3972)

Co-authored-by: Ngo Quoc Dat <datlechin@gmail.com>

* feat(display-format): decode Protobuf schemas and Schema Registry values (#3979)

Co-authored-by: Ngo Quoc Dat <datlechin@gmail.com>

* refactor(core): move the Linux core crates into a shared core workspace (#3986)

* fix(security): import TableProSQLGrammar where the Flux classifier reads its tiers

* docs(core): record the bindings, grid data, plugin ABI and SQL Server decisions (#3987)

* fix(licensing): call the paid plan Pro, as the site and server now do (#3989)

* fix(parsing): stop malformed replies and files from crashing, hanging or filling memory (#3991)

---------

Co-authored-by: Nguyễn Nam Long <j2teamnnl@gmail.com>
datlechin added a commit that referenced this pull request Nov 2, 2026
* fix(security): tier SQL with one classifier for macOS and iOS

Discovered by Anthropic's OSS Scanner, as vulnerability ANT-2026-7WQZ7IBV.
Discovered by Anthropic's OSS Scanner, as vulnerability ANT-2026-85FM8CNF.
Discovered by Anthropic's OSS Scanner, as vulnerability ANT-2026-BO7086MJ.
Discovered by Anthropic's OSS Scanner, as vulnerability ANT-2026-DXB5GXHF.
Discovered by Anthropic's OSS Scanner, as vulnerability ANT-2026-RSL7CZQC.
Discovered by Anthropic's OSS Scanner, as vulnerability ANT-2026-TUTSEW23.
Discovered by Anthropic's OSS Scanner, as vulnerability ANT-2026-VEO0AP8A.
Advisory: GHSA-3x6q-h487-r38w

* fix(security): run external statements as approved and read only in the engine

Discovered by Anthropic's OSS Scanner, as vulnerability ANT-2026-AK0DTFEV.
Discovered by Anthropic's OSS Scanner, as vulnerability ANT-2026-2N5FN7X4.
Discovered by Anthropic's OSS Scanner, as vulnerability ANT-2026-A6ENWH1O.
Discovered by Anthropic's OSS Scanner, as vulnerability ANT-2026-AWAVHFV6.
Discovered by Anthropic's OSS Scanner, as vulnerability ANT-2026-DRI3Z3MD.
Discovered by Anthropic's OSS Scanner, as vulnerability ANT-2026-G0F69IT6.
Discovered by Anthropic's OSS Scanner, as vulnerability ANT-2026-KIUNV39X.
Discovered by Anthropic's OSS Scanner, as vulnerability ANT-2026-KYD5T4TJ.
Discovered by Anthropic's OSS Scanner, as vulnerability ANT-2026-LG65P706.
Discovered by Anthropic's OSS Scanner, as vulnerability ANT-2026-P5IW81B3.
Discovered by Anthropic's OSS Scanner, as vulnerability ANT-2026-PCUF2R3V.
Discovered by Anthropic's OSS Scanner, as vulnerability ANT-2026-UHVIMOF0.
Advisory: GHSA-3xf7-86qj-4546

* fix(l10n): add the iOS strings the MongoDB and ClickHouse refusals use

* fix(linux-clickhouse): hold user SQL and read-only connections to readonly=1

readonly=2 still let INSERT INTO FUNCTION file() write a file on the server (measured on 26.8).

Advisory: GHSA-3xf7-86qj-4546

* feat(plugin-influxdb): add an InfluxDB 2 and 3 driver over HTTP (#3976)

Co-authored-by: Ngo Quoc Dat <datlechin@gmail.com>

* feat(plugin-qdrant): add a Qdrant driver over the REST API (#3972)

Co-authored-by: Ngo Quoc Dat <datlechin@gmail.com>

* feat(display-format): decode Protobuf schemas and Schema Registry values (#3979)

Co-authored-by: Ngo Quoc Dat <datlechin@gmail.com>

* refactor(core): move the Linux core crates into a shared core workspace (#3986)

* fix(security): import TableProSQLGrammar where the Flux classifier reads its tiers

* fix(security): import TableProSQLGrammar where the gates read the shared classifier's tier

* docs(core): record the bindings, grid data, plugin ABI and SQL Server decisions (#3987)

* fix(licensing): call the paid plan Pro, as the site and server now do (#3989)

* fix(parsing): stop malformed replies and files from crashing, hanging or filling memory (#3991)

---------

Co-authored-by: Nguyễn Nam Long <j2teamnnl@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants