feat(ingestion): bronze shape comes from Airbyte — append_dedup, retire RMT promotion - #2882
Open
cyberantonz wants to merge 4 commits into
Open
feat(ingestion): bronze shape comes from Airbyte — append_dedup, retire RMT promotion#2882cyberantonz wants to merge 4 commits into
cyberantonz wants to merge 4 commits into
Conversation
|
Important Review skippedToo many files! This PR contains 210 files, which is 110 over the limit of 100. To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. This review ran on the open-source allowance, not this organization's plan, because the pull request author doesn't have an assigned seat. Waiting won't change this — ask an organization admin to assign them a seat, or add seats in Billing if every seat is already assigned, then retry. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (210)
You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
ktursunov
approved these changes
Aug 27, 2026
Contributor
|
Are we sure we want to do it in the current release? |
…re RMT promotion (constructorfabric#2877) The destination-clickhouse 2.x connector performs no destination-side dedup work (upsertTable is a stub; append_dedup is the same append-only insert path) — it just creates the table as ReplacingMergeTree ORDER BY the primary key. So the bronze RMT shape moves to the connector: - reconcile builds catalogs with destinationSyncMode=append_dedup and primaryKey=[[unique_key]] for every keyed stream (append fallback for keyless ones); normalize_catalog_to_append.py renamed to normalize_catalog.py accordingly - bootstrap-db generation mirrors the same catalog rule, so the dumped snapshot is byte-identical to what a real sync creates - the promote_bronze_to_rmt macro, all 21 <connector>__bronze_promoted bootstrap models, their depends_on lines, schema.yml entries and the generation-time promotion step are deleted - snapshot regenerated: unique_key becomes NOT NULL String (the destination strips PK nullability), allow_nullable_key is gone, and the two previously-unpromoted tables (claude_team_invites, support_ticket_ids) converge to the uniform RMT shape - audit_rmt_read_dedup treats every bronze_* source read as RMT (verified against main: identical findings) - conventions updated (architecture rules 4 and 8, check-dbt-conventions checks 4 and 5, connector validate workflow); the obsolete constructorfabric#1886 promotion-selector regression test is removed with the model it guarded Warm environments converge on the first sync after the switch: the destination detects the primary-key nullability change and rebuilds the table itself (temp + copy + EXCHANGE), verified lossless on a disposable instance. Prerequisite audits and verification are recorded on constructorfabric#2877. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Anton Zelenov <antonz@constructor.tech>
…ilder A stream without a unique_key schema property would land as an ever-duplicating bronze table — a defect, not a degraded mode. All three catalog builders (reconcile normalize_catalog.py, bootstrap-db create-connector-tables.sh, the declarative-connector dev tool) now fail loudly naming the offending streams instead of falling back to plain append; the append fallback is removed and the conventions text updated to match. No current stream is affected: all 162 bronze tables carry unique_key. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Anton Zelenov <antonz@constructor.tech>
The 2.1.29 destination is the version instances should run (it adds the replicated-engine options needed for clustered ClickHouse). Snapshot regenerated with it is byte-identical to the committed one, so the CDK bump between 2.1.25 and 2.1.29 changed no type mappings and no snapshot diff accompanies the pin. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Anton Zelenov <antonz@constructor.tech>
Tables added on main since the branch diverged (compass, the new github streams and their silver/gold descendants) converge to the append_dedup shape: unique_key NOT NULL, allow_nullable_key dropped. Also sweeps the promotion references main introduced (compass__bronze_promoted and its schema.yml entry, depends_on headers in the new github/bitbucket models, connector-authoring skill instructions). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Anton Zelenov <antonz@constructor.tech>
cyberantonz
force-pushed
the
feat/2877-bronze-append-dedup
branch
from
September 1, 2026 11:28
9934d56 to
ae3318a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #2877 — bronze table shape moves from our promotion machinery to the Airbyte destination itself.
What changed
destinationSyncMode=append_dedup+primaryKey=[["unique_key"]]for every stream whose schema carriesunique_key(plainappendfallback + stderr warning for keyless streams).normalize_catalog_to_append.py→normalize_catalog.py; theab_create_connectioncontract comment and the dev-toolgenerate-catalog.shfollow the same rule (overwriteis gone everywhere).promote_bronze_to_rmtmacro, all 21<connector>__bronze_promotedbootstrap models, ~120-- depends_onlines, schema.yml entries, README/comment references. The obsolete Jira enrich crashes on every real sync (FINAL on a MergeTree bronze table) — no task metrics ever appear #1886 promotion-selector regression test goes with the model it guarded.dump-ddl.sh):unique_keybecomes NOT NULLStringon all 162 bronze tables (the destination strips PK nullability),allow_nullable_keyis dropped, engines/ORDER BY otherwise byte-identical — and the two previously-unpromoted tables (bronze_claude_team.claude_team_invites,bronze_zendesk.support_ticket_ids) converge to the uniformReplacingMergeTree(_airbyte_extracted_at) ORDER BY unique_keyshape.audit_rmt_read_dedup.pynow treats everybronze_*source read as RMT (behavior verified identical against main: same advisory findings) + pathlib/ruff cleanup.Why it's safe
All four prerequisite audits/verifications are recorded on #2877 (no connector emits a null
unique_key; every stream's version column falls back to_airbyte_extracted_atexactly as promotion pinned; all 1220 test fixture rows already comply). Warm environments converge on the first sync after the switch: the destination detects the primary-key nullability change and rebuilds the table itself (temp + copy +EXCHANGE) — verified lossless on a disposable ClickHouse 25.7.5 with the stock destination image.dbt parsegreen; field parity: 0 failures; pre-commit green.Closes #2877
🤖 Generated with Claude Code