Skip to content

feat(data-quality): add W65 durable lifecycle store - #253

Merged
akhiabanchian merged 1 commit into
mainfrom
w65-data-quality-durable-store
Oct 4, 2026
Merged

akhiabanchian merged 1 commit into
mainfrom
w65-data-quality-durable-store

Conversation

@ammarheidari

@ammarheidari ammarheidari commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Authority

W65 / #214 Slice 4A is protected-main admitted via PR #252 at baseline 9572726f9880e82aaf1b836b0a8c1e330c5ca568.

Slice 4B scope

  • provider-neutral ADO lifecycle/evidence store;
  • SQLite standalone connection factory with WAL/foreign-key/busy-timeout settings;
  • PostgreSQL connection factory with HA-capable migration locking;
  • schema versioning under kafdeck_schema_info;
  • create vs revision-guarded replace/state update semantics;
  • stable bounded policy pagination;
  • idempotent aggregate/progress evidence append;
  • bounded evidence time-range queries;
  • SQLite round-trip/idempotency/pagination tests.

Safety

  • no Kafka reads or consumer-group membership;
  • no offset commits;
  • no raw key/value/header/payload persistence;
  • durable evidence remains aggregate/progress only;
  • no unbounded scans: policy and evidence queries retain contract ceilings;
  • no worker/API/UI activation in this slice.

Final corrective exact head: db3feabe59f8b4139b5dcf4900afe9094ceae952.

Refs #214 #209.

Corrective closure

  • policy scope is part of durable evidence identity;
  • state transitions use atomic CAS with UPDATE ... RETURNING;
  • evidence queries use a complete deterministic identity ordering;
  • shared kafdeck_schema_info creation is serialized with the same advisory lock as historical metrics;
  • policy persistence now uses an explicit persistence DTO rather than deserializing domain constructors directly.
  • historical metric maintenance now uses the same shared schema advisory lock, closing the remaining startup DDL race.
  • mutation operation, fleet mutation state, and connect auto-restart initializers now also join the shared kafdeck_schema_info advisory-lock protocol.
  • corrected the fleet initializer so shared schema transaction scope is confined to InitializeAsync; no schema transaction leaks into fleet conflict backfill.

Copy link
Copy Markdown
Contributor Author

@codex review

Please review exact head ccbcc033559df9a77bf324ae357dfffe2dbc19ad only. Focus on SQLite/PostgreSQL provider-neutral SQL, optimistic revision correctness, stable pagination, idempotent evidence identity, schema initialization concurrency, and proving no raw Kafka material can become durable.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 4, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-04T19:43:19.037749Z db3feab Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ccbcc03355

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

akhiabanchian
akhiabanchian previously approved these changes Oct 4, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

Please review corrective exact head 24d8958783ea8e32dfceba2e410d25c9d65cf9bf only. Verify the prior four findings and CI serialization failure are closed: scope-qualified evidence identity, atomic state CAS returning the winning definition, deterministic evidence ordering, shared schema migration lock, and explicit persistence DTO round-trip.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 24d8958783

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ammarheidari
ammarheidari force-pushed the w65-data-quality-durable-store branch from ac67e7d to 81407f9 Compare October 4, 2026 19:29

Copy link
Copy Markdown
Contributor Author

@codex review

Please review final corrective exact head 81407f9c9771bbfae8cd70909dcca8379107d584 only. Verify all prior findings are closed, including the remaining shared-schema startup race: data-quality, historical-metric, and historical-metric-maintenance initializers now serialize kafdeck_schema_info creation with the same advisory lock.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 81407f9c97

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ammarheidari
ammarheidari force-pushed the w65-data-quality-durable-store branch from 7f26e0a to 5ae2c21 Compare October 4, 2026 19:38

Copy link
Copy Markdown
Contributor Author

@codex review

Please review final exact head 5ae2c218963fea7c3faf6673fe73fc1e1d2e06a7 only. Verify all prior findings are closed and every PostgreSQL initializer that creates kafdeck_schema_info participates in the same shared advisory-lock protocol, including mutation operations, fleet mutation state, and connect auto-restart.

Signed-off-by: Ammar Heidari <ammar@arad-itc.org>
@ammarheidari
ammarheidari force-pushed the w65-data-quality-durable-store branch from 9708cae to db3feab Compare October 4, 2026 19:39

Copy link
Copy Markdown
Contributor Author

@codex review

Please review final exact head db3feabe59f8b4139b5dcf4900afe9094ceae952 only. Verify all previous persistence/concurrency findings remain closed, all shared-schema creators join the common advisory-lock protocol, and the fleet initializer compile regression from the prior head is fully corrected without leaking schema transaction scope into conflict backfill.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: db3feabe59

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@akhiabanchian
akhiabanchian merged commit 8d066a2 into main Oct 4, 2026
9 checks passed
@akhiabanchian
akhiabanchian deleted the w65-data-quality-durable-store branch October 4, 2026 20:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants