diff --git a/.dev-loop/INGEST_REPORT.md b/.dev-loop/INGEST_REPORT.md index 55ccfd1..f4d1b82 100644 --- a/.dev-loop/INGEST_REPORT.md +++ b/.dev-loop/INGEST_REPORT.md @@ -1,53 +1,214 @@ -# Knowledge consolidation — 15 open PRs (#17–#40) → one reconciled state +# Knowledge flush — 4 insight(s): 2 ingested, 1 folded into #73, 1 dropped as an in-flight duplicate -The 15 open `knowledge/*` PRs (created 2026-08-04 → 2026-08-05, before the -harvest processed-store dedupe fix in #41) contained 123 file-versions of ~75 -unique pages, with the same insight landing at up to 3 different paths across -up to 8 PRs. Per-PR review would re-import those duplicates, so — as with the -#6–#13 consolidation — this branch carries the reconciled end-state and the 15 -PRs are closed in its favor. +> A fourth candidate was appended to the queue by the harvester while this flush was +> running; it was picked up on the pre-retirement re-read and processed here rather +> than left pending. It is insight 4 below. ## Verified best-practice -Every adopted page's sources were carried from its originating PR's flush, where -they were live-verified at flush time; no new URLs were introduced during -consolidation (checked mechanically: every `http(s)` URL in every merged page -appears in a source PR's diff; every added body line in amended pages traces to -a source PR hunk — orphan-line verification). Confidence fields were kept as the -originating flushes set them, except client-side-rate-limiting where the union -of provider-doc citations (Okta, Auth0, GitHub, OpenAI, RFC 6585) supports -`verified` for the load-bearing claims. One subagent's fabricated content (12 -files matching neither main nor any PR, with invented source URLs) was detected -by the same verification and replaced with true PR content. +### 1. PostgreSQL catalog statistics are not evidence about a table's current contents — **verified** (ingested) + +**Claim.** To find the newest rows of a large unindexed table, start the `ctid` tail +scan at `pg_relation_size(rel)/current_setting('block_size')::int`, not at +`pg_class.relpages`; bound the range probe by aggregating (`DISTINCT`/`max`/`count`) +rather than with `LIMIT n`; and read `last_analyze`/`n_mod_since_analyze` before +citing `pg_stats` most-common values as the value set. + +Every URL below was opened in this session and the quoted text read out of the +fetched page (they are reproduced verbatim in the page's Sources block): + +| Sub-claim | Source | What it says | +|---|---|---| +| `relpages` is a stale snapshot | postgresql.org/docs/current/catalog-pg-class.html | "This is only an estimate used by the planner. It is updated by `VACUUM`, `ANALYZE`, and a few DDL commands such as `CREATE INDEX`"; `reltuples` is `-1` when never vacuumed/analyzed | +| `pg_relation_size` measures actual bytes | postgresql.org/docs/current/functions-admin.html | "Computes the disk space used by one 'fork' … With one argument, this returns the size of the main data fork"; results "measured in bytes" | +| Block size is build-time, not 8192 by law | postgresql.org/docs/current/runtime-config-preset.html | `block_size` "is determined by the value of `BLCKSZ` when building the server. The default value is 8192 bytes" | +| `pg_stats` is a sample from the last analyze | postgresql.org/docs/current/sql-analyze.html | "For large tables, `ANALYZE` takes a random sample of the table contents"; "the statistics are only approximate" | +| Analyze recency is observable | postgresql.org/docs/current/monitoring-stats.html | `last_analyze`, `last_autoanalyze`, `n_mod_since_analyze`, `n_live_tup` definitions | +| ctid = (block, offset), and it moves | postgresql.org/docs/current/datatype-oid.html + ddl-system-columns.html | "A tuple ID is a pair (block number, tuple index within block)"; "a row's `ctid` will change if it is updated or moved by `VACUUM FULL`" | +| Efficient range scans need PG 14+ | postgresql.org/docs/release/14.0/ | "Allow efficient heap scanning of a range of `TIDs` … Previously a sequential scan was required for non-equality `TID` specifications" | + +**Field observation carried from the session** (PRD, `external_data.collected_from_seumter`, +read-only): `relpages` 43992 vs `pg_relation_size` 44897 blocks; `ctid > '(43970,0)' +LIMIT 20` returned only `20260619` while `DISTINCT` over `ctid > '(44880,0)'` +returned `20260719`; the `pg_stats` MCV list topped out at `20260427`; +`reltuples` 1,069,782 vs `n_live_tup` 1,092,465. + +**Corrections made to the raw candidate during verification** (the candidate is a +draft, not the page): the candidate hard-coded `/8192`, which the `block_size` +preset contradicts → the page divides by `current_setting('block_size')`. The +candidate also stated the tail-scan technique unconditionally; the `ctid` doc's +update/`VACUUM FULL` caveat and the PG14 release note bound it to append-mostly +tables on PG 14+, both now edge-case rows. No local Postgres was available in this +environment (no `psql`, docker daemon down), so the mechanism rests on the official +docs plus the field observation — no reproduction was fabricated. + +### 2. A mechanism fixture equal to the shipped default — **verified** (folded into #73) + +**Claim.** When a config knob's test fixture repeats the code's default value, +"override honoured" and "override ignored" produce the same output, so +`cfg.get("k", DEFAULT) → DEFAULT` is unkillable by any assertion. + +Mechanism is deductive (two paths with an identical observable) and was reproduced +in the field. Evidence opened and re-read in this session rather than taken from the +candidate's prose: `heal/heal_detector.py:30` `RENOTIFY_SEC_DEFAULT = 21600` and +`:225` `renotify = det.get("renotify_sec", RENOTIFY_SEC_DEFAULT)`; the tests now +carry `{"renotify_sec": 100}` (`heal/test_heal_detector.py:420,435`), the post-fix +state the candidate describes. Reported result: at fixture `21600` the +knob-deleting mutant left 36 tests GREEN; at fixture `100`, no assertion changed, +the same mutant went RED in 2 cases. The existing citation base of the target page +(PIT "Survived", Stryker mutator set) already covers the mutation vocabulary used. + +### 3. Python bytecode cache invalidates a mutation run — **verified but already carried** (dropped) + +The claim (equal-size same-second edits reuse cached bytecode; purge `__pycache__` +and run under `-B`; require a surviving no-op control) is correct and sourced — and +open PR **#52** already carries it in +`backend/python/language/bytecode-cache-staleness`, including the exact nuance the +candidate adds: "Both settings govern writing only … so a `.pyc` left on disk by an +earlier run is still validated and reused … Purge `__pycache__` once before the run +and keep `-B` set for the rest of it", cited to docs.python.org/3/using/cmdline.html. +Nothing in the candidate is absent there. + +### 4. A branch whose bookkeeping another writer on the same path repeats — **verified** (ingested) + +**Claim.** When the branch you are pinning writes a flag or counter that a later +loop, retry, or error handler on the same execution path also writes, assert it from +an input that leaves those other writers inert (empty collection, no error injected). +With both active, the observable is identical whether or not the branch ran, so +deleting the branch keeps the suite green while coverage and assertion counts rise. + +Mechanism is deductive (two writers, one observable) and reproduced in the field. +Evidence re-read from the source this session rather than from the queue row: +`heal/heal_detector.py:306-308` — the `secret_source == "none"` arm sets +`run_ok = False; send_failed = True` *before* the message loop, and the loop's own +failure handling sets the same two. Reported result: deleting the branch left all 42 +tests passing, because both `none` cases supplied trigger messages that fail inside +the loop; adding one `messages=[]` case (`heal/test_heal_detector.py:539,549`) turned +the same deletion RED in 1 test with no assertion changed. The target page's existing +citation base (PIT kill attribution, Google's mutation-testing post, Fowler on +coverage) already carries the vocabulary; no new external source was needed, and none +was invented. ## Existing-layer check -- Merged-main near-dup scan before consolidation: pairwise Jaccard over - title + "When this applies" across all 141 merged pages → **0 flagged pairs**; - previously merged content carries no duplication. -- Cross-PR dedup during consolidation: 10 duplicate clusters collapsed to one - canonical page each (rate limiting 8→1, call-site enumeration 7→folded into - the canonical merged in #20, stderr/exit-0 diagnostics 4→1, sysroot 2→1, - env-off-switch 2→1, completion predicates 2→1, robots.txt 2→1, - harness-mediated results 2→1, leaked artifacts 2→1, orchestration category - naming unified). Three near-pairs kept distinct after trigger comparison, - with mutual `related:` links (differential setup vs interpretation; expansion - semantics vs off-switch design; import-time tactics vs level choice). -- 24 existing pages received union-merged amendments; additions already present - in main (from #16/#20) were skipped, and all non-canonical `related:` ids - were remapped to canonical page ids (post-merge broken-link scan: 0). +Routed via `INDEX.md` → databases ("surveying live data to derive a rule") and +testing ("cases/assertions, test data"). + +Pages read: databases-data-survey-surveying-live-data-for-a-rule, +databases-query-optimization-reading-execution-plans, +databases-operations-autovacuum-and-wraparound, +databases-query-optimization-existence-and-count-checks, +databases-query-optimization-keyset-pagination, +testing-quality-harness-reverse-controls, +backend-python-language-bytecode-cache-staleness, +testing-data-test-data-and-isolation, +testing-quality-tests-that-cannot-fail + +Findings: + +- **Overlap, kept distinct:** `surveying-live-data-for-a-rule` also deals with + drawing conclusions from a survey, but its trigger is *empty-result ambiguity* + when deriving a mapping/enum rule (`GROUP BY` over zero rows). The new page's + trigger is *stale catalog estimates* standing in for heap contents. Different + question, different remedy; cross-linked both ways (the existing page's `related:` + now names the new one). +- **No duplicate found:** `grep` for `degenerate|equal to the default|same as the + default` across the merged wiki returned nothing; there is no page on catalog + statistics as an evidence source. +- **No conflicts flagged.** The new page's directives do not contradict any + directive on the pages read; `reading-execution-plans` is referenced inline for + the "probe runs against production" edge case. +- **Merged-page edit:** `surveying-live-data-for-a-rule` frontmatter `related:` only + (no body change). +- **Merged before created (insight 4):** `tests-that-cannot-fail` already owns the + "prove a test can fail" trigger and its mutation-outcome table already splits + "exactly the expected test reddens" / "the file reddens, the target test does not" + / "no test reddens". Insight 4 is the fourth outcome in that same table — an + assertion on the observable *exists* and still nothing reddens — so it became one + table row, one edge case, one Instead-of row and one Sources entry on that page + rather than a new page. `surviving-mutant-equivalence-triage` (open PR #52) was + read and considered as the alternative home; its step-1 table classifies a survivor + as uncovered / weak-test / equivalent, and this case is a fourth class, but placing + it there would put the row on an unmerged branch and split one table across two + pages. The two pages are already cross-linked. +- Candidate 1's target page (`bytecode-cache-staleness`) was read in both its merged + and its #52 form — the merged form alone would have justified an append, and the + #52 form makes even that redundant. That distinction is why the open-PR check + below, not the merged-layer check, decided this candidate. + +## Open-PR check + +Listed all 19 open `knowledge/*` heads (`gh pr list --search "head:knowledge/"`) and +took each PR's changed-file list, then fetched and read the heads whose files +overlapped a candidate's trigger space. Note: heads #72–#76 live on the fork +`dch0202-rsquare/dev-loop`, so `git fetch origin ` fails on them with +`couldn't find remote ref` — they were fetched from the `fork` remote instead. + +| Head (PR) | Overlapping files | Verdict for our candidates | +|---|---|---| +| #52 `…20260807-100149` | `backend/python/language/bytecode-cache-staleness`, `testing/quality/surviving-mutant-equivalence-triage`, `harness-reverse-controls` | **drop** candidate 3 — this branch already carries it, including the `-B`-governs-writing-only row | +| #73 `…co-kr-20260810-163633` | `testing/quality/default-values-under-test` (new page), `tests-that-cannot-fail`, `minimum-case-set` | **fold** candidate 2 — same subject, complementary side; pushed as commit `74215dc` to that branch and noted on the PR | +| #61 `…20260807-213244` | `testing/data/harness-vs-run-path-fixtures`, `harness-reverse-controls`, `test-data-and-isolation` | read in full — its fixture topic is an *absent* operand in a synthesized input, not a value equal to a default; no overlap | +| #49, #47 | `tests-that-cannot-fail`, `harness-reverse-controls` | file-level overlap only; neither touches fixture-vs-default or catalog statistics | +| #74 (`comparing-two-execution-plans`), #73 (`trigram-index-short-patterns`, `index-selection`) | databases pages | read the file lists; no page on catalog statistics or `ctid` scanning — candidate 3's target area is untouched by every open head | +| #76, #72, #69, #68, #66, #64, #62, #58, #57, #56, #55, #51, #50 | none in our trigger space | **new** — no bearing on candidate 1 | + +The late-arriving insight 4 was diffed against the same set: the only heads touching +`tests-that-cannot-fail` are #52, #49, #47 and #73, and none of them adds a +masked-observable case. Checked by retrieving each head's own version of that file +and searching it for the concept (`inert|also writes|same observable|only this +branch`): 0 hits on all four, with the file present on each (121/121/124/125 lines, +so the empty result is an absence of the content, not a failed retrieval). Verdict +**new**. + +Per-candidate verdicts: **2 new** (catalog statistics; masked branch bookkeeping), +**1 fold** (#73), **1 drop** (pending duplicate of #52). No sibling duplicate PR was +opened for either the folded or the dropped candidate. ## Routing decision -- New categories: `infrastructure/agent-orchestration` (5 pages; unified the - competing `orchestration`/`agent-orchestration` names), `databases/data-survey` - (1), `qa/deliverables` (1). All other pages route into existing categories. -- Canonical-path decisions: rate limiting → `backend/common/reliability/` - (sits beside timeouts-and-retries; 6 of 8 variants chose it); stderr - diagnostics → `platforms/processes/` (concern spans beyond shells); leaked - artifacts → `testing/data/artifact-leakage-from-a-suite`; call-site - enumeration → the existing `backend/common/change-impact/` page. -- All 38 new pages listed in their domain indexes (nearest-index rule; backend - routes via its python sub-index for bytecode-cache-staleness); INDEX.md domain - summaries updated for infrastructure/qa/databases. Full-wiki lint: frontmatter, - ids, related-links, index coverage, size, qualifiers, staleness → 0 findings. +| Insight | Target | Rationale | +|---|---|---| +| Catalog statistics as current state | `databases` / `data-survey` / `catalog-statistics-as-current-state.md` (new page, existing category) | The task is "survey a live table to establish a fact", which is what `data-survey` owns. `operations` was considered and rejected: that category is about *running* VACUUM/ANALYZE (bloat, wraparound tuning), while this page is about reading their leftovers as evidence. `query-optimization` was rejected because the `ctid` scan here is a probe, not a query being made faster. No new category needed | +| Mechanism fixture vs shipped default | `testing` / `quality` / `default-values-under-test.md` — **on PR #73's branch, not here** | The page is the mirror subject and is still in flight; a second page with an adjacent trigger is exactly the pile-up the flush protocol warns about | +| Masked branch bookkeeping | `testing` / `quality` / `tests-that-cannot-fail.md` (merged into the existing page) | Merge-before-create: the page owns this trigger and its mutation-outcome table needed a fourth row rather than a second page. No new category | +| Python bytecode cache in a mutation harness | none — retired | Already on PR #52 | + +Plumbing updated on this branch: `wiki/databases/index.md` (new `data-survey` row +with its load-when line), `log.md` (dated ingest entry recording all four verdicts), +`surveying-live-data-for-a-rule` `related:` back-link. Page body is 93 lines +(limit 120); all four `related:` ids and the one inline `[page-id]` reference were +resolved against `wiki/` before commit. + +Cross-Check: the six load-bearing claims of this report were re-verified against +their artifacts rather than from memory — (a) each cited PostgreSQL URL was fetched +in-session and the page's quoted strings match the fetched text; (b) `relpages` vs +`pg_relation_size` semantics come from the catalog doc, not from the candidate's +prose, which was corrected on two points (`/8192`, unbounded applicability); +(c) the #52 duplicate verdict was taken from the branch diff, not from its PR title; +(d) the #73 fold evidence was read out of `heal_detector.py:30,225` and +`test_heal_detector.py:420,435` in the worktree, not copied from the queue row; +(e) every `Pages read:` id was resolved to a file with `grep -rl "^id: …"`; +(f) the "no open head carries insight 4" claim was run, not assumed — each head's own +copy of the file was retrieved and searched, and each retrieval was confirmed +non-empty first. Not +verified: no reproduction of the catalog staleness was run locally — no `psql` and +no running Docker daemon in this environment — so that mechanism rests on the +official docs plus the recorded field observation, and the page says so. + +## Decision Log + +**의도** +- 큐의 3건을 각각 다른 결말로 보냈다 — 신규 페이지 1, 인플라이트 PR 폴드 1, 폐기 1. 형제 중복 PR을 만들지 않는 것이 이 플러시의 1순위 제약이었다. +- 신규 페이지를 `databases/data-survey`에 둔 이유: "살아있는 테이블을 조사해 사실을 확정한다"가 그 카테고리의 소유 영역이고, 기존 `surveying-live-data-for-a-rule`(빈 결과 해석)과는 트리거가 다르다. 양방향 `related` 링크로 연결했다. +- 후보 원문을 그대로 싣지 않고 문서 검증 중 2곳을 교정했다: `/8192` 하드코딩 → `current_setting('block_size')`(BLCKSZ는 빌드타임), 그리고 무조건 적용 → append-mostly + PG14+ 전제를 edge case로 명시. + +**배제한 대안** +- 후보 3(파이썬 bytecode 캐시)로 페이지를 만들거나 머지된 페이지에 append하는 안 → 배제. 열린 PR #52가 `-B`는 쓰기만 막고 기존 `.pyc`는 계속 읽힌다는 핵심 뉘앙스까지 이미 담고 있어 새로 추가할 델타가 0이었다(브랜치 diff로 확인, PR 제목이 아니라). +- 후보 2를 이 PR에 신규 페이지로 넣는 안 → 배제. 대상 페이지(`default-values-under-test`)가 PR #73에 인플라이트라 트리거가 인접한 페이지가 두 개 생긴다. #73 브랜치에 커밋 `74215dc`로 폴드하고 PR에 코멘트를 남겼다. +- 신규 페이지를 `databases/operations`에 두는 안 → 배제. 그 카테고리는 VACUUM/ANALYZE를 *운영*하는 쪽이고, 이 페이지는 그 산물을 *증거로 읽는* 쪽이다. + +**리뷰어가 볼 곳** +- `wiki/databases/data-survey/catalog-statistics-as-current-state.md` — 인용문이 실제 PostgreSQL 문서 문장과 일치하는지(8개 URL 전부 이 세션에서 fetch), 그리고 `ctid` 꼬리 스캔의 전제(append-mostly, PG14+)가 edge case로 충분히 좁혀졌는지. +- 로컬 재현은 없다 — 이 환경에 `psql`이 없고 Docker 데몬이 죽어 있어 문서 검증 + 현장 관측(PRD read-only)으로만 뒷받침했고, 페이지와 리포트 양쪽에 그렇게 적었다. [추정] 아님, 미실행 사실 그대로. +- `wiki/databases/index.md`의 load-when 줄이 페이지의 "When this applies"와 어긋나지 않는지(drift 방지). diff --git a/log.md b/log.md index c930fc2..fd3e289 100644 --- a/log.md +++ b/log.md @@ -43,3 +43,4 @@ Append-only. Format: `## [YYYY-MM-DD] '(,0)'::tid` — aggregate over the range, so every block in it contributes | +| Does any row newer than X exist? | `SELECT max(col) FROM t WHERE ctid > '(,0)'::tid`, widening N until the answer stops moving | +| How many rows sit in this range? | `count(*)` over the same predicate — a zero here is unambiguous, an empty `DISTINCT` is not | + +4. **When you bound a range probe with `LIMIT n`, read the result as "the first n + rows at the start of the range", not as "the range's contents".** The scan walks + the range in ascending block order, so `LIMIT` truncates at its oldest end — the + newest blocks are exactly what it drops. + +5. **Before citing `pg_stats` (most-common values, histogram) as the value set, read + `last_analyze`, `last_autoanalyze`, and `n_mod_since_analyze` from + `pg_stat_all_tables`.** ANALYZE "takes a random sample of the table contents, + rather than examining every row" and its output is "only approximate"; when the + last analyze is old or `NULL`, values that arrived since are absent from the MCV + list by construction, not by absence in the table. + +6. **Cross-check `reltuples` against `n_live_tup` and treat a gap as unmeasured + inflow.** `reltuples` moves only at VACUUM/ANALYZE/DDL, while `n_live_tup` is the + cumulative-statistics estimate; the difference is the signal that the catalog view + of the table is behind the table. + +7. **Report which probe produced the answer, with the block range and the analyze + timestamps you read.** A tail-scan answer is a statement about the blocks scanned; + without the range it reads as a statement about the table. + +## Edge cases + +| Case | Then | +|------|------| +| The table takes UPDATEs or DELETEs, not append-only inserts | Physical order stops tracking insert order — a row's `ctid` "will change if it is updated or moved by `VACUUM FULL`", and freed space is refilled by later writes. Confirm the append-mostly assumption (`n_tup_upd`/`n_tup_del` near zero) before reading the tail as "newest" | +| The server is PostgreSQL 13 or older | Range predicates on `ctid` fall back to a sequential scan; efficient TID range scanning arrived in 14 ("Previously a sequential scan was required for non-equality `TID` specifications"). Bound the cost another way, or accept the scan | +| `reltuples` is `-1` | The table "has never yet been vacuumed or analyzed" and the row count is unknown — no catalog-derived count exists to compare against | +| The last blocks are empty of visible rows | The tail can hold dead tuples or free space; widen the range downward rather than concluding the table stopped receiving rows | +| The size you need includes TOASTed values | `pg_relation_size` with one argument returns the main fork only; `pg_table_size` adds TOAST, FSM, and visibility map. Block arithmetic for `ctid` uses the main fork | +| You may run maintenance on the table | `ANALYZE ` refreshes the statistics and makes `pg_stats`/`relpages` answer the question directly — take this path when the table is not so large that the analyze cost matters | +| The probe runs against production | Keep every range bounded by `ctid` and read `EXPLAIN` before executing; an unbounded probe on the same column is the full scan you were avoiding ([databases-query-optimization-reading-execution-plans]) | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Start a tail scan at `relpages` | Start at `pg_relation_size(rel)/current_setting('block_size')::int` | `relpages` is the planner's estimate from the last VACUUM/ANALYZE; blocks appended since sit above it and are never scanned | +| Read "the MCV list's newest value is April" as "nothing arrived after April" | Read `last_analyze`/`last_autoanalyze` first, and re-probe the heap when they are old or `NULL` | The MCV list is a sample from the moment of the last analyze, so recency of data and recency of statistics are different facts | +| Bound a `ctid` range probe with `LIMIT 20` and read the values as the range's | Aggregate the range (`DISTINCT`, `max`, `count`) or narrow it to the last blocks | The scan returns the range's oldest rows first, so `LIMIT` yields the end you were not asking about | +| Hard-code `/ 8192` in the block arithmetic | Read `current_setting('block_size')` | `BLCKSZ` is a build-time choice; the arithmetic silently points at the wrong block on a non-default build | +| Conclude "the ingest stopped" from one cheap catalog probe | Name the probe and its range in the finding, then confirm with a second probe of a different kind | Catalog estimates and heap contents are different sources; agreement between two of them is what makes the conclusion an observation | + +## Sources + +- https://www.postgresql.org/docs/current/catalog-pg-class.html — `relpages`: "Size of the on-disk representation of this table in pages (of size `BLCKSZ`). This is only an estimate used by the planner. It is updated by `VACUUM`, `ANALYZE`, and a few DDL commands such as `CREATE INDEX`"; `reltuples` carries the same estimate/update wording and "If the table has never yet been vacuumed or analyzed, `reltuples` contains `-1` indicating that the row count is unknown" +- https://www.postgresql.org/docs/current/functions-admin.html — `pg_relation_size(relation regclass [, fork text]) → bigint` "Computes the disk space used by one 'fork' of the specified relation ... With one argument, this returns the size of the main data fork"; results "are measured in bytes"; `pg_table_size` "Computes the disk space used by the specified table, excluding indexes (but including its TOAST table if any, free space map, and visibility map)" +- https://www.postgresql.org/docs/current/runtime-config-preset.html — `block_size`: "Reports the size of a disk block. It is determined by the value of `BLCKSZ` when building the server. The default value is 8192 bytes" +- https://www.postgresql.org/docs/current/sql-analyze.html — "For large tables, `ANALYZE` takes a random sample of the table contents, rather than examining every row"; "the statistics are only approximate, and will change slightly each time `ANALYZE` is run"; the collected statistics are "a list of some of the most common values in each column and a histogram showing the approximate data distribution" +- https://www.postgresql.org/docs/current/monitoring-stats.html — `pg_stat_all_tables`: `n_live_tup` "Estimated number of live rows", `n_mod_since_analyze` "Estimated number of rows modified since this table was last analyzed", `last_analyze` "Last time at which this table was manually analyzed", `last_autoanalyze` "Last time at which this table was analyzed by the autovacuum daemon" +- https://www.postgresql.org/docs/current/ddl-system-columns.html — `ctid`: "The physical location of the row version within its table. Note that although the `ctid` can be used to locate the row version very quickly, a row's `ctid` will change if it is updated or moved by `VACUUM FULL`" +- https://www.postgresql.org/docs/current/datatype-oid.html — "`tid`, or tuple identifier (row identifier) ... A tuple ID is a pair (block number, tuple index within block) that identifies the physical location of the row within its table" +- https://www.postgresql.org/docs/release/14.0/ — Optimizer: "Allow efficient heap scanning of a range of `TIDs` (Edmund Horner, David Rowley) ... Previously a sequential scan was required for non-equality `TID` specifications" +- Field observation 2026-08-12 (PostgreSQL, `external_data.collected_from_seumter`, ~1.07M rows, no index on the batch-date column): `relpages` = 43992 while `pg_relation_size` gave 44897 blocks — 905 blocks appended since the last maintenance. `ctid > '(43970,0)' LIMIT 20` returned only `20260619`, while `SELECT DISTINCT` over `ctid > '(44880,0)'` returned `20260719`. The `pg_stats` MCV list topped out at `20260427`, missing three months of arrivals; `reltuples` (1,069,782) trailed `n_live_tup` (1,092,465) by ~22.7k rows, which was the cross-signal that the catalog was behind the heap diff --git a/wiki/databases/data-survey/surveying-live-data-for-a-rule.md b/wiki/databases/data-survey/surveying-live-data-for-a-rule.md index 604279f..49baaa6 100644 --- a/wiki/databases/data-survey/surveying-live-data-for-a-rule.md +++ b/wiki/databases/data-survey/surveying-live-data-for-a-rule.md @@ -8,7 +8,7 @@ sources: - https://www.postgresql.org/docs/current/functions-aggregate.html - https://greatexpectations.io/blog/exploring-data-quality-volume/ last_verified: 2026-08-05 -related: [databases-query-optimization-existence-and-count-checks, databases-schema-design-requirements-to-tables, databases-schema-design-nullability-and-defaults] +related: [databases-data-survey-catalog-statistics-as-current-state, databases-query-optimization-existence-and-count-checks, databases-schema-design-requirements-to-tables, databases-schema-design-nullability-and-defaults] --- # Surveying Live Data to Derive a Mapping or Normalization Rule diff --git a/wiki/databases/index.md b/wiki/databases/index.md index 66083a7..9661a8e 100644 --- a/wiki/databases/index.md +++ b/wiki/databases/index.md @@ -50,6 +50,7 @@ Match your situation to a "load when" line; load only matching pages. | Page | Load when | |------|-----------| +| [catalog-statistics-as-current-state](data-survey/catalog-statistics-as-current-state.md) | Finding a large PostgreSQL table's newest rows when the column has no index, so you reach for `pg_class.relpages`/`reltuples`, `pg_stats` most-common values, or a `ctid` range scan of the last blocks; about to conclude "no recent data" from a cheap catalog probe; choosing the scan's starting block or how to bound a `ctid` range | | [surveying-live-data-for-a-rule](data-survey/surveying-live-data-for-a-rule.md) | A task says to sample real data to decide a mapping table, normalization/canonicalization rule, enum value set, or parsing rule; a `GROUP BY`/`DISTINCT` survey came back with zero rows; deciding what evidence replaces the data when the table is empty; recording in the deliverable which evidence a rule was actually derived from | ## sqlite diff --git a/wiki/testing/quality/tests-that-cannot-fail.md b/wiki/testing/quality/tests-that-cannot-fail.md index b0534af..0b90b06 100644 --- a/wiki/testing/quality/tests-that-cannot-fail.md +++ b/wiki/testing/quality/tests-that-cannot-fail.md @@ -15,7 +15,7 @@ sources: - https://www.gnu.org/software/sed/manual/html_node/Exit-status.html - https://git-scm.com/docs/git-checkout - https://git-scm.com/docs/git-restore -last_verified: 2026-08-05 +last_verified: 2026-08-12 related: [testing-quality-minimum-case-set, testing-quality-behavior-not-implementation, testing-mocking-what-to-mock, testing-async-async-testing, testing-quality-checks-that-cannot-pass, testing-quality-spec-artifact-checks, testing-quality-harness-reverse-controls, testing-quality-schema-additions-under-a-golden-gate, testing-quality-differential-run-agreement, testing-quality-completion-predicates, testing-quality-guard-shape-vs-consequence, testing-quality-injected-clock-duration-assertions, testing-quality-write-path-assertions, backend-common-change-impact-call-site-enumeration, platforms-shells-portable-shell-scripts, qa-document-verification-spec-document-gates] --- @@ -45,6 +45,7 @@ suite reported as covered, or you are auditing a suspiciously green suite. | Exactly the expected test reddens | That assertion discriminates on that input | Record the pair (mutation → test) and move on | | The file reddens but the target test stays green | The target assertion is unproven; another one caught the mutation | Mutate what this assertion reads, not what its name suggests | | No test reddens | Nothing asserts that behavior | Add the assertion, then re-run the mutation | +| No test reddens, and an assertion on that exact observable already exists | Another writer on the same execution path sets the same observable, so no case can attribute it to this branch | Add a case where only this branch acts — for a branch that runs before a loop, an input with zero items to process — then re-run and require red | 3. **Choose the restore mechanism by whether the work under test is committed, before you mutate anything.** `git checkout -- ` replaces the file with @@ -86,6 +87,7 @@ suite reported as covered, or you are auditing a suspiciously green suite. | Auditing a whole suite, not one test | Run an automated mutation-testing tool (PIT, Stryker) and treat surviving mutants in changed code as missing or defective tests | | The mutation run is your own script rather than PIT/Stryker | Prove the harness discriminates before citing its score — a semantics-preserving no-op must survive ([testing-quality-harness-reverse-controls]) | | A test intentionally has no outcome assertion (smoke test: module loads, page renders) | Keep it only when the regression it guards manifests as a throw; name it as a smoke test so reviewers do not count it as behavior coverage | +| The branch under test writes a flag or counter that a later loop, retry, or error handler on the same path also writes | Assert it from an input that leaves the other writers inert (empty collection, zero retries, no error injected); with both active the observable is the same whether or not the branch ran, so deleting the branch entirely keeps the suite green | | The always-green test is a snapshot approved without reading | Snapshot rules → [testing-quality-behavior-not-implementation] | | Assertions were just pulled into a shared contract (base class, mixin, parameterised suite) so several subjects now run them | Re-prove each assertion against each subject: the assertion's scope stayed where it was written while its name now speaks for every subject — one mutation per (assertion, subject) pair is the granularity | | The two implementations under comparison model different amounts of state | An agreement verdict on the default input cannot fail for the unmodelled dimension → [testing-quality-differential-run-agreement] | @@ -103,6 +105,7 @@ suite reported as covered, or you are auditing a suspiciously green suite. | Add `expect(result).toBeDefined()` to give a test "an assertion" | Assert the specific value/shape the behavior guarantees | `toBeDefined` on an always-defined value passes for every behavior, including broken | | Prove an error path with `try { await f() } catch (e) { expect(e.message)... }` alone | Use `rejects`/`assertThrows`-style assertion, or add `expect.assertions(1)` above the try | When `f()` succeeds, the catch never runs and the test passes with zero assertions | | Trust "green suite + high coverage" as proof an area is tested | Break the behavior once and require a red run | Coverage counts execution, not detection; high numbers are reachable with assertion-free tests | +| Read "the branch's tests pass and its lines are covered" as the branch being guarded | Delete the branch and require a test to redden; when none does, add an input that isolates it | A co-occurring writer of the same flag makes coverage and assertion counts rise while nothing pins the branch's own contribution | | Delete a suspicious always-green test to clean up | Fix it via the table above, then re-verify it can fail | The test names a behavior someone meant to guard; deletion drops the intent along with the defect | | Prove a test file can fail by seeding one mutation and watching the file go red | Seed one mutation per assertion and require exactly the owning test to redden | A file-level red is produced by whichever assertion happens to be strictest; the silent ones remain unproven | | Pick a mutation from what the test's name says it covers | Pick it from what the assertion body actually reads | An assertion inherited into a shared contract keeps its original narrow scope, so a reasonable-looking mutation sails past it | @@ -119,3 +122,4 @@ suite reported as covered, or you are auditing a suspiciously green suite. - https://man7.org/linux/man-pages/man2/execve.2.html — the shebang is honoured only on direct execution, not when a file is passed to an interpreter - https://www.gnu.org/software/sed/manual/html_node/Exit-status.html — a `sed` expression that matches nothing still exits 0 - https://git-scm.com/docs/git-checkout, https://git-scm.com/docs/git-restore — `checkout -- ` restores the index copy, discarding unstaged changes; measured 2026-08-05: with the fix unstaged the checkout removed fix and mutation together, and the lost import surfaced as `Ran 1042 … errors=1` where the intact tree ran 1098 +- Field reproduction 2026-08-12 (a Python health-check daemon, `heal_detector.py`): the `secret_source == "none"` branch sets `run_ok = False; send_failed = True` before the message loop, and the loop's own failure handling sets the same two. Deleting the branch entirely left all 42 tests passing, because both `none` cases supplied trigger messages that fail in the loop. Adding one case with `messages=[]` — the branch's only active writer — turned the same deletion RED in 1 test, with no assertion changed