Skip to content

[CBRD-XXXXX] Keep the group-by input page fixed while its bytes are borrowed - #166

Closed
xmilex-git wants to merge 5 commits into
developfrom
cbrd-groupby-peek-page-fix
Closed

xmilex-git wants to merge 5 commits into
developfrom
cbrd-groupby-peek-page-fix

Conversation

@xmilex-git

@xmilex-git xmilex-git commented Aug 7, 2026 •

Copy link
Copy Markdown
Owner

Problem

Sort-based GROUP BY returns silently wrong rows: char/varchar group-by key columns come back as binary garbage while the row count and the aggregate values stay correct. No error, no warning.

It needs a join, a GROUP BY on columns from the joined side, and COUNT(DISTINCT ...) — the DISTINCT aggregate makes the query hash-ineligible, so it falls to the sort path. It only shows once the group-by sort spills.

'SMALL BURNISHED NICKEL'           19                    62
'\x00\x00\x01\x00\x00\x00\x08\x00\x00\r'          '\x00\x00\x01\x00\x00\x00\x10...'

Exactly 10 bytes of non-string data in a CHAR(10) column. Non-deterministic: the same statement returns clean rows on one execution and corrupt rows on the next, and the corrupt rows differ every time.

Root cause

qexec_gby_put_next() fixes an input list-file page and passes a pointer into it to qexec_gby_agg_tuple() with peek = PEEK, so fetch_val_list() makes the group-by DB_VALUEs borrow the page rather than copy it. The page is then unfixed at the end of the same loop iteration:

qexec_gby_agg_tuple (thread_p, info, data, peek);
info->input_recs++;
#if 1  /* SortCache */
  if (page)
    qmgr_free_old_page_and_init (thread_p, page, list_idp->tfile_vfid);
#endif

Those values are not consumed there. They are the group's output values, read later by qexec_gby_finalize_group_dim() — when a subsequent tuple opens a new group, or, for the final group, after sort_listfile() returns in qexec_groupby(). Once the sort spills, the unfixed frame is recycled before that read.

The discriminator is spill, not buffer size. Same plan, same query, ;trace on:

clean:   GROUPBY (hash: false, sort: true, page: 128,   ioread: 0,     rows: 27840)
corrupt: GROUPBY (hash: false, sort: true, page: 12112, ioread: 80765, rows: 1000)

data_buffer_size only decides whether the sort spills, which is why the effect is not monotonic in it.

Fix

Hold the borrowed page until its values have been consumed. struct groupby_state already carries fixed_page / fixed_vpid for exactly this, disabled behind #if 0 /* SortCache */; this restores them and moves the release to:

  • just before the next tuple's values overwrite the borrowed ones (after that iteration's possible finalize);
  • after the final qexec_gby_finalize_group_dim() in qexec_groupby();
  • on the error path in qexec_clear_groupby_state().

At most one page stays fixed at a time. The original VPID is kept in a separate page_vpid because QFILE_GET_OVERFLOW_VPID() overwrites vpid.

The COPY paths — overflow tuples and A_sort_key — are unchanged; they already materialise their own copy.

Forcing peek = COPY was the one-line alternative. It was not taken: extending the fix range also lets consecutive tuples that share a page skip the re-fix, and measured faster.

Verification

OptDebug build (11.5.0.2405-a2c3e03), data_buffer_size=16M, part_s 1,000,000 rows joined to partsupp_s 4,000,000 rows, 1,000 groups.

corrupt runs
before 6/6
after 0/8

TPC-H SF10 Q16 at data_buffer_size=512M:

corrupt runs
before, a2c3e03 4/10
before, 607f1ee9 3/10
after 0/5

Correctness: the fixed build's 27,840 Q16 rows are SHA-256 identical to the known-good result produced at data_buffer_size=8192M, where the sort does not spill.

No assertion fires in the OptDebug build before or after — the dangling read lands in a live buffer frame, so nothing detects it.

Performance, same OptDebug build and identical conf, 3 runs of the reproducer:

wall
before 44.11 / 44.47 / 44.84 s
after 34.16 / 35.89 / 36.95 s

Release-build performance not measured.

Not a regression

Reproduces on 607f1ee9 (2026-07-24) as well as on a2c3e03. The peek = PEEK shortcut and the per-tuple unfix are long-standing. In particular this is unrelated to the loaddb --no-logging-index parallel index build (PR CUBRID#7504 / CBRD-27071): that code is not in 607f1ee9, the affected database was never loaded through that path, and a full heap scan of the base table returns no bad rows.

Reproducer

select p_brand, p_type, p_size, count(distinct ps_suppkey)
from partsupp_s, part_s
where p_partkey = ps_partkey
group by p_brand, p_type, p_size;

Full setup is in the linked issue's Repro section. Run it several times — it is non-deterministic.

soheejung-cs and others added 5 commits August 5, 2026 17:40
…of float noise (CUBRID#7572)

http://jira.cubrid.org/browse/CBRD-27139

Two defects made qo_plan_cmp() asymmetric, so the chosen plan depended
on visit order:
- Total-cost equality required both fixed and variable components to
  match exactly; equal totals with different splits compared LT from
  both argument orders.
- Tie tests used exact float equality, so near-ties were decided by
  sub-epsilon noise instead of the tie-break rules.

Compare total cost under a relative epsilon (QO_COST_EQ, 1e-6) and
gate the rule-based index tie-breaks with the same epsilon. Verified
with a 400-pair antisymmetry probe (LT<->GT / EQ<->EQ only); RBO band
constants are unchanged.
…#7504)

http://jira.cubrid.org/browse/CBRD-27071
Purpose
When an index is created on a table containing data, the entire image of each B-tree page is recorded in the redo log (RVBT_COPYPAGE) whenever a page is created. For large tables, hundreds of thousands or more leaf pages may be generated, making this logging a bottleneck during index creation.

To address this issue, this change introduces a parallel bulk index build that skips redo logging for index pages only. It is enabled only when the new hidden option --no-logging-index is specified for loaddb in CS mode. loaddb without this option and regular CREATE INDEX or ALTER statements continue to use the existing logged build. For a table with 341,000,000 rows, approximately 20 GB in size, the loaddb index loading time was reduced by a factor of 2.48 compared with develop (609.2 seconds to 245.5 seconds).

Implementation
load_db.c, util_admin.c: Adds the hidden --no-logging-index option to loaddb and passes it as a flag in the index loading request (BTREE_LOADINDEX). The server determines whether the flag is valid, and ignores it for requests from client types other than loaddb-related clients.

btree_load.c: Adds the no-redo parallel build path. It reuses the existing external sort infrastructure to divide the sorted key space into as many ranges as there are workers. Each worker merges its assigned range from all runs and writes the leaf pages. The main thread then links the range boundaries and builds the upper levels. Completed pages are flushed immediately using write-through, allowing the build process and disk writes to overlap.

px_parallel.cpp: The server determines the parallel degree based on the number of logical cores using the INDEX_BUILD type in compute_parallel_degree, with an internal maximum of 32. If there are fewer sorted runs than the calculated degree, the degree is reduced accordingly. If it is less than 2, the build falls back to the serial path.

log_manager.c, log_recovery.c, recovery.c: Immediately after flushing the buffers and synchronizing the volume before commit, the system writes a payload-free barrier record (RVBT_BULK_BUILD_DURABLE). Crash recovery passes through the barrier as a no-op, while full replay by restoredb rejects replay beyond the barrier and displays a dedicated message in both English and Korean.

Remarks
After loading indexes with --no-logging-index, a new full backup must be taken. The previous backup chain cannot replay beyond the corresponding loaddb segment.

SA mode, indexes created with WITH ONLINE, and builds that are determined to be serial, such as those with too few sorted runs or involving small or empty tables, use the existing logged build regardless of the option and do not write the barrier record.

The three hunks in page_buffer.c contain the same fix as PR [CBRD-27084], issue CUBRID#7487 (CBRD-27084), which addresses an infinite page-fix spin or hang caused by waiter_exists not being cleared in the page buffer after waking the flush-waiting thread. If that PR is merged into develop first, these hunks will disappear during the rebase.
…CUBRID#7365)

http://jira.cubrid.org/browse/CBRD-26976

Previously a GROUP BY ... LIMIT n query could not stop right after the
limit was reached: it kept consuming input rows to accumulate and
finalize the next group, only to discard it. This removes that wasted
work by terminating group processing as soon as the LIMIT is filled.

Also add a "readrows" field to the GROUPBY line of the trace output to
show how many input rows were consumed by the GROUP BY.
…nds precede a query error (CUBRID#7605)

<http://jira.cubrid.org/browse/CBRD-27162 >

* Fixed a line number error when using the csql -i option.
* Resolved an issue where the error line number was printed incorrectly when a session command was specified in the input file (via the -i option) and an error occurred in the SQL statement immediately following it.
…orrowed

qexec_gby_put_next() hands a pointer into a fixed list-file page to
qexec_gby_agg_tuple() with peek = PEEK, so fetch_val_list() makes the
group-by DB_VALUEs borrow that page instead of copying. The page was then
unfixed at the end of the same loop iteration, but those values are not
consumed there: they are the group's output values and are read later by
qexec_gby_finalize_group_dim(), when a subsequent tuple opens a new group
or -- for the final group -- after sort_listfile() returns.

Once the group-by sort spills, the unfixed frame is recycled before that
read, so the group-by keys are materialised from unrelated bytes. The
result is silent: row count and aggregate values stay correct while
char/varchar group keys come back as binary garbage, and it is
non-deterministic, so the same statement can return correct rows on one
execution and corrupt rows on the next.

Restore the fixed_page/fixed_vpid bookkeeping that struct groupby_state
already carries and hold the borrowed page until its values have been
consumed, releasing it just before they are overwritten, after the final
finalize in qexec_groupby(), and on the error path in
qexec_clear_groupby_state(). At most one page stays fixed at a time.
Consecutive tuples of a sorted input frequently share a page, so the
reuse check also removes redundant fix/unfix calls.

The COPY paths -- overflow tuples and A_sort_key -- are unchanged; they
already materialise their own copy.
@xmilex-git

Copy link
Copy Markdown
Owner Author

Recreating: the fork's develop was 4 upstream commits behind, so the diff included unrelated commits. develop has been fast-forwarded to a2c3e03 and the PR is reopened with the single commit.

@xmilex-git xmilex-git closed this Aug 7, 2026
@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown

⚠️ TC Branch Finalize Failed for cubrid-testcases-private-ex

Failed to process tc/pr-166.
Please check the workflow run for details.

@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown

⚠️ TC Branch Finalize Failed for cubrid-testcases

Failed to process tc/pr-166.
Please check the workflow run for details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants