[CBRD-XXXXX] Keep the group-by input page fixed while its bytes are borrowed - #166
Closed
xmilex-git wants to merge 5 commits into
Closed
xmilex-git wants to merge 5 commits into
xmilex-git wants to merge 5 commits into
Conversation
…of float noise (CUBRID#7572) http://jira.cubrid.org/browse/CBRD-27139 Two defects made qo_plan_cmp() asymmetric, so the chosen plan depended on visit order: - Total-cost equality required both fixed and variable components to match exactly; equal totals with different splits compared LT from both argument orders. - Tie tests used exact float equality, so near-ties were decided by sub-epsilon noise instead of the tie-break rules. Compare total cost under a relative epsilon (QO_COST_EQ, 1e-6) and gate the rule-based index tie-breaks with the same epsilon. Verified with a 400-pair antisymmetry probe (LT<->GT / EQ<->EQ only); RBO band constants are unchanged.
…#7504) http://jira.cubrid.org/browse/CBRD-27071 Purpose When an index is created on a table containing data, the entire image of each B-tree page is recorded in the redo log (RVBT_COPYPAGE) whenever a page is created. For large tables, hundreds of thousands or more leaf pages may be generated, making this logging a bottleneck during index creation. To address this issue, this change introduces a parallel bulk index build that skips redo logging for index pages only. It is enabled only when the new hidden option --no-logging-index is specified for loaddb in CS mode. loaddb without this option and regular CREATE INDEX or ALTER statements continue to use the existing logged build. For a table with 341,000,000 rows, approximately 20 GB in size, the loaddb index loading time was reduced by a factor of 2.48 compared with develop (609.2 seconds to 245.5 seconds). Implementation load_db.c, util_admin.c: Adds the hidden --no-logging-index option to loaddb and passes it as a flag in the index loading request (BTREE_LOADINDEX). The server determines whether the flag is valid, and ignores it for requests from client types other than loaddb-related clients. btree_load.c: Adds the no-redo parallel build path. It reuses the existing external sort infrastructure to divide the sorted key space into as many ranges as there are workers. Each worker merges its assigned range from all runs and writes the leaf pages. The main thread then links the range boundaries and builds the upper levels. Completed pages are flushed immediately using write-through, allowing the build process and disk writes to overlap. px_parallel.cpp: The server determines the parallel degree based on the number of logical cores using the INDEX_BUILD type in compute_parallel_degree, with an internal maximum of 32. If there are fewer sorted runs than the calculated degree, the degree is reduced accordingly. If it is less than 2, the build falls back to the serial path. log_manager.c, log_recovery.c, recovery.c: Immediately after flushing the buffers and synchronizing the volume before commit, the system writes a payload-free barrier record (RVBT_BULK_BUILD_DURABLE). Crash recovery passes through the barrier as a no-op, while full replay by restoredb rejects replay beyond the barrier and displays a dedicated message in both English and Korean. Remarks After loading indexes with --no-logging-index, a new full backup must be taken. The previous backup chain cannot replay beyond the corresponding loaddb segment. SA mode, indexes created with WITH ONLINE, and builds that are determined to be serial, such as those with too few sorted runs or involving small or empty tables, use the existing logged build regardless of the option and do not write the barrier record. The three hunks in page_buffer.c contain the same fix as PR [CBRD-27084], issue CUBRID#7487 (CBRD-27084), which addresses an infinite page-fix spin or hang caused by waiter_exists not being cleared in the page buffer after waking the flush-waiting thread. If that PR is merged into develop first, these hunks will disappear during the rebase.
…CUBRID#7365) http://jira.cubrid.org/browse/CBRD-26976 Previously a GROUP BY ... LIMIT n query could not stop right after the limit was reached: it kept consuming input rows to accumulate and finalize the next group, only to discard it. This removes that wasted work by terminating group processing as soon as the LIMIT is filled. Also add a "readrows" field to the GROUPBY line of the trace output to show how many input rows were consumed by the GROUP BY.
…nds precede a query error (CUBRID#7605) <http://jira.cubrid.org/browse/CBRD-27162 > * Fixed a line number error when using the csql -i option. * Resolved an issue where the error line number was printed incorrectly when a session command was specified in the input file (via the -i option) and an error occurred in the SQL statement immediately following it.
…orrowed qexec_gby_put_next() hands a pointer into a fixed list-file page to qexec_gby_agg_tuple() with peek = PEEK, so fetch_val_list() makes the group-by DB_VALUEs borrow that page instead of copying. The page was then unfixed at the end of the same loop iteration, but those values are not consumed there: they are the group's output values and are read later by qexec_gby_finalize_group_dim(), when a subsequent tuple opens a new group or -- for the final group -- after sort_listfile() returns. Once the group-by sort spills, the unfixed frame is recycled before that read, so the group-by keys are materialised from unrelated bytes. The result is silent: row count and aggregate values stay correct while char/varchar group keys come back as binary garbage, and it is non-deterministic, so the same statement can return correct rows on one execution and corrupt rows on the next. Restore the fixed_page/fixed_vpid bookkeeping that struct groupby_state already carries and hold the borrowed page until its values have been consumed, releasing it just before they are overwritten, after the final finalize in qexec_groupby(), and on the error path in qexec_clear_groupby_state(). At most one page stays fixed at a time. Consecutive tuples of a sorted input frequently share a page, so the reuse check also removes redundant fix/unfix calls. The COPY paths -- overflow tuples and A_sort_key -- are unchanged; they already materialise their own copy.
Owner
Author
|
Recreating: the fork's develop was 4 upstream commits behind, so the diff included unrelated commits. develop has been fast-forwarded to a2c3e03 and the PR is reopened with the single commit. |
|
Failed to process |
|
Failed to process |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Sort-based
GROUP BYreturns silently wrong rows:char/varchargroup-by key columns come back as binary garbage while the row count and the aggregate values stay correct. No error, no warning.It needs a join, a
GROUP BYon columns from the joined side, andCOUNT(DISTINCT ...)— theDISTINCTaggregate makes the query hash-ineligible, so it falls to the sort path. It only shows once the group-by sort spills.Exactly 10 bytes of non-string data in a
CHAR(10)column. Non-deterministic: the same statement returns clean rows on one execution and corrupt rows on the next, and the corrupt rows differ every time.Root cause
qexec_gby_put_next()fixes an input list-file page and passes a pointer into it toqexec_gby_agg_tuple()withpeek = PEEK, sofetch_val_list()makes the group-byDB_VALUEs borrow the page rather than copy it. The page is then unfixed at the end of the same loop iteration:Those values are not consumed there. They are the group's output values, read later by
qexec_gby_finalize_group_dim()— when a subsequent tuple opens a new group, or, for the final group, aftersort_listfile()returns inqexec_groupby(). Once the sort spills, the unfixed frame is recycled before that read.The discriminator is spill, not buffer size. Same plan, same query,
;trace on:data_buffer_sizeonly decides whether the sort spills, which is why the effect is not monotonic in it.Fix
Hold the borrowed page until its values have been consumed.
struct groupby_statealready carriesfixed_page/fixed_vpidfor exactly this, disabled behind#if 0 /* SortCache */; this restores them and moves the release to:qexec_gby_finalize_group_dim()inqexec_groupby();qexec_clear_groupby_state().At most one page stays fixed at a time. The original VPID is kept in a separate
page_vpidbecauseQFILE_GET_OVERFLOW_VPID()overwritesvpid.The
COPYpaths — overflow tuples andA_sort_key— are unchanged; they already materialise their own copy.Forcing
peek = COPYwas the one-line alternative. It was not taken: extending the fix range also lets consecutive tuples that share a page skip the re-fix, and measured faster.Verification
OptDebug build (
11.5.0.2405-a2c3e03),data_buffer_size=16M,part_s1,000,000 rows joined topartsupp_s4,000,000 rows, 1,000 groups.TPC-H SF10 Q16 at
data_buffer_size=512M:a2c3e03607f1ee9Correctness: the fixed build's 27,840 Q16 rows are SHA-256 identical to the known-good result produced at
data_buffer_size=8192M, where the sort does not spill.No assertion fires in the OptDebug build before or after — the dangling read lands in a live buffer frame, so nothing detects it.
Performance, same OptDebug build and identical conf, 3 runs of the reproducer:
Release-build performance not measured.
Not a regression
Reproduces on
607f1ee9(2026-07-24) as well as ona2c3e03. Thepeek = PEEKshortcut and the per-tuple unfix are long-standing. In particular this is unrelated to theloaddb --no-logging-indexparallel index build (PR CUBRID#7504 / CBRD-27071): that code is not in607f1ee9, the affected database was never loaded through that path, and a full heap scan of the base table returns no bad rows.Reproducer
Full setup is in the linked issue's Repro section. Run it several times — it is non-deterministic.