Antalya 26.6: Antalya 26.3 port - improvements for cluster requests - #2133
Closed
zvonand wants to merge 2 commits into
Closed
Antalya 26.6: Antalya 26.3 port - improvements for cluster requests#2133zvonand wants to merge 2 commits into
zvonand wants to merge 2 commits into
Conversation
…next commit) --- Original cherry-pick message follows: Merge pull request #1687 from Altinity/feature/antalya-26.3/pr-1414-1 Antalya 26.3 port - improvements for cluster requests # Conflicts: # src/Common/ProfileEvents.cpp # src/Core/Settings.cpp # src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.cpp # src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFileIterator.cpp # src/Storages/ObjectStorage/StorageObjectStorageCluster.cpp # src/Storages/ObjectStorage/StorageObjectStorageSource.cpp # src/Storages/StorageFileCluster.cpp # src/Storages/StorageURLCluster.cpp # tests/integration/test_s3_cluster/test.py # tests/queries/0_stateless/03413_experimental_settings_cannot_be_enabled_by_default.sql
Kept both sides where antalya-26.6 and the port added independent code (ProfileEvents rows, settings declarations, obsolete settings, test cases, experimental-settings exception list). Adapted for antalya-26.6: - Dropped the moved local `deserializeFieldFromBinaryRepr` copy in ManifestFileIterator.cpp (the port moves it to IcebergFieldParseHelpers); the Variant early-return of the local copy is unreachable because the call site already skips Variant columns. - `StorageObjectStorageSource::createReader` on antalya-26.6 uses a `while (true)` loop with explicit `continue`/`break` (query condition cache handling) instead of the port's `do/while`, so the port's retry command handling uses `continue` directly and the `not_a_path` flag is not needed. - Skipped re-adding `extern const SettingsBool input_format_parquet_use_native_reader_v3` in StorageObjectStorageSource.cpp: it is an obsolete setting on antalya-26.6. - `StorageFileCluster`/`StorageURLCluster`: the new `FileTaskIterator`/ `UrlTaskIterator` classes get virtual columns from the storage metadata snapshot (`metadata->virtuals.getSampleBlock(...)`), since `getVirtualsList()` no longer exists on antalya-26.6. - Moved the antalya-26.6 `storage_cluster_read_sleep` failpoint from the removed task-iterator lambda into `TaskDistributor::operator()` in StorageObjectStorageCluster.cpp. - `test_read_constant_columns_optimization.py`: the profile-event expectation assumed the 26.1 behaviour where every Iceberg data file was fetched twice (`expected * 2`) because `ReadFileMetadata` did not use the local file cache. On antalya-26.6 each data file is fetched exactly once, so the expectation is `expected + 1` (local) / `expected + 4` (cluster, one metadata.json per subquery). The per-query read counts the test asserts (7 without the optimization; 5/0/4/3/0 with it) are unchanged.
zvonand
added
releasy
Member
Author
|
Superseded: this PR ported a single member of the Per the port graph (#2048) that group ports as one combined PR covering #1687, #1751, #1991, #1759. The group definition had been lost from the session config, so each member was queued as its own unit — that's now fixed and the combined PR will be opened by the next Closing in favour of the combined port. |
zvonand
added a commit
to Altinity/RelEasy
that referenced
this pull request
Jul 31, 2026
A group written to the deps overlay was silently deleted from every file `run` reads on the next `graph discover`, so its members each ported as a separate PR while the graph report and the issue kept displaying the group. On antalya-26.6 all six groups were lost this way: #1687 went out alone as Altinity/ClickHouse#2133 instead of as a 4-PR combined port, #1718 alone instead of 23, and pr-1843 / pr-1984 were left permanently gated behind `auto-grp-pr-1718`, a unit that no longer existed. The bit that distinguishes an overlay group from a hand-curated one didn't survive the round-trip: `PRGroupConfig.auto_discovered` had nowhere to go in `FeatureUnit`, so `_build_candidate_unit_set` set `is_user_group` from `is_group` alone and every overlay group came back marked user-declared. After that the group belonged to nobody — `_write_session_overlay` skipped it ("user groups live in the main session"), `_reconcile_session_user_groups` couldn't find it in `pr_sources.groups` and only warned, and `_collapse_components_to_groups` refused to re-merge its component. The discover summary reported the loss as `refresh: N removed since last run`, which reads like PRs landing in target. Carry the flag through `FeatureUnit.auto_discovered`, and split `_CandidateUnit.is_group` (cherry-picks several PRs as one) from `is_user_group` (hand-curated, don't rewrite). Overlay groups are now `is_group=True, is_user_group=False`, so the trial pick still runs combined and the overlay writer keeps re-emitting them. `_reconcile_session_user_groups` now creates a missing session group instead of warning: the overlay refuses to carry user groups, so skipping left the group in the report and the issue but in neither file `run` reads. It inherits `pr_policy.if_exists` rather than pinning PRGroupConfig's "skip". Group ids get an idempotent `_auto_group_id`. Now that a re-read auto group can be merged into a larger component again — the intended "the group simply grows" path, suppressed by this bug — the lead-id scheme would otherwise produce `auto-grp-auto-grp-pr-1687` and orphan its cache branch and state. `pr_membership.remove_pr` no longer claims `dropped from group 'X'` for an overlay group, where `save_session` filters the edit out. The veto is unaffected; it lands via exclude_prs, which discovery applies to members. 6 tests in tests/test_graph_update.py: the overlay round-trip (auto stays auto, user groups stay out of the overlay), group creation on reconcile, and `_auto_group_id`. Reverting just the `is_user_group` line reproduces the original failure — the group missing from the overlay. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changelog category (leave one):
CI/CD Options
Exclude tests:
Regression jobs to run:
Cherry-picked from #1687.
Cherry-picked from #1414, also has changes from #1597.