[ConfigNode] Fence Region creation across database deletion#18285
Open
CRZbulabula wants to merge 5 commits into
Open
[ConfigNode] Fence Region creation across database deletion#18285CRZbulabula wants to merge 5 commits into
CRZbulabula wants to merge 5 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR hardens ConfigNode’s database/RegionGroup lifecycle against races between Region creation/dispatch and database deletion by introducing a database-scoped lifecycle lock shared across requests and Procedures, adding durable fencing/cleanup for queued RegionCreateTasks, and strengthening RegionMaintainer scheduling/backoff to avoid dispatching stale work.
Changes:
- Added a database-name-scoped lifecycle lock manager and a base procedure type that holds database locks for the full procedure lifetime.
- Made DeleteDatabaseProcedure and CreateRegionGroupsProcedure fence/validate work under the lifecycle lock and introduced a durable consensus plan to batch-remove queued RegionCreateTasks for pre-deleted databases.
- Updated PartitionInfo/PartitionManager to validate batched CreateRegionGroups plans atomically, deduplicate and fence region-maintenance tasks, and apply batched scheduling with per-DataNode limits and backoff.
Reviewed changes
Copilot reviewed 25 out of 25 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| iotdb-core/confignode/src/test/java/org/apache/iotdb/confignode/procedure/scheduler/DatabaseLifecycleLockManagerTest.java | Adds unit tests for database lifecycle lock scoping, reentrancy, and procedure/request interaction. |
| iotdb-core/confignode/src/test/java/org/apache/iotdb/confignode/procedure/impl/schema/DeleteDatabaseProcedureTest.java | Extends delete-database procedure tests for child region-group deletion and transient consensus retry behavior. |
| iotdb-core/confignode/src/test/java/org/apache/iotdb/confignode/procedure/impl/CreateRegionGroupsProcedureTest.java | Adds tests for fenced validation, idempotent retry, cleanup behavior, and mutual exclusion with delete procedures. |
| iotdb-core/confignode/src/test/java/org/apache/iotdb/confignode/persistence/PartitionInfoTest.java | Adds tests for task cancellation idempotency, snapshot framing, atomic validation, and recreated-db safety. |
| iotdb-core/confignode/src/test/java/org/apache/iotdb/confignode/manager/ProcedureManagerTest.java | Adds test coverage for detecting unfinished database-lifecycle procedures. |
| iotdb-core/confignode/src/test/java/org/apache/iotdb/confignode/manager/ClusterSchemaManagerTest.java | Adds test ensuring setDatabase waits for lifecycle admission before checking unfinished procedures. |
| iotdb-core/confignode/src/test/java/org/apache/iotdb/confignode/consensus/request/ConfigPhysicalPlanSerDeTest.java | Adds SerDe tests for CreateRegionGroupsPlan and BatchRemoveRegionCreateTasksPlan. |
| iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/scheduler/DatabaseLifecycleLockManager.java | Introduces a unified database-scoped lock table for procedures and synchronous requests. |
| iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/schema/DeleteDatabaseProcedure.java | Converts to database-lock-holding procedure, adds durable task cancellation, and runs region-group deletions as children under the lock. |
| iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/region/RemoveRegionGroupProcedure.java | Updates procedure documentation to reflect new parent/lock ownership model. |
| iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/region/CreateRegionGroupsProcedure.java | Converts to database-lock-holding procedure, adds pre-state validation fencing and child-procedure cleanup under lock. |
| iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/AbstractDatabaseProcedure.java | Adds a base class that acquires/releases database lifecycle locks and holds them until procedure completion. |
| iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/env/ConfigNodeProcedureEnv.java | Wires lifecycle lock manager into the env and exposes validation/cancellation helpers. |
| iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/persistence/partition/PartitionInfo.java | Adds atomic CreateRegionGroups plan validation, task offer fencing/dedup, durable batch cancellation, and snapshot queue atomicity. |
| iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/persistence/executor/ConfigPlanExecutor.java | Routes the new BatchRemoveRegionCreateTasks plan to PartitionInfo. |
| iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/schema/ClusterSchemaManager.java | Serializes set/alter database operations against lifecycle procedures using database lifecycle locks. |
| iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/ProcedureManager.java | Guards delete-database submission and region-group procedure submission with lifecycle locks; adds helpers for lifecycle detection. |
| iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/partition/PartitionManager.java | Locks RegionMaintainer dispatch by database, adds batching limits and exponential backoff/jitter, and adds durable task-cancellation write path. |
| iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/consensus/request/write/region/BatchRemoveRegionCreateTasksPlan.java | Introduces a new idempotent consensus plan to remove queued RegionCreateTasks for pre-deleted databases. |
| iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/consensus/request/ConfigPhysicalPlanType.java | Adds a new physical plan type for task batch removal. |
| iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/consensus/request/ConfigPhysicalPlan.java | Extends plan factory to deserialize the new batch-removal plan. |
| iotdb-core/confignode/src/main/i18n/zh/org/apache/iotdb/confignode/i18n/ManagerMessages.java | Adds zh-CN i18n constants for lifecycle-procedure fencing messages. |
| iotdb-core/confignode/src/main/i18n/zh/org/apache/iotdb/confignode/i18n/ConfigNodeMessages.java | Adds zh-CN i18n constants for CreateRegionGroupsPlan rejection reasons. |
| iotdb-core/confignode/src/main/i18n/en/org/apache/iotdb/confignode/i18n/ManagerMessages.java | Adds en i18n constants for lifecycle-procedure fencing messages. |
| iotdb-core/confignode/src/main/i18n/en/org/apache/iotdb/confignode/i18n/ConfigNodeMessages.java | Adds en i18n constants for CreateRegionGroupsPlan rejection reasons. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
CRZbulabula
force-pushed
the
yongzao/confignode-region-create-fence
branch
from
July 23, 2026 04:24
08fb79a to
be7e361
Compare
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #18285 +/- ##
============================================
+ Coverage 42.83% 43.06% +0.23%
Complexity 374 374
============================================
Files 5362 5365 +3
Lines 381409 381859 +450
Branches 49509 49594 +85
============================================
+ Hits 163368 164448 +1080
+ Misses 218041 217411 -630 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Region creation and retry dispatch could race with database deletion, allowing stale Region creation work to survive after DROP or mutate missing metadata.
This PR fixes the lifecycle entirely in ConfigNode:
The implementation is ConfigNode-only. It does not introduce database lifecycle generations and does not change the historical Procedure, plan, or PartitionInfo snapshot serialization formats.
This also covers the useful ConfigNode race fixes discussed in #18273—atomic CreateRegionGroupsPlan validation, RegionGroup ID high-water-mark advancement, database lifecycle mutual exclusion, and pre-persistence cleanup—without adding generation fields to persisted formats.
Tests
The tests cover durable batch cancellation, multiple failed replicas of one Region, same-name database recreation, stale/missing/pre-deleted database validation, child RegionGroup cleanup, transient Consensus retries, task-offer idempotency, database lock mutual exclusion/reentrancy, snapshot compatibility, and procedure serialization compatibility.
This PR has: