Skip to content

Recover interrupted topic deletion on the 3.0 bridge - #596

Open
bringhurst wants to merge 2 commits into
3.0-li-bridge/stop-response-fencingfrom
3.0-li-bridge/interrupted-deletion-recovery
Open

bringhurst wants to merge 2 commits into
3.0-li-bridge/stop-response-fencingfrom
3.0-li-bridge/interrupted-deletion-recovery

Conversation

@bringhurst

Copy link
Copy Markdown
Member

Problem

A 3.0 controller can stop during recursive topic deletion. The topic parent can retain reassignment fields after its leader/ISR znode is removed. The next controller then repeatedly fails reassignment startup. In the failed recovery run, producer-ID allocation timed out while the controller looped.

The new native startup regression also exposed a gap in #595: combined responses preserve deletion intent, but direct native StopReplica responses did not set the v4 deletion bit. The stricter callback therefore dropped genuine direct deletion acknowledgements.

Change

Under li.protocol.bridge.topic.deletion.state.cleanup.enable, and only while deletion is enabled:

  • Recover reassignments only for marked deletions with missing leader/ISR state.
  • Preserve every assigned replica. Clear reassignment flags, not replica membership.
  • Fence ZooKeeper writes and publish each in-memory assignment only after its write succeeds.
  • Remove only the selected legacy reassignment entries, then let acknowledged deletion finish.
  • Echo the request's deletion bit in native v4 StopReplica responses, including failed deletions.

Cleanup off retains the old behavior. Older wire responses remain unchanged. No deadline or record assertion is relaxed.

Evidence

  • Both generations reproduced the missing-state controller loop before the repair.
  • The maintained native startup case failed again until the direct response bit was repaired. With both fixes, the 3.0 case finishes deletion, recreates the name, and verifies exact records using the unchanged client jar.
  • The native response regression fails before (expected true, got false) and passes after a real v4 serialization round trip. It covers cleanup off/on, deletion success/failure, and non-delete failure.
  • KafkaApisTest, BridgeInterruptedDeletionTest, ControllerChannelManagerTest and TopicDeletionManagerTest: 171 tests, no failures/errors, one existing disabled testAlterReplicaLogDirs case. Actual RAT and archive build pass on Scala 2.12.
  • Separate 3.9 repair and mandatory process evidence will follow in the 3.9 stack. This is not full-pair qualification or rollout approval.

Based on #595. Keep this in the 3.0 history; do not merge release lines.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant