diff --git a/docs/ops/li-bridge-review-comments.md b/docs/ops/li-bridge-review-comments.md index d5b3799d80bc2..ac78681450335 100644 --- a/docs/ops/li-bridge-review-comments.md +++ b/docs/ops/li-bridge-review-comments.md @@ -17,7 +17,7 @@ limitations under the License. # Review comment dispositions -All 62 original threads have replies with published source decisions. A fresh GraphQL readback across 47 PRs verified every expected reply, found no mismatches and found no new review threads. Resolved status alone was not accepted as proof. Re-fetch after the final publication and check the actual source/test coverage before closing the review. +All 62 original threads have replies with published source decisions. A fresh GraphQL readback across 48 PRs verified every expected reply, found no mismatches and found no new review threads. Resolved status alone was not accepted as proof. Re-fetch after the final publication and check the actual source/test coverage before closing the review. ## Later qualification findings @@ -33,6 +33,8 @@ All 62 original threads have replies with published source decisions. A fresh Gr | F27: interrupted deletion breaks controller startup | [596](https://github.com/linkedin/kafka/pull/596)/[597](https://github.com/linkedin/kafka/pull/597) retain all replicas, clear only marked missing-state reassignments, and finish acknowledged deletion under cleanup. Both generations pass the maintained native startup and exact-record cases; scenario revision 6 rejects missing evidence. | Scoped bundles are not full-pair qualification. | | F28: old producer times out obtaining recreated-topic metadata | Latest PR 594 CI failure is retained. The unchanged client jar reproduces a 60-second metadata stall in a deterministic network fixture. The existing zero-expiry setting also passes a separate real-broker experiment with both survivor generations. The migration profile and deadlines are unchanged. | Client-floor decision and full final-pair qualification remain open. | +The verifier isolation finding is tracked in [598](https://github.com/linkedin/kafka/pull/598): all Gradle invocations remain single-use, and the global daemon-stop commands are removed. Its command-plan regression fails before and passes after. No functional qualification check is removed. + ## PR 541 | Comment | Decision | Code/test evidence | Reply | diff --git a/docs/ops/li-bridge-review.md b/docs/ops/li-bridge-review.md index 461030cefd5b0..bb0864ab5903c 100644 --- a/docs/ops/li-bridge-review.md +++ b/docs/ops/li-bridge-review.md @@ -19,14 +19,14 @@ limitations under the License. ## Current verdict -**Do not deploy this candidate without final-source qualification and the release approvals.** The retained scenario-5 full-verifier pass predates F25–F27. The interrupted-deletion repairs and mandatory scenario-6 checks are now published, but only their scoped before/after process tests have passed. A separate old-client metadata timeout remains open. The inventory covers 47 open PRs. Earlier passing or failed bundles retain their original source and coverage limits. +**Do not deploy this candidate without final-source qualification and the release approvals.** The retained scenario-5 full-verifier pass predates F25–F27. The interrupted-deletion repairs and mandatory scenario-6 checks are now published, but only their scoped before/after process tests have passed. A separate old-client metadata timeout remains open. The inventory covers 48 open PRs. Earlier passing or failed bundles retain their original source and coverage limits. The findings below record what the initial review and later tests found. Instructions in an original finding describe the repair that was needed; use the current disposition and evidence sections for status. This is not a line-by-line approval of every Kafka change. ### Reviewed revisions - Workspace plan: `LI-3.0-TO-3.9-ROLLING-UPGRADE-PLAN.md`. Canonical runbook: `docs/ops/li-bridge-upgrade.md`. -- Current 3.9 implementation and qualification: PR 597, `3.9-li-bridge/interrupted-deletion-recovery`, including F25 and mandatory scenario-6 checks. +- Current 3.9 implementation and qualification: PR 598, `3.9-li-bridge/verifier-process-isolation`, including F25, F27 and mandatory scenario-6 checks. - Current 3.0 source: PR 596, `8086d1796816bd2383844b7b9f9b05ebd8b33c9b`. - CI: PR 558, `3e799b08ea`; PR 559, `a86214e2da`. - Wrapper: `1764cc95bfa21e19d3ff89e0164e5896808b5507`, including the diagnostic opt-in and the earlier ACL test fix. @@ -333,6 +333,12 @@ Six diagnostic replays passed, but old-client logs exposed a 30-second interval This is not yet a proven resolution of the 60-second CI failure. No client binary/profile or test deadline has been changed to hide it. The existing zero setting also passed a separate real-broker experiment: the unchanged client jar produced and verified recreated-topic records with each broker generation as the survivor while the other bootstrap broker was offline. This is not the migration profile or full qualification. Decide whether this existing setting may be a pre-upgrade client prerequisite, then record the deployed floor and owner decision before closing the finding. A config opt-out is not automatic approval to change deployed clients. +### F29 — P2: Verifier cleanup can stop unrelated Gradle builds + +The verifier ran global `gradlew --stop` commands in both checkouts. Those commands address a shared daemon registry, not just processes owned by this verification. Every verifier/stager Gradle command already uses `--no-daemon`, so the global stops are unnecessary. + +PR 598 removes them and their evidence-row requirement. A regression renders the actual full command plan, requires `--no-daemon`, and rejects `--stop`; it fails before and passes after. All 85 Python tests and actual RAT pass. No functional source, wrapper, archive, phase or record check is removed. This is a verifier-only change, not a broker/client behavior change. + ## PR dispositions and dependency audit Every PR below has a distinct migration or CI purpose. Keep these scopes, but do not treat publication, a resolved thread or a green job as release approval. Publication does not mean that a layer is approved. The controller/ZooKeeper, security, storage and operational changes still need the corresponding owners' review. @@ -386,12 +392,13 @@ Every PR below has a distinct migration or CI purpose. Keep these scopes, but do | [593](https://github.com/linkedin/kafka/pull/593) | mandatory native offline-deletion checks, scenario revision 5 — verification | | [594](https://github.com/linkedin/kafka/pull/594) | default-off placement, controller diagnostics and instrumentation audit — operations/verification | | [597](https://github.com/linkedin/kafka/pull/597) | 3.9 interrupted deletion and mandatory scenario-revision-6 checks — controller/verification | +| [598](https://github.com/linkedin/kafka/pull/598) | verifier process isolation — verification | Merge CI 558/559 into their own release branches first. Then retarget the upgrade stack bottoms as described in the runbook. Never force a Git dependency between the 3.0 and 3.9 CI branches. When reordering again, change PR bases before pushing a head that becomes an ancestor of its former base; GitHub can otherwise auto-close and delete that branch. -The oversized original scopes were split. Control wire definitions are in 565 (503 lines), handlers in 545 (845); storage metrics are in 566 (221), broker metrics/watchdog wiring in 551 (910); workload helpers precede runner 553 (754). The new evidence auditor, verifier and release gate are separate layers. All 47 PR diffs were below 1,000 changed lines at the latest readback. This documentation follow-up stays separate from the 863-line original runbook PR. +The oversized original scopes were split. Control wire definitions are in 565 (503 lines), handlers in 545 (845); storage metrics are in 566 (221), broker metrics/watchdog wiring in 551 (910); workload helpers precede runner 553 (754). The new evidence auditor, verifier and release gate are separate layers. All 48 PR diffs were below 1,000 changed lines at the latest readback. This documentation follow-up stays separate from the 863-line original runbook PR. -The original 62 review threads now have replies with published source decisions and code/test references. See `docs/ops/li-bridge-review-comments.md`. The static `LeaderTransferManager.noOp()` call is valid: javap confirms the forwarder, and the Java builder compiles. Empty, malformed, negative and unordered metric bucket lists are rejected. The earlier claim that empty lists were supported has been corrected against the actual source and test assertion. The latest readback covers 47 PRs and finds all 62 expected replies, no mismatches and no new review threads. The later F18/F19 issue comments are retained and have follow-up code and qualification records. Re-fetch after the final publication; comment status is not proof that the code is correct. +The original 62 review threads now have replies with published source decisions and code/test references. See `docs/ops/li-bridge-review-comments.md`. The static `LeaderTransferManager.noOp()` call is valid: javap confirms the forwarder, and the Java builder compiles. Empty, malformed, negative and unordered metric bucket lists are rejected. The earlier claim that empty lists were supported has been corrected against the actual source and test assertion. The latest readback covers 48 PRs and finds all 62 expected replies, no mismatches and no new review threads. The later F18/F19 issue comments are retained and have follow-up code and qualification records. Re-fetch after the final publication; comment status is not proof that the code is correct. ## Verification and remaining requirements @@ -401,24 +408,24 @@ This prompt-to-artifact checklist separates observed results from open requireme | Requirement | Artifact and verification surface | Evidence / open work | |---|---|---| -| Review the named plan and every open public PR | `LI-3.0-TO-3.9-ROLLING-UPGRADE-PLAN.md`; canonical runbook; GitHub inventory | 47 open PRs are listed in both tables. Recheck the exact PR set after any further publication. | -| Explain scope and dependencies | PR responsibility table, heads/bases, stack membership | Separate protocol, handlers, storage metrics, runner, auditor, verifier and release-gate layers. GitHub stack 582 has 36 upgrade PRs; stack 581 has nine. Both new members were appended and read back. CI 558/559 remain on independent release histories. No release branch was merged. | -| Apply the requested label | GitHub labels | All 45 upgrade PRs have `kafka-upgrade-august-2026`; CI 558/559 do not. | +| Review the named plan and every open public PR | `LI-3.0-TO-3.9-ROLLING-UPGRADE-PLAN.md`; canonical runbook; GitHub inventory | 48 open PRs are listed in both tables. Recheck the exact PR set after any further publication. | +| Explain scope and dependencies | PR responsibility table, heads/bases, stack membership | Separate protocol, handlers, storage metrics, runner, auditor, verifier and release-gate layers. GitHub stack 582 has 37 upgrade PRs; stack 581 has nine. New members were appended and read back. CI 558/559 remain on independent release histories. No release branch was merged. | +| Apply the requested label | GitHub labels | All 46 upgrade PRs have `kafka-upgrade-august-2026`; CI 558/559 do not. | | Keep PRs below 1,000 changed lines, preferably near 500 | Additions plus deletions, not file length | Established diffs passed the limit; recheck the new metrics follow-ups after publication. The largest established diffs are 981 and 963 lines. | | Use plain, direct English | Plan, review, comment replies and workflow comments | Final wording/link review remains required. Historical findings are not current deployment instructions. | | Gate every Kafka behavior change | `KafkaConfig`, `DynamicBrokerConfig`, runtime call sites, metrics, wrapper mapping | 24 default-off 3.9 gates. F18/F19 use the cleanup gate; ISR retry repair uses bridge mode; F22 separately gates diagnostic registration. Dedicated disabled/activation tests pass. The gate-audit matrix records runtime call sites, non-Boolean configuration opt-ins and native behavior; verify it against final published source and final-source test results. | | Select symmetric v2/v5/v1 control | Schemas, controller selectors, wire fixtures and retained logs | Both generations have fixtures and real-process coverage. F20 makes rotated log checks fail closed too. Final-source process qualification remains required. | | Fence activation and preserve callbacks | `RequestSendThreadBridgeTest` | Blocked dequeue, sustained queue and admitted-deletion callback tests pass. Controller restart remains mandatory. | -| Enforce all six phases and unchanged clients | `li_bridge_contract.py`, preflight, persistent client/Streams/Connect and private-API helpers | 84 Python tests pass. F28's cold-bootstrap qualification remains open. Native control stays at IBP 3.0 before the separate IBP roll. One 3.0 archive is not the deployed client/tool floor. | +| Enforce all six phases and unchanged clients | `li_bridge_contract.py`, preflight, persistent client/Streams/Connect and private-API helpers | 85 Python tests pass. F28's cold-bootstrap qualification remains open. Native control stays at IBP 3.0 before the separate IBP roll. One 3.0 archive is not the deployed client/tool floor. | | Prove persisted rollback and recovery | Process runner, record helper, timings and JUnit | Historical full run covers canary/all-3.9 rollback, cancellation, crashes and truncation. Repeat against the final source; registration or ISR alone is not proof. | | Prove deletion and name reuse | `TopicDeletionManager`, `ZkMetadataCache`, `ReplicaManager`, `BridgeTopicIdentity` | Gated F12/F13/F15/F17/F18/F19 repairs have tests. Revision 6 retains all four post-promotion record checks and adds both generations' interrupted-deletion startup cases. Scoped before/after checks pass but do not replace complete qualification. | | Handle version-changing ISR retries | `BridgeAlterPartitionRetryTest` | One queued builder crosses versions 3/1 and activation without reusing mutated request data. Earlier failed logs remain failed evidence. | | Collect real runtime/configuration/state | Live inventory, runtime probe and negative-input tests | Disposable-cluster collection passes. Production binaries, settings, state dispositions, owners and client/tool floor remain required inputs. | | Qualify pagination with the loaded runtime | `KafkaZkClient`, five vendor-client tests and release runtime probe | Startup rejects an unsupported client. Vendor tests pass. The actual deployed client/Jute/server pairing remains a release gate. | -| Follow Google shell style, including comments | Four wrappers, all extracted workflow Bash blocks, ShellCheck, shfmt, syntax/length checks | The refreshed audit checks exact reviewed source revisions: 54 changed-workflow blocks and four wrappers pass ShellCheck, shfmt, syntax, no-tab and 80-column checks. Unmodified upstream Docker workflows are outside these PRs; the broader diagnostic results are retained separately. | +| Follow Google shell style, including comments | Four wrappers, all extracted workflow Bash blocks, ShellCheck, shfmt, syntax/length checks | The audit covers 54 changed-workflow blocks, four wrappers and nine Bash documentation examples. The broader example check found three continuation-indentation issues and one overlong line; formatting-only fixes preserve their assignment/argument tokens. All examples now pass ShellCheck, shfmt, syntax, no-tab and 80-column checks. Unmodified upstream Docker workflows are outside these PRs. | | Preserve wrapper/API compatibility | Factory mapping tests, ACL tests, complete wrapper suite and jar comparison | The current 133-test suite passes with matching main jars, stable main/test-classifier hashes and unchanged source. Wrapper commit 1764cc95 is published. The required Mint refresh produced a fresh dependency spec; no TTL or artifact-identity bypass was used. | | Qualify real archives and reject incomplete evidence | `verify_li_bridge.sh`, `audit_li_bridge_evidence.py`, `verify_li_bridge_release.sh` and negative fixtures | Earlier full verifier passed but predates F18/F19. The metrics-gated full run failed during the process phase after its earlier stages passed. Eight new release-input tests cover missing inputs, malformed identifiers, mismatched/unknown archive metadata, dirty/wrong checkouts, forced full mode and rejection before launch. A real-archive identity-only check also passes; neither it nor the fixtures grant release approval. | -| Address every review comment with evidence | Comment ledger, source/test decisions, GraphQL readback | All original 62 replies verified across 47 PRs; no new review threads. Later issue findings have published fixes and explicit qualification limits. Re-fetch after final publication. | +| Address every review comment with evidence | Comment ledger, source/test decisions, GraphQL readback | All original 62 replies verified across 48 PRs; no new review threads. Later issue findings have published fixes and explicit qualification limits. Re-fetch after final publication. | | Keep the three documents consistent | Workspace files and `docs/ops/li-bridge-{upgrade,review,review-comments}.md` | This follow-up synchronizes the records. Verify relative links and actual PR/source/evidence state before calling the review complete. | | Preserve side-task PR 2039 | ADU worktree, commit history and formatting checks | Rebased/pushed on master at `ae007420`; formatting is one separate commit and is idempotent. Functional patches remain unchanged. | @@ -441,7 +448,7 @@ Later evidence supersedes the inventory and coverage limits of those historical - `/tmp/li-scenario-4-batch-fixed`: all four offline-reuse record checks passed, but the complete run failed at the native checkpoint. The old metadata-churn helper exited on `ControllerMovedException` during controller movement. Its progress had advanced to 221 cycles. The unchanged upstream fence and helper retry gap are covered by F21; do not waive this failed run. The summary records `passed=false` and unchanged source. Rotated logs omitted by that older collector are retained separately in `/tmp/li-scenario-4-batch-fixed-rotated-logs.tgz`. - `/tmp/li-log-rotation-before.log`: both new rotation regressions fail before F20. `/tmp/li-log-rotation-restacked.log`: all 70 Python tests pass afterward. - `/tmp/li-churn-retry-before-{3.0,3.9}.log`: real client error classes expose the missing controller retry and incorrectly retryable storage/corruption errors. `/tmp/li-churn-retry-after-{3.0,3.9}.log` and `/tmp/li-churn-retry-restacked-python.log` pass with the repair. -- `/tmp/li-review-readback-result.json`: 47 PRs, 62 original threads, no missing/mismatched replies and no new threads at that readback. +- `/tmp/li-review-readback-result.json`: 48 PRs, 62 original threads, no missing/mismatched replies and no new threads at that readback. The complete revision-4 process run `/tmp/li-scenario-4-churn-fixed` passed on clean `d8f255f8a4` / `1a5d02403b`, with all four name-reuse checks, unchanged source and an issue-free process audit. It includes F20/F21 but predates F22. The real `mint --no-metrics dependency create-dependency-spec --detect-variant --overwrite` command has now refreshed the wrapper metadata successfully. An invocation without `--overwrite` returned success without refreshing the expired file; that no-op was not accepted as freshness evidence. @@ -455,6 +462,8 @@ The complete revision-4 process run `/tmp/li-scenario-4-churn-fixed` passed on c `/tmp/li-f27-native-echo-fixed` passes the maintained interrupted-deletion cases for both generations on 8086d17968 / c1091c6d27. Its summary remains an incomplete migration. `/tmp/li-f27-contract-before.log` retains six expected negative-fixture failures; `/tmp/li-f27-contract-after.log` passes all 84 tests. The latest PR 594 failed artifact is `/tmp/li-ci-594-f26-small-evidence`; `/tmp/li-reuse-metadata-diagnostic`, `/tmp/li-bootstrap-expiry-probe.log` and `/tmp/li-bootstrap-selection-probe.log` and `/tmp/li-bootstrap-progress-probe.log` record the bounded F28 investigation. `/tmp/li-expiry-off-profile-probe` retains the successful separate config experiment and its exact one-property helper diff; its complete-migration status remains false. +`/tmp/li-verifier-isolation-regression-before.log` preserves the global-stop regression; `/tmp/li-verifier-isolation-after.log` passes all 85 Python tests. The removed evidence rows represented unnecessary daemon cleanup, not functional coverage. `/tmp/li-document-shell-audit/report.json` retains the failing documentation-example audit; `/tmp/li-document-shell-audit-fixed/report.json` passes all nine examples after formatting-only fixes. + No complete bundle yet covers all final default-off, response-fencing and interrupted-deletion repairs together. Failed runs remain failed. A source, archive or scenario change must be checked against the fingerprint before reuse. ### Inputs still required before production diff --git a/docs/ops/li-bridge-upgrade.md b/docs/ops/li-bridge-upgrade.md index 304e223d3b2a5..55d721742bea4 100644 --- a/docs/ops/li-bridge-upgrade.md +++ b/docs/ops/li-bridge-upgrade.md @@ -23,7 +23,7 @@ limitations under the License. The implementation base is Apache **3.9.2** with the reviewed LI bridge stack. Pin the final internal `3.9.2.N`, matching `3.0.1.N`, wrapper commit, archive checksums, JDKs and ZooKeeper runtime in the release record. A maintenance-baseline change requires a new qualification run; do not substitute a newer tag during rollout. -The current implementation is the split stack through `3.9-li-bridge/interrupted-deletion-recovery`, with the companion `3.0-li-bridge/interrupted-deletion-recovery` branch. It is not the closed aggregate PR 542. The canonical mergeable runbook is `docs/ops/li-bridge-upgrade.md`; the workspace copy is `LI-3.0-TO-3.9-ROLLING-UPGRADE-PLAN.md`. The `3.9-li-bridge/review-refresh` branch updates the documentation after the behavior fixes. Historical experiments are evidence, not current acceptance criteria. +The current implementation is the split stack through `3.9-li-bridge/verifier-process-isolation`, with the companion `3.0-li-bridge/interrupted-deletion-recovery` branch. It is not the closed aggregate PR 542. The canonical mergeable runbook is `docs/ops/li-bridge-upgrade.md`; the workspace copy is `LI-3.0-TO-3.9-ROLLING-UPGRADE-PLAN.md`. The `3.9-li-bridge/review-refresh` branch updates the documentation after the behavior fixes. Historical experiments are evidence, not current acceptance criteria. **Clients do not change.** The supported producer, consumer, transactional client, Streams application, Connect worker, LI AdminClient and operational-tool artifacts/configuration must remain unchanged across every phase. Discovery must name their deployed version floor and owners. One 3.0 test archive is not proof for every externally deployed client. The latest qualification still has an unresolved cold-producer metadata timeout with an offline bootstrap broker. Record the deployed LI clients' `li.client.cluster.metadata.expire.time.ms` setting and test cold startup with unavailable bootstrap entries. Diagnostic replays and an existing config opt-out are not release approval or permission to change clients during the roll. @@ -176,16 +176,16 @@ Repeat config arguments for **every** live broker. Reports distinguish `configur ## Qualification before the first canary -Run qualification against independently built archives, then repeat it against the actual published archives and official wrapper dependencies. Retain checksums, source identities, commands, phase results and logs. +Run qualification against independently built archives, then repeat it against the actual published archives and official wrapper dependencies. Retain checksums, source identities, commands, phase results and logs. Gradle commands use `--no-daemon`; the verifier must not stop unrelated builds through a shared daemon registry. ```bash JAVA_HOME=/path/to/jdk17 \ -LI_BRIDGE_JAVA_30_HOME=/path/to/jdk11 \ -KAFKA_30_TGZ=/releases/kafka-30.tgz \ -KAFKA_39_TGZ=/releases/kafka-39.tgz \ -SKIP_LOCAL_STAGE=1 WRAPPER_ROOT=/checkouts/kafka-server \ -BRIDGE_VERIFY_FULL=1 EVIDENCE_DIR=/evidence/bridge \ -tests/bin/verify_li_bridge.sh + LI_BRIDGE_JAVA_30_HOME=/path/to/jdk11 \ + KAFKA_30_TGZ=/releases/kafka-30.tgz \ + KAFKA_39_TGZ=/releases/kafka-39.tgz \ + SKIP_LOCAL_STAGE=1 WRAPPER_ROOT=/checkouts/kafka-server \ + BRIDGE_VERIFY_FULL=1 EVIDENCE_DIR=/evidence/bridge \ + tests/bin/verify_li_bridge.sh ``` For the protected release gate, use `tests/bin/verify_li_bridge_release.sh` with published `KAFKA_30_SHA256`, `KAFKA_39_SHA256`, approved full `KAFKA_30_COMMIT`, `KAFKA_39_COMMIT`, and `WRAPPER_COMMIT` in addition to the inputs above. It rejects dirty/unapproved checkouts, archive/source/checksum mismatches and partial verification, and runs a strict final audit. @@ -240,7 +240,7 @@ Automatically stop for unexpected control versions, post-fence API 1001 traffic, ## PR inventory and merge order -All 47 open public PRs are covered below. Upgrade PRs carry `kafka-upgrade-august-2026`; CI foundations 558 and 559 do not. All current diffs are below 1,000 changed lines. These checks do not grant approval to deploy. +All 48 open public PRs are covered below. Upgrade PRs carry `kafka-upgrade-august-2026`; CI foundations 558 and 559 do not. All current diffs are below 1,000 changed lines. These checks do not grant approval to deploy. Closed PRs 542 and 555 are superseded. GitHub automatically closed 563 and 564 during the dependency reorder because their new heads were contained in their former base branches. No release branch was merged. Their restored, separate reviews are 579 and 578. @@ -293,12 +293,13 @@ Closed PRs 542 and 555 are superseded. GitHub automatically closed 563 and 564 d | [593](https://github.com/linkedin/kafka/pull/593) | mandatory native offline-deletion checks, scenario revision 5 — verification | | [594](https://github.com/linkedin/kafka/pull/594) | default-off placement, controller diagnostics and instrumentation audit — operations/verification | | [597](https://github.com/linkedin/kafka/pull/597) | 3.9 interrupted deletion and mandatory scenario-revision-6 checks — controller/verification | +| [598](https://github.com/linkedin/kafka/pull/598) | verifier process isolation — verification | Merge 558 into `3.9-li` and 559 into `3.0-li` first. They have different release bases, so do not put them in one dependent Git stack. Rebase/retarget 575 to `3.0-li` and 543 to `3.9-li`; do not merge feature work into temporary CI branches. The GitHub stack rooted at PR 575 has the 3.0 order **575 → 541 → 577 → 583 → 585 → 588 → 592 → 595 → 596**. The stack rooted at PR 543 has the 3.9 order: -**543 → 544 → 565 → 545 → 546 → 547 → 548 → 560 → 549 → 550 → 561 → 566 → 551 → 567 → 576 → 568 → 578 → 579 → 552 → 569 → 570 → 571 → 553 → 572 → 554 → 573 → 574 → 584 → 586 → 587 → 589 → 590 → 591 → 593 → 594 → 597**. +**543 → 544 → 565 → 545 → 546 → 547 → 548 → 560 → 549 → 550 → 561 → 566 → 551 → 567 → 576 → 568 → 578 → 579 → 552 → 569 → 570 → 571 → 553 → 572 → 554 → 573 → 574 → 584 → 586 → 587 → 589 → 590 → 591 → 593 → 594 → 597 → 598**. Retarget remaining layers after each independent merge. Wrapper `1764cc95` contains the diagnostic opt-in, ACL test repair (`6ddf2a87`) and cleanup mapping/tests (`1a9ecccf`); its source suite passes 133 tests. Add the approved wrapper/dependency/security PR and named deployment-gate owner to the release record. diff --git a/tests/bin/README.li-bridge.md b/tests/bin/README.li-bridge.md index 0aa7263dc7471..2e107f593a204 100644 --- a/tests/bin/README.li-bridge.md +++ b/tests/bin/README.li-bridge.md @@ -23,10 +23,10 @@ Use Java 17 and Python 3.9 or newer. Run the verifier from the current split-sta ```sh JAVA_HOME=/path/to/jdk17 \ -KAFKA_30_TGZ=/releases/kafka_2.12-3.0.1.83.tgz \ -WRAPPER_ROOT=/checkouts/kafka-server \ -EVIDENCE_DIR=/evidence/local-bridge \ -tests/bin/verify_li_bridge.sh + KAFKA_30_TGZ=/releases/kafka_2.12-3.0.1.83.tgz \ + WRAPPER_ROOT=/checkouts/kafka-server \ + EVIDENCE_DIR=/evidence/local-bridge \ + tests/bin/verify_li_bridge.sh ``` The verifier stages Scala 2.12 jars for the wrapper and builds the 3.9 archive. `LI_BRIDGE_VERSION` selects the local artifact version. Its default comes from `gradle.properties`. @@ -37,19 +37,19 @@ Download both archives from approved releases and verify their published SHA-256 ```sh JAVA_HOME=/path/to/jdk17 \ -KAFKA_30_TGZ=/releases/kafka_2.12-3.0.1.83.tgz \ -KAFKA_39_TGZ=/releases/kafka_2.12-3.9.2.17.tgz \ -SKIP_LOCAL_STAGE=1 \ -WRAPPER_ROOT=/checkouts/kafka-server \ -EVIDENCE_DIR=/evidence/released-bridge \ -tests/bin/verify_li_bridge.sh + KAFKA_30_TGZ=/releases/kafka_2.12-3.0.1.83.tgz \ + KAFKA_39_TGZ=/releases/kafka_2.12-3.9.2.17.tgz \ + SKIP_LOCAL_STAGE=1 \ + WRAPPER_ROOT=/checkouts/kafka-server \ + EVIDENCE_DIR=/evidence/released-bridge \ + tests/bin/verify_li_bridge.sh ``` `KAFKA_39_TGZ` disables the local archive build. It requires `SKIP_LOCAL_STAGE=1` and a 3.0 archive. The verifier checks the version inside each archive and retains the exact input bytes before compilation starts. The wrapper's declared Kafka and Scala versions must match the 3.9 archive. Every resolved main Kafka jar must match a jar in that archive byte for byte. Test classifiers must use the selected version. Their hashes are recorded. The verifier checks the resolved files before and after wrapper testing. -Source compilation and focused source tests still run. `BRIDGE_VERIFY_FULL=1` adds the complete clients, server, and storage suites. The mixed-process test runs the retained archives. +Source compilation and focused source tests still run. `BRIDGE_VERIFY_FULL=1` adds the complete clients, server, and storage suites. The mixed-process test runs the retained archives. Gradle commands use `--no-daemon`; the verifier does not run global `--stop` commands that could interrupt unrelated builds. ## Diagnostic metrics opt-in @@ -116,8 +116,8 @@ Optional scenario inputs and defaults: To check a standalone process result, use the same scenario environment as the run: ```sh -python3 tests/bin/audit_li_bridge_evidence.py --process-only --require-archives \ - --evidence-dir /evidence/process-run +python3 tests/bin/audit_li_bridge_evidence.py \ + --process-only --require-archives --evidence-dir /evidence/process-run ``` This checks all phase reports, record checks, process identities and archive hashes. It does not claim that wrapper tests, full suites or release approval passed. Failures retain thread dumps and the relevant ZooKeeper state before cleanup. The process test also writes a JUnit report. diff --git a/tests/bin/audit_li_bridge_evidence.py b/tests/bin/audit_li_bridge_evidence.py index 2b286144ded83..10f713ccab988 100755 --- a/tests/bin/audit_li_bridge_evidence.py +++ b/tests/bin/audit_li_bridge_evidence.py @@ -48,8 +48,6 @@ "wrapper-tests", "wrapper-artifacts-unchanged", "release-39", - "stop-kafka-gradle", - "stop-wrapper-gradle", "mixed-process", ) diff --git a/tests/bin/verify_li_bridge.py b/tests/bin/verify_li_bridge.py index 8f69fa4dfbe69..fe337e3663f36 100644 --- a/tests/bin/verify_li_bridge.py +++ b/tests/bin/verify_li_bridge.py @@ -233,8 +233,6 @@ def run(self): tasks.append("--offline") self.command("wrapper-tests", tasks, self.wrapper, {"KAFKA_30_TGZ": str(self.legacy)}) self.wrapper_artifacts(after=True) - self.gradle("stop-kafka-gradle", ["--stop"]) - self.gradle("stop-wrapper-gradle", ["--stop"], self.wrapper) self.command("mixed-process", [self.bin / "li_bridge_mixed_cluster_smoke.sh"], extra_env={ "KAFKA_30_TGZ": str(self.legacy), "KAFKA_39_TGZ": str(self.broker), "EVIDENCE_DIR": str(self.evidence / "mixed-process")}) diff --git a/tests/unit/li_bridge_commands_test.py b/tests/unit/li_bridge_commands_test.py index 64c91f98c935b..07a7caa86fe14 100644 --- a/tests/unit/li_bridge_commands_test.py +++ b/tests/unit/li_bridge_commands_test.py @@ -88,6 +88,24 @@ def check_candidate(*args, **kwargs): self.assertEqual(passed, json.loads((root / "verification-summary.json").read_text())["passed"]) self.assertEqual("passed" if passed else "failed(1)", verification.commands.results["evidence-audit"]) + def test_verifier_uses_single_use_gradle_without_stopping_other_builds(self): + root = Path(__file__).parents[2] + with tempfile.TemporaryDirectory() as directory: + wrapper = Path(directory) / "wrapper" + wrapper.mkdir() + environment = dict(os.environ, BRIDGE_VERIFY_DRY_RUN="1", BRIDGE_VERIFY_FULL="1", + EVIDENCE_DIR=str(Path(directory) / "evidence"), WRAPPER_ROOT=str(wrapper), + KAFKA_30_TGZ=str(Path(directory) / "synthetic-unused.tgz")) + environment.pop("KAFKA_39_TGZ", None) + with mock.patch("verify_li_bridge.java_17"), mock.patch.object(Commands, "run") as commands: + Verification(root, environment).run() + gradle = [call.args[1] for call in commands.call_args_list + if Path(call.args[1][0]).name == "gradlew"] + self.assertTrue(gradle) + for command in gradle: + self.assertIn("--no-daemon", command) + self.assertNotIn("--stop", command, "Global stop can terminate an unrelated build") + def test_selection_names_tests_and_requires_all_suites(self): path = Path(__file__).parents[1] / "bin/li_bridge_test_selection.ini" selected = selection(path)