fix(evmigration): cross-module runtime continuity for account/validator migration (PR2) - #199
Conversation
Behavior change --------------- 1. VerifyMigrationProofsForAnte now enforces the canary allowlist, not just EnableMigration and the migration window. Migration txs are fee-free and zero-signer, so before this change canary mode still admitted unlimited txs from arbitrary (non-allowlisted) sources into the mempool and into block proposals, each paying full multisig proof verification in CheckTx, only to be rejected by preChecks in DeliverTx. The activation decision is now a single shared helper, CheckMigrationActivation, called by both the ante admission gate and message execution, so the mempool filter can no longer diverge from the consensus decision (spec s4 "one activation policy helper shared by account and validator messages"). preChecks remains authoritative; the ante is a best-effort mempool filter, unchanged in that role. 2. Repairs five tests left red by 0d3685b (enable_migration default false). They live in app/ and app/evm, outside the previously verified package set, and asserted enabled-by-default behavior. They now open the activation gate explicitly so they still exercise the proof / admission / proposal behavior they name instead of short-circuiting on "migration is disabled". Disabled-by-default remains pinned in x/evmigration/keeper/ante_test.go. Rationale --------- Invariant I19 (activation) requires the disabled/canary/open policy to hold at every message entry point. The ante is a message entry point for zero-fee, zero-signature txs and was the one place the canary list was not consulted. Tests ----- RED-then-GREEN per invariant: - TestVerifyMigrationProofsForAnte_CanaryGate (I19): unlisted source rejected, allowlisted source admitted, empty list leaves open mode unchanged, validator migration honours the same list. Confirmed failing before the production change. - TestEVMigration_BaseAppLateFailureRollsBackEveryStore (I20): drives the production BaseApp FinalizeBlock path and fails in the LAST migration step (step 7, MigrateActions) via a dangling creator index, so rollback is exercised only after distribution, staking, auth, bank, retained SDK state, feegrant, audit and supernode writes have already occurred in the tx cache. Asserted differentially against an empty control block on the same app; no migration-owned or identity-bearing store may appear in the delta, and no committed key or value may reference either identity. - TestEVMigration_CheckTxReCheckTxSimulateAreStateNeutral (spec s5): all three phases individually and in sequence leave committed state byte- identical. Qualified ante side effect: the failing-tx block legitimately commits wasmd CountTXDecorator's per-block tx counter at wasm key 0x08. It carries an 8-byte height and 4-byte count only, no identity, and is asserted explicitly rather than allow-listed away, per the spec requirement to assert ante effects separately instead of claiming the whole app store is unchanged. The rollback assertion was mutation-tested (forcing a surviving migration record makes it fail), confirming it is load-bearing. Risks ----- Narrows mempool admission only. No new state keys, no proto change, no migration, no module-version change. A previously-admitted-then-rejected tx is now rejected earlier and never reaches a block, which is the intent. Rollback -------- Revert this commit; the canary list remains enforced in consensus by preChecks, so reverting weakens mempool filtering only, not the state machine. Observability ------------- No new events. Rejections surface as the existing ErrMigrationNotCanary. Verification ------------ go test -tags=test ./app/... .............. PASS go test ./x/... ........................... PASS go test -tags='integration test' ./tests/integration/evmigration/... PASS make lint ................................. 0 issues git diff --check .......................... clean
…ntinuity Adds the continuity fixtures called for by spec s6.5-6.6 and GOLDEN s7/s10 that had no coverage. Test-only; no production behavior changes. Everlight (I4, GOLDEN s7) ------------------------- TestEverlightNextTickContinuityAfterValidatorMigration asserts that the NEXT distribution after a validator-address migration pays exactly what it would have paid with no migration, as a differential against a control chain seeded identically. Proving the rdist row was copied is not sufficient: distributePool reads SNDistState keyed by validator address and feeds PrevRawBytes into the growth cap, SmoothedBytes into the EMA and PeriodsActive into the ramp-up weight. A plan that moved the bytes but left the consumer reading the old key, or that silently reset the accumulator, still produces a wrong payout. The test drives the real tick, then asserts the accumulator advances under the destination validator and is not re-created under the source. Two properties make the assertion non-vacuous, both found by initial RED runs rather than assumed: - params enable every state-dependent lever (ramp-up 4, smoothing 4, 10% growth cap). With ramp-up off and smoothing 1 a lost accumulator pays the same amount and the test proves nothing. - a second, never-migrated supernode shares the pool. A single SN receives 100% regardless of weight, which also makes payout equality trivial. TestEverlightDiscontinuityWouldChangeNextPayout is the negative control: a LOST accumulator (the pre-fix behavior) must change the payout. If it does not, the continuity assertion above is meaningless. Audit cohort matrix (I7, I8) ---------------------------- TestEpochReportContinuityAcrossMigrationCohorts runs none/one/some/all migrated over a 4-node cohort. The asymmetry is the dangerous case: an epoch's active set is frozen under the accounts that existed at anchor time, while submission authenticates against who is registered now. A mixed cohort splits if those identities are conflated, and that is invisible in a homogeneous all-or-nothing test. Each actor must report exactly once, the row must land under the epoch-logical account, and current_submitter must record the live signer. TestEpochReportCohortRejectsDoubleReportAcrossIdentities proves a migrated node cannot occupy two slots in one epoch by reporting under both its old and new accounts - otherwise "some migrated" is a participation-inflation vector, not just a continuity risk. TestNextEpochReportsUseCurrentIdentityAfterTransition proves the other boundary: from the effective epoch onward the current account IS the logical identity, no new rows accrue under the retired account, and historical epochs are never back-filled onto the new identity. TestCohortMatrixIsExhaustive fails loudly if the matrix silently shrinks. Evidence -------- Mutation-tested, not just green: replacing AccountForEpoch with the raw Creator in msg_submit_epoch_report.go makes the cohort suite fail on the migrated actors, confirming these tests detect the exact identity-split defect they are written for. Source restored and verified clean afterward. Risks ----- None to consensus: no production file is touched by this commit. Rollback -------- Revert; coverage returns to its prior state.
Test-only. Completes the repair of tests invalidated by 0d3685b (enable_migration default false) and clears one staticcheck finding introduced by the cohort matrix. Fixed ----- 1. tests/integration/evm/mempool/evmigration_zero_signer_test.go - three end-to-end tests spin up real nodes and exercise the zero-signer mempool/ante path, which sits behind the activation gate. They were short-circuiting on "migration is disabled": TestEVMigrationZeroSignerTxBroadcastSyncWithMempoolEnabled TestEVMigrationZeroSignerTxBroadcastSyncAfterLegacyMainnetConfigMigration TestEVMigrationProofValidNonexistentLegacyAccountRejectedByAnte Confirmed pre-existing: all three fail identically on the unmodified handoff head 267fcba. Because these boot from genesis rather than an in-process app, the gate is opened via a new enableMigrationInGenesis helper that edits evmigration genesis params before StartAndWaitRPC, with Params.Validate() asserted so a malformed edit fails loudly. 2. staticcheck QF1002 in identity_continuity_cohort_test.go - converted an expression switch to a tagged switch on migrateCount. This is the third and final location of the same class. The first two (app/, app/evm) were repaired in a1ead82. All three were invisible to the branch's original verified package set, which covered only ./x/... and app/upgrades/v1_20_0. Disabled-by-default remains deliberately pinned in x/evmigration/keeper/ante_test.go and x/evmigration/types/params_test.go, so opening the gate inside these consumers does not weaken that guarantee. Risks ----- None to consensus: no production file is touched by this commit. Rollback -------- Revert; the three e2e tests return to failing and lint returns one finding.
Fixes the two CI `system` job failures on this stack: TestStorageTruth_ScoreDecay_TriggersRecovery TestStorageTruth_MultipleRecheckEvidence_AccumulatesScore Both failed with: out of gas in location: ReadFlat; gasWanted: 500000, gasUsed: 500331 Cause ----- The audit account-transition lineage added by 55822e7 makes every epoch-report submission resolve logical-vs-current identity, which reads the forward/reverse transition index. That adds a small but unavoidable amount of gas to a path these tests drive with a hardcoded `--gas 500000`. Observed usage is 500331 - 331 over the cap. The tests were not asserting a gas bound; the literal is test scaffolding, and it had already been raised once before (200000 -> 500000, see the CP3.5 F-B comment) for the same reason when recheck secondary indexes landed. Not a production regression: no gas limit, block limit or fee parameter changes, and no consensus path is altered. This is a test-scaffolding cap. All five occurrences are raised together so the suite cannot fail piecemeal as different tests approach the boundary, and each now carries the observed figure so the next person does not have to re-derive it. Evidence -------- GitHub Actions `system` job, which runs `make install` on a clean runner and therefore builds the binary from this exact commit: d750427 (before) ... FAIL gasWanted: 500000, gasUsed: 500331 <this commit> ... PASS That before/after on CI-built binaries is the load-bearing evidence. Local systemtests are deliberately NOT cited here. On this host /root/go/bin/lumerad is a symlink to a devnet build from a different checkout, so `make install` fails with "Text file busy" and the suite silently exercises a stale binary that predates 55822e7 - which cannot reproduce the gas regression at all. Any local systemex run on this host is therefore not evidence for or against this change. Verify with: ls -l $(which lumerad) # must not be a symlink into another checkout Rollback -------- Revert; the two tests return to failing on the old cap.
3664422 to
c02ad4f
Compare
CI:
|
|
| Network | chain_id | app_version | on-chain audit |
|---|---|---|---|
| Mainnet | lumera-mainnet-1 |
1.12.0 (pre-EVM) | v2 |
| Testnet | lumera-testnet-2 |
1.20.1 | v2 |
Mainnet confirmed by 4 independent RPCs (polkachu, stakerhouse, ibs.team, linknode); module versions read from each chain's LCD.
The problem
This PR raises x/audit ConsensusVersion 2 → 3 and correctly registers the migration (module.go:103, RegisterMigration(types.ModuleName, 2, NewMigrateV2ToV3())).
But RunMigrations only fires from an upgrade handler, and this stack adds no new handler. Its entire diff to app/upgrades/ is comment-only:
app/upgrades/v1_20_0/upgrade.go | 5 +++--
app/upgrades/v1_20_0/upgrade_test.go | 2 +-
Both v1.20.0 and v1.20.1 have already executed on testnet and will never run again.
- Testnet: on-chain audit stays at v2 while this binary declares 3, with no migration path between them. Testnet cannot take this release as-is.
- Mainnet: still at 1.12.0, so
v1.20.0has not run yet and would carry audit 2→3 as part of the EVM bring-up. Mainnet is accidentally fine — which is precisely why a mainnet-only rehearsal would have missed this.
Required fix
Add a v1_20_2 (or v1_21_0) handler whose job is to run RunMigrations so the audit bump lands legally. No StoreUpgrades needed. The migration is a genuine state no-op — NewMigrateV2ToV3 returns nil, legacy reports decode with empty current_submitter and identity indexes start empty — so the handler is thin, but it has to exist.
Second, lower-severity item
The v1_20_0 comment change documents a real behavioral shift: enable_migration now defaults false. Since v1.20.0 already ran on testnet with the old default but has not run on mainnet, the same binary will produce different evmigration params on the two networks. Before rollout we should read testnet's live params and, if migration is enabled there, submit an explicit MsgUpdateParams rather than relying on a code default to fix an already-migrated chain.
(Standing rule this PR already follows: never change an executed handler's logic — comments only. Any behavior change belongs in a new handler.)
Also worth fixing separately
The official mainnet endpoints are unhealthy: rpc.lumera.io does not respond and lcd.lumera.io returns 502, while community RPCs serve fine. Not a consensus issue — the chain is producing blocks — but our own endpoints should be up before an upgrade window when operators are watching them.
Full plan (including the mainnet 1.12.0 → EVM jump, which has never been rehearsed) is written up separately and I'll share it with the team.
…ion 2->3 RELEASE BLOCKER FIX ------------------- This stack raises x/audit ConsensusVersion 2 -> 3 and registers the 2->3 migration, but RunMigrations only executes from inside an upgrade handler and no new handler was added. Verified live on 2026-07-30: lumera-testnet-2 app_version 1.20.1 audit module version 2 lumera-mainnet-1 app_version 1.12.0 audit module version 2 Testnet had ALREADY executed both v1.20.0 and v1.20.1, so neither can run again. Shipping without this, the binary declares audit 3 while committed testnet state says 2, with no path between them. Mainnet masked the defect: still at 1.12.0, it has not run v1.20.0 yet, so the EVM bring-up would have carried the bump for free. A mainnet-only rehearsal would have passed and shipped a broken testnet release. WHAT THIS ADDS -------------- app/upgrades/v1_20_2 exposes only UpgradeName = "v1.20.2". The upgrade is wired with the shared standardUpgradeHandler, which runs RunMigrations and nothing else. No StoreUpgrades by design: testnet already mounted the EVM store keys in v1.20.0 and re-mounting an existing key is an error. No bespoke handler logic. Every behavioral change belongs in the module migration so a chain reaching this version by any path gets identical state. TESTS ----- TDD, RED confirmed first (the package did not compile until the handler existed): - TestAuditConsensusVersionHasCarryingUpgrade pins audit ConsensusVersion and asserts the newest registered upgrade is strictly newer than v1.20.1, which is already live on testnet. Mutation-verified: removing v1.20.2 from upgradeNames makes it fail. - TestV1202IsRegisteredAndMigrationOnly asserts registration on mainnet, testnet and devnet chain-ids and that StoreUpgrade is nil. - TestV1202IsRecognizedAsKnownUpgrade proves SetupUpgrades resolves the plan name to a real handler, i.e. a node built from this tree will not stop with "upgrade plan not registered". - TestV1202UpgradeNameMatchesDirectory guards the UpgradeName != git tag trap. - TestUpgradeNamesOrder updated for the new entry. DEVNET REHEARSAL (Phase 1, testnet-shaped) ------------------------------------------ Ran on the canonical 5-validator devnet, FROM the real v1.20.1 release artifact (sha256 a150df59..., tarball checksum verified against the published release_checksum) TO a binary built from this commit (sha256 b0b88821...). Pre-upgrade state matched live testnet exactly: audit v2 with the full EVM stack present. gov proposal 1 ......... PASSED (4000000000000 yes, 0 no) upgrade boundary ....... height 185, "UPGRADE v1.20.2 NEEDED" binary swap ............ a150df59... -> b0b88821... on all 5 validators resume ................. height 187 q upgrade applied ...... height 185 module_versions ........ audit v2 -> v3; every other module unchanged audit params ........... readable, no corruption bank send .............. code 0 Risks ----- Adds a new upgrade name; no store changes, no proto change, no state mutation beyond the module version bump. The migration itself is a no-op (NewMigrateV2ToV3 returns nil; legacy reports decode with empty current_submitter and identity indexes start empty). Rollback -------- Revert. Because the upgrade has not executed on any live network, reverting pre-activation is safe. Observability ------------- standardUpgradeHandler logs upgrade start, migration completion and success. Verification ------------ make lint .............................. 0 issues go test -tags=test ./app/... ........... PASS go test ./x/... ........................ PASS make integration-tests NOCACHE=1 ....... PASS git diff --check ....................... clean
Phase 0 complete + Phase 1 devnet rehearsal PASSED
Phase 1 rehearsal — testnet-shaped (
|
| Gate | Result |
|---|---|
| gov proposal 1 | PASSED — 4,000,000,000,000 yes / 0 no |
| upgrade boundary | height 185, UPGRADE "v1.20.2" NEEDED |
| binary swap | a150df59… → b0b88821… on all 5 validators |
| resume | height 187 |
q upgrade applied v1.20.2 |
height: 185 |
module_versions |
audit v2 → v3; every other module unchanged |
q audit params |
readable, no corruption |
| bank send | code: 0 |
That is the exact transition that could not have happened without this commit.
Finding: the rehearsal independently reproduced the testnet param divergence
Post-upgrade, q evmigration params on the devnet returned enable_migration: true. Root-caused to genesis: the devnet was seeded by the v1.20.1 binary, whose InitGenesis wrote the old default. v1.20.2 is migration-only and deliberately does not touch params.
This is the same condition I measured on live testnet (enable_migration: true, migration_end_time 1790940497 = 2026-10-02). So it is confirmed from two independent directions, and it means:
A MsgUpdateParams setting enable_migration=false is required on testnet before rollout. The code default cannot retroactively fix an already-initialized chain. The Phase 4 sequence is therefore enabled → disabled first, then the controlled canary → open ramp — not a clean start from disabled.
CI: system job flake
The system check failed on this commit. It is the documented port-collision flake, not a regression. All three criteria from our CI-parity rules are met:
- Explicit collision in the log —
err> listen tcp 127.0.0.1:39981: bind: address already in use, followed by--- FAIL: TestAuditEmptyActiveSetBootstrap_NonCompliantHostStaysPostponedandpanic: Fail in goroutine after ... has completed. systempassed on the immediately preceding commitc02ad4f4.- The named test passes locally on this exact commit —
PASS (53.69s), binary sha256-verified as built from1a64adbe.
Nothing in this commit touches systemtests, ports, or the audit bootstrap path; the diff is one new upgrade package plus its registration and tests.
Local gate on 1a64adbe
make lint ........................... 0 issues
go test -tags=test ./app/... ........ PASS
go test ./x/... ..................... PASS
make integration-tests NOCACHE=1 .... PASS
git diff --check .................... clean
Devnet torn down; no containers or state left behind.
The Phase 2 mainnet-shaped devnet rehearsal caught two defects in v1.20.2 that no amount of testnet-shaped testing could have surfaced. Both would have taken mainnet down at the upgrade height. REHEARSAL SETUP --------------- 5-validator devnet built from the real v1.12.0 release artifact (tarball sha256 f64f4a31... verified against the published release_checksum), with devnet-genesis.json trimmed to the v1.12.0 module set so the chain was a faithful mainnet replica: audit v2, 30 modules, evmigration/evm/erc20/feemarket/precisebank ABSENT which is exactly what lumera-mainnet-1 reports today. DEFECT 1 — no StoreUpgrades --------------------------- v1.20.2 originally declared none, reasoning that testnet already mounted the EVM store keys in v1.20.0 and that re-adding a mounted key is an error. Correct for testnet, wrong for mainnet. Every validator crash-looped: panic: failed to load latest version: version of store evmigration mismatch root store's version; expected 155 got 0; new stores should be added using StoreUpgrades Fixed by declaring the same five EVM store additions as v1.20.0/v1.20.1 and routing v1.20.2 through the existing AddOnlyStoreLoader, which mounts only keys missing from committed state and never deletes one. The declaration and the add-only loader are a MATCHED PAIR — changing one without the other breaks exactly one network while leaving the other green. DEFECT 2 — migrations-only handler on a pre-EVM chain ----------------------------------------------------- With the stores mounted, the next run got further and then panicked: panic: error initializing evm coin info: denom metadata aatom could not be found standardUpgradeHandler runs RunMigrations and nothing else. A pre-EVM chain also needs the v1.20.0 bring-up work: bank denom metadata upsert, Lumera EVM param finalization, and InitEvmCoinInfo. Without it cosmos/evm falls back to the upstream atom denom, which does not exist on Lumera. Fixed by making the handler state-driven, mirroring v1.20.1 so the two cannot drift: EVM modules absent -> delegate to the full v1.20.0 bring-up EVM modules present -> migrations only partially present -> fail closed (not producible by any correct path) Routing is on STATE, never chain-id, so a network arriving by an unexpected path still converges on the same result. EVIDENCE — third run, all gates green ------------------------------------- q upgrade applied v1.20.2 ...... height 114 validators ..................... 200,200,200,200,201 (lockstep) module_versions ................ 30 -> 35 modules audit .......................... v2 -> v3 evm/erc20/feemarket/precisebank/evmigration ... all now present q evm params ................... evm_denom "ulume" (NOT aatom) q evmigration params ........... enable_migration FALSE q audit params ................. readable, uncorrupted bank send ...................... code 0 enable_migration=false on the mainnet path is the security-relevant result: it confirms a chain arriving from 1.12.0 gets the safe default, while testnet (which ran v1.20.0 under the old default) keeps enable_migration=true in committed state and needs an explicit MsgUpdateParams before rollout. TESTS ----- Both defects are now regression-guarded in v1_20_2_store_test.go, written RED-first (StoreUpgrades was undefined until the fix): - TestV1202MountsEVMStoresForMainnetOneHop asserts all five store keys are declared on every chain-id, and that nothing is deleted or renamed. - TestV1202UsesAddOnlyStoreLoader asserts the loader pairing, with adaptive mode off so the path cannot depend on an env flag. TestV1202IsRegisteredAndMigrationOnly was renamed to TestV1202IsRegistered and its StoreUpgrade==nil assertion removed — that assertion encoded the exact wrong premise this rehearsal disproved. The rename keeps the mistake visible in history rather than silently deleting it. RISKS ----- Adds store mounts on a path that previously declared none. Mitigated by the add-only loader (no-op when keys exist) and by the rehearsal covering both arrival shapes. No proto change, no new state keys, no migration beyond the audit module-version bump. ROLLBACK -------- Revert. The upgrade has not executed on any live network, so pre-activation revert is safe.
Correction: testnet should KEEP
|
EnableMigration |
CanaryLegacyAddresses |
Meaning |
|---|---|---|
false |
any | closed — mainnet's post-upgrade default |
true |
empty | open — testnet today |
true |
non-empty | canary — opt-in narrowing |
Canary is a narrowing of an already-open network, entered by adding addresses. It is not a stage every network must pass through, and an open network does not need to be closed to reach it.
So the testnet/mainnet divergence is benign — and correct for each
- Testnet stays open and keeps migrating across the upgrade. This PR lands the audit 2→3 migration, the ante-level canary enforcement and the identity-continuity fixes underneath a live migration flow, with no interruption and no params change.
- Mainnet arrives closed, because it runs
v1.20.0's bring-up for the first time under the new default. Correct posture for a chain that has never had migration enabled — governance opens it deliberately.
The only real requirement was that this asymmetry be known and intentional rather than discovered in production. It now is, and it is test-pinned.
Test added
x/evmigration/keeper/canary_semantics_test.go pins all three states, so nobody "hardens" the empty-list case into a deny and silently breaks every open network at the next upgrade:
TestCheckMigrationActivationEmptyCanaryAllowsEveryone— empty canary admits any address;EnableMigration=falsedominates the allowlist; non-empty canary restricts to listed addresses.TestCanaryIsOptIn— documents closed / open / canary as a three-state model.
Both pass. The revised Phase 4 plan is now "upgrade testnet, change no params, verify migrations continue across the boundary" rather than the disable-then-ramp sequence I originally wrote.
…is ALLOW ALL
Locks down the three-state model that decides whether a live network keeps
migrating across this upgrade.
WHY
---
CheckMigrationActivation (keeper/ante.go:123) reads:
if !params.EnableMigration { return ErrMigrationDisabled }
if len(params.CanaryLegacyAddresses) == 0 { return nil } // ALLOW ALL
An empty allowlist means allow-all, NOT deny-all. Combined with
EnableMigration=true that is "migration fully open", which is exactly the state
lumera-testnet-2 is in today (verified live 2026-07-30: enable_migration=true,
migration_end_time 1790940497 = 2026-10-02, no canary addresses set).
CORRECTION
----------
I had recommended flipping testnet to enable_migration=false before rollout.
That recommendation was wrong and is withdrawn. Migration being open is the
POINT of this release; the continuity work exists to make migration safe, not
to switch it off. Closing a working testnet migration would have been a
self-inflicted regression with no upside.
The resulting three states:
EnableMigration=false -> closed (mainnet post-upgrade default)
EnableMigration=true, canary empty -> open (testnet today, unchanged)
EnableMigration=true, canary non-empty -> canary (opt-in narrowing)
Canary is a NARROWING of an already-open network, entered by ADDING addresses.
It is not a stage every network must pass through, and an open network does not
need to be closed first to reach it.
The testnet/mainnet asymmetry is therefore real but benign, and is the correct
outcome for each: testnet stays open and keeps migrating while this upgrade
lands the audit 2->3 migration, the ante-level canary enforcement and the
identity-continuity fixes underneath it; mainnet arrives closed because it runs
v1.20.0's bring-up for the first time under the new default, which is the right
posture for a chain that has never had migration enabled.
TESTS
-----
- TestCheckMigrationActivationEmptyCanaryAllowsEveryone: empty canary admits any
address and does not discriminate between addresses; EnableMigration=false
dominates even a listed address; a non-empty canary admits listed and rejects
unlisted with ErrMigrationNotCanary.
- TestCanaryIsOptIn: documents closed/open/canary as a three-state model so the
operational meaning of each params combination is unambiguous.
These exist so nobody "hardens" the empty-list case into a deny and silently
breaks every open network at the next upgrade.
RISKS
-----
Test-only. No production code touched.
VERIFICATION
------------
make lint ......................... 0 issues
go test ./x/evmigration/... ....... PASS
go test ./x/... ................... PASS
go test -tags=test ./app/... ...... PASS
git diff --check .................. clean
Phase 2b complete — two-hop rehearsed, and the two mainnet paths produce DIFFERENT stateBoth mainnet-shaped paths are now rehearsed end-to-end on a 5-validator devnet built from the real v1.12.0 release artifact (tarball sha256
Module state converges. evmigration params do not. Why
The absent Not a defect in either path. It is the direct consequence of "params are initialized once, by the binary that performs the bring-up". Recommendation: SINGLE-HOP for mainnet
If two-hop is ever operationally required it is safe, but it is not complete without a follow-up Testnet is unaffectedTestnet stays at The mainnet/testnet difference is intentional and correct for each network: mainnet opens deliberately, testnet never stops. Rehearsal status
Written up in full at |
Correcting myself, and flagging a release-blocking collision with #198First, I was wrong. I said the multi-SuperNode gate was blocked on this PR merging, a chain artifact being cut, and #318 bumping its Measured just now, not assumed: The SuperNode repo already carries five Release blocker: #198 and #199 both define
|
| Status | |
|---|---|
| Chain upgrade correctness (both shapes, both paths) | ✅ Phases 0/1/2/2b |
| Static gates on #199 | ✅ 5/5 CI |
| Local cross-repo build + SN suite | ✅ zero failures |
| SuperNode history preservation | ❌ unproven |
| Pending-SN migration per operator docs | ❌ unproven |
| Validator + multisig migration, mainnet shape | ❌ unproven |
| Everlight continuity (metrics/rewards/payouts) | ❌ unproven |
| Audit module continuity across the boundary | ❌ unproven |
| LEP-6 continuity | ❌ unproven |
| Cascade / reporting / P2P post-migration | ❌ unproven |
| AppHash equality across validators | ❌ never asserted |
Phases 0-2b proved the chain upgrades. They proved nothing about SuperNodes, Everlight, LEP-6, Cascade, or the operator docs. I should not have implied we were close to done.
That last row matters: height lockstep is not divergence-freedom. Two nodes can sit at the same height with different state. We should be asserting byte-equal AppHash across all five validators at N post-upgrade heights, and we currently assert it nowhere.
Plan
Full gate-by-gate plan written up in ROAD-TO-RELEASE-CONFIDENCE.md: one integration branch (#196 + #198 + #199 + #197), two devnet shapes (testnet replica and mainnet replica), ten gates each with a falsifiable pass criterion and required evidence — including negative controls so we can prove a test can detect a difference at all, and literal execution of #197's runbooks rather than paraphrase.
Order: resolve the #198/#199 collision → integration branch green → T-shape gates → M-shape gates → fix docs as bugs surface → only then cut the tag.
Found on a mainnet-shaped devnet while migrating a real fixture cohort: any
legacy account holding an ACTIVE governance deposit could not migrate.
buildGovernancePlan called proto.Clone on a govv1.Deposit. Deposit.Amount is
[]sdk.Coin, whose Amount is an sdkmath.Int wrapping *big.Int. gogoproto's
reflective table-merge descends into big.Int's unexported 'abs []big.Word',
finds no registered merger for big.Word, and panics:
ERR panic recovered in runTx err="recovered: merger not found for type:big.Word
gogoproto/proto.(*mergeInfo).computeMergeInfo table_merge.go:662
x/gov/types/v1.(*Deposit).XXX_Merge gov.pb.go:212
gogoproto/proto.Clone clone.go:52
keeper.buildGovernancePlan migrate_retained.go:417
msgServer.ClaimLegacyAccount msg_server_claim_legacy.go:106
The tx aborts so no state is corrupted, but the account stays unmigratable while
the deposit exists, and the operator-facing error names neither governance nor
deposits - undiagnosable in the field.
Likelihood is high: any account that has submitted a proposal whose deposit is
still held is affected, which includes the governance participants most likely
to migrate first.
Replaced both proto.Clone sites with an explicit cloneGovDeposit deep copy.
Coin.Amount is an immutable sdkmath.Int, so element-wise copy is a correct deep
copy and avoids reflection entirely.
Tests (x/evmigration/keeper/migrate_retained_clone_test.go):
- TestProtoCloneOnGovDepositPanics: RED test pinning that proto.Clone still
panics upstream, so the helper is not 'simplified' back into it later.
- TestCloneGovDepositIsCorrectDeepCopy: value equality AND independence - a
shallow copy would let later mutation leak into the plan's source record,
which is what rollback/verification compares against.
- TestCloneGovDepositEdgeCases: nil Amount, empty slice, and a multi-word
big.Int (2^200) - the exact shape that makes the reflective walk touch
big.Word at all.
Verified live: rebuilt binary sha256 019e427ed4cbc433..., hot-swapped into the
running mainnet-shaped devnet, restarted, re-ran migration -> zero
'merger not found' occurrences in validator logs. ./x/evmigration/... and
-tags=test ./app/... both green.
Known follow-up (NOT fixed here): with the panic gone the same account now
fails with 'stale governance deposit source for proposal 2: value changed' from
the plan's staleness guard. Fails closed, so it is safe, but it needs its own
investigation - tracked in BUGS-AND-FINDINGS.md as BUG-17.
Found and fixed a real chain bug during devnet migration testingRunning the full The bug
ImpactAny legacy account holding an active governance deposit could not migrate. The tx aborts, so there is no state corruption — but:
Likelihood is high, not theoretical: any account that has submitted a proposal whose deposit is still held is affected. On mainnet that includes the governance participants most likely to migrate early. FixReplaced both Tests (
|
| Failure | Verdict |
|---|---|
merger not found for type:big.Word |
this bug — fixed |
stale governance deposit source for proposal 2: value changed |
surfaced behind it, open, under investigation |
legacy address is a validator operator; use MsgMigrateValidator instead |
correct rejection — validators go through migrate-validator |
Known follow-up, not fixed here
With the panic gone, the same account now fails cleanly with stale governance deposit source for proposal 2: value changed from the plan's precondition guard. It fails closed, so it is safe. But nothing mutated that deposit between plan-build and apply in this run, so it looks like a false staleness positive. I am not claiming it is a defect until I have a focused reproduction that distinguishes "guard is over-strict" from "something genuinely rewrites the deposit mid-tx". It blocks the same population this bug did, so it is the next investigation.
Two defects in the retained-state staleness guard, both found on a
mainnet-shaped devnet while migrating a real fixture cohort. Each one
independently prevented a legacy account holding a governance deposit from
ever migrating.
1) proto.Equal is unreliable for Coin-bearing messages
verifyCollectionValue compared the re-read on-chain value against the plan's
expectation with proto.Equal. gogoproto cannot compare sdkmath.Int:
proto: don't know how to compare 2000000000
so proto.Equal returns FALSE for two byte-identical gov Deposits. Migration
failed with
stale governance deposit source for proposal 2: value changed
on a deposit nothing had touched. Same reflection family that panics outright
in proto.Clone.
Replaced with marshalled-bytes comparison (protoBytesEqual), which is the
deterministic, consensus-relevant notion of "unchanged" - it is what the store
actually holds - and avoids reflection entirely. Applied to the authz-grant and
destination-vote comparisons too, which had the same exposure.
2) typed-nil interface defeated the optional-destination check
verifyOptionalCollectionValue took `expected proto.Message` and tested
`expected == nil` to mean "the plan saw no destination entry". Callers pass a
concrete typed pointer, and a nil *govv1.Deposit boxed in an interface is NOT
== nil, so the "absent is fine" branch never ran. The guard fell through to the
comparison path and failed on ErrNotFound:
stale governance deposit destination for proposal 2: collections: not found:
key '("2","lumera1k7del...")' of type ...gov.v1.Deposit
It demanded the destination exist, then failed because it did not - blocking
every account whose target address had no pre-existing deposit, i.e. the normal
case. Added isNilMessage, which detects typed-nil via reflection.
Both fail closed, so no state was corrupted. But they permanently blocked
legitimate migrations and the errors pointed at concurrent mutation that never
happened.
Tests (migrate_retained_staleness_test.go):
- TestProtoEqualIsBrokenForGovDeposits: characterization test pinning the
upstream defect, so the byte comparison is not "simplified" back later.
- TestProtoBytesEqualFixesTheFalseStaleness: identical deposits compare equal,
including via cloneGovDeposit.
- TestProtoBytesEqualDetectsRealChanges: NEGATIVE CONTROL - a comparator that
always returned true would also "fix" the bug while destroying the guard.
Covers changed amount, depositor, proposal id, extra denom, emptied amount.
- TestProtoBytesEqualNilVsEmptyCoins: nil and empty Coins marshal identically
and must not trip the guard.
- TestIsNilMessageCatchesTypedNil: the typed-nil trap for both Deposit and Vote,
asserting the raw `== nil` is false while isNilMessage is true.
Verified live on the mainnet-shaped devnet (v1.12.0 -> v1.20.2 single-hop, 5
validators, 5 supernodes, 19-account fixture cohort):
before: 16/19 migrated, blocked by the panic
then: 16/19, blocked by "source: value changed"
then: 16/19, blocked by "destination: not found"
now: 17/19 - 17 migration records on chain
The 2 remaining failures are validator operators correctly rejected with
"use MsgMigrateValidator instead".
Gates: ./x/evmigration/... green, -tags=test ./app/... green,
-tags='integration test' ./tests/integration/evmigration/... green,
make lint 0 issues.
Two more defects in the same guard family — found by continuing past the first green runFollowing the BUG-17 —
|
| State | Result |
|---|---|
| baseline | 16/19 — panic: merger not found for type:big.Word |
| + BUG-16 fix | 16/19 — source: value changed |
| + BUG-17 fix | 16/19 — destination: not found |
| + BUG-19 fix | 17/19 — 17 migration records on chain |
The 2 remaining failures are validator operators correctly rejected with use MsgMigrateValidator instead — that is the validator-migration path, not a defect.
Tests
TestProtoEqualIsBrokenForGovDeposits— characterization test pinning the upstream defect, so the byte comparison is not "simplified" back later. If gogoproto ever fixes this, the test fails and tells us.TestProtoBytesEqualFixesTheFalseStaleness— identical deposits compare equal, including viacloneGovDeposit.TestProtoBytesEqualDetectsRealChanges— negative control. A comparator that always returnedtruewould also make the symptom vanish while destroying the guard entirely. Covers changed amount, depositor, proposal id, extra denom, emptied amount.TestProtoBytesEqualNilVsEmptyCoins— nil and emptyCoinsmarshal identically and must not trip the guard.TestIsNilMessageCatchesTypedNil— the typed-nil trap for bothDepositandVote, asserting the raw== nilis false whileisNilMessageis true. (Worth noting: testify'srequire.NotNiluses reflection and sees through typed-nil, so it cannot demonstrate this — the raw comparison has to be asserted directly.)
Gates
./x/evmigration/... green · -tags=test ./app/... green · -tags='integration test' ./tests/integration/evmigration/... green · make lint 0 issues · verified live on the mainnet-shaped devnet.
Reviewer note
The general lesson, and why I kept going after the first fix went green: fail-closed is not the same as correct. A guard that cannot be satisfied is an outage with good manners. Unit tests passed throughout all three defects — only a mainnet-shaped devnet with a real fixture cohort (19 legacy accounts with delegations, unbondings, redelegations, third-party withdraw addresses, authz, feegrants, claims, actions, multisig and permanent-locked) surfaced them. Stopping at the first green run would have shipped two of these.
…migration
Closes a coverage hole where nothing - unit or devnet - proved that a
supernode's evidence history and metrics survive an identity migration.
The gap was vacuous-green in both places:
- devnet fixtures report evidence=0 and has_metrics=false on every
supernode, so "evidence preserved" passed because there was nothing
to preserve;
- the keeper fixture rawTestSuperNode() carries no evidence either.
This test supplies the non-empty state both lack: three evidence entries with
distinct reporters, types and heights, plus metrics with a non-zero
ReportCount.
Contract note (corrected after reading the existing test): the primary
supernode record is VALIDATION-ONLY in ApplyIdentityMigrationPlan and is
deliberately NOT moved -
// Primary/account/history are owned by PR196 and are validation-only here.
require.Equal(t, sourcePrimaryRaw, store.Get(types.GetSupernodeKey(source)))
require.Nil(t, store.Get(types.GetSupernodeKey(destination)))
so the correct invariant at this layer is not "evidence moves" but "evidence is
left byte-identical at the source, untouched" while the continuity state the
plan does own (metrics, distribution, payout history) relocates. The test also
asserts no record is conjured at the destination.
Asserted:
- evidence count, reporter, type and height unchanged, in order
- metrics follow the identity with exact values, ReportCount included
- metrics do NOT remain at the source (duplicate state would double-count
in audit/payout aggregation)
Proven non-vacuous by mutation testing (evidence/g4_mutation_evidence.sh),
3/3 mutants detected:
- metrics move dropped -> "metrics must follow the migrated identity"
- metrics ReportCount zeroed -> "report count must be preserved"
- one evidence entry silently dropped -> "should have 3 item(s), but has 2"
Baseline passes before and after; the script restores the source tree and
re-verifies. Note a first attempt at the third mutant injected into
ApplyIdentityMigrationPlan where sourceSN is out of scope: it failed to
compile, which is INCONCLUSIVE rather than a pass, and was fixed to inject
into BuildIdentityMigrationPlan.
Gates: ./x/supernode/... green, ./x/evmigration/... green,
golangci-lint ./x/supernode/... 0 issues.
G1b (AppHash equality) and G2 evidence/metrics — two more gates closedG1b — AppHash equality was NEVER asserted anywhereEvery upgrade rehearsal to date verified only that blocks were produced after the upgrade. That proves liveness, not agreement — a validator can keep producing blocks while holding divergent state, and CometBFT only halts once the divergence is actually voted on. So "the upgrade succeeded" has never meant "all validators computed the same state". Added At the upgrade block itself (2753) — where a non-deterministic handler would diverge: Sweep across the critical window — 9/9 PASS, 0 fail, 0 inconclusive:
Non-vacuity proven: This establishes that the G2 — evidence/metrics preservation was vacuous in TWO placesNot just the devnet (
Contract corrected by reading the code. My first draft asserted evidence moves to the destination — and failed. The existing test states the real contract: // Primary/account/history are owned by PR196 and are validation-only here.
require.Equal(t, sourcePrimaryRaw, store.Get(types.GetSupernodeKey(source)))
require.Nil(t, store.Get(types.GetSupernodeKey(destination)))
Mutation-tested, 3/3 detected (
Where the branch stands5 of 9 release gates now green, each backed by a falsifiable check plus a negative control:
Gates on this commit: Not a ship recommendation. G3–G8 need running supernode daemons and remain untested; the devnet's SNs are all |
Operator runbook addendum — every trap that actually bit during rehearsalFull mainnet-shaped rehearsal is done (v1.12.0 → v1.20.2 single-hop, 5 validators, 19-account legacy fixture). Beyond the three chain bugs fixed on this branch, the rehearsal surfaced a set of operator-facing traps that will cost downtime if they aren't documented. None of them are hypothetical — each one bit me. Written up as 1. Release order is rigidSuperNode v2.6.4 cannot build against any released chain — it needs 2. Migrating an SN account is not the whole job
Good news, and worth crediting: this fails loudly. The daemon's 3.
|
enable_migration |
canary_legacy_addresses |
Effect |
|---|---|---|
false |
any | Closed |
true |
empty | OPEN to everyone (not deny-all) |
true |
non-empty | Canary — only listed |
An empty allowlist with migration enabled is allow-all.
7. Verifying an upgrade: block production ≠ agreement
A validator can keep producing blocks while holding divergent state; CometBFT only halts once the divergence is voted on. Assert AppHash equality across validators at the upgrade height — more than one distinct hash is consensus divergence even with every node producing. An unreachable validator is inconclusive, not a pass.
Gate status on this branch: 5 of 9 green (G1 upgrade correctness, G1b AppHash equality, G2 SN history, G9 collision, G10 integration), each with a negative control or mutation suite.
G3–G8 (Everlight, audit continuity, LEP-6, Cascade) remain unproven — they need a live supernode, and per item 3 above the devnet's supernodes have never been able to start. That is a pre-existing provisioning defect, now root-caused, but it means those gates are blocked rather than passing. I'd treat this branch as production-gated on the migration/upgrade core only.
G3 + G5: supernodes brought up, audit continuity PROVEN across migrationTwo updates: the supernode blocker is resolved (without touching chain state), and that unlocked the first real proof of audit-path continuity. G3 — supernodes are runningThe devnet's supernodes had never started — all 5 sat The supernode daemon uses its own keyring ( Fixed config-only, no chain change. The key holding each migrated SN account already existed as So no G5 — audit continuity across EVM migration 🟢 PASSThe invariant: after an SN account migrates legacy → EVM, the audit module must keep accepting that node's epoch reports under the new identity with no coverage gap. If migration broke the audit path, reports would stop at the migration height. 5/5 PASS: ~20 consecutive epochs per node, every report under the post-migration account, zero gaps. Live on-chain, actively advancing ( Non-vacuity: 2/2 mutants detected — injected coverage gap → Scope, stated plainly: this proves the host-report path end-to-end (daemon signature → audit acceptance → storage). It does NOT prove storage-challenge or storage-proof — every report carries Observation for the audit-module owner (not filed as a bug)4/5 supernodes stay POSTPONED, and the cause is protocol logic, not config. Every health gate passes ( // Bootstrap exception: when the epoch's anchored active set is empty, no
// probers exist by construction, so the peer-port recovery rule below is
// unsatisfiable and would deadlock the chain (all SNs POSTPONED → 0
// probers → 0 peer reports → no SN can ever recover).
if ... len(anchor.ActiveSupernodeAccounts) == 0 { return true, nil }It fires only at exactly zero. This devnet has one active supernode, so the exception doesn't apply — but a single active node cannot peer-attest for anyone but itself, leaving four healthy nodes permanently unrecoverable. Not a mainnet defect (healthy active sets are >> 1), but a genuine recovery edge case at active-set size 1 — exactly the degraded state you'd most want to recover from. Question: should the exception trigger when the active set is too small to supply any peer prober ( Not filed as a defect — it needs the module owner's intent on whether size-1 recovery is supported. Gate status: 6 of 9
Chain gates on |
…testing)
Class-A storage-truth fault accounting had NO test coverage. Mutation testing
proved it: forcing `isClassA = false` in updateNodeSuspicionHistoryFields - so
HASH_MISMATCH and RECHECK_CONFIRMED_FAIL are never counted - left the entire
storage-truth suite GREEN.
ClassACountWindow / LastClassAEpoch / CleanPassCount gate band escalation and
recovery. Without these assertions, a regression could let a node with repeated
hash mismatches (the strongest evidence of storage dishonesty) accumulate no
suspicion and recover as if clean, with nothing failing.
Adds two tests:
TestUpdateNodeSuspicionHistoryFields_ClassAFaultAccounting
HASH_MISMATCH and RECHECK_CONFIRMED_FAIL each must
- increment ClassACountWindow (added to a pre-existing count, not replaced)
- stamp LastClassAEpoch
- reset CleanPassCount (recovery requires clean passes with no new Class A)
- leave ClassBCountWindow untouched
TestUpdateNodeSuspicionHistoryFields_ClassBDoesNotTouchClassAGates
TIMEOUT_OR_NO_RESPONSE must increment ClassB counters ONLY and must not
touch Class-A gates. The code comment states TIMEOUT-on-INDEX "must not
reset Class-A recovery gates or increment ClassACountWindow"; that
separation was previously unasserted in both directions.
Verified the new coverage is real - with the mutant injected the suite now
FAILS on both Class-A classes, and passes once restored:
--- FAIL: ..._ClassAFaultAccounting/STORAGE_PROOF_RESULT_CLASS_HASH_MISMATCH
--- FAIL: ..._ClassAFaultAccounting/STORAGE_PROOF_RESULT_CLASS_RECHECK_CONFIRMED_FAIL
Two fixture requirements worth noting for future tests here:
- the counters live inside an `isFailure` branch keyed on BucketType /
ArtifactClass, so a bare {ResultClass} result does not exercise them;
- WindowStartEpoch must be fresh, otherwise the stale-window reset zeroes
ClassACountWindow before the increment and masks the assertion.
Gates: ./x/audit/... green, golangci-lint ./x/audit/... 0 issues.
1 participant
PR2 — EVM migration runtime continuity core
Stacked on #196 (
a79d3628, untouched). Merge #196 first.Chain head:
d7504279· SuperNode companion:LumeraProtocol/supernodematee/evmigration-continuityWhat this changes
PR #196 fixed the destructive SuperNode-history rewrite and hardened source-account ownership. It is not a complete cross-module identity migration. This PR implements forward runtime continuity for account/validator migrations, and keeps migration disabled by default until the full gate passes.
Per the GOLDEN analysis, migration currently loses or splits identity-keyed state across seven independent surfaces. Each commit closes one.
32f38d3fsnm_/rdist/, keepsrhist/immutable, collision + orphan detection55822e710d3685bcdisabled/canary/open);enable_migrationnow defaults false9015af7e2c5818c51864d947RemoveUnbondingDelegation/RemoveRedelegationdo not clean timeslices, so old UBD/RED queue tuples were never removed and could wedge staking EndBlock267fcba2StakeAuthorizationallow/deny valoper rewrite, gov votes/deposits, third-party withdraw addrs, bounded caps, field-7 zero-value compata1ead8281b8f1b84d7504279State keys
Moved exactly once (source deleted, destination asserted absent first):
snm_<val>latest metrics ·rdist/<val>Everlight accumulator · audit live singletons (suspicion, reliability, action/storage markers) · staking delegation/UBD/RED primaries and their maturity-queue timeslices · authz grants · gov votes/deposits · distribution withdraw addresses · action Creator (pre-finalization).Immutable, never rewritten:
rhist/payout history · audit anchors/reports/evidence/facts/transcripts · action SuperNodes after finalization · everything in terminal action states.Added:
AccountTransitionlineage (forward + reverse index) ·EpochReport.current_submitter(field 7) ·Params.canary_legacy_addresses(6),Params.max_retained_state_entries(7).ABCI phases
Upgrade requirement
A coordinated consensus activation is required. Old and new binaries execute the same migration transaction differently, so a mixed-binary rollout while migration is executable is consensus-unsafe.
enable_migrationnow defaults to false, so the safe posture is: ship with migration disabled, then activate via governancedisabled → canary → openonly after the full gate passes.New persisted state (audit transition lineage, report field 7, two new params) means audit and evmigration module-version migrations are included. Field 7 zero-values decode correctly for params serialized before it existed (
EffectiveMaxRetainedStateEntries()).Risks
1864d947). It touches shared timeslices consumed by staking EndBlock; a wrong write wedges the chain. Mitigated by exact-tuple preconditions (oldCount == 1 && newCount == 0), re-verification of every snapshot immediately before the first write, and destination-collision + delegation starting-info preflight.max_retained_state_entries(default 10000), rejecting at cap+1 before any write.Rollback
Migration is disabled by default, so the deployed-but-inactive state is the rollback state. Emergency governance
MsgUpdateParamssettingenable_migration=falsehalts all further migrations; it cannot reverse already-committed ones. Reverting the branch pre-activation is safe because no migration can have executed.Non-goals
Repairing already-migrated testnet state · rewriting historical reports/evidence/payouts/anchors/transcripts · generic Wasm/EVM contract-state ownership transfer · ICA/group/circuit authority transfer · broad query/UI aggregation.
Observability
Migration events carry source/destination identity, per-family migrated counts, collision-rejection family, audit lineage activation, Everlight state move, and migration record ID. No secrets or proof material. Blocker codes are a stable enum shared by preflight and operator output.
Evidence
Local CI parity on the exact head
d7504279, immutable tree:Cross-repo, via a throwaway
go.work(never committed): the SuperNode companion builds against this chain head, andpkg/lumera/modules/audit,supernode/host_reporter,supernode/storage_challengeall pass.Mutation-tested, not just green — two assertions were verified to fail when the behavior they guard is broken:
AccountForEpochwith the rawCreatormakes the cohort suite fail on migrated actors;Notable coverage added here:
none/one/some/allmigrated cohort matrix; current-epoch and next-epoch report continuity; Everlight next-tick payout equality against a never-migrated control, with a negative control proving a lost accumulator does change the payout; BaseApp late-failure rollback (failing in step 7 of 7, after every other module has written); CheckTx/ReCheckTx/Simulate state neutrality.One qualified ante side effect is asserted explicitly rather than hidden: wasmd's
CountTXDecoratorcommits a per-block tx counter at wasm key0x08for any block-admitted tx. It carries an 8-byte height and 4-byte count, no identity.Not yet done — do not enable canary before these
FROM_TAG→TO_TAGupgrade rehearsal on mainnet-shaped statemake system-tests/make systemex-tests