Skip to content

fix(evmigration): preserve SuperNode history during migration - #196

Open
mateeullahmalik wants to merge 21 commits into
masterfrom
matee/evmigration-history-repair
Open

fix(evmigration): preserve SuperNode history during migration#196
mateeullahmalik wants to merge 21 commits into
masterfrom
matee/evmigration-history-repair

Conversation

@mateeullahmalik

@mateeullahmalik mateeullahmalik commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Short version

This PR fixes the SuperNode ownership record during EVM account and validator migration.

For example, suppose Account A owned a SuperNode from height 100 and the ownership moved to Account B at height 500. The old migration code could rebuild the record with Account B in both history entries. The result looked as though Account B had owned the SuperNode since height 100.

After this change, the existing Account A entry is left unchanged and one Account B entry is added at the migration height.

This PR fixes that ownership-history bug. It does not yet move every other piece of chain state that may refer to the old account or validator address. The known gaps are listed below so this PR is not mistaken for a complete state-continuity fix.

What this PR changes

Account migration

If the old account owns a SuperNode, the migration now:

  • loads the existing SuperNode record instead of constructing a replacement;
  • changes the current SupernodeAccount to the new account;
  • keeps the previous account history exactly as it was;
  • appends one history entry for the new account at the current block height;
  • updates the SuperNode account index without changing the validator-keyed primary key.

Validator migration

If the old validator has a SuperNode, the migration now:

  • moves the existing SuperNode record from the old validator key to the new validator key;
  • keeps the SuperNode lifecycle history;
  • moves the latest SupernodeMetricsState to the new validator key;
  • updates validator addresses stored in the SuperNode's embedded evidence;
  • changes the SuperNode account only when the validator was using its own account as the SuperNode account;
  • leaves an independently owned SuperNode account unchanged.

A validator migration can involve two different relationships at the same time: a SuperNode owned by the validator's account, and a SuperNode registered against the validator but owned by another account. The migration now validates and preserves both instead of assuming they are the same record.

Ownership checks

Before writing migration state, the code now:

  • decodes account addresses and compares the underlying address bytes instead of comparing Bech32 text;
  • checks both the primary SuperNode records and the account index;
  • rejects stale, duplicate, malformed, or conflicting ownership;
  • rejects migration when the destination account already owns a SuperNode.

This matters because two valid Bech32 strings can decode to the same account bytes. A string-only lookup is not enough to prove that the destination is unused.

What this PR does not fix

Everlight continuity when the validator address changes

Everlight stores per-SuperNode distribution state under the validator address:

  • rdist/<validator> contains the EMA, previous raw usage, eligibility start height, and active-period count;
  • rhist/<validator>/... contains payout history.

rdist/ IS migrated by this PR. BuildIdentityMigrationPlan re-keys it from the old to the new validator address, preserving smoothed_bytes, prev_raw_bytes, eligibility_start_height, and periods_active verbatim (x/supernode/v1/types/identity_migration_plan.go). Smoothing baseline and ramp-up therefore do not reset. This was corrected after review; an earlier revision of this description incorrectly listed rdist/ as unmigrated.

rhist/ is NOT migrated. Payout history remains split across the old and new validator addresses. PayoutHistoryPrefixForValidator is read only by query_get_payout_history.go, so the effect is confined to the payout-history query returning a partial series for a migrated validator. It does not feed eligibility or weight, so it has no effect on distribution amounts.

This gap applies when the validator address changes. An account-only SuperNode migration that leaves the validator address unchanged does not re-key these Everlight records.

LEP-6 and audit continuity when the SuperNode account changes

The audit module stores many records using the SuperNode account string. This includes epoch reports and their indexes, evidence subject indexes and epoch counts, frozen epoch participant sets, suspicion and reporter-reliability state, postponement markers and reasons, storage-truth fact indexes, heal operations, verifier votes, and recheck records.

PR #196 does not rewrite those records when SupernodeAccount changes. The new account may be the current owner in x/supernode while the supporting audit state still refers to the old account.

The highest-risk case is migration during an active audit epoch. The epoch anchor contains a frozen participant list. Replacing the old account in live SuperNode state does not automatically replace it in that frozen list, and blindly rewriting a frozen epoch would also change the meaning of an epoch already in progress. We need an explicit rule for whether migration is blocked, deferred, or carried across an epoch boundary before implementing this part.

This audit gap applies when the SuperNode account changes. If only the validator address changes and an independently owned SuperNode keeps the same account, its account-keyed audit identity does not change.

Existing damaged history

This PR prevents new ownership-history corruption. It does not repair records already changed by an earlier migration.

Other address-keyed state

Everlight and audit/LEP-6 are confirmed gaps, not necessarily the complete list. The remaining modules and secondary indexes still need a migration audit before we call the overall account/validator migration complete.

Why the migration rejects conflicts

Choosing an arbitrary record when ownership is ambiguous would make the result depend on corrupt or duplicate state. The safer result is to reject the transaction before the first write. The operator can inspect the conflicting records and correct the cause instead of committing a migration that overwrites another SuperNode owner.

State changes in this PR

On a successful account migration:

  • the validator-keyed SuperNode primary key stays the same;
  • the old SuperNode account index is removed;
  • the destination account index is added;
  • the existing SuperNode value is updated with the new current account and one new history entry.

On a successful validator migration:

  • the old validator-keyed SuperNode primary key is removed;
  • the preserved SuperNode record is written under the new validator key;
  • the latest SuperNode metrics value is moved to the new validator key;
  • the SuperNode account index is updated only if the account itself changes.

The existing EVM migration flow still handles auth, bank, staking, Cosmos SDK x/distribution, delegations, fee grants, and action references. This PR does not add writes to Everlight's rdist/ or rhist/ keys or to the x/audit store.

Transaction and replay behavior

  • CheckTx: no new state writes. Existing message, proof, ante, and mempool checks still apply.
  • DeliverTx: ownership and destination checks run before migration writes. Any later error rolls back with the Cosmos SDK transaction cache.
  • BeginBlock / EndBlock: unchanged.
  • Replay is deterministic. The code uses decoded address bytes and deterministic store iteration; it does not use time, network calls, randomness, files, or other off-chain input.

Migration and upgrade impact

Risks

  • The strict ownership check scans SuperNode primary records during migration. The scan is deterministic and gas-metered, but costs more than one index lookup.
  • State that was previously tolerated despite a stale or conflicting account index will now cause the migration to fail. This is intentional; committing an ambiguous ownership change would be worse.
  • The remaining Everlight and audit gaps can split state between old and new identities. Operators should not treat this PR alone as proof that all module state survives migration.
  • Validator downtime and multisig coordination are still operational procedures. This PR does not make those external steps atomic.

Rollback

Before any migration transaction is committed, the code can be rolled back normally.

After a successful migration transaction, rolling back the binary does not reverse the committed state. Reversal would require an explicit corrective state transition or restoration from a pre-migration snapshot. For that reason, broad rollout should wait until the remaining module-continuity rules are either implemented or explicitly accepted.

How this was verified

The focused keeper and integration tests cover:

  • preserving the existing account timeline and adding one migration-height entry;
  • self-owned and independently owned SuperNode relationships;
  • stale and conflicting indexes;
  • destination ownership collisions, including alternate valid Bech32 encodings of the same address bytes;
  • rollback on validation failure.

The full script suite and lint pass on the stacked source tree.

A historical devnet rehearsal also completed from the v1.12.0 lineage through v1.20.2 with:

  • 5 SuperNodes registered using v2.5.3-testnet;
  • 5 standalone SuperNode account migrations;
  • 4 single-signature validator migrations;
  • 1 two-of-three multisig validator migration;
  • 10/10 migration records and 10/10 delegations verified;
  • 5/5 validators bonded and none jailed;
  • the core x/supernode records and account histories preserved;
  • block production continuing after the final candidate rollout.

That rehearsal proves the ownership-record behavior fixed here. It did not pre-populate and verify every Everlight or audit/LEP-6 state family across migration, so those continuity gaps remain open.

Observability

Operators can verify migration results through the migration transaction, migration record, emitted events, validator and SuperNode queries, delegation queries, and node logs. This PR adds no new off-chain telemetry dependency.

Stack order

  1. This PR: preserve and validate the core SuperNode ownership record
  2. app: register v1.20.2 migration-only upgrade handler #198: add the v1.20.2 migration-only upgrade handler
  3. docs(ops): add fail-closed EVM migration runbooks #197: add operator tooling and runbooks

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR hardens x/evmigration’s SuperNode handling so account/validator migrations preserve SuperNode ownership history and fail closed on ambiguous or corrupt ownership state, aligning migration execution and estimation with the true ownership model in x/supernode.

Changes:

  • Add strict SuperNode ownership resolution that scans both primary records and the account index using decoded address bytes, detecting duplicates/stale/malformed state.
  • Rework account and validator migration flows to prevalidate SuperNode ownership against pristine state, preserve existing ownership history verbatim, and append a single migration-height entry when the effective owner changes.
  • Update DI wiring and mocks/tests (unit + integration + devnet checks) to use the strict ownership path and cover failure/rollback scenarios.

Reviewed changes

Copilot reviewed 17 out of 18 changed files in this pull request and generated no comments.

Show a summary per file
File Description
x/supernode/v1/module/depinject.go Exposes the concrete Supernode keeper via depinject for modules needing strict ownership behavior.
x/supernode/v1/keeper/supernode_raw.go Implements strict ownership resolution by scanning primary + index and comparing decoded address bytes.
x/supernode/v1/keeper/supernode_raw_internal_test.go Adds internal tests for strict scanning behavior, terminal iterator errors, and state immutability.
x/evmigration/types/expected_keepers.go Extends the expected Supernode keeper interface to include strict ownership lookup.
x/evmigration/module/depinject.go Switches Supernode dependency to the concrete keeper to access strict ownership APIs.
x/evmigration/mocks/expected_keepers_mock.go Updates mocks to include StrictGetSuperNodeByAccount.
x/evmigration/keeper/query.go Makes MigrationEstimate use strict ownership logic (and validator dual-dimension validation) for correct HasSupernode reporting.
x/evmigration/keeper/query_test.go Updates/extends estimate tests to match strict ownership behavior and error propagation.
x/evmigration/keeper/msg_server_migrate_validator.go Adds strict preflight validation for validator SuperNode dimensions before the first mutation, and uses validated plans during V5.
x/evmigration/keeper/msg_server_migrate_validator_test.go Updates validator migration msg-server tests for strict preflight + validated SuperNode migration.
x/evmigration/keeper/msg_server_claim_legacy.go Adds strict preflight ownership resolution + destination collision checks and passes validated SN data into migration.
x/evmigration/keeper/msg_server_claim_legacy_test.go Updates claim tests to reflect strict preflight behavior and early rejection before mutation.
x/evmigration/keeper/migrate_validator.go Implements validator SuperNode ownership validation plan + preserves history while re-keying validator-associated records and metrics.
x/evmigration/keeper/migrate_test.go Updates migration unit tests to assert history preservation and alternate-encoding behavior.
x/evmigration/keeper/migrate_supernode.go Refactors supernode migration into a “validated record” path and preserves prior history verbatim.
tests/scripts/devnet-makefile.bats Ensures dry-run targets don’t recurse into real make during tests by overriding MAKE.
tests/integration/evmigration/supernode_ownership_execution_test.go Adds integration coverage for fail-closed ownership validation and no-mutation guarantees on failure.
devnet/tests/evmigration/migrate_validators.go Updates devnet verification to assert SuperNode history immutability (except for one appended migration entry).
Files not reviewed (1)
  • x/evmigration/mocks/expected_keepers_mock.go: Generated file

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

mateeullahmalik and others added 12 commits August 1, 2026 22:37
…tor migration

Validator migration moved the SuperNode primary and latest metrics but never
touched the validator-keyed Everlight accumulator at `rdist/<validatorText>`.
The distribution loop reads and writes that key (x/supernode/v1/keeper/
distribution.go:124-133, 264-272), so after an operator-address change the
next usable observation initialises a fresh SNDistState:

  - SmoothedBytes (EMA baseline) resets
  - PrevRawBytes = 0, bypassing usage_growth_cap_bps_per_period for the
    first observation
  - PeriodsActive resets, re-entering the new_sn_ramp_up_periods ramp
  - EligibilityStartHeight resets
  - the old rdist/<oldValidator> row is orphaned permanently

This is mutable accounting state that feeds payout weight. Unlike the audit
and query-lineage gaps, it cannot be reconstructed by a lineage-aware query
or an off-chain indexer, so it has to be carried in consensus.

Observed on testnet: two validators that migrated under stock v1.20.1 have
payout rows on both the old and the new validator prefix, and the post-
migration row re-enters at ramp_weight 0.25 despite the lineage having
already accumulated ramp periods. Mainnet is mid-ramp right now (payout
height 6057000: 5 payees at ramp_weight 0.50 with smoothed != raw), so the
same migration there would reset live EMA baselines.

Introduce a SuperNode-owned immutable Build/Apply plan that moves exactly the
two validator-keyed families it owns — `snm_` latest metrics and `rdist/`
SNDistState. The plan is built before the first write and fails closed on a
destination collision or a malformed source row; malformed bytes are an error,
never treated as absence. It is built unconditionally rather than gated on a
validator-associated SuperNode primary, because distribution residue can exist
under `rdist/` with no primary present.

The ad-hoc inline metrics move in migrateValidatedValidatorSupernode is
removed; the plan is now the single owner of both families, so evmigration no
longer choreographs SuperNode-internal writes directly.

State keys: moves `snm_<oldVal>` -> `snm_<newVal>` and `rdist/<oldVal>` ->
`rdist/<newVal>`. Payout history (`rhist/`) is deliberately untouched --
those rows are immutable historical facts and stay under the validator that
earned them.

ABCI phases: DeliverTx only. No BeginBlock/EndBlock logic is added. The
resulting state is consumed by the existing EndBlock distribution loop.

Determinism: deterministic KV iteration over a bounded, capped prefix scan;
no wall clock, network, randomness, or map ordering.

No proto change, no new store, no module consensus-version bump.

Ten existing strict-mock tests in x/evmigration/keeper still expect the
removed inline GetMetricsState/SetMetricsState/DeleteMetricsState sequence
and fail on this commit. They are repaired in the following commit.
The previous commit moved the latest-metrics move out of migrate_validator.go
and into the SuperNode-owned identity migration plan, which also carries the
Everlight SNDistState move. Ten strict-mock tests still asserted the removed
inline GetMetricsState/SetMetricsState/DeleteMetricsState sequence and failed.

Repair them by asserting the new contract rather than deleting the coverage:

  - migrate_test.go gains expectIdentityMigrationPlan, which pins the source
    and destination validator, feeds a realistic marshalled metrics payload,
    and asserts the plan handed to Apply is the exact plan returned by Build.
  - the msg-server fixtures gain expectIdentityMigrationPlanBuilt (pre-V1,
    MaxTimes(1)) and expectIdentityMigrationPlanApplied (V5, exact Times(1)).

Build and Apply are asserted SEPARATELY and asymmetrically, on purpose. A
single combined MaxTimes(1) allowance for both was written first and a mutant
proved it vacuous: deleting the Apply call at V5 still passed, because
MaxTimes permits zero. Splitting them, with Apply pinned to exactly one call
in the tests that reach V5, closes that hole. Tests that abort before V5
install no Apply expectation at all, so a premature apply is caught as an
unexpected call.

The build matcher also asserts source == old validator operator address and
source != destination. A second mutant proved this necessary: transposing the
arguments to Build(ctx, new, old) -- which would move continuity state in the
wrong direction -- passed cleanly while the addresses were matched with
gomock.Any().

Deliberately NOT using a blanket .AnyTimes() stub for the plan calls. That
would make every one of these tests silently tolerant of the exact regression
class this commit exists to catch.

Mutation testing, 5/5 detected:

  drop Apply at V5 (msg-server path)      -> 6 tests fail
  drop Apply (direct keeper path)         -> 9 tests fail
  transpose source/destination, msg-server-> 9 tests fail
  transpose source/destination, keeper    -> 9 tests fail
  apply the plan twice                    -> 6 tests fail

A sixth mutant, removing the pre-V1 Build entirely, is caught at compile time.

go test ./x/evmigration/... ./x/supernode/... -count=1  -> all packages ok
Restores the v1.20.2 upgrade that commit a79d362 reverted, in the shape the
revert was needed for.

# WHY THE UPGRADE IS REQUIRED

This PR changes DeliverTx outcomes for the SAME migration transaction:

  - old code rewrote existing PrevSupernodeAccounts rows from legacy to
    destination; new code preserves them and appends one transition;
  - old code resolved SuperNode ownership by literal text and tolerated stale
    or duplicate index entries; new code resolves canonically and rejects them;
  - old code orphaned the validator-keyed Everlight SNDistState on an operator
    address change; new code moves it with the validator.

Two validators on different binaries therefore commit different state for the
same block. This release must not be rolled out node-by-node while migration
transactions can execute; it needs one named halt height. There is no store
migration and no module consensus-version bump here -- making the behavior
change atomic across the validator set is the entire purpose, and that is a
sufficient and standard reason for a Cosmos upgrade boundary.

# WHY THE REVERTED VERSION COULD NOT BE RESTORED AS-IS

The reverted commit registered a standard migrations-only handler with no
StoreUpgrades:

    case upgrade_v1_20_2.UpgradeName:
        return UpgradeConfig{Handler: standardUpgradeHandler(...)}, true

That is correct for testnet and fatal for mainnet. Live state, verified
2026-08-01:

    lumera-mainnet-1   app_version 1.12.0   audit v2   EVM stack ABSENT
    lumera-testnet-2   app_version 1.20.1   audit v2   EVM stack PRESENT

Mainnet arrives from 1.12.0 with no EVM stores mounted and has run neither
v1.20.0 nor v1.20.1, so a migrations-only v1.20.2 panics twice over:

    panic: failed to load latest version: version of store evmigration
    mismatch root store's version; expected N got 0

    panic: error initializing evm coin info: denom metadata aatom could not
    be found

The handler is therefore state-driven, exactly like v1.20.1: it inspects
fromVM, never chain-id, and serves both arrival shapes from one binary --
full v1.20.0 EVM bring-up when the stack is absent, migrations only when it is
present. Partial EVM state cannot arise from any correct path and fails closed
rather than guessing a branch.

Because both surviving shapes are precisely what v1.20.1 already implements
and has been rehearsed on, the handler delegates to it instead of duplicating
the branch, so the two cannot drift apart in a later edit. StoreUpgrades is
aliased to v1_20_0.StoreUpgrades for the same reason -- one declaration, not a
hand-copied list.

# STORE LOADER ROUTING

store_loader_selector.go routes v1.20.2 to the add-only store loader alongside
v1.20.1, on every network and regardless of the adaptive env flag. The
add-only loader mounts declared keys missing from committed state and never
deletes a store: a no-op on testnet, the full EVM mount on the mainnet
one-hop. Omitting this routing reintroduces the store-version panic above.

# TESTS

  app/upgrades/v1_20_2/upgrade_test.go
    - upgrade name pinned (governance --name / cosmovisor dir / q upgrade applied)
    - both live arrival shapes plus the inconsistent one
    - partial EVM state aborts, returns no version map, and names the missing
      modules so an operator can act on it
    - StoreUpgrades is the v1.20.0 declaration, includes evmigration, and
      deletes/renames nothing
  app/upgrades/upgrades_test.go
    - v1.20.2 registered with a handler AND store upgrades on mainnet, testnet
      and devnet
    - add-only store loader selected with adaptive both off and on

Mutation testing, 4/4 detected:

  drop StoreUpgrade from the registry entry   -> 2 tests fail
  remove v1.20.2 from upgradeNames            -> 1 test fails
  remove the partial-EVM fail-closed branch   -> 1 test fails
  route EVM-present state to the bring-up     -> 1 test fails

Dropping the add-only loader routing is caught at compile time.

go test ./app/upgrades/... -count=1 -> all packages ok

Pre-existing and unrelated: TestEVMMempoolDisabledWhenMaxTxsIsNegative in
package app fails identically on untouched PR #196 head a79d362.
Exercises the real v1.20.2 handler through SetupUpgrades against a real app
with real keepers, once per live network shape, instead of asserting only on
the fromVM partitioning helper.

Live state these tests encode (verified 2026-08-01):

    lumera-mainnet-1   1.12.0   audit v2   EVM ABSENT   -> full bring-up
    lumera-testnet-2   1.20.1   audit v2   EVM PRESENT  -> migrations only

Mainnet one-hop: asserts the handler applies Lumera EVM params over
cosmos/evm's upstream "aatom" defaults (params are deliberately clobbered
first, so a pass proves the handler re-applied them rather than finding them
already correct), mounts the evmigration store, and sets migration_end_time to
block time + 3 months.

Testnet: seeds the live migration_end_time observed on lumera-testnet-2
(1790940497) and asserts the upgrade LEAVES IT UNTOUCHED. This is the
assertion that matters most for testnet -- both v1.20.0 and v1.20.1 are
already spent there, and if v1.20.2 were to re-run the bring-up it would
recompute the deadline and stomp the window governance is currently running
against.

Also covered:
  - store declaration is purely additive on all three networks (no Deleted,
    no Renamed) -- the destructive direction the add-only loader must never see
  - replaying the handler against already-upgraded state is idempotent, which
    is what a validator that crashes mid-upgrade and restarts actually does

Note for anyone extending this file: these tests REQUIRE -tags=test. Without
it lumeraapp.Setup skips at test_helpers.go:133 and every one of them reports
RUN with no PASS -- they pass vacuously. That is how they were first written
and it was caught by checking for the missing PASS lines.

Mutation testing, 3/3 detected:

  route mainnet (EVM absent) to migrations-only  -> 2 tests fail
  route testnet (EVM present) to full bring-up   -> 1 test fails (deadline stomp)
  add a Deleted entry to StoreUpgrades           -> 2 tests fail

Also removes an unused expectIdentityMigrationPlanBuildOnly helper left behind
by the previous commit; golangci-lint's unused check flagged it.

go test -tags=test ./app/upgrades/ -run TestV1202 -> 6/6 PASS
v1.20.2 carries NO module consensus-version bump. It is a behavior activation
boundary, not a state migration. Nothing enforced that, and the failure mode is
severe: if someone later raises a ConsensusVersion without registering the
matching migration, RunMigrations fails at the halt height -- on a live chain,
with every validator already stopped and waiting.

Pinned against what the chains actually report
(/cosmos/upgrade/v1beta1/module_versions, 2026-08-01):

    lumera-mainnet-1   audit 2   supernode 1   evmigration absent
    lumera-testnet-2   audit 2   supernode 1   evmigration 1

Three assertions:

  - the binary declares audit 2 / supernode 1 / evmigration 1, matching both
    live chains, and the audit module agrees with its own types constant;
  - on testnet's real arrival shape RunMigrations returns a version map EQUAL
    to the input, which is what makes "migrations only" a verified claim rather
    than a comment in the handler;
  - a mainnet node arriving via the 1.12.0 one-hop lands on the SAME module
    versions as a testnet node arriving via 1.20.1, so the two networks do not
    end up on different state machines.

Two modelling traps worth recording, both hit while writing this:

  - fromVM for mainnet is NOT module.VersionMap{}. An empty map tells
    RunMigrations every module is new, so it calls InitGenesis on all of them
    and panics with "groups: sequence: already initialized". A live 1.12.0
    chain reports auth/bank/staking/group/audit/supernode normally and is
    missing ONLY the EVM stack.
  - the EVM module registers as "evm", while its store key is "vm". Deleting
    "vm" from the version map leaves "evm" behind, producing partial EVM state.
    The handler's fail-closed branch caught this and rejected the upgrade,
    which is precisely its purpose -- the guard proved itself on a real mistake
    rather than only on a synthetic mutant.

Mutation testing, 3/3 detected -- bumping audit, evmigration, or supernode
ConsensusVersion without a registered migration fails 2 tests each.

go test -tags=test ./app/upgrades/ -run TestV1202 -> 9/9 PASS

Reminder for future edits: these tests REQUIRE -tags=test. Without it
lumeraapp.Setup skips and they pass vacuously.
…dings

Test-harness and documentation only. No chain logic, no state machine, no
protobuf. Prepares the v1.20.2 devnet validation of both upgrade shapes.

# Makefile.devnet: devnet-upgrade-1202

There was no way to drive a v1.20.2 upgrade on devnet -- targets existed for
1110/1111/1120/1201 and devnet-evm-upgrade hardcodes v1.12.0 -> v1.20.1.

Modelled on devnet-upgrade-1201 because both upgrade to the LOCALLY BUILT
binary rather than a pre-downloaded release, which is required while v1.20.2 is
unreleased.

One target serves both rehearsal shapes, because the v1.20.2 handler is
state-driven (inspects fromVM) rather than chain-id driven:

    testnet-shaped   1.20.1 -> 1.20.2   migrations only, EVM already present
    mainnet-shaped   1.12.0 -> 1.20.2   full EVM bring-up + add-only store mount

The comment records why the coordinated governance halt is mandatory: v1.20.2
changes evmigration DeliverTx outcomes, so a rolling node-by-node restart would
fork the network.

# docs: supernode-migration.md

Folds in findings from the earlier mainnet-shaped rehearsal that were sitting in
an out-of-tree addendum. Every item below cost real debugging time; leaving them
undocumented means each operator rediscovers them.

Prerequisites gains:
  - keyring passphrase must be >= 8 characters (fails mid-way through key
    creation with an unrelated-looking error otherwise)
  - explicit upgrade ORDER: chain first, then supernode. v2.6.x refuses to start
    against a pre-EVM chain by design, so upgrading early takes the node offline
    until the chain catches up. Includes the verbatim fatal text.
  - the sn-manager auto-update gate, with both verbatim log lines:
        Automatic update to <ver> blocked: supernode.evm_key_name is missing or empty
        Automatic update to <ver> blocked: cannot read SuperNode evm_key_name: <err>
    Framed as protective and correct, with an explicit "do not work around it,
    downgrade, or disable the updater" -- an operator who reads this as a bug
    will do exactly the wrong thing. Notes that already-v2.6 nodes are not gated
    because successful migration clears the field.

Step 4 (Verify) gains:
  - verify against chain state, not the daemon's logs
  - the old address stops resolving and that is EXPECTED; prev_supernode_accounts
    is provenance, not an alias. Includes the verbatim NotFound.
  - migration is NOT repeatable: the legacy key is deleted on success, so a
    re-run only ever retries a failure. Record the destination mnemonic first.
  - the two-keyring trap: the daemon uses ~/.supernode/keys, separate from the
    validator keyring, so the SAME key name resolves to two different addresses.
    A mismatch here prevents startup and looks like a migration fault.
  - a migrated eth_secp256k1 key cannot be moved between keyrings: export
    appears to succeed, import fails. Generate it in the target keyring.

Troubleshooting gains a destructive-operation guardrail: never delete a key
before its replacement is proven, and never suppress stderr on a destructive
step -- that is how a recoverable import failure became permanent key loss
across 5 nodes.

Verified: 64 code fences (balanced), 595 lines.
Devnet harness only. No chain logic, no state machine, no protobuf. Found while
driving real cascade traffic for the v1.20.2 validation.

# 1. KEY_NAME: unbound variable

lumera-uploader-setup.sh runs `set -euo pipefail` and reads ${KEY_NAME} in
validator_funding_address() to locate the genesis account that funds the
uploader's accounts -- but never assigns it, and common.sh does not either.

start.sh:304 launches this script standalone via nohup, so it does NOT inherit
supernode-setup.sh's shell where KEY_NAME="${MONIKER}_key" is set
(supernode-setup.sh:95). With -u the first funding lookup aborted setup:

    line 591: KEY_NAME: unbound variable

So the uploader could never start on a fresh devnet; setup died before writing
any config. Derived the same way supernode-setup.sh derives it, override-friendly.
MONIKER is already guaranteed by the assertion at line 44.

# 2. add_dir_to_scanner produced invalid, wrongly-typed TOML

After fix 1 the uploader started and panicked:

    toml: line 89 (last key "scanner"): expected '.' or '=', but got ']'

Two independent defects, both caused by driving a multi-line inline-table value
through crudini:

  a) Orphan bracket. The template value spans lines:

         directories = [
           { srcPath = "...", processedPath = "...", isPublic = "random" }
         ]

     `crudini --get` returns only the FIRST physical line -- the bare `[`. The
     code then stripped a trailing `]` that was not on that line and `--set`
     wrote a fresh single-line array, leaving the template's own closing `]`
     behind on its own line. Invalid TOML.

  b) Wrong type. Even with balanced brackets, ["/path"] is an array of STRINGS.
     The schema is an array of inline TABLES (config/scanner.go):

         Directories []ScannerDirectory `toml:"directories"`
         SrcPath       string `toml:"srcPath"`
         ProcessedPath string `toml:"processedPath"`

     NormalizeScannerDirectories requires srcPath; strings cannot unmarshal.

Replaced crudini with an awk rewrite that consumes every physical line of the
existing value and emits correctly-typed entries. It accumulates across calls
rather than overwriting, is idempotent on re-add, and fails loudly rather than
leaving a config that makes the uploader panic at startup.

Verified live:

    directories = [
      { srcPath = "/shared/nm-files", processedPath = "/shared/nm-files/processed", isPublic = "random" },
      { srcPath = "/root/nm-files", processedPath = "/root/nm-files/processed", isPublic = "random" },
      { srcPath = "~/.lumera-uploader/drop", processedPath = "~/.lumera-uploader/drop/processed", isPublic = "random" }
    ]

Tested by ops/v1202-validation/scripts/test_add_dir_to_scanner.sh, which extracts
the function from this script (no copy-paste drift) and asserts on the exact
multi-line fixture that broke crudini: one closing bracket, srcPath present,
balanced brackets, trailing section intact, accumulation, idempotency, and a
tomllib parse with per-entry type assertions. Mutation-checked by feeding the
suite the old broken output -- it is killed by the TOML parser, so the suite is
not vacuous.

shellcheck -S error clean.

Also adds devnet/config/config-phase1-nohermes.json: the default config with
hermes disabled and nothing else changed. config-no-hermes.json is missing
sn-account-mnemonics, api, rpc and json-rpc and carries a stale network-maker
key, so booting from it yields a devnet with no supernode accounts and no LCD.
…hy upgrade

Devnet harness only. No chain logic. Found during the v1.20.2 rehearsal, where a
textbook-correct upgrade was reported as a failure.

Observed on a real v1.20.1 -> v1.20.2 run: all five validators halted at exactly
the plan height 2429 with the expected panic, yet upgrade.sh printed

    ⚠️  Chain at 2429 passed 2429 without halting; proceeding to guard.

and exited 1. Three independent bugs combined to produce that.

# 1. The halt marker regex could never match

detect_upgrade_halt() built the pattern with a literal quoted version:

    UPGRADE.*"v1.20.2".*NEEDED

but the logger escapes the quotes, so the bytes on disk are:

    err="failed to apply block; error UPGRADE \"v1.20.2\" NEEDED at height: 2429: "

`"v1.20.2"` therefore never matches `\"v1.20.2\"`, and the dots were unescaped
regex wildcards besides. Halt detection was broken unconditionally, independent
of how much log was searched. Now matches the version without surrounding quotes
and escapes the dots.

# 2. It searched a window the supernode floods

The pattern was applied to `docker compose logs --tail=100`. The container
entrypoint multiplexes several files onto stdout:

    tail -F /root/logs/validator.log /root/logs/supernode.log ...

so per-block supernode chatter pushes the one-time halt panic out of a shallow
window. Measured on the halted chain: --tail=100 found 0 matches while --tail=400
found 3. Now greps /root/logs/validator.log directly (bounded, cannot be flooded
out) and falls back to a --tail=5000 scan if that file is not readable.

# 3. The "sailed past" guard misfired on the correct halt state

The guard used `height >= UPGRADE_HEIGHT`. A halted node does NOT go dark: it
panics in the consensus routine, stops advancing, and keeps serving
`lumerad status` with latest_block_height == UPGRADE_HEIGHT indefinitely. All
five validators served height 2429 for the entire halt window.

So `>=` is true in the normal, correct halt state and the warning fired
immediately on a healthy upgrade. Only a height STRICTLY GREATER than the plan
height means blocks were produced beyond it, i.e. the upgrade did not take
effect. Changed to `>`, and corrected the stale comment claiming the node stops
serving RPC and never reaches UPGRADE_HEIGHT.

Net effect of 1+2: the primary success signal was unreachable, so the script
always fell through to the height guard, which then misread the correct halt as
a failure.

Tested by ops/v1202-validation/scripts/test_upgrade_halt_detection.sh, which
builds a fixture with the real escaped-quote panic buried under 300 lines of
supernode chatter and asserts: the shallow window misses it, the original quoted
pattern matches nothing even against the full file, whole-file and deep-fallback
greps both find it, `>=` misfires at height == plan while `>` does not, and `>`
still detects a genuine overrun and stays silent at plan-1. 8/8 pass.

shellcheck -S error clean.
…rsal

Devnet fixtures only. No chain logic. Needed to rehearse the mainnet arrival
shape for v1.20.2: a chain born on PRE-EVM v1.12.0 with 100% legacy secp256k1
accounts, upgraded one-hop to v1.20.2 so the handler takes the full EVM
bring-up branch (len(present)==0) rather than the migrations-only branch a
1.20.1 -> 1.20.2 devnet exercises.

genesis-setup2-preevm.json is devnet-genesis.json with two changes:

  1. `evmigration` removed. v1.12.0 has the audit module but NOT evmigration,
     and genesis must match the binary's module set exactly. Booting with an
     extra module leaves the chain a hair away from the real mainnet shape;
     booting with a MISSING one fails hard:

         failed to validate genesis state:
         failed to unmarshal audit genesis state: EOF

     (that is what devnet-genesis-orig.json produces — it predates audit, so
     it is NOT the right pre-EVM base despite the name.)

  2. `claim.total_claimable_amount` -> 0. No claims.csv is staged and v1.12.0
     has no --skip-claims-check, so a non-zero total aborts devnet-build.

config-setup2-mainnet-shape.json is config-phase1-nohermes.json with the
uploader disabled. Keeping `chain.evm_from_version: v1.20.0` is deliberate and
load-bearing: common.sh's lumera_supports_evm() compares the RUNNING binary
version against that cutover, so on v1.12.0 it correctly reports no EVM support
and the setup scripts provision legacy secp256k1 keys.

Verified staged artifacts rather than trusting build output: lumerad reports
1.12.0 with ZERO feemarket/precisebank strings, and all keyring entries are
/cosmos.crypto.secp256k1.PubKey.

Note for anyone reusing this: `make devnet-build BIN_DIR=...` is silently
ignored. The variable is DEVNET_BIN_DIR, and DEVNET_BUILD_LUMERA defaults to 1
which rebuilds from source and overwrites whatever you staged. Correct form:

    make devnet-build DEVNET_BIN_DIR=devnet/bin-v1.12.0 \
      DEVNET_BUILD_LUMERA=0 DEVNET_BUILD_TESTS=0 \
      CONFIG_JSON=config/config-setup2-mainnet-shape.json \
      EXTERNAL_GENESIS_FILE=devnet/config/genesis-setup2-preevm.json

Also note primary_validator_setup() requires external_genesis.json
unconditionally (validator-setup.sh:847) even though the Makefile prints
"Using default initialization..." — there is no working default-init path.

Rehearsal result: 27/27 assertions pass. Handler logged "EVM not yet
initialized, running full v1.20.0 bring-up" and "add-only EVM bring-up",
modules grew 30 -> 35, migration_end_time was DERIVED as exactly the 2-day
devnet window from the upgrade block time, denom metadata gained the 18-decimal
alume unit, and 5 real MsgClaimLegacyAccount migrations executed post-upgrade.
Single app hash across all 5 validators throughout.
Devnet fixture only. No chain logic.

Needed to validate 97f696d (feat(feemarket): raise base fee fivefold) on the
v1.20.1 -> v1.20.2 arrival shape. That commit bumps the handler to write the
configured base fee on both arrival shapes AND bumps
devnet-genesis-evm.json's feemarket base_fee from 0.0025 to 0.0125.

Booting the rehearsal from the updated genesis would make the obvious assertion
"base_fee is 0.0125 after the upgrade" pass trivially, 0.0125 -> 0.0125, which
is true whether the handler writes the value, skips it, or ignores the field
entirely. A gate that cannot fail is not a gate.

This fixture is devnet-genesis-evm.json with two changes:

  1. feemarket.params.base_fee pinned back to 0.0025 — the value a real
     v1.20.1 chain carries. In practice EIP-1559 decayed it to the
     min_gas_price floor 0.0005 before the halt height, so the pre-state ended
     up 25x below the target, which is stronger still.
  2. claim.total_claimable_amount -> 0. No claims.csv is staged and the
     v1.20.1 binary has no --skip-claims-check, so a non-zero total aborts
     devnet-build.

Rehearsal result (binary built at 37a4721, verified by string-scan to contain
all four new feemarket symbols that are absent from the pre-change build):
13/13 assertions pass.

Worth recording for whoever writes the next assertion: base_fee is a DYNAMIC
EIP-1559 value, not a config constant. The feemarket EndBlocker decays it every
block, including the block the handler writes it:

    h=131 (pre)   0.000500000000000000
    h=132 (halt)  0.011718750000000000
    h=133         0.010986328125000000

0.0125 * 15/16 = 0.01171875 exactly (base_fee_change_denominator = 16), so
`q feemarket params --height <halt>` can NEVER return 0.0125 and asserting
equality against it would FAIL on a correct chain. Assert instead on the
handler log line, on the discontinuous jump at the halt height, and on the
exact 15/16 relationship.

Both arrival shapes were confirmed to converge byte-identically:

    offset   from v1.20.1 (h=132)     from v1.12.0 (h=109)
    halt+0   0.011718750000000000     0.011718750000000000
    halt+1   0.010986328125000000     0.010986328125000000
    halt+2   0.010299682617187515     0.010299682617187515

Supernode fee impact measured rather than assumed: burn rose from ~3969 to
~5069 ulume per epoch (~28%, not 5x, because min_gas_price is unchanged at
0.0005 and an idle chain decays back to that floor). All 5 SNs stayed ACTIVE
with epoch reports landing; lowest balance was ~993x the documented 10k
dead-prober threshold. No liveness risk at devnet funding levels, but the
runway did shorten, which is worth an operator note since a drained SN account
is a known cause of fleet-wide POSTPONED.

Also note the validator log is ANSI-coloured: the bytes between "base fee" and
"base_fee=" are ESC[0m ESC[36m, so a contiguous grep pattern silently matches
nothing. Strip escapes before asserting on log lines.
… raise

Commit 97f696d raised FeeMarketDefaultBaseFee 0.0025 -> 0.0125. That change is
intentional, but five test/CI locations had the OLD value baked in as literals,
so the integration and determinism-pipeline jobs went red on this branch while
master stayed green.

Fixes the stale assumptions, not the shipped behavior. Every value is now
derived from config/evm.go so the next retune cannot silently break them.

feemarket: the drain loop was capped at 20 empty blocks. Decaying from the
default to the floor takes log(default/floor)/log(den/(den-1)) blocks, which is
~50 at 0.0125. The loop exited at ~3.4x the floor, leaving the high-load phase
no headroom to demonstrate an increase. The model reproduces CI's reported
start_height=31 exactly. Budget is now computed from config.

mempool: the affordable-gas ceiling was a fixed 2.2 gwei, below the 12.5 gwei
start, so it could never be reached. Anchored on the configured default rather
than a multiple of the floor, because earlier subtests in the suite submit load
and push the base fee back up (a 3x-floor ceiling still failed). The receiver is
funded for the resulting cost by construction. Runtime 95s -> 15s.

jsonrpc: flat "1000ulume" over 200k gas is 0.005ulume/gas vs 0.00746 required.
Switched the default cosmos-tx path to --gas-prices, which scales with the base
fee; callers passing an explicit fee (e.g. asserting rejection below the floor)
keep their literal. EVM-side funding of 2e14 wei was also genuinely too small
for a 21k tx at 12.5 gwei, so it is now sized from the live gas price.

contracts: same flat-fee rejection on the funding tx, but it was swallowed --
the helper ignored the CheckTx code, so the account stayed at zero and the
failure surfaced much later as a misleading "insufficient funds" on the tx under
test. Now uses --gas-prices, asserts the CheckTx code, waits for the tx to
commit, and asserts the EVM balance actually landed. The 1e13 ulume amount was
never wrong (= 1e25 wei) and is unchanged.

consensus-determinism.yml: flat --fees 500ulume is 0.0025ulume/gas vs 0.010986
required. Derives GAS_PRICES from config/evm.go. Added
tests/scripts/check-determinism-gas-price.sh, which asserts the derived value
clears the observed floor AND that the old value still fails, so the check
cannot go vacuous.

Verification (local, real runs):
  go test -tags='integration test' ./tests/integration/evm/...
    ante 49.7s, contracts 273.3s, feemarket 363.3s, ibc 6.1s, jsonrpc 277.2s,
    mempool 238.6s, precisebank 98.7s, precompiles 135.8s, vm 80.4s -- all ok
  make lint -> 0 issues (incl. shellcheck)
  bash tests/scripts/check-determinism-gas-price.sh -> PASS

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 57 out of 57 changed files in this pull request and generated no new comments.

Suppressed comments (4)

app/upgrades/v1_20_2_bringup_external_test.go:42

  • newV1202Params claims to build “real keeper wiring”, but it sets ModuleManager to module.NewManager() and Configurator to a stub. That means the upgrade handlers’ RunMigrations path isn’t exercised here and these tests can pass even if the real upgrade would fail with the app’s ModuleManager/Configurator. Use the app’s real module wiring so the test meaningfully models a chain upgrade.
// newV1202Params builds the real keeper wiring a coordinated upgrade would have.
func newV1202Params(app *lumeraapp.App, chainID string) appParams.AppUpgradeParams {
	return appParams.AppUpgradeParams{
		ChainID:           chainID,
		Logger:            log.NewNopLogger(),
		ModuleManager:     module.NewManager(),
		Configurator:      module.NewConfigurator(nil, nil, nil),
		BankKeeper:        app.BankKeeper,

app/upgrades/v1_20_2_bringup_external_test.go:88

  • This test passes an empty fromVM into the upgrade handler, but a real v1.12.0 chain’s fromVM contains versions for all existing (non‑EVM) modules. Using an empty map can mask bugs (e.g., InitGenesis re-runs) and doesn’t model the actual mainnet one-hop shape. Build fromVM from app.ModuleManager.GetVersionMap() and delete only the EVM + evmigration modules.
	// fromVM is EMPTY: mainnet carries no EVM module versions at 1.12.0.
	newVM, err := config.Handler(sdk.WrapSDKContext(ctx), upgradetypes.Plan{}, module.VersionMap{})
	require.NoError(t, err, "the mainnet 1.12.0 -> 1.20.2 one-hop must succeed")

tests/integration/evmtest/feeconfig.go:91

  • MinCosmosGasPriceWithHeadroom is documented as safe to pass directly to --gas-prices, but it currently returns only the decimal (e.g. "0.0125") without the denom suffix. Cosmos CLI expects a full coin string like "0.0125ulume"; this helper should append the denom (or the comment/name should be adjusted).
    x/supernode/v1/types/identity_migration_plan.go:72
  • PR description says Everlight continuity is not fixed and that neither rdist/<validator> nor rhist/<validator>/... are moved on validator address changes. This plan explicitly migrates rdist/ state via SNDistStateKey(...), so the PR description (and any related operator expectations) should be updated to reflect that only payout history (rhist/) remains unmigrated.

@mateeullahmalik mateeullahmalik self-assigned this Aug 5, 2026
@mateeullahmalik
mateeullahmalik requested a review from a-ok123 August 5, 2026 19:20
…romVM, denom suffix

All four suppressed Copilot comments on PR #196 were verified against the code
and all four were correct. Fixes below.

1) v1_20_2_bringup_external_test.go: newV1202Params claimed "real keeper wiring"
   but passed module.NewManager() and a nil-stub Configurator. v1.20.2 delegates
   to the v1.20.1 handler, which calls
   p.ModuleManager.RunMigrations(ctx, p.Configurator, fromVM) — with an empty
   manager that is a no-op, so the tests could pass even if the real upgrade
   failed. Now uses app.ModuleManager and app.Configurator(), matching what
   app.go:500-501 wires in production.

   Verified non-vacuous by mutation: corrupting the RunMigrations input in
   v1_20_0/upgrade.go now fails TestV1202MainnetOneHopRunsFullEVMBringup. Under
   the old stub that mutation was undetectable.

2) Same file: the mainnet one-hop passed an EMPTY module.VersionMap. A real
   v1.12.0 chain carries versions for every existing non-EVM module.
   mainnetPreEVMVersionMap() now derives fromVM from
   app.ModuleManager.GetVersionMap() and deletes only the EVM stack plus
   evmigration.

   This immediately surfaced a real defect the stub had been hiding: with the
   empty map, RunMigrations re-runs InitGenesis for EVERY module and x/group
   panics with "sequence: already initialized: unique constraint violation".
   TestV1202IsIdempotentAcrossReplay was passing only because the stub manager
   never reached that code.

   Added TestV1202MainnetSuppressesEVMInitGenesis to pin the InitGenesis
   suppression guard and assert post-upgrade consensus-version parity with the
   app's module manager. Mutation-verified: removing
   fromVM[evmtypes.ModuleName] = 1 fails the bring-up test. Documented in the
   test that the feemarket guard is NOT independently detectable this way,
   because v1.20.2 deliberately re-applies BaseFee after RunMigrations
   (v1_20_2/upgrade.go:157) — that masking is by design, not a test gap.

3) tests/integration/evmtest/feeconfig.go: MinCosmosGasPriceWithHeadroom is
   documented as safe to pass to --gas-prices but returned a bare decimal
   ("0.0125") with no denom. The Cosmos CLI rejects that. Now returns a full
   coin string ("0.0125ulume"). The helper had no callers yet, so this was
   latent rather than breaking.

4) PR description contradicted the code: it listed rdist/ as unmigrated, but
   BuildIdentityMigrationPlan re-keys it via SNDistStateKey and preserves all
   four numeric fields verbatim. Description corrected — only rhist/ (payout
   history) remains unmigrated, and rhist/ is read solely by
   query_get_payout_history.go, so it does not affect eligibility or weight.

Verification:
  go test -tags=test ./app/upgrades/...           all ok (9/9 TestV1202* pass)
  go vet -tags='integration test' ./...           clean
  make lint                                      0 issues
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants