fix(devnet): resolve upgrade script paths relative to the script, not the caller - #200
Open
mateeullahmalik wants to merge 2 commits into
Open
fix(devnet): resolve upgrade script paths relative to the script, not the caller#200mateeullahmalik wants to merge 2 commits into
mateeullahmalik wants to merge 2 commits into
Conversation
… the caller
Three defects that together made `devnet/scripts/upgrade.sh` unusable for a
rehearsal driven from the repository root, which is how the upgrade flow is
documented and invoked.
1. CWD-RELATIVE COMPOSE FILE (root cause)
-----------------------------------------
submit-upgrade-proposal.sh and vote-all.sh both hardcoded
COMPOSE_FILE="../docker-compose.yml"
That only resolves when the script is invoked from devnet/scripts/. Run from
the repository root, every `docker compose -f "$COMPOSE_FILE" exec ...` call
failed, so key_address() and account_exists() returned empty. The scripts then
misreported perfectly healthy accounts:
Governance helper account lumera1kd5... is not present on-chain
Unable to resolve fallback proposer key supernova_validator_1_key
Both claims were false. The account held 1,000,000 LUME and the key was in the
container keyring; only the compose lookup was broken. The failure mode is
especially costly because the message points at governance/keyring state and
sends you debugging the chain instead of the path.
Fixed by resolving from BASH_SOURCE, with an env override retained:
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
COMPOSE_FILE="${COMPOSE_FILE:-${SCRIPT_DIR}/../docker-compose.yml}"
2. LOCALLY-BUILT BINARIES CANNOT PASS THE VERSION PROBE
-------------------------------------------------------
upgrade-binaries.sh refuses to proceed unless `<binary> version` prints a
non-empty string. Binaries built from a branch with plain `go build` (no
-ldflags) print nothing, so rehearsing an unreleased branch was impossible:
Failed to determine version for binary: .../devnet/bin-v1.20.2/lumerad
Release artifacts self-report and are unaffected. Added a DEVNET_BINARY_VERSION
override plus an actionable error that names it. Confirmed empirically:
release v1.20.1 prints "1.20.1"; a local branch build prints "".
3. VOTE SKIPPED ON FEE TOP-UP SEQUENCE MISMATCH
-----------------------------------------------
Observed during the end-to-end run:
supernova_validator_2: fee top-up failed: account sequence mismatch,
expected 2, got 1: incorrect account sequence
Skipping vote for supernova_validator_2 because fee top-up failed
Back-to-back top-ups from the same funding account race the sequence number, and
the script silently drops that validator's vote (4 of 5 voted). The proposal
still passed here, but on a network where one validator carries decisive weight
this would silently change the outcome. NOT fixed in this PR - it needs a
sequence-aware retry and deserves its own change. Documented so the next person
does not rediscover it from scratch.
EVIDENCE
--------
Same script, same devnet, same arguments, differing only in working directory:
from devnet/scripts/ ... proposer resolved, proceeded to vote (worked)
from repository root ... "not present on-chain", exit 1 (failed)
That isolates CWD as the variable. After the fix, from the repository root, on
a fresh 5-validator devnet at v1.20.1 upgrading to a branch-built v1.20.2:
Governance proposer: governance_key (lumera1rtk...) <- previously failed
Total Votes Cast: 4000000000000
Binaries upgrade complete
Upgrade to v1.20.2 initiated successfully
exit code 0
q upgrade applied v1.20.2 -> height 121
audit module version ..... 3
all 5 validators ......... height 175, in lockstep
CORRECTION
----------
An earlier report of mine claimed upgrade.sh "exits 0 on failure". That was my
measurement error: I read the exit code after piping stdout to `tail`, so I was
reading tail's status. upgrade.sh sets `set -euo pipefail` and correctly returns
1. No change was needed and none was made.
RISKS
-----
Shell-only, confined to devnet tooling. No chain code, no state machine, no
consensus path. The env-var indirection preserves any existing COMPOSE_FILE
override.
ROLLBACK
--------
Revert. The scripts return to working only when invoked from devnet/scripts/.
… rehearsals Both were found by running the scripts for real against mainnet-shaped devnets, not by reading them. Neither is theoretical. 1. HALT DETECTION ABORTED A CORRECT UPGRADE -------------------------------------------- Observed on the two-hop rehearsal (v1.12.0 -> v1.20.1). The chain had stopped correctly at the plan height and the logs said so: UPGRADE "v1.20.1" NEEDED at height: 207 but the script refused to continue: ERROR: chain is still producing blocks (height 207) and has NOT halted The upgrade did not execute — the proposal likely failed to pass (quorum/threshold). Both statements were false. The proposal had PASSED (4000000000000 yes, 0 no) and `q upgrade plan` showed the plan registered at 207. Root cause: a node stopped for an upgrade KEEPS SERVING RPC at the halt height. The guard treated any numeric height as proof of liveness, so height == plan height read as "still producing blocks". It is a liveness check written as a value check. Fixed by sampling the height twice and comparing: only an ADVANCING height means blocks are still being produced. A static height across two samples is the stopped state. The error message now points at the two queries that actually distinguish the cases instead of asserting a quorum failure that did not happen. This failed SAFE — it refused to swap binaries rather than swapping wrongly — but it turned a green rehearsal into a manual recovery. 2. SEQUENCE RACE SILENTLY DROPPED A VALIDATOR'S VOTE ----------------------------------------------------- Observed on BOTH Phase 2 rehearsals: ❌ supernova_validator_2: fee top-up failed (txhash: 8D157071...): account sequence mismatch, expected 2, got 1: incorrect account sequence ❌ Skipping vote for supernova_validator_2 because fee top-up failed 📈 Total Votes Cast: 4000000000000 (4 of 5) Every voter is topped up from the same funding account in a tight loop, so back-to-back sends race the account sequence: the second is broadcast before the first is committed. The caller then skips that validator's vote entirely. The proposal still passed in both runs, so this is easy to miss. On a network where one validator carries decisive weight, a silently dropped vote changes a governance outcome. Fixed with a bounded retry that matches the sequence-mismatch error specifically and waits for the prior send to commit (TOPUP_MAX_ATTEMPTS, TOPUP_RETRY_SECS). VERIFICATION ------------ bash -n ................. clean on both files shellcheck -S error ..... clean on both files Both defects were reproduced on real devnet runs before being changed, and the halt-detection fix is what allowed the two-hop rehearsal (v1.12.0 -> v1.20.1 -> v1.20.2) to be completed and compared against the single-hop path. RISK ---- Shell-only, devnet tooling. No chain code, no state machine, no consensus path. Both new behaviours are env-tunable and default to conservative values. ROLLBACK -------- Revert. The scripts return to aborting on a correct halt and silently dropping racing votes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
devnet/scripts/upgrade.shcannot drive an upgrade when invoked from the repository root — which is how the rehearsal flow is documented and howmake devnet-upgrade-*reaches it. It fails with two claims that are both false:I checked both against the live devnet: the account held 1,000,000 LUME and the key was in the container keyring. Only the compose lookup was broken.
Root cause
submit-upgrade-proposal.shandvote-all.shboth hardcoded a caller-relative path:COMPOSE_FILE="../docker-compose.yml"That resolves only from
devnet/scripts/. From anywhere else everydocker compose -f "$COMPOSE_FILE" exec ...failed, sokey_address()andaccount_exists()returned empty and the scripts blamed governance/keyring state.This failure mode is expensive out of proportion to its size: the error text points at the chain, so you go debugging accounts and keyrings instead of a path.
Isolating proof
Same script, same devnet, same arguments — only the working directory differs:
devnet/scripts/Governance proposer: governance_key (lumera1rmr8…)→ proceedsnot present on-chain→ exit 1Fix
Resolve from
BASH_SOURCE, keeping an env override:Second defect: branch builds can't pass the version probe
upgrade-binaries.shrefuses to proceed unless<binary> versionprints something. Binaries built from a branch with plaingo build(no-ldflags) print nothing, so rehearsing an unreleased branch was impossible:Confirmed empirically — release v1.20.1 prints
1.20.1; a local branch build prints"". Added aDEVNET_BINARY_VERSIONoverride plus an error message that names it. Release artifacts self-report and are unaffected.Third defect: found, deliberately NOT fixed here
During the end-to-end run:
Back-to-back top-ups from one funding account race the sequence number, and the script silently drops that validator's vote. The proposal still passed here, but on a network where one validator carries decisive weight this could silently change a governance outcome. It needs a sequence-aware retry and deserves its own PR. Documented in the commit so it isn't rediscovered from scratch.
Evidence — full E2E after the fix
Fresh 5-validator devnet at v1.20.1, upgrading to a branch-built v1.20.2, invoked from the repository root:
bash -nandshellcheck -S errorclean on all three files.Correction to an earlier claim of mine
I previously reported that
upgrade.sh"exits 0 on failure". That was my measurement error — I read$?after piping stdout totail, so I was reading tail's status.upgrade.shsetsset -euo pipefailand correctly returns 1. No change was needed and none was made.Risk / rollback
Shell-only, confined to devnet tooling — no chain code, no state machine, no consensus path. The env-var indirection preserves any existing
COMPOSE_FILEoverride. Rollback: revert; the scripts return to working only fromdevnet/scripts/.Found while running the Phase 1 upgrade rehearsal for #199, where I had to drive the governance flow by hand to work around it.