Skip to content

fix(devnet): resolve upgrade script paths relative to the script, not the caller - #200

Open
mateeullahmalik wants to merge 2 commits into
masterfrom
matee/devnet-upgrade-script-cwd
Open

fix(devnet): resolve upgrade script paths relative to the script, not the caller#200
mateeullahmalik wants to merge 2 commits into
masterfrom
matee/devnet-upgrade-script-cwd

Conversation

@mateeullahmalik

Copy link
Copy Markdown
Contributor

Problem

devnet/scripts/upgrade.sh cannot drive an upgrade when invoked from the repository root — which is how the rehearsal flow is documented and how make devnet-upgrade-* reaches it. It fails with two claims that are both false:

⚠️  Governance helper account lumera1kd5... is not present on-chain; using primary validator proposer.
❌ Unable to resolve fallback proposer key supernova_validator_1_key

I checked both against the live devnet: the account held 1,000,000 LUME and the key was in the container keyring. Only the compose lookup was broken.

Root cause

submit-upgrade-proposal.sh and vote-all.sh both hardcoded a caller-relative path:

COMPOSE_FILE="../docker-compose.yml"

That resolves only from devnet/scripts/. From anywhere else every docker compose -f "$COMPOSE_FILE" exec ... failed, so key_address() and account_exists() returned empty and the scripts blamed governance/keyring state.

This failure mode is expensive out of proportion to its size: the error text points at the chain, so you go debugging accounts and keyrings instead of a path.

Isolating proof

Same script, same devnet, same arguments — only the working directory differs:

Invoked from Result
devnet/scripts/ Governance proposer: governance_key (lumera1rmr8…) → proceeds
repository root not present on-chain → exit 1

Fix

Resolve from BASH_SOURCE, keeping an env override:

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
COMPOSE_FILE="${COMPOSE_FILE:-${SCRIPT_DIR}/../docker-compose.yml}"

Second defect: branch builds can't pass the version probe

upgrade-binaries.sh refuses to proceed unless <binary> version prints something. Binaries built from a branch with plain go build (no -ldflags) print nothing, so rehearsing an unreleased branch was impossible:

Failed to determine version for binary: .../devnet/bin-v1.20.2/lumerad

Confirmed empirically — release v1.20.1 prints 1.20.1; a local branch build prints "". Added a DEVNET_BINARY_VERSION override plus an error message that names it. Release artifacts self-report and are unaffected.

Third defect: found, deliberately NOT fixed here

During the end-to-end run:

❌ supernova_validator_2: fee top-up failed: account sequence mismatch, expected 2, got 1
❌ Skipping vote for supernova_validator_2 because fee top-up failed
📈 Total Votes Cast: 4000000000000     (4 of 5 validators)

Back-to-back top-ups from one funding account race the sequence number, and the script silently drops that validator's vote. The proposal still passed here, but on a network where one validator carries decisive weight this could silently change a governance outcome. It needs a sequence-aware retry and deserves its own PR. Documented in the commit so it isn't rediscovered from scratch.

Evidence — full E2E after the fix

Fresh 5-validator devnet at v1.20.1, upgrading to a branch-built v1.20.2, invoked from the repository root:

Governance proposer: governance_key (lumera1rtk...)   <- previously failed here
Total Votes Cast: 4000000000000
Binaries upgrade complete
Upgrade to v1.20.2 initiated successfully
exit code 0

q upgrade applied v1.20.2 ...... height 121
audit module version ........... 3
all 5 validators ............... height 175, in lockstep

bash -n and shellcheck -S error clean on all three files.

Correction to an earlier claim of mine

I previously reported that upgrade.sh "exits 0 on failure". That was my measurement error — I read $? after piping stdout to tail, so I was reading tail's status. upgrade.sh sets set -euo pipefail and correctly returns 1. No change was needed and none was made.

Risk / rollback

Shell-only, confined to devnet tooling — no chain code, no state machine, no consensus path. The env-var indirection preserves any existing COMPOSE_FILE override. Rollback: revert; the scripts return to working only from devnet/scripts/.


Found while running the Phase 1 upgrade rehearsal for #199, where I had to drive the governance flow by hand to work around it.

… the caller

Three defects that together made `devnet/scripts/upgrade.sh` unusable for a
rehearsal driven from the repository root, which is how the upgrade flow is
documented and invoked.

1. CWD-RELATIVE COMPOSE FILE (root cause)
-----------------------------------------
submit-upgrade-proposal.sh and vote-all.sh both hardcoded

    COMPOSE_FILE="../docker-compose.yml"

That only resolves when the script is invoked from devnet/scripts/. Run from
the repository root, every `docker compose -f "$COMPOSE_FILE" exec ...` call
failed, so key_address() and account_exists() returned empty. The scripts then
misreported perfectly healthy accounts:

    Governance helper account lumera1kd5... is not present on-chain
    Unable to resolve fallback proposer key supernova_validator_1_key

Both claims were false. The account held 1,000,000 LUME and the key was in the
container keyring; only the compose lookup was broken. The failure mode is
especially costly because the message points at governance/keyring state and
sends you debugging the chain instead of the path.

Fixed by resolving from BASH_SOURCE, with an env override retained:

    SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
    COMPOSE_FILE="${COMPOSE_FILE:-${SCRIPT_DIR}/../docker-compose.yml}"

2. LOCALLY-BUILT BINARIES CANNOT PASS THE VERSION PROBE
-------------------------------------------------------
upgrade-binaries.sh refuses to proceed unless `<binary> version` prints a
non-empty string. Binaries built from a branch with plain `go build` (no
-ldflags) print nothing, so rehearsing an unreleased branch was impossible:

    Failed to determine version for binary: .../devnet/bin-v1.20.2/lumerad

Release artifacts self-report and are unaffected. Added a DEVNET_BINARY_VERSION
override plus an actionable error that names it. Confirmed empirically:
release v1.20.1 prints "1.20.1"; a local branch build prints "".

3. VOTE SKIPPED ON FEE TOP-UP SEQUENCE MISMATCH
-----------------------------------------------
Observed during the end-to-end run:

    supernova_validator_2: fee top-up failed: account sequence mismatch,
    expected 2, got 1: incorrect account sequence
    Skipping vote for supernova_validator_2 because fee top-up failed

Back-to-back top-ups from the same funding account race the sequence number, and
the script silently drops that validator's vote (4 of 5 voted). The proposal
still passed here, but on a network where one validator carries decisive weight
this would silently change the outcome. NOT fixed in this PR - it needs a
sequence-aware retry and deserves its own change. Documented so the next person
does not rediscover it from scratch.

EVIDENCE
--------
Same script, same devnet, same arguments, differing only in working directory:

    from devnet/scripts/ ... proposer resolved, proceeded to vote  (worked)
    from repository root ... "not present on-chain", exit 1        (failed)

That isolates CWD as the variable. After the fix, from the repository root, on
a fresh 5-validator devnet at v1.20.1 upgrading to a branch-built v1.20.2:

    Governance proposer: governance_key (lumera1rtk...)   <- previously failed
    Total Votes Cast: 4000000000000
    Binaries upgrade complete
    Upgrade to v1.20.2 initiated successfully
    exit code 0

    q upgrade applied v1.20.2 -> height 121
    audit module version ..... 3
    all 5 validators ......... height 175, in lockstep

CORRECTION
----------
An earlier report of mine claimed upgrade.sh "exits 0 on failure". That was my
measurement error: I read the exit code after piping stdout to `tail`, so I was
reading tail's status. upgrade.sh sets `set -euo pipefail` and correctly returns
1. No change was needed and none was made.

RISKS
-----
Shell-only, confined to devnet tooling. No chain code, no state machine, no
consensus path. The env-var indirection preserves any existing COMPOSE_FILE
override.

ROLLBACK
--------
Revert. The scripts return to working only when invoked from devnet/scripts/.
… rehearsals

Both were found by running the scripts for real against mainnet-shaped devnets,
not by reading them. Neither is theoretical.

1. HALT DETECTION ABORTED A CORRECT UPGRADE
--------------------------------------------
Observed on the two-hop rehearsal (v1.12.0 -> v1.20.1). The chain had stopped
correctly at the plan height and the logs said so:

	UPGRADE "v1.20.1" NEEDED at height: 207

but the script refused to continue:

	ERROR: chain is still producing blocks (height 207) and has NOT halted
	The upgrade did not execute — the proposal likely failed to pass
	(quorum/threshold).

Both statements were false. The proposal had PASSED (4000000000000 yes, 0 no)
and `q upgrade plan` showed the plan registered at 207.

Root cause: a node stopped for an upgrade KEEPS SERVING RPC at the halt height.
The guard treated any numeric height as proof of liveness, so height == plan
height read as "still producing blocks". It is a liveness check written as a
value check.

Fixed by sampling the height twice and comparing: only an ADVANCING height means
blocks are still being produced. A static height across two samples is the
stopped state. The error message now points at the two queries that actually
distinguish the cases instead of asserting a quorum failure that did not happen.

This failed SAFE — it refused to swap binaries rather than swapping wrongly —
but it turned a green rehearsal into a manual recovery.

2. SEQUENCE RACE SILENTLY DROPPED A VALIDATOR'S VOTE
-----------------------------------------------------
Observed on BOTH Phase 2 rehearsals:

	❌ supernova_validator_2: fee top-up failed (txhash: 8D157071...):
	   account sequence mismatch, expected 2, got 1: incorrect account sequence
	❌ Skipping vote for supernova_validator_2 because fee top-up failed
	📈 Total Votes Cast: 4000000000000        (4 of 5)

Every voter is topped up from the same funding account in a tight loop, so
back-to-back sends race the account sequence: the second is broadcast before the
first is committed. The caller then skips that validator's vote entirely.

The proposal still passed in both runs, so this is easy to miss. On a network
where one validator carries decisive weight, a silently dropped vote changes a
governance outcome. Fixed with a bounded retry that matches the sequence-mismatch
error specifically and waits for the prior send to commit
(TOPUP_MAX_ATTEMPTS, TOPUP_RETRY_SECS).

VERIFICATION
------------
bash -n ................. clean on both files
shellcheck -S error ..... clean on both files

Both defects were reproduced on real devnet runs before being changed, and the
halt-detection fix is what allowed the two-hop rehearsal (v1.12.0 -> v1.20.1 ->
v1.20.2) to be completed and compared against the single-hop path.

RISK
----
Shell-only, devnet tooling. No chain code, no state machine, no consensus path.
Both new behaviours are env-tunable and default to conservative values.

ROLLBACK
--------
Revert. The scripts return to aborting on a correct halt and silently dropping
racing votes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant