feat!: rebuild the POLYX record as a double-entry ledger (redesign phase 04) - #349
Open
prashantasdeveloper wants to merge 22 commits into
Open
prashantasdeveloper wants to merge 22 commits into
prashantasdeveloper wants to merge 22 commits into
Conversation
Moves the database from postgres 12 to 18. Postgres 13 and later let a boolean column be part of the range index the indexer builds when historical tracking is on, which postgres 12 rejected. The data volume is now mounted at /var/lib/postgresql because postgres 18 stores its files in a version folder underneath. BREAKING CHANGE: the database is a new major version and its volume path changed, so any existing data volume has to be recreated and the chain reindexed from the start.
The redesign reindexes the whole chain from block zero, so the incremental schema-migration files no longer apply and would only run stale changes on restart.
When resolving which identity an address belongs to, the code always called a storage item that only exists on newer runtimes, so it threw on early blocks. It also created an identity without its default portfolio, which made a later identity-created event fail because the portfolio was missing. Both are now handled: the older storage item is used when the new one is absent, and the default portfolio is created alongside the identity.
Replaces the single PolyxTransaction table with two: PolyxEntry, one append-only row per account per side of a movement, each carrying the account's balance right after it; and AccountBalance, the running balance that can be read at any past block. Every balances event is decoded, turned into a move between the free and reserved pools, and written as entries plus an updated balance. This fixes a set of long-standing gaps: - the events that create and destroy reserved balance on the current runtime were never indexed, so reserved balance and staking bonds were invisible; - account reaping (dust) was not recorded; - a transfer-with-memo produced a spurious second row instead of just adding the memo to the real transfer; - an absolute "set balance" event was recorded as if it were a movement, corrupting every later total; it is now a checkpoint that records only the difference; - frozen balance is now the largest active lock rather than a sum, which the old model could not represent; - pre-upgrade staking bonds moved no balance (they were a lock) and no longer produce a movement row; on the current runtime the bond's balance effect comes from the paired hold/release events, so the staking events themselves are recorded as state only and not double-counted; - for rewards paid before the upgrade the event only named the stash, so the actual payee is now read from chain storage at the reward block; - opening balances are seeded from chain state at genesis, without which every derived balance would be off by the genesis allocation. BREAKING CHANGE: polyxTransactions is replaced by polyxEntries. Queries that matched an account across the from/to columns now filter on a single accountId. BalanceTypeEnum is gone; use PolyxEntry.pool, .kind and .direction. The same staking event name now means different things before and after the v8 upgrade (a lock versus a hold).
An offline script samples accounts, oversampling the ones involved in the riskier cases, and compares the ledger's free, reserved and frozen balances against the chain at blocks on either side of each runtime upgrade, then prints a breakdown of any differences by cause. A second script measures how often pre-upgrade rewards were paid somewhere other than the stash. The running indexer also checks touched accounts against chain state periodically and after a set-balance or dust event, recording and correcting any drift.
`memo` is the trailing parameter of `InstructionCreated` and was added after the event first shipped. Register the shape with `optionalFrom: 7` so a 7-parameter event decodes with `memo` undefined rather than failing an arity check. `handleInstructionCreated` already tolerates an absent memo.
A stripped `ClaimRevoked` carrying a zero issuer DID and a `NoData` claim matches no indexed claim and is not an attributable revocation. Return early instead of recording a MissingReferencedEntity anomaly for it.
Balance attribution:
- TreasuryReimbursement(payerDid, amount) is the treasury's share of a
fee, not a refund to the payer. Credit the treasury pallet account; the
payer's fee is already debited by protocolFee.FeeCharged /
transactionPayment.TransactionFeePaid.
- A pre-v8 RewardDestination::Staked payee auto-restakes: the reward is
credited to free and immediately locked, with no staking.Bonded event.
Raise the staking lock in handleReward, and have the reconciler pin the
staking lock to the on-chain frozen amount so later staking events
adjust a real base.
- staking.payee(stash).toJSON() renders a unit variant as a lower-cased
single-key object ({ staked: null }), not the bare string 'Staked'.
Normalise both forms so a Staked payee is classified correctly.
- The in-flight reconciler ignores drift below 100 POLYX: the pre-v5.4
weight fee is applied with no event, and a sample taken mid-block on a
multi-touch account compares partial derived state against the block's
final on-chain state. Neither is a handler defect.
Derivation cost:
- Cache staking.payee per stash; it rarely changes and the reconciler
corrects any stale-entry drift.
- Reconcile every 2000 blocks rather than 500.
- Memoise system.account per block, so an account touched N times in a
sampled block costs one read rather than N.
snapshotFromMetadata keyed events and calls by the metadata pallet spelling
(Asset, ExternalAgents) while eventDrift, arityFixtureFor, the arity fixtures,
CAPTURED_MODULES and project.ts all use the event.section spelling (asset,
externalAgents). Exact-key lookups therefore missed and fell back to {} -
eventDrift reported every fixture event as removed (81 against mainnet 8000020)
and arityFixtureFor wrote "modules": {} under --write.
Normalise the pallet name once, on the way in, via sectionId(); the metadata
spelling never leaves snapshotFromMetadata. snapshot.modules is unaffected -
section.toLowerCase() is the same value either way.
The unit tests keyed both fixture and snapshot "Balances", so the suite stayed
green over the broken script; they now use the real convention, plus a
snapshotFromMetadata block against real v15 metadata from
@polkadot/types-support.
Co-authored-by: Francis <francis@polymesh.network>
stateTrieMigration.Migrated / AutoMigrationFinished / Halted were not in EventIdEnum, so the generic event recorder logged an UnknownEnumValue anomaly and stored them as "Unknown". They carry no balance effect - just recorded like any other event now.
… spec @subql/node can hand a mapping the previous runtime's spec version for the one block a runtime upgrade takes effect on. At the v5->v6 boundary that made asset.AssetBalanceUpdated resolve no tuple shape and the throwing decode proxy killed the worker. resolveShapeTolerant retries once with api.runtimeVersion.specVersion (read from the block's own runtime) when the reported version resolves nothing - but only when it is newer and within one release line, so neither a correct reported version nor a stale/HEAD api.runtimeVersion pulls in a wildly wrong shape.
Cognitive complexity: postTransition and handleBalanceSet split into named helpers; the reconcile-polyx and measure-a15-payees scripts' main loops each extract their inner body. Plus a bigint maxBig helper (Math.max cannot take bigint), toHaveLength over .length assertions, one extracted ternary, an optional chain, and .at(-1).
… flags validators.AutomaticPayoutFinished / ValidatorPayoutFailed reach the generic event recorder and would log an UnknownEnumValue anomaly; mmr / mmrleaf are runtime pallets absent from ModuleIdEnum. sync-metadata.ts now reports zero missing members across all three enums.
The rewardedStashes helper took data: unknown[], so data[1].toString() looked
to Sonar (S6551) like it could stringify a plain object. The values are codecs;
type the param as { toString(): string }[].
`Enum#toJSON()` camel-cases the variant name, so an 8.x `Rewarded` with an
`Account` payee arrives as `{ account: "5..." }` and never matched the
`'Account'` comparison: the row stored `rewardDestination: 'account'` with no
destination account. Every object form was affected, `{ staked: null }`
included, so no destination account was resolved for any of them.
The parse now lives once in `utils/staking.ts`, shared with the pre-v8
`staking.payee` read. It takes `AnyJson` and narrows rather than casting
`toJSON()` to a hand-written shape, and `rewardDestination` is a union of the
variant names instead of `string`.
`.toJSON()` stays in place of the generated `Option<PalletStakingRewardDestination>`
accessors: those describe the current metadata only, while both callers read
older runtimes, and the reason is now recorded at the helper.
The '\''staking '\'' lock was accumulated from Bonded / Withdrawn / restaked-Reward deltas, which never see the max-bond cap, the rounding of a compounded RewardDestination::Staked reward, unbonding chunks, or a slash - so it drifted from the real lock across the whole validator set. readStakingLock(stash) reads staking.ledger(controller).total from chain - the exact value pallet-staking passes to Currency::set_lock, hence what miscFrozen reports. syncStakingLock is wired into handleBonded / handleWithdrawn / handleReward (restaked) / handleStakingSlash, and falls back to the delta accumulator when the ledger cannot be read. Controller is resolved via staking.bonded(stash) and cached.
NftHolder.nftIds is a JSON array and historical mode versions the whole array on every save. A bulk mint - hundreds of NFTPortfolioUpdated for one holder in one block - made that Sigma(1..n) array serialisations and poisoned the store cache with 30k-element arrays, degrading the indexer to a crawl on later empty blocks. Holder mutations are now buffered per block and each holder is saved once, on block change or from a new handleBlock (Block handler, filter modulo 100). getNftHolder reads the buffer first. Nothing inside the indexer reads NftHolder - it is written for external queries only - so a holder being at most one block-handler interval stale is acceptable.
getAsset threw a plain Error on a missing asset, which crashed the worker for a block whose AssetCreated was skipped upstream. getAssetOrAnomaly records a MissingReferencedEntity and returns undefined instead; handleAssetBalanceUpdated uses it and returns early.
…was right CHANGES.md, entity-review.md and 02-polyx-ledger.md each asserted [V] that get8xStakingEventDetails resolved the RewardDestination correctly. PR #350 showed it did not - the object form never matched. Corrected in the repo's usual style.
The v5-v7 `Currency::set_lock("staking ", ...)` bond becomes a `RuntimeHoldReason::Staking`
hold on v8. `pallet_balances` runs the conversion as a no-extrinsic two-pass storage
migration: pass 1 emits `Upgraded` + `Held{Staking}`, pass 2 emits `Unlocked` for the
lingering currency lock. `handleBalanceHeld` already turns pass 1 into the `free -> reserved`
movement; the lock side was unhandled, so `frozen` over-reported for the ~420k blocks
between the two passes.
`handleBalanceUnlocked` now clears the account's `"staking "` lock on a v8 `Unlocked` when
one is present - post-v8 nothing re-creates a currency staking lock, so it can only be the
migration removing it. Pre-v8 and non-staking unlocks keep the generic `"balances"` path.
Confirmed against real testnet v8 blocks that `staking.Bonded` / `Unbonded` / `Withdrawn`
pair with `balances.Held` / `Released` in the same extrinsic (and `rebond` correctly emits
neither), so the docs' [I] on that pairing becomes [V].
Base automatically changed from
redesign/03-infrastructure
to
docs/architecture
September 10, 2026 10:52
… committee `TreasuryDisbursement`'s first parameter is the identity that authorised the spend, not the source of funds — `treasury.disbursement` always moves POLYX out of the treasury pallet account, the mirror of the reimbursement handler's credit. Debiting the authorising committee's primary key instead left the treasury drifting high forever and the committee account low, on every spec version that emits no paired `balances.Transfer` (pre-5.0.0). The paired-transfer relabel now covers both sides of the movement too.
`getAssetId` chose between a 12-byte ticker and a 16-byte asset ID on `is7xChain(block)`. That is wrong whenever the block misreports its runtime: `@subql/node` has been seen serving the pre-upgrade spec for a long run of blocks after v7.0.0 actually activated (~169k blocks reported as spec 6003050 instead of 7000003), and an already-migrated 16-byte asset ID in one of those blocks was then run through `getAssetIdForLegacyTicker` and blake2-hashed into a bogus id, so its `Asset` lookup missed. A `Ticker` is `[u8; 12]`, so a 16-byte value is a migrated asset ID whatever spec the block claims. `isMigratedAssetId` tests that, and `getAssetId` / `getAssetIdWithTicker` / the `handleAssetCreated` ticker derivation / `getAssetIdForStatisticsEvent` all check it before falling back to the spec gate.
|
prashantasdeveloper
marked this pull request as ready for review
September 11, 2026 12:30
prashantasdeveloper
added this pull request to stack #358
September 15, 2026 08:23
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 4 of the indexer redesign — plan
docs/implementation/02-polyx-ledger.md.Replaces the single-column
PolyxTransactionlist with an entry-centricPolyxEntryledger plus a materialised
AccountBalance, so "what was account X's balance atblock N" becomes an index scan instead of an un-answerable question.
Chained on
redesign/03-infrastructure(decode layer +IndexerAnomaly).Why
PolyxTransactionhad four structural faults, each fatal to point-in-time balances:typecolumn for a two-sided movement —ReservedisFree → Reserved;a single row records half of it.
frozenis a MAX of active locks, not a SUM — no summing of movement rowsreproduces it.
BalanceSetis an absolute value recorded as a delta — corrupts every runningtotal after it.
BalanceTypeEnumconflated three systems — real pools (Free/Reserved), alock floor, and staking-ledger states whose mechanism inverted at v8.
Plus: several v8
balancesevents were unregistered, leavingreservedunindexed on v8.What changed
Model
PolyxEntry— append-only, one row per(account, movement side), carryingfreeAfter/reservedAfter/frozenAftersnapshots; sibling entries of oneon-chain movement share
movementId. NewMovementKind/EntryDirection/PolyxPool/HoldReasonenums replaceBalanceTypeEnum.AccountBalance— running balance, height-versioned.frozen = MAX(locks),bonded= staking lock (≤ v7.4) orStakinghold (v8), plustransferable,otherReserved, and lifetime aggregates.PolyxTransactionandBalanceTypeEnumremoved.Handlers —
mapPolyxLedger.tsreplacesmapPolyxTransaction.tsbalances.*movement (transfer / endow / reserve / repatriate / hold /release / burn / mint / dust /
BalanceSet) writes entries and advances thebalance, at both the pre-v8 tuple shape and the v8 struct shape.
set_lockbonding,RewardDestination::Stakedauto-restake(credited and locked, no
Bondedevent), v8Bonded ↔ balances.Held{Staking}.Reconciliation
reconcilePolyx.ts) — periodically checks the derivedAccountBalanceagainst
system.accounton chain, records aBalanceReconciliationDriftanomalyand corrects, so one missed event can't compound.
scripts/reconcile-polyx.ts) — full-history audit.Decode shapes — pre-v8 tuple shapes for
balances.*andstaking.*.Postgres 18 — the height-versioned indexes need
btree_gist's boolean operatorclass (PG 13+); image
postgres:12-alpine → 18-alpine, data mount adjusted for thePG 18 subdirectory layout.
Full resync from genesis (decision D5) —
db/migrations/*deleted; accountresolution fixed for reindex-from-zero.
Follow-up commits
Three
chorecommits refine the new code (all pre-merge, none touch released code):settlement.InstructionCreatedaccepts its pre-memo7-param arity.identity.ClaimRevokedwith a zero issuer (stripped early-chain event).Stakedrestake lock,staking.payee().toJSON()variant normalisation, a 100-POLYX reconciler noise floor; plus aper-stash payee cache, a coarser reconcile interval and per-block
system.accountmemoisation.
Breaking changes
PolyxTransaction,BalanceTypeEnumremoved;PolyxEntry,AccountBalance,PolyxPool,MovementKind,EntryDirection,HoldReasonadded.Testing
yarn codegen && yarn typecheck && yarn lint && yarn test:unit(396 unit tests; new suites
mapPolyxLedger,reconcilePolyx,seedAccountBalances,rewardDestinationA15,extract8xStakingAmount).follow-up commits above.
Known follow-up
The pre-v8 staking lock is derived by accumulating restaked rewards; on chain it is
staking.ledger.active(max-bond capped, and MAX'd against other locks). The in-flightreconciler bounds the resulting drift to ≤ ~1 era per validator and corrects it at each
checkpoint; a precise fix (read
ledger.activeat the reward block) is tracked for afollow-up.
Merge
Rebase-merge onto
redesign/03-infrastructure. Do not squash.