GRANDPA Finality Explained, With Real Production War Stories

What is GRANDPA finality?

GRANDPA (GHOST-based Recursive ANcestor Deriving Prefix Agreement) is a deterministic finality gadget from the Polkadot ecosystem, specified in the GRANDPA paper. Its defining trick is that validators vote on chains, not on individual blocks: a vote for a block implicitly votes for all its ancestors, so the protocol can finalise many blocks in one round. Voting proceeds in rounds of prevotes and precommits, and when more than two-thirds of voting weight agrees on a common ancestor, that block and everything below it becomes final, meaning it can never be reverted without slashing at least a third of validators. GRANDPA runs alongside a separate block-production mechanism, which in QBC's case is VQE useful-work mining. This split, production for liveness and GRANDPA for safety, is the standard Substrate architecture and the reason a chain can keep growing even while finality is temporarily stalled.

How rounds, prevotes, and precommits actually work

Each GRANDPA round has a primary proposer and two voting phases. Validators first broadcast prevotes for the best block they can see on the chain they consider canonical. Once a validator observes a supermajority prevote prefix, it precommits to it. A supermajority of precommits produces a commit, and the round finalises. The Polkadot protocol specification covers the full estimator logic. The practical takeaway for operators is that a round only completes when 2/3 of voters can both see and agree on the same blocks. With 3 GRANDPA voters, one node down is survivable; two is a halt. Anything that makes a single voter's view of the chain diverge, database pruning, a stuck sync, a phantom fork vote, stalls everyone, because rounds are a global rendezvous, not independent per-validator progress. That is where every one of our production incidents began. A useful mental model: block production is a local decision each author makes alone, while finality is a distributed agreement the whole voter set must reach together, which is why the two fail in such different ways.

War story: the pruned-block prevote stall

Our worst recurring incident traced to a subtle interaction between GRANDPA and state pruning. A voter would attempt to prevote for its best block, but chain selection failed with an UnknownBlock error because the block it wanted to reference had already been pruned from its database. The voter could not produce a valid prevote, the round could not gather a supermajority, and finality halted while block production continued happily on top. The unfinalized backlog then grew until someone intervened. The empirically reliable recovery was almost embarrassingly simple: restart any one voter, which forces a fresh round with a clean chain-selection view, and the stall clears. The durable fix belongs in node code, in the chain-selection path used by the voter. Lesson for anyone running custom consensus on Substrate: monitor the gap between best block and finalised block, and alert when it grows, because production liveness will mask a dead finality gadget for hours.

War story: equivocation risk when restoring voters

GRANDPA punishes equivocation, meaning one voter signing two conflicting votes in the same round, and rightly so, since equivocation is how safety breaks. But the same rule creates a trap for operators. If a stuck voter is restored from an old database snapshot, or wiped and resynced while its keys stay live, it can re-enter a round it already voted in and sign again, looking exactly like an attacker. We hit a variant of this with a voter casting a phantom vote for a fork it had seen before recovery. The safe pattern we converged on: take the voter fully offline, copy or warp-sync a database from a known-good live node, flush any stale voter state, and only then rejoin. Never run two instances that share a GRANDPA key, even briefly. If you operate a validator, treat vote-signing keys with the same never-twice discipline as any slashable signing key, because the protocol cannot distinguish your incident from an attack.

Session keys, key rotation, and upgrade pitfalls

GRANDPA voter identity lives in session keys, and session key management is where several of our sharpest edges appeared. Adding a new key type to a running chain, as we did when enabling BEEFY for light-client finality proofs, required an explicit storage migration to upgrade every validator's key bundle; without it, key decoding fails silently and voters drop out. Rotating keys must respect session boundaries, or a validator can end up voting with a key the current session does not recognize. And any change to native host functions requires rolling out new node binaries across the fleet, staggered, with the block author restarted last, before the runtime upgrade activates, or nodes fail closed and the network forks. None of this is in the quickstart docs. All of it is survivable with one habit: rehearse every consensus-adjacent change on a forked copy of production state before touching the live chain.

War story: shipping the finality fix without stopping the chain

Diagnosing the pruned-block stall was half the incident; deploying the fix was the other half. The durable correction lived in the node's chain-selection path, native code, which means forkless runtime upgrades could not carry it: every validator needed a new binary. Rolling new binaries across a fleet that is actively voting is its own hazard, because restarting too many voters at once drops the round below the 2/3 threshold and causes exactly the stall you are trying to cure. The choreography that worked: stage the binary everywhere, restart validators one at a time while watching peer counts and the best-minus-finalised gap recover between each step, and restart the current block author last so block production never pauses. The QuantumAI Blockchain incident record is public: the chain-selection correction is written up in the engineering log of the SelectChain fix, and the runtime-side hardening that accompanied it in the finality stall fix writeup. Fleets that cannot rehearse this on forked state should assume their first live attempt will teach them the threshold arithmetic the hard way.

The dashboard that predicts a stall

After enough incidents, our monitoring converged on a short list of numbers that predict finality trouble before users feel it. The best-minus-finalised gap is the headline metric: on a healthy chain it hovers near constant, and any monotonic growth means rounds are failing, whatever block height is doing. Peer count per voter matters because a partitioned voter votes on a view nobody shares. Session-key health deserves an explicit check after every upgrade, doubly so on a post-quantum chain whose bundles carry extra key types, since a silently undecodable key bundle looks identical to an offline validator; we learned this enabling BEEFY, whose compact signed commitments now back light-client proofs and whose rollout is documented in the BEEFY and MMR launch log. Equivocation monitoring closed the loop: rather than trusting operators to never double-sign, the network ships automated detection and slashing, described in the autonomous equivocation slashing writeup. None of these metrics require exotic infrastructure; all of them require deciding, in advance, that finality is a service with its own alerting rather than a background detail nobody owns until it breaks.

GRANDPA vs probabilistic finality

The comparison that matters for builders is deterministic versus probabilistic finality, covered in depth in deterministic finality explained. Nakamoto-style chains offer probabilistic finality: reorg risk decays with confirmations but never reaches zero. GRANDPA offers a proof: a finalised block carries a commit signed by 2/3 of validators, an artifact a bridge or light client can verify without trusting anyone, which is exactly what QBC's BEEFY layer packages into compact proofs. The price is liveness coupling: a Nakamoto chain degrades gracefully when nodes drop, while GRANDPA halts finality outright below the 2/3 threshold. On QBC, a live network of 4 validators on 3 machines, we have operated both failure modes and prefer the honest halt: an explicitly stalled finality gadget is diagnosable and recoverable, silently reorganized history is neither. New nodes join without replaying the full chain via warp sync, verifying finality proofs instead.