Ethereum Beacon Chain Finality Incident

Quick Take

• An exceptional scenario caused certain Consensus Layer (CL) nodes to crash and miss attestations, leading to a failure in obtaining enough votes for an updated pair of checkpoints (finality). Transactions were still processed and finality has been restored. Prysm and Teku issued patch releases.

Get the original hardware wallet (affiliate link).

This is our best recollection of events. We are open to any corrections or feedback.

Beacon Chain Stopped Finalizing Blocks

On May 11th at 8:13 PM UTC, the Ethereum Beacon Chain failed to finalize starting at epoch 200,552. Finality was restored roughly 20 minutes later without any intervention. The network suffered finality issues across three epochs and experienced validator participation as low as 40%. The issue was pinned to consensus layer clients.

Prysm Users Experience High Resouce Load

According to Prysm developer Terence Tsao, Prysm nodes received attestations for older epochs where the block did not reflect the latest checkpoint in fork choice. As a result, Prysm nodes experienced high resource usage while attempting to catch up with the latest attestations. Prysm users reported increased CPU load, missed slot proposals, out-of-memory errors, and crashes.

Terence said most consensus clients had similar issues with the exception of Lighthouse, which uses a different design that filters unviable attestations. Since a high number of validators missed attestations due to an issue with CL clients, a majority vote for a pair of checkpoints was not reached, causing blocks to not reach finality.

According to robplust, more than half of the validators who missed slots proposals during the incident were running a Prysm consensus client. Robplust also noted that attester effectiveness across all clients started to degrade as soon as three hours prior to the incident.

Ethereum Transaction Finalization

Finalization is a state where transactions in a block cannot be reverted without an attacker committing to losing at least one-third of the total staked ether. Ethereum transactions have finality after validators attest, in a timely manner, to the correct head of the chain and to a pair of checkpoints blocks. A checkpoint occurs at the start of each epoch.

Participation from validators representing at least two-thirds of all staked ether is required for a vote on a pair of checkpoints to be considered valid. Since a pair of checkpoints consist of two epochs and an epoch occurs every 6.4 minutes, Ethereum transactions typically finalize every 12.8 minutes.

Finalization Is “Nice To Have”

Finalization provides an economic guarantee that transactions will not be reverted. Finalization is not required for the network to remain live. In fact, Ethereum continued to process transactions during the unfinalized state. The chain does not pause and end users are not affected unless there is a block reorganization.

During an unfinalized state, the most recent blocks are not yet attested by the majority of validators. When blocks are not finalized, there is a possibility that all blocks, since the previous finalized block, can be reorganized. The network remains in a state similar to Bitcoin and Ethereum PoW, where a block reorganization is possible.

This is a developing incident, follow the latest updates on ethstaker.

• Second incident across 8 epochs (from 200751 to 200758).
• Second incident caused an inactivity leak.
• Second incident finalization restored.
• Prsym v4.0.3-hotfix release.
• Teku v23.5.0 release.