Welcome to our eth2 insights series, presented by Elias Simos–Protocol Specialist at Bison Trails. In this series, Elias reveals insights he uncovered in his joint research effort with Sid Shekhar (Blockchain Research lead at Coinbase), eth2 Medalla—A journey through the underbelly of eth2’s final call before takeoff.
In this second post, Elias looks at slashings in eth2 along with their correlates, probable causes, and the most common sources of slashing malaise in Medalla.
Slashing is a core protocol function and mechanism enabler in proof of stake networks. In very crude terms, if block rewards are the “carrot,” slashing is the “stick”—a negative incentive to keep network participants from misbehaving.
In Phase 0 of eth2, slashing is the fate of those validators that either: (i) double attest or double propose, or (ii) surround vote. To better understand slashing and the conditions that define its various forms, it’s important to remember how consensus is achieved in eth2. At a very high level, the body of validators broadcasts votes on the state of the chain (attestations) and subsequently these votes are aggregated and then passed on to the block producer to include in a block. But what do these votes include exactly?
When an attester votes, they must provide two types of votes: (i) a vote towards the head of the chain, and (ii) a vote towards finality. As such, an attester’s vote is described by the following core components/fields:
Target slot– this signifies the slot in which the attester has been allocated to attest in by the RANDAO, in a given epoch. It also signifies the vote towards the head of the chain, by LMD-GHOST.
Committee index– this is the group of validators that the attester has been assigned to attest with. Committees help with aggregating votes together, which end up on-chain as “grouped” attestations.
Source checkpoint [s]– a checkpoint is the first (proposed) block of an epoch. The source checkpoint is a vote towards what the attester recognizes as the last justified epoch boundary, by Casper FFG. A checkpoint is considered justified if ⅔ of validators in the active set have voted for it.
Target checkpoint [t]– this is a vote towards what should be the next justified checkpoint, by Casper FFG. The source and target votes help towards getting the chain state to finalize.
As soon as ⅔ of active validator votes are included on-chain for a said pair of source (s) and target (t) checkpoints related to an epoch, they become justified. The previous epoch’s pair become finalized—meaning that the state of the chain before that checkpoint boundary becomes unchangeable.
The Casper FFG paper outlines the two slashing conditions as follows:
Double vote– when two conflicting attester votes from the same validator that share the same target checkpoint (t1 = t2) get included on-chain. For proposers, a double vote translates to the proposer signing two different beacon blocks in the same slot.
Surround vote– an attester vote with source and target surrounding or surrounded by previously made votes, such that:
s1 < s2 < t2 < t1
Slashing in Phase 0 not only triggers a balance reduction (which partly depends on how many other validators get slashed around the same time), but also an exit from the active set. This means that validators will join the exit queue and will not earn back the right to participate in consensus and earn rewards. It also means that their ETH will be frozen until Phase 1.5.
The entity responsible for calling out a slashable offence and enacting protocol level penalties is known as the “whistleblower.” The whistleblower collects the evidence that outlines the offence, and then commits the proof(s) to a block proposer for inclusion on-chain. While beyond Phase 0 the whistleblower stands to earn the majority of those rewards (⅞), for Phase 0 the block proposer will earn the full whistleblower reward.
Finally, slashable offences need not be discovered immediately after they actually take place. The protocol allows for a window of time between an offence being committed and the inclusion of a slashing proof, known as the “weak-subjectivity” period. This can span for a maximum of 54k epochs—a long time!
This window of time implies that the whistleblower needs to store an extensive record of chain history, and use that to survey for violations of protocol rules. Given the size of the chain’s state and the large share that attestations occupy, it follows that a leaner aggregated attestations state leads to a leaner overall history, which might then lead to better detection of rule violations!
Over the sample of 14,500 epochs we surveyed in eth2data.github.io, we observed approximately 5,100 attester slashings and 50 proposer slashings. Note that the overall number of slashings observed—taken from the beaconcha.in API—does not agree with the number reported by beaconscan.com which stands at ~1,900.
The misreporting is likely on beaconcha.in and it likely relates to beaconcha.in tracking “submitted slashing proofs” and not actual slashing events. Proofs for the same event can be submitted multiple times, but it is only the first that enforces the punishment.
Regardless, while the nominal values might be misleading, there is still value in surfacing the overall trends.
The crushing majority of the attester violations in Medalla took place around epoch 2,200—at the beginning of the roughtime incident. To provide a little context, during roughtime Prysm client clocks went out of sync with network time. Because of this, validators incorrectly proposed blocks and submitted attestations for future slots.
While the offences were centered around the beginning of roughtime, only about 20% of the total slashing proofs were included around the same time. 94% of the whole set of slashing proofs was included by epoch 4,000. The remainder kept rolling in up until the very end of the sample of 14,500 epochs we looked at—arriving with a significant delay.
For clarity and ease of use, let’s call the delta between the offence taking place and the proof getting actually included on-chain, “detection delay.”
On average, the attester offence detection delay stood at 750 epochs. The top 10% of slashers, however, managed to get their proofs included in under 20 epochs! When trying to link those fast slashers to client choice, no deviation from the network-wide client distribution was found among the ⅓ that we were able to identify.
Now, things get really interesting when comparing the aggregate attestations surplus over time, with the average detection delay in slashable offences, measured on a per epoch basis.
The two trends appear to correlate! When the attestations surplus was at its highest early on in Medalla (up to epoch 2,200), slashings took on average 3000 epochs to get detected. As the Testnet matured things improved—with the average detection delay dropping to approximately 500 epochs. However, even then it appears that as the attestations surplus grew, the detection delay followed suit. This is a very important insight to keep in mind, as it means the poor aggregation performance in Medalla has a direct effect on the enactment of slashings!
Now, getting a little more granular with respect to the nature of attester violations, we observed that all enacted slashings link back to double votes! More specifically, 90% were down to double votes with the same source and target epochs but different target slots. Another 8% of the double vote slashings were attestations that disagreed both in the target slot and source epoch fields, while double vote slashings due to pure asynchrony in checkpoint values was a much less frequent phenomenon.
Note that 6.23% of the reported slashings do not appear to be slashable offences—likely due to a bug in the beaconcha.in API.
Moreover, none of the observations satisfy the surround vote condition. This doesn’t necessarily mean that no surround vote violations took place. However, surfacing these violations is orders of magnitude more complex than it is for simple double votes—and is an area of active research within the technical Ethereum community.
Finally, when investigating the client types of the proposers that included the slashing proofs, we found that a greater proportion of those were running Lighthouse than is representative of the overall client software distribution among validators in Medalla. There are two reasons why this might be the case.
The first is that as a lot of Prysm validators went down during roughtime, validators running Lighthouse represented a larger share of the population!
When looking at proofs submitted after epoch 4,000—when the network returned to “normality”—Lighthouse’s share appears even greater! Caveating for small sample size, this could indicate that the second reason for Lighthouse’s overperformance on the slashing front, is that the slashing software packed in the Lighthouse client might have an edge over its competition!
Proposer slashings are a little more straightforward than attester slashings and center around double proposals of blocks. In Medalla, these have been few and far between (47 events in total), with the majority concentrated around epoch 13,700. Our initial thought was that this was the case either due to a bug, or a client software upgrade that included a better “slasher” module.
After inspecting both the proposers that included the proofs and the proposers that got slashed, we think that the former is more likely as there was no evident pattern in the whistleblower/proposer client type.
We were able to map the slashed proposers to an overwhelming majority running the Nimbus client (~60%). When zooming in to the high density proposer slashing epoch range, that percentage grew to over 85%! Given that Nimbus was subject to syncing issues as attention of stakers turned from Medalla to Spading/Zinken around the same time, we think it’s likely that this was the cause of the proposer slashings here.
It’s also worth noting that the detection delay in proposer slashings was 1 slot for ~85% of the cases observed. This is a stark contrast to the average 750 epochs detection delay observed in attester slashings and a testament to the fact that detecting attester slashings is orders of magnitude more complex than detecting proposer slashings!
Overall, over the course of our research, as we were able to attribute the majority of slashings in Medalla to their most probable root causes, it becomes apparent that client syncing issues (specifically with Prysm during roughtime and Nimbus around epoch 13,700) have been the most common source of malaise for validators in the Testnet.
While we didn’t have enough time to survey the attestations log for slashable offences that weren’t enacted, in this post we showed that–indeed–it appears that poor aggregation performance has a direct effect on how quickly slashable offences are detected.
Given our thinking around the attestations surplus and how it can potentially degrade the performance of whistleblowers, it’s reasonable to expect that correlating attestations surplus over time with unpunished slashable offences might hold even more interesting insights!
And that’s perhaps a story for another time...
For custodial staking, use Coinbase's eth2 retail staking solution powered by Bison Trails.
It’s not too late to be an eth2 Pioneer. Learn more about the eth2 Pioneer Program for enterprise. We want you to have early access to build on the Beacon Chain!
Bison Trails is a blockchain infrastructure platform-as-a-service (PaaS) company based in New York City. We built a platform for anyone who wants to participate in 25 new chains effortlessly.
We also make it easy for anyone building Web 3.0 applications to connect to blockchain data from 33 protocols with Query & Transact (QT). Our goal is for the entire blockchain ecosystem to flourish by providing robust infrastructure for the pioneers of tomorrow.
In January, 2021, we announced Bison Trails joined Coinbase to accelerate our mission to provide easy-to-use blockchain infrastructure, now as a standalone product line. The Bison Trails platform will continue to support our customers. With Coinbase’s backing, we will enhance our infrastructure platform and make it even easier to participate in decentralized networks and build applications that connect to blockchain data.
Guide to OsmosisOct 21 2021
Bison Trails Provides Secure Infrastructure for Current to Build Hybrid Financial Applications on the Polkadot EcosystemOct 21 2021
Who’s who in eth2: Luke Youngblood from CoinbaseOct 8 2021
Colchis Capital to Run Provenance Validators on the Bison Trails Platform to Drive Innovation in Real Estate FinancingOct 5 2021
Bison Trails newsletter 017Oct 1 2021
Today we’re announcing the new Substrate staking dashboard.Sep 24 2021
Bison Trails newsletter 016Aug 27 2021
Guide to Binance Smart ChainAug 19 2021
eth2 update 015Aug 4 2021
Bison Trails newsletter 015Jul 22 2021
Bison Trails and CoinList: supporting the growth of innovative networksJul 22 2021
Launching Solana Query & TransactJul 19 2021
View more →