Welcome to our eth2 insights series, presented by Elias Simos–Protocol Specialist at Bison Trails. In this series, Elias reveals insights he uncovered in his joint research effort with Sid Shekhar (Blockchain Research lead at Coinbase), eth2 Medalla—A journey through the underbelly of eth2’s final call before takeoff.
In this fourth post, Elias reviews how Medalla developed from beginning to close, introduces metrics that will help participants gauge overall network health and highlights some key takeaways ahead of Mainnet launch!
In this fourth and final post of eth2 insights stemming from the data that the Medalla Testnet produced, I will be looking into overall network health metrics. As we edge closer to Mainnet, with over 50% of the ETH required committed at the time of writing, this feels like the right send-off. My goal here is to equip you–the reader–with the right metrics and enough background information so that you are better able to contextualize activity on the network come Mainnet.
The analysis in this post spans between epochs [0, 14572] which map to slots [0, 466,335]—a time window that corresponds with approximately the beginning of Medalla (Aug 4th, 2020) to the first half of October 2020 (8th).
Over the course of this window over 78,846 unique validator indices became active on Medalla. The maximum number of activated validators observed stood at 72,554, and the maximum amount of staked Testnet ETH stood at 2.35M.
27% of the validators that were active over the Testnet’s life cycle were activated on epoch 0. This means that the threshold for finality at the start of Medalla stood at ~13.3k valuable attestations.
From that point onwards it was a steady rise for the active validator count; roughly 10k validators were added to Medalla every 2000 epochs (~9 days). Interestingly, though while more validators were deployed, no new significant operator groups entered the Testnet at its later stages. In fact, from the Top-20 operators, most entered before epoch 5,000 and a large majority were active since Genesis.
A glaring exception in the steady growth pattern of activated validators is the window between epoch 2,000 and epoch 4,000—the period in which the roughtime incident took place.
The count of ETH deposits over time gives us a clearer view of the network interruption during the incident; no deposits took place approx. between epochs [2200, 3500], while we are observing a similar density of deposits at Testnet start and at Testnet “re-start.”
Now, the above already hints clearly towards the fact that not all network states are made equal! Given this, there is merit in thinking along the lines of “normal” and “abnormal” network conditions.
During the roughtime incident, and for a period of approx. 1,500 epochs (1 week), 75% of the slots available to block proposers were missed. Without enough blocks being proposed, the network could not achieve finality for a prolonged period of time.
In a perfect world, 100% of the block slots available would be proposed—helping the network to continue moving in perpetuity, and all activated attesters would vote and so on; alas, the world is not perfect—but it can still be “normal!”
When plotting out the distributions of missed and proposed block slots over the set of epochs we based this analysis on we found that they both follow approximately normal distributions, with fat tails left and right that capture the essence of “abnormal” network conditions, like the roughtime incident.
When excluding the roughtime incident (the fat tail) from the set of observations—broadly epochs (2000, 4000)—we found that under “normal” conditions Medalla clocked at 3:1 proposed : missed slots! This means that, in every epoch, 24 slots were proposed and 7 were missed on average.
In and of itself, the fact that so many slots were missed under normal network conditions should raise some eyebrows. With a lot of slots being missed, the chain is slower to reach finality and the load of information included per slot increases—likely making it harder to survey past chain states for misconduct.
What could be different come Mainnet, however, is that since being negligent will be punished, a lot of the “absenteeism” we saw in Medalla is likely to course-correct. Indeed, as Pintail uncovered in their research, approximately 10% of validators in Medalla were entirely dormant or abandoned for the majority of the Testnet’s duration.
At this point, it’s also worth noting that under “normal” conditions the uncle rate for Medalla stood at roughly 1% of all block slots per epoch—a stark contrast to the ~5% on average that eth1.x produces.
The rate of orphaned blocks (correct blocks that do not make it on the canonical chain due to latency issues in the P2P layer of the network—i.e. the message doesn’t make it on time) remained stable over the course of the Testnet, with notable spikes around the beginning of the roughtime incident and at around epoch 13,700, when the network appears to have experienced another short spurt of high latency. This second spurt of high latency led to both 70% of the blocks being orphaned for the duration of an epoch and more than 60% of all block slots being missed a few epochs later.
The inclusion delay is a key moderating factor for the reward an attester stands to achieve by voting on blocks. For every additional block that the inclusion is delayed, the rewards reduce further. The reason why this is the case is that the network prefers that information relating to its state to be committed sooner rather than later, thus enhancing liveness and security.
In the aggregated view above, we looked at the whole set of attestations that were submitted on the Tesnet and their respective inclusion delays, calculating the average delays per epoch, per validator.
While pre-roughtime the aggregate inclusion delay ranged between 5 and 15 block slots, post-roughtime the aggregate inclusion delay smoothed out over time—from an average of 9 slots in epoch 4,000 to approximately 4 slots by epoch 14,700.
This view highlights that kickstarting a network is hard. The improvement we saw in Medalla over time took place as multiple client upgrades were released and then subsequently adopted by the network. Translating this into a learning for Mainnet, it means that client teams need to be ready with their releases and give operators enough headway pre-genesis to adopt them.
As a reminder finality is a state the chain achieves where a block (and its contents—e.g. transactions) can no longer be changed without a large percentage of stake getting destroyed. Eth2 guarantees finality, with a desired target of 2 epochs to reach it, at any given point in time.
Over the course of our research in eth2data.github.io we thought it would be interesting to come up with a visualization that surfaces a sense of timing in the chain’s efforts to reach finality. We did this by benchmarking for the target slot and counting backwards for when the attestations necessary to reach finality actually arrived.
From the above, it is clear that pre-roughtime Medalla was able to finalize within the target 2 epochs for over 80% of the time. However, as roughtime kicked in, the problem becomes crystal clear: between epochs [2500, 3200] Medalla could barely get 15% of the total valuable attestations necessary to achieve finality included!
When the network re-started, and for a period roughly 4500 epochs long, Medalla was finalizing within 3+ epochs consistently. It took until about epoch 11800 to start finalizing in the 2 epoch desired window again.
It looks as if pace/rhythm really matters for the network—as an interruption in momentum (roughtime) hampered the network’s ability to finalize over the desired window for much longer than the interruption window.
Medalla was a vibrant Testnet, with valuable learnings for participants and observers alike. Zooming out on the analysis we did in eth2data.github.io and the eth2 Insights Series, it seems as if Medalla truly served its purpose; we saw a clear improvement in overall network conditions as the Testnet matured, and also had the chance to take learnings from a doomsday-level fire-drill (roughtime).
Given that economic incentives will be activated in Mainnet, I also fully expect as Phase 0 takes hold we will see some of the baseline indicators (such as blocks proposed : blocks missed) improve drastically.
At the same time, high complexity, incomplete tooling, and frozen ETH will be deterring factors for players that have not been active in the Testnets. As such, Phase 0 adoption over time will probably be a case of “gradually and then suddenly,” with a chorus of familiar voices active from the start, but with the second and third cohorts trickling in gradually over time.
This will likely lead to a relatively concentrated Phase 0 in its early stages, both on the “ETH at stake” level as well as the deeper layers of the network (infrastructure providers, client software etc); but it also spells a very attractive opportunity (from a rewards perspective) for the brave early adopters that choose to swim alongside the innovators.
For all intents and purposes, we are still VERY early in what is slated to be the most exciting network launch in years.
Buckle up buckaroos—we’re in for a hell of a ride!
For custodial staking, use Coinbase's eth2 retail staking solution powered by Bison Trails.
It’s not too late to be an eth2 Pioneer. Learn more about the eth2 Pioneer Program for enterprise. We want you to have early access to build on the Beacon Chain!
Bison Trails is a blockchain infrastructure platform-as-a-service (PaaS) company based in New York City. We built a platform for anyone who wants to participate in 25 new chains effortlessly.
We also make it easy for anyone building Web 3.0 applications to connect to blockchain data from 33 protocols with Query & Transact (QT). Our goal is for the entire blockchain ecosystem to flourish by providing robust infrastructure for the pioneers of tomorrow.
In January, 2021, we announced Bison Trails joined Coinbase to accelerate our mission to provide easy-to-use blockchain infrastructure, now as a standalone product line. The Bison Trails platform will continue to support our customers. With Coinbase’s backing, we will enhance our infrastructure platform and make it even easier to participate in decentralized networks and build applications that connect to blockchain data.
Guide to OsmosisOct 21 2021
Bison Trails Provides Secure Infrastructure for Current to Build Hybrid Financial Applications on the Polkadot EcosystemOct 21 2021
Who’s who in eth2: Luke Youngblood from CoinbaseOct 8 2021
Colchis Capital to Run Provenance Validators on the Bison Trails Platform to Drive Innovation in Real Estate FinancingOct 5 2021
Bison Trails newsletter 017Oct 1 2021
Today we’re announcing the new Substrate staking dashboard.Sep 24 2021
Bison Trails newsletter 016Aug 27 2021
Guide to Binance Smart ChainAug 19 2021
eth2 update 015Aug 4 2021
Bison Trails newsletter 015Jul 22 2021
Bison Trails and CoinList: supporting the growth of innovative networksJul 22 2021
Launching Solana Query & TransactJul 19 2021
View more →