protocols

Who’s who in eth2: Ben Edgington from ConsenSys Teku


In the second edition of our Who’s who in eth2 series, Elias Simos interviews Ben on the key elements of building eth2 and surprises discovered along the way.

Who’s who in eth2: Ben Edgington from ConsenSys Teku

By Elias Simos · Mar 7 2021

Welcome to Who’s who in eth2, presented by Elias Simos, Protocol Specialist at Bison Trails. In this series, Elias interviews key contributors at eth2’s four client teams—Prysmatic Labs, Sigma Prime, Nimbus, and Teku—to explore their involvement in eth2, visions for the future, and insider perspectives on what eth2 means for the world.

In this second post, Elias interviews Ben Edgington, Product Owner for the ConsenSys Teku eth2 client, on the speed of eth2’s origins, what motivates him to work on public blockchains, the key elements to bringing eth2 to fruition, and surprises discovered along the way.


Why don’t we start with a little bit of background about yourself, and how you came to spearhead the development of Teku.

I fell into crypto in early 2016. Before that I spent 20 years working for a massive Japanese multinational (Hitachi), and during that time ascended to a Head of Engineering role. Over time, it felt like I was getting deskilled. I was just filling in Excel spreadsheets and making PowerPoint presentations all day, and wasn't allowed to do anything interesting or fun anymore! In that role we did a lot of work in financial services and blockchain kept coming up.To cut a long story short, I just went down the rabbit hole from that point onwards.

I joined ConsenSys in October of 2017 and ended up in the protocol engineering team. At the time, quite a lot of the team’s focus was on Enterprise Ethereum. I spent the first three or four months working in that space, but my first true love was always public, permissionless Ethereum.

So I set out on what was a lonely journey of digging deeper into the critical questions around the future Ethereum–sharding, proof of stake, etc.—back at a time when the Beacon Chain didn’t even exist as a concept.


Vitalik Buterin (left) and Ben Edgington (right)
Figure 1: Vitalik Buterin (left) and Ben Edgington (right)


Things really kicked into gear for me in 2018 when we killed off some older ideas and started with a clean sheet of paper working on the Beacon Chain as the foundation for eth2. In October of that year we recruited a brilliant team at ConsenSys and started building what was called Artemis at the time. Artemis was our eth2 software, but we weren't committed to making it a viable client. It was mostly a research project for us.

A year later, around September 2019, we took stock of what we had built and found that we had the seeds of a viable mainnet client. So we transitioned it to a product development team—the same capable people who had worked on Hyperledger Besu and some of ConsenSys’ enterprise clients—and really spent the following year productizing it.


What's fascinating to me in this story is how everything came together in what seems like a very short time. Two years, from the inception of the Beacon Chain as an idea to having a production-grade Phase 0 with ~98% participation, is not long.

This is what I like to remind people! Pundits have been saying that eth2 is delayed by years. If you don't understand the background, I guess it can look like that. But actually, everything before June of 2018 was preparatory work. What we've accomplished in the two and a half years since then is incredible to me.


What is it that motivates you to devote your time to working on public blockchains, and to come at the problems you are tackling from the angle that you are currently?

Good question! It’s a lot about doing something new. I like to be breaking new ground. I think a lot of my motivation comes from wanting to understand things from the bottom up. I’m much more interested in bits and bytes and protocols than I am in applications and higher level abstractions.

When I started in Ethereum in 2016, the first thing I did was build a bytecode disassembler. I started downloading contracts from the chain and trying to work out what they did without the source code, just for fun. That's the kind of thing that I enjoy—understanding things at the foundational level. So getting into protocols felt very natural.


Going back to the eth2 chronicles–I’d love to learn more about the early community around the eth2 research effort and how that evolved over time.

Let’s start from the first sharding workshop we had in Taipei. I think about 50 or 60 people attended there. But from all of those people, I think there's probably six or seven who are still around, including Jacek from Status (Nimbus), the Prysmatic Labs guys (Raul Jordan, Preston Van Loon, and Terence Tsao), myself, and a colleague from ConsenSys, and that's about it! Everyone else moved on to other things.

Since then, there's been a huge influx of new people to the broader Ethereum scalability space. There's definitely more overlap of the people involved today with those that attended the Berlin workshop in mid-2018.

More recently, one of the seminal events of the eth2 development effort was an offsite we did in Ontario in September 2019. We hired some cabins by a lake and gathered 30+ members of the various teams and the Ethereum Foundation, and basically did a lock-in for a week. It was so remote, there was nothing to do except write code. We spent a week there getting seven or eight different client implementations talking to each other for the first time.

That was the moment when we all realized, “Yeah, we can do this. This is going to work!”


Figure 2: eth2 off-site by Skeleton Lake in Muskoka, Ontario
Figure 2: eth2 off-site by Skeleton Lake in Muskoka, Ontario


So we had 30 or so people there, but that wasn't everybody, because not everybody could get to Canada. If you go and look at the spec repo there's about 70 people or so contributing to it at the time. It might only be a few lines here or there, but that just speaks to the breadth of the whole effort.

Overall, I’d say over a hundred people have had some involvement in developing the eth2 spec. If we include people who engage in debates and discussions, and inform the community about the latest developments, it's probably evolved into a broader community of 200+ people.


What happened to the other clients that didn’t make it through to Mainnet?

Parity was working on an eth2 client—and actually got to a point where it was functional—but at the end of 2019, they abandoned it.

The Trinity client from the Ethereum Foundation that was built in Python was active until quite recently. They ceased development in Q3 2020. It just wasn't performant enough and my understanding is they thought the team's time would be better spent working on some of the tooling and other aspects of supporting eth2.

The Harmony client is interesting. This was a Russian team who came from EthereumJ: four Ethereum OGs. They eventually joined ConsenSys. Three of them are now working in R&D, and one of them we've adopted into the Teku dev team. Their implementation was in Java, and given that Teku is also in Java, it felt like a natural fit.

In distributed software, initial implementations tend to assume a benign environment. But, in fact, the most valuable part of the whole undertaking is making the software robust against attacks—thinking adversarially, not making too many assumptions, and just being really, really cautious. A lot of that stuff is so much harder to get right with single client implementations.


Let’s talk about the process of evolving the spec from that lock-in in Ontario to the present day. What were some of the key moments that defined the direction of it all?

The spec started out as a HackMD document and then it was transitioned to GitHub. The process was open-source from the beginning. I’d say that transparency has been incredibly helpful in drawing in a lot of mindshare and evolving the project.

Since early on though, the Ethereum Foundation Research Team was responsible for the majority of the heavy lifting in the spec, with Justin Drake, Vitalik Buterin, and Hsiao-Wei Wang initially being the key figures, and various others that have come on board over time. It’s worth noting, however, that all their work was taking place in full public view. Having this continuous open dialogue between theoreticians and implementers has been invaluable in making the spec more robust. But it wasn’t all smooth!

The process involved numerous revisions, rewrites, and a lot of going down dead ends. For what felt like a long time a lot was in flux, with the implementers having to go back and forth on a lot of the code that we wrote. I think it was only in June of 2019 that the spec was frozen to allow client implementers to work on a stable basis and execute without interruptions.

The spec unfroze again in January of 2020 and evolved much more slowly after that.The initial Phase 0 spec had various issues, like cross-links and leaky abstractions around sharding. It was cleaned up in January of 2020, the sharding stuff was taken out, and we ended up with a pure proof of stake Beacon Chain that can support sharding in future.

For the rest of the year, the main emphasis was on iterating on the networking protocol, which is a very pragmatic engineering challenge: a very iterative process, with a good degree of trial and error. To an extent we're still in that process.


What would you say the key people, skills and/or ideas that intermingled and brought eth2 to life were, both relating to the broader effort as well as to Teku specifically?

I feel that Justin Drake coming into the picture was critical. He appeared at a time when the whole vision had become a little stagnant. Nobody was quite sure how to deliver eth2. Justin came in and brought a slew of fresh ideas to the table–the BLS signatures, for example, which made the whole thing practical as a Beacon Chain–and really catalyzed a lot of new research streams. I think he and Vitalik had a very good dynamic.

Also, Danny Ryan coming in as a coordinator was transformational. We would never be where we are now without that function. It's truly extraordinary how he's managed to herd the cats on this.

Internally, within ConsenSys, we started Teku as an R&D project and things changed a lot as we moved it into production. We brought on people with skills across networking, Java development, and so on. The team has evolved from a research-focused set-up to a production-focused one, building at the enterprise level and working in a more systematic, deliberate, and feature-based way. It has been a huge privilege to work with so many brilliant team members over the last couple of years.


What were the key decisions that you made early on in the project?

The first one was actually whether we would build a client or not, and the whole thought process around what the best way to support the eth2 effort was. We concluded that building a client was the best way to get involved. We made that decision with a lot still unknown at the time.

Number two was around Q3 2019. At that point we had a solid codebase, but the question remained: “Do we productize it?” It is a long journey from a strong research project to a product that people are going to stake scary amounts of ETH on. After a round of stakeholder reviews, the decision came back as a resounding “Yes, we need to do this.”

The next one was our choice of Java as the programming language we would implement Teku in. With Java there's a huge toolkit available to help us build that includes testing frameworks, world class libraries, static analysis tools, performance monitoring–it’s all so rich! Java comes out of this networking client server world, and it just fits together very nicely with the problem at hand. Performance is also great. It's super fast and efficient. One challenge, however, is memory management, which can be tricky when people are trying to run on constrained devices.

Finally, another key decision for us was our market positioning, and the process around fashioning our message. I think it's good that clients differentiate themselves into different markets. If there's a monoculture it's potentially harmful to the network. As we saw in the Medalla incident, a bug in one client can bring the network to its knees.


Figure 3: Ben Edgington’s long standing newsletter “What’s New in Eth2”
Figure 3: Ben Edgington’s long standing newsletter “What’s New in Eth2”: https://eth2.news


One good way to achieve sufficient client diversity is to differentiate the clients with regard to their target users. Prysm, for example, has targeted the individual staker and they've been fabulously good at that community building. That's just not our strength. Instead, we have consciously focused Teku on institutional staking.

Within Pegasys we have huge experience of Enterprise Ethereum, and a base of developers who have typically come from enterprise Java backgrounds. We can do product support, we've got a strong dev-ops culture, and our goal is to give institutions confidence that we can support them now and into the future.


Are there any surprising findings that you stumbled upon as you were implementing the spec, and how did those influence the spec in return?

I think one of the things that we had expected when we first started was that the state transition stuff would be 80% of the work. Actually, this ended up being 5-10% of the work. 90-95% was all of the surrounding stuff: the storage, the networking, the threading models, and the internal architecture of the client. Getting all of that running correctly and efficiently took the vast majority of the time.

Beside that, it was fairly easy getting Teku optimized for the “happy flow” when things run well. However, when we ran into trouble on Medalla and the network wasn't finalizing, we started exploring some of the “unhappy” territory. I think that applies to all the implementation teams.

During the Medalla incident the network fragmented into dozens of forks and the software started running into edge-cases that we hadn't optimized for. So, memory use exploded, a lot of CPU time was spent exploring redundant forks, and so on. Dealing with all of that was a whole new challenge. All the teams did incredible work during those weeks and the benefits of that fed back into the standard “happy flow” implementations. This speaks to the value of all the extensive testing we did before rolling out Mainnet, making the software even more robust.

In distributed software, initial implementations tend to assume a benign environment. But, in fact, the most valuable part of the whole undertaking is making the software robust against attacks—thinking adversarially, not making too many assumptions, and just being really, really cautious. A lot of that stuff is so much harder to get right with single client implementations.


We touched on client diversity being an important goal for the eth2 implementation. How do we get closer to the desired balance?

This is tricky because once there are real returns involved, if one client starts becoming dominant—in terms of effective rewards, uptime, and so on—people will adopt that client. They won't adopt an inferior client due to some abstract notion of diversity.

Thankfully, I think there is a natural correction to that adverse dynamic (as far as network-wide goals are concerned). If one client gets ahead, the other client teams will just work harder on optimizing their software in order to leapfrog ahead. And, so far in some internal benchmarking, this seems to be the case.

I think it's good that clients differentiate themselves into different markets. If there's a monoculture it's potentially harmful to the network. As we saw in the Medalla incident, a bug in one client can bring the network to its knees.


Going back to what we talked about earlier in the conversation, do you think it is everything “around” the actual software–positioning and so on–that will eventually help drive client diversity in a broader sense?

I think so. Nimbus has got its natural constituency with low-powered devices. Prysm has the community focus. The really interesting one for me is Lighthouse, and the niche they will carve out. I don't think they have said anything about that yet. The reality of it is, however, that I don't know the answer. Only time will tell. Maybe it takes an incident. Maybe it takes one of the clients having trouble, forcing the network to adopt others. But my sense is that market positioning will eventually play out to be the largest differentiator.

When I speak to potential users, I say “You're not adopting a client so much as adopting a team.” Over the next couple of years there's going to be a huge amount of change in eth2. I think that as a prospective user of a client, it all boils down to your degree of confidence in the team to deliver the roadmap and to tackle all the emergent challenges. And I can’t speak too highly of the team behind Teku!


Looking to the future, what are the biggest challenges that lay ahead? And what are the biggest opportunities both for Teku and for client teams more broadly?

In the grand scheme of things, we've barely started on this journey. One of the exciting things about working in the Ethereum ecosystem is that the roadmap is constantly in flux as new information continues emerging. “Agile” would be the positive way to put it.

As it stands today, we have three independent streams to work on for the next year or so, which are (i) light-client implementation, (ii) sharding implementation, and (iii) the eth1-eth2 merge. There's a lot of work to be done on all of those. My goal for Q1 is to emerge with a plan for the rest of the year.

From a pure Teku point of view, there are two main things that I see as the big challenges and opportunities. The first is that we want to start differentiating and delivering on the ”institutional staking” focus, not just on the soft side of things, but actually on features. We have our Web 3.0 signer production-ready, but we need to be working harder on specific pain points that our customers face.

The other one is that—I believe—we're the only team that really has an in-house eth1 client. As we move towards an eth1-eth2 merge there will be an architecture whereby you can mix and match your eth1 and eth2 clients, and they'll communicate over an API. One thing we're exploring at ConsenSys is actually taking Besu (our eth1 client), integrating it more tightly with Teku, and seeing if that gives us any advantage in terms of manageability, performance, and so on. It's exploratory at the moment, and there's quite a lot of work needed on Besu, but that’s a direction we’re very excited to keep exploring.


Final question: what is your favorite detail about another client?

I think the sheer number of lines of code that the Prysm team has written and rewritten over the last three years is just astonishing. I have huge respect for them, for the sheer perseverance they’ve shown in sticking with this!


Interview by Elias Simos


About Bison Trails


Enterprise infrastructure to power the crypto ecosystem.

Pioneering Blockchain Infrastructure®

Bison Trails is a blockchain infrastructure platform-as-a-service (PaaS) company based in New York City. We built a platform for anyone who wants to participate in 22 new chains effortlessly.

We also make it easy for anyone building Web 3.0 applications to connect to blockchain data from 30 protocols with Query & Transact (QT). Our goal is for the entire blockchain ecosystem to flourish by providing robust infrastructure for the pioneers of tomorrow.

In January, 2021, we announced Bison Trails joined Coinbase to accelerate our mission to provide easy-to-use blockchain infrastructure, now as a standalone product line. The Bison Trails platform will continue to support our customers. With Coinbase’s backing, we will enhance our infrastructure platform and make it even easier to participate in decentralized networks and build applications that connect to blockchain data.


bison cool

THIS DOCUMENT IS FOR INFORMATIONAL PURPOSES ONLY. PLEASE DO NOT CONSTRUE ANY SUCH INFORMATION OR OTHER MATERIAL CONTAINED IN THIS DOCUMENT AS LEGAL, TAX, INVESTMENT, FINANCIAL, OR OTHER ADVICE. THIS DOCUMENT AND THE INFORMATION CONTAINED HEREIN IS NOT A RECOMMENDATION OR ENDORSEMENT OF ANY DIGITAL ASSET, PROTOCOL, NETWORK OR PROJECT. HOWEVER, BISON TRAILS (INCLUDING ITS AFFILIATES AND/OR EMPLOYEES) MAY HAVE, OR MAY IN THE FUTURE HAVE, A SIGNIFICANT FINANCIAL INTEREST IN, AND MAY RECEIVE COMPENSATION FOR SERVICES RELATED TO, ONE OR MORE OF THE DIGITAL ASSETS, PROTOCOLS, NETWORKS, ENTITIES, PROJECTS AND/OR VENTURES DISCUSSED HEREIN.

THE RISK OF LOSS IN CRYPTOCURRENCY, INCLUDING STAKING, CAN BE SUBSTANTIAL AND NOTHING HEREIN IS INTENDED TO BE A GUARANTEE AGAINST THE POSSIBILITY OF LOSS. THIS DOCUMENT AND THE CONTENT CONTAINED HEREIN ARE BASED ON INFORMATION WHICH IS BELIEVED TO BE RELIABLE AND HAS BEEN OBTAINED FROM SOURCES BELIEVED TO BE RELIABLE BUT BISON TRAILS MAKES NO REPRESENTATION OR WARRANTY, EXPRESS OR IMPLIED, AS TO THE FAIRNESS, ACCURACY, ADEQUACY, REASONABLENESS OR COMPLETENESS OF SUCH INFORMATION.

ANY USE OF BISON TRAILS’ SERVICES MAY BE CONTINGENT ON COMPLETION OF BISON TRAILS’ ONBOARDING PROCESS, INCLUDING ENTRANCE INTO APPLICABLE LEGAL DOCUMENTATION AND WILL BE, AT ALL TIMES, SUBJECT TO AND GOVERNED BY BISON TRAILS’ POLICIES, INCLUDING WITHOUT LIMITATION, ITS TERMS OF SERVICE AND PRIVACY POLICY, AS MAY BE AMENDED FROM TIME TO TIME.

Latest news

help

Contact us

Get in touch

General
Sales
Media
Legal