Welcome to Who’s who in eth2, presented by Elias Simos, Protocol Specialist at Bison Trails. In this series, Elias interviews key contributors at eth2’s four client teams—Prysmatic Labs, Sigma Prime, Nimbus, and Teku—to explore their involvement in eth2, visions for the future, and insider perspectives on what eth2 means for the world.
In this first post, Elias interviews Raul Jordan, Co-lead Ethereum Developer at Prysmatic Labs, on the team’s origin story, ethos, community building efforts, future outlook, and more.
Absolutely, and thank you once again for doing this. My name is Raul Jordan. I'm one of the maintainers of Prysmatic Labs. I'm originally from Honduras and first came to the US–where I currently live–to study CompSci at Harvard.
I was always fascinated with what you could do with a computer. The fact that you can use a keyboard and just type in magic words and all of a sudden create something that people on the other side of the world can use was pretty mind blowing to me as a student.
During my time as a student, I was introduced to Ethereum. I met a few people from the community (e.g. the teams at Augur and Aragon) and was immediately intrigued.
Towards the end of 2017, after the whole hype of the bull run, the conversations around the future of Ethereum were really ramping up. At the time Vitalik [Buterin] put out a lot of content on the topic and also published a grant for work related to Ethereum scaling.
Back then, I was still an “outsider” wanting to break into the ecosystem and scaling seemed like one of the most interesting problems to apply myself to. But at the time, it was really not obvious who was working on these problems. The only information available was something like one FAQ page and a bunch of scattered blog posts.
I'm somebody that likes to take problems into my own hands. I really think that if you work with the right people, you can really accomplish anything. So I just gave it a shot. I put out a call on Reddit that went something like, “Hey, I’m a developer looking into Ethereum. Anyone want to collaborate with me on prototyping this sharding thing Vitalik talks about?”
And then I actually got a bunch of responses. The first one was from Preston Van Loon (co-founder at Prysmatic Labs) saying he was interested. Next thing you know we are meeting up at his office in Google, where he was working full-time. We bounced ideas back and forth and settled on giving this a shot. And then suddenly we had around six or seven volunteers on the same boat.
Looking back, I think it was all about timing and being at the right place at the right time. We eventually managed to get a grant from the Ethereum Foundation to do this on a more incentivized basis. And that's how Prysmatic Labs came to be.
"I'm somebody that likes to take problems into my own hands. I really think that if you work with the right people, you can really accomplish anything."
Absolutely! I think if you want to pull off an organic open source initiative with people from around the world, you need to have people that are really motivated by a shared goal.
I started this effort with Preston while he was at Google. And two other people that we had from day one that are still with us today are Terrence (Tsao) and Nishant (Das), both of whom are just stellar individuals. They were there from the beginning, always learning, curious, and wanting to improve.
Most of the early team initially were basically working two jobs. They would work as software engineers at their day jobs, and then come back at five and work until midnight on this volunteer project. We then started getting a few more grants from the foundation, and that's when things really took off.
I remember I was with Preston in an Ethereum meetup and then he told me “the only reason I would leave Google is if I had a 10x opportunity.” And I think that, in terms of the opportunity that shipping eth2 would provide, it was more than enough for Preston and the other guys to jump ship and join full-time.
We want to showcase to people that we are grassroots. We are builders that like to bring good software engineering practices to the mix and be very community driven, which we have been from day one.
That's something that I noticed personally was lacking in the eth1 development space. People complained a lot about things happening behind closed doors. So we really wanted this to be an open forum, and I think that drove the point home. I think by now we must have roughly 10k people on our discord server.
"I think that our company culture is 'no nonsense.' We just want to build code that works and has been proven to work."
I think the openness is incredible–thinking about what it brings to the robustness of protocols. If you only have one implementation of the spec, in my view, you're 99% more likely to have serious bugs, meaning that you don't properly implement the specification. And you will never know because there's no other implementation that could point you in the direction of production level disagreements (e.g. blocks, state roots). Having multiple implementations is a big strength.
On the flipside, I would say wrangling interoperability between teams that are distributed all over the place can be challenging. Things improved as time went by, but initially–in part because of fragmented communication–we had to rewrite our code from scratch almost two or three times because the specification changed under our feet.
And when there are disagreements, without a clear leader in the process, how do you resolve that conflict? You end up having this deadlock, and it feels like things are moving really slowly.
Overall, however, the relationship between all the teams has been really cordial. The Ethereum Foundation had a big role to play here, driving the research and project managing the development at a high level. None of us would be here if we didn't have help from the likes of Danny Ryan and protolambda from the Foundation. They are really the heroes of this.
At the end of the day, I think all the difficulties along the way have been for the better. We have a really stable mainnet so far.
I would say it's probably more of a nexus approach. The EF is basically in charge of the research, but that doesn't mean that they dictate it. The field was open from the beginning for broader members of the core implementer community to come in with feedback and ideas. Actually, I believe the top contributor outside of the core research team is Terrence from our team.
During the latter months of development, maybe six or seven months leading up to mainnet launch, everything was driven pretty much by client implementers as the spec was frozen, meaning that there were no major changes on the Phase 0 spec. In that time, all the teams were basically just communicating with each other, doing multi-client testing, and such.
"If you only have one implementation of the spec, in my view, you're 99% more likely to have serious bugs, meaning that you don't properly implement the specification. And you will never know because there's no other implementation that could point you in the direction of production level disagreements (e.g. blocks, state roots). Having multiple implementations is a big strength."
What that means is that this really represents a breakthrough technology in computer science, and leaving a mark as one of the major implementations running the protocol is really what drives us to do what we do. The team has had incredible chemistry. We want to prove to people that we can ship this, that we can be a part of history.
Thinking further down the line I think we can go in so many directions after the roadmap is complete. And I would love to do it with the current team that we have today.
The way we approached the problem from very early on was very hands-on. We wanted to take problems into our own hands and really build something useful.
When picking a programming language, often it comes down to personal preference or your past experience. But for us, we didn't have long experience in Go development (Prysm is built in Go). But we wanted the software to be practical and made the intentional decision to build it in Go.
I think that our company culture is “no nonsense.” We just want to build code that works and has been proven to work.
So we picked Go as the programming language because I would say it's the language of the cloud for the last four or five years. A significant portion of cloud services and large scale applications that serve billions of users today are written in Go. And because of the large developer base that builds with Go there is a very long library of tools that we can draw from in the development process; that means we don’t need to go write an API or a json RPC from scratch.
As an example, in Prysm we use protocol buffers, which are a transport mechanism for API payloads. For that we use GRPC, which is an RPC library, for implementing these RPC services at scale.
These things serve, quite literally, billions of users on a day-to-day basis. They run all of Uber, they run all of Google, for example, and countless other companies. So we know these things work. We don't feel we need to reinvent the wheel.
One of the biggest things that helped us get to where we are today was deciding to launch a single client public testnet for people to try out very early on. We didn't even have a peer-to-peer networking specification at the time we launched the first Prysm testnet. And we were like–screw it! We're going to build our own, because people want to try this out. They wanted to see what staking was like. They wanted to test out the limits of the software.
We also were very deliberate about building a lot of documentation, onboarding guides etc., so that people can engage with the software. Having users trying it out and giving us continuous feedback really helped to push the envelope. It wasn't until a year later that we had a multi-client testnet.
Writing is for sure appreciated by the community. All of us on our team are always answering stakers’ questions. People jump in the Discord server and they're like, “What does this mean? can you guys explain it?” And, somebody always pops in and answers.
Over time, we built up a strong culture of DevOps. For example, when the Medalla incident happened, we followed our standard practice of doing post-mortems. Then we go through, we try to understand what happened and take out learnings, identify where we got lucky, and ask “how can this never happen again?” We built that culture of post-mortems and we made them public. In fact, the Medalla Medium post was actually just us basically copy pasting that post-mortem into a blog post. Thankfully, now the eth2 initiative has a dedicated response team–the Blue Team–where you have people not just from our team, but from other teams and the foundation that are able to respond to incidents if/when they happen.
Totally, I would say the same. I think it's really nice to be in a position where a lot of people are using your software, but it also comes with a lot more responsibility. So if something were to go wrong with Prysm, for example, we would really depend on all of the other clients to step up and keep the network going. Diversity helps the software become a lot more resilient.
Even if a client has a small share of the entire set of operators, it's still critical that that project is maintained and that it's supported by the parties that need to support it. If there's a serious issue in the network and those clients are still well-maintained they can help remediate these problems and people can migrate those clients.
The relationship we have with all the other teams has improved a lot over time. We appreciate each other and we appreciate the improvements that everyone is making to their software.
"The key to the success that Prysm has had so far is that we focused a lot on the average staker."
The key to the success that Prysm has had so far is that we focused a lot on the average staker. I will tell you, the average staker right now most likely runs Windows, is somewhat technical and as such is aware of the risks and what it means to stake, but is not a guru on the command line—doesn't have a Linux set up in their living room, so to speak. So the first thing I’d say is to really focus on the user.
Another one is really good documentation, good communication, making sure you respond to feature requests. And eventually, making the software much easier to use through point-and-click interfaces.
So that question is a difficult one. I think there's a lot of ways that you can try, but the problem is that you game those metrics. It's really difficult to create an incentive for beacon nodes that can not be gamed–that's something to keep in mind. Honestly, the best we can do really is to just focus on improving the bottom line.
Now, we have a ticking time bomb on our hands, in the sense that eth2 is live. We still have to ship Phase 1, add sharding, etc. That, in and of itself, is a separate beast. And these things need to be done in parallel. So, how do we prioritize our time as a team to accomplish this?
So far, we’ve relied a lot on the wisdom of the Ethereum Foundation on the research side for what to do next. They've been incredibly insightful. And now we need to ship the next phases ASAP–ideally this upcoming year (2021).
That's the main challenge I see ahead. We’re on a treadmill that is not going to stop. We need to get going and to implement this stuff.
I want to see Prysmatic Labs as a highly regarded core implementation team. We are a team that thrives on experimentation, with a core focus on improving the Layer 1. And we want to keep going, even once that roadmap is done. I think that our team can succeed at any future challenge. It will be a dream come true for me if we can build something on top of eth2 in the future. So whether that is a really interesting smart contract based application or tackling scalability, I'll be really thrilled to do that.
I thought it was super cool when Nimbus posted a photo of themselves running a small connected network of Raspberry Pis or something, running perfectly on like some large scale testnet. And just how little resources the client consumes is really amazing. Like any sort of device can probably run Nimbus–that is really cool. I think those guys are really pushing the envelope of what's possible on lower-end devices.
And like I said, if there's a team/client that remains well-maintained, and is aligned with the specification and works on mainnet, we need it to be used by people, we need it to keep going and to be supported. Hopefully one day they can get it running on a smart fridge or something similar. I think those guys have the ability to do it.
Interview by Elias Simos
Bison Trails is a blockchain infrastructure platform-as-a-service (PaaS) company based in New York City. We built a platform for anyone who wants to participate in 22 new chains effortlessly.
We also make it easy for anyone building Web 3.0 applications to connect to blockchain data from 30 protocols with Query & Transact (QT). Our goal is for the entire blockchain ecosystem to flourish by providing robust infrastructure for the pioneers of tomorrow.
In January, 2021, we announced Bison Trails joined Coinbase to accelerate our mission to provide easy-to-use blockchain infrastructure, now as a standalone product line. The Bison Trails platform will continue to support our customers. With Coinbase’s backing, we will enhance our infrastructure platform and make it even easier to participate in decentralized networks and build applications that connect to blockchain data.
Substrate ecosystem update 006Apr 10 2021
eth2 update 013Apr 9 2021
Why are people excited about NFTs?Apr 8 2021
Celo governance proposal 0024: Q&A with Protocol Specialist Viktor BuninApr 1 2021
Who’s who in eth2: Jacek Sieka from NimbusMar 28 2021
Guide to Crypto.org ChainMar 26 2021
Who’s who in eth2: Paul Hauner from Sigma PrimeMar 18 2021
eth2 update 012Mar 18 2021
Bison Trails announces support for MinaMar 18 2021
An introduction to Protocol Specialists at Bison TrailsMar 15 2021
Meet the Herd: Engineering Manager Arthur BurkartMar 12 2021
Bison Trails newsletter 012Mar 9 2021
View more →