Welcome to Who’s who in eth2, presented by Elias Simos, Protocol Specialist at Bison Trails. In this series, Elias interviews key contributors at eth2’s four client teams—Prysmatic Labs, Sigma Prime, Nimbus, and Teku—to explore their involvement in eth2, visions for the future, and insider perspectives on what eth2 means for the world.
In this third post, Elias interviews Paul on how the Sigma Prime band got together, the impact of security-mindedness, how competition between clients is analogous to race car driving, and more reflections on eth2’s journey.
I was born in Australia, and have lived here for most of my life. I guess I was always interested in computers. I lived in a smallish city and since I was a kid I spent a lot of time just playing with my dad's computers and so on.
While in high school, I met Adrian and Luke, who later became my co-founders at Sigma Prime (two out of four), and together we spent hours on end fidgeting with electronics and computers. You know—from putting together hardware, to getting up into the school's network in ways that were “not so legal” (laughs).
Coming out of school, I didn’t get into computer science straight away. Instead, I spent a good amount of time working in audio-visual. Eventually, I got pulled over to a department that was heavily oriented towards programming: specifically the automation of large audio-visual systems like in auditoriums, or boardrooms via conferencing systems, and stuff like that.
As I was on that track, I ended up falling down the proverbial rabbit hole that is crypto. And it turned out that my friends, Adrian Manning, Luke Anderson and Mehdi Zerouali, were also on a similar path around the same time.
Given our backgrounds and interest in programming, Ethereum became our landing pad. Initially we started working together on an Ethereum mining project, and after some time a few people approached us and asked if we would do some consulting for them. Starting a company felt pretty natural. And that’s how Sigma Prime was born!
Sigma Prime is the symbol used in the Ethereum yellow paper to represent the change in the world’s state. We thought it was pretty cool, meaningful—and we also got a free logo with it! Initially we ran with it “for fun” for maybe six months before we started getting a bit more formally into a bunch of smart contract auditing work; this would have been around October, 2016. That’s when we incorporated Sigma Prime.
Then I think we chopped through for about maybe six months until the infamous 2017 ICO craze, which brought in a lot of consulting and auditing work for us. That became the core breadwinner for Sigma Prime.
After spending a solid year dedicated to building up the security side of the business, we decided to continue our software development work again. And we also felt very much drawn to the whole proof of stake idea, mainly because of its environmental properties. So, our involvement in eth2 started back in early 2018. I initially started working with Danny (Ryan) in an open source way—Sigma Prime spare time.
Sure! Back in 2017, when the auditing business started really picking up, Solidity was still in its pretty early days. I can't remember which language revision it was in, but it was pretty low. To give you some context, I think before 2017 there really weren’t any smart contracts out there. It was basically just the DAO and a handful of chicken lotto games that people were just gambling on.
The ICO era brought about a lot of change. Instead of either small scrappy contracts or really big fancy ones, you had these in-between contracts that were sufficiently complex to be risky, and these contracts held increasingly more money. Given how new everything was, there weren’t really any established security services as it was quite a unique skill to understand Solidity and the execution environment working inside of Ethereum.
We just picked it up kind of naturally. Mehdi was working for one of the big consulting firms as an Information Security professional, and Adrian and I were hands-on in terms of Solidity development. Luke was a lecturer in Information Security at the University of Sydney. We got our start with a local company that wanted us to review some of their contracts. We did it for an absurdly low price given today's rates, just to get our heads around it.
Our team’s background in Information Security gave us a big advantage, as far as bringing best practices to the game goes. We can’t just “declare things to be safe.” We have to articulately communicate the risks. So, over time, project by project, we developed a bunch of assets—testing suites, report formats, and so on. Then we picked up some bigger clients. Actually, Haven (the precursor to Synthetix) was one of our early big customers, and received a lot of really good feedback from both our clients and from the rest of the community. Gradually, as the industry expanded, we realized that the scope for doing this kind of work was much larger than just working with people in Australia.
From that point, we started to get quite serious, started expanding the team and so on. The good thing with the auditing business is that it is one of the few places in this industry that you can run a kind of a typical, very sustainable business model of professional services, with a large market to serve and not a lot of competition. This allowed us to bootstrap with our own capital and then to start doing more interesting things.
It was a passion for the foundation of it all, if anything! One of the founding goals of Sigma Prime was to work with the Ethereum Foundation. Back in 2017 we watched Vitalik Buterin and Vlad Zamfir and all the researchers presenting ideas on sharding and proof of stake; I think we all really looked up to them, and were really inspired to be part of that group of core developers working on Ethereum.
We were all mining ETH back in the day and making a steady amount of income from it, so when we started Sigma Prime none of us was particularly concerned about really trying to put bread on the table. That gave us the ability to do the things that really interested us.
If I recall correctly, the story starts with me spending some time out of town and just poking around the internet looking for things to do. I came across the eth2 sharding manager contract or something; it was something like the deposit contract for the old version of eth2. This would have been in early 2018, before the pivot to the whole eth2 beacon chain. Before I knew it, I had started contributing to the repo—picking up issues, testing frameworks, and trying to find bugs and drive optimizations. That’s how I first got connected with Danny Ryan,and we got along really well! Soon after I was going on holiday in Japan and he happened to be there at the same time—a lucky strike.
We spent some time together, and he got to know that I wasn’t a single developer but instead had this company that was interested in devoting more resources to the eth2 development effort. I think that’s the moment when the idea of us building a client started taking shape.
In racing, you don't win by overtaking the person in front of you. You win by driving the track faster than them.
Soon after, I started researching and we eventually made a grant proposal to the Ethereum Foundation. After a couple of back and forths—the spec was changing very frequently at the time—I started laying down the first few lines of code for Lighthouse in Rust. “Why Rust?” you’ll ask. Well, I hadn't worked with Rust a lot before, and I thought it would be a good language to get some more experience on. Also, it’s so “strict” that it would make it easier to find the logic bugs in the early spec.
At that time I managed to put together a basic state transition implementation. Then we went back to the EF and asked if they would be interested in helping us develop this into a more formal client implementation, as there was no other team at the time developing one on Rust. And they said “Yes!”
Since the beginning, given our backgrounds, we were always really keen to make the client very security-focused. We wanted Lighthouse to be—in a way—an audit of the specification as well as an implementation. We were committed to ensuring that we tested our own code way beyond what the Ethereum Foundation required.
We also really wanted Lighthouse to be performant. While when you say “We want to be safe and fast,” it kind of sounds like you shouldn't really have both of those things, I think they do go hand in hand. Put simply, when you are able to fend off attacks on the network and reject invalid things quickly, you gain on performance while also raising the bar on security!
These kinds of decisions meant that we always took a little bit longer to deliver things. Our client didn’t always come with the latest and greatest features that people wanted.
From a technical perspective, I think the greatest difficulty we had was that the specification changed quite a lot. We ended up throwing away quite a bit of code, which is understandable in the end. It's a typical problem really. The way that we work in Ethereum is in this very decentralized and open way, and it has its costs. Just like decision making in a democracy is more tedious than in a dictatorship. But, ultimately, I think it works well.
Besides that, given the rate at which the spec was changing, there were often gaps that we would then have to go and fill without a blueprint to develop on. That involved a lot of trial and error, which at times was hard to deal with, especially as different teams filled those gaps in different ways. We would then have to find a way to agree and interoperate. Danny Ryan’s role was extremely helpful in that aspect. He was the person who would make the decision when all we needed was someone to make a decision, which was great.
Now, from a management and business perspective, the most challenging aspect of things was the viability of the whole endeavor. Working off of grants is fine, but you always end up wondering “At what point will the grants stop coming?” That was a real concern, and we used to do a lot of thinking about how to diversify our revenue.
Finally, I guess the competition between clients was a bit stressful too. You're always trying to catch up with your peers. In that regard, I think you have to master the mental game. Like in racing, you don't win by overtaking the person in front of you. You win by driving the track faster than them.
Overall, I think the security-mindedness of Sigma Prime and our auditing capabilities brought a lot to the table in the eth2 development process. Now, do users think of that when they are thinking about which client they will use? I'm not so sure, but I know this capability we brought with us has had a very positive impact in the spec.
It was hard initially, but got a lot better over time. Being based out of Australia when most of your peers are working out of CET or EST/PST can be taxing. To give you an example, the eth2 sync calls often took place at midnight in Australia. I’m an early bird, so staying up that late really affects my sleeping schedule. But after a while it got better.
And remember, the issues that client implementation teams need to work on are generally pretty difficult and lengthy ones. That is to say that we're not all like teams working in one business bouncing questions off of each other quickly. Our work sprints were long and independent, and leveraging async tools like Github and so on really helped.
Plus, I think that for the work that we do, the asynchronous flow gives you a nice time-buffer in order to really think things through and be deliberate about what you communicate. Most—if not all—of the people involved in eth2 at a core level are really intelligent and reasonable people, and once a “best idea” surfaces, everybody accepts it.
Ultimately, the offsite we did in Canada back in 2019 was a real turning point. That was the moment when we were finally able to put a face to the names we had been working with for a while, and really got to know them as people. After breaking bread and staying together for a while it was much easier to deal with with people online.
Bugs—lots of bugs! An interesting point to note here is that the security team of Sigma Prime runs a “fuzzer.” This is a piece of software that sprays random inputs to a program and it attempts to make it crash—that's a basic fuzzer. And then a “differential fuzzer” is a more advanced version that sprays the same random inputs at multiple programs, and then surveys the outputs for discrepancies. So we ended up deploying that on the different client implementations, and I think it has found maybe 30 or 40 critical bugs on clients.
Overall, I think the security-mindedness of Sigma Prime and our auditing capabilities brought a lot to the table in the eth2 development process. Now, do users think of that when they are thinking about which client they will use? I'm not so sure, but I know this capability we brought with us has had a very positive impact in the spec.
Another stream of interesting findings came from our work optimizing Lighthouse. Early on the specification was written in a way that made the software easy to read, but made it very slow to execute runtime as the network scaled. To give you an example, we understood that it would take 2 seconds to run eth2 state transitions for 10 validators, 20 seconds for 20 validators, and something like 20 years for 5000 validators. So, in our work we uncovered ways to improve on these issues that in turn made their way into the spec.
We have always shared our findings with our peers. There’s no optimization that we “kept for ourselves,” so to speak. In fact, frequently we’ll comment our code in a way that makes it easier for other clients to learn from, or write test sectors for other clients to implement.
Oh, this is a good question. In fact, we had to define this for one of the grant questions. When first faced with this question, I remember thinking “If a hundred people will run our client we'll feel like we did a good job.” Suffice to say, we’ve done more than a good job by that standard.
Looking ahead, success for us will be doing our part in keeping Phase 0 stable and in getting Phase 1 out the door. We’re looking to add more monitoring capabilities on Lighthouse to understand what's happening even better, and when things aren't happening when they really should be. We’ll also be working to maximize validator effectiveness and profitability, while maintaining the high level of robustness and security that we have so far.
At the same time, we’re starting to work on how Phase 1 is going to operate, and exploring the unsolved engineering challenges around it. That's one part of success. And then the other part is figuring out how to deploy the Phase 1 logic onto Lighthouse without disrupting the smooth operation of its Phase 0 parts. If we do this successfully, I’ll be very happy!
I think maintaining Lighthouse as one of the leading clients and carving out our core capabilities in developing secure, stable, and performant software continues to be our main challenge, and what we’ll continue to be optimizing towards—always in a way that’s benevolent to the protocol.
Ultimately, we want to continue being able to maintain Lighthouse at a high standard, maintain a solid base of employment for the people that work at Sigma Prime, keep providing them with a great place to work at, and then continuing to output a product that is really useful for people. To me personally, even if it turns out that, say, two years from now, for whatever reason, Lighthouse fails because we did a terrible job at it, but eth2 has taken off, I will still feel satisfied. Even if the market has completely switched to using another client, knowing that we have made significant contributions to the development effort of eth2 and getting it to succeed is really what I feel success in this endeavor is all about.
I have a ton of respect for all of the work that Nimbus does. The way that Nimbus has had to work on their compiler and the Nim language as they go, and the way they have come to things in a different manner than everyone else, is something I admire. Sometimes it has gone their way, perhaps others it hasn’t, but they've always thought about things differently. And by that, they tend to come to the table with ideas that expand the breadth of the hive-mind, which is really interesting and a big value add for the collective effort.
Interview by Elias Simos
Bison Trails is a blockchain infrastructure platform-as-a-service (PaaS) company based in New York City. We built a platform for anyone who wants to participate in 22 new chains effortlessly.
We also make it easy for anyone building Web 3.0 applications to connect to blockchain data from 30 protocols with Query & Transact (QT). Our goal is for the entire blockchain ecosystem to flourish by providing robust infrastructure for the pioneers of tomorrow.
In January, 2021, we announced Bison Trails joined Coinbase to accelerate our mission to provide easy-to-use blockchain infrastructure, now as a standalone product line. The Bison Trails platform will continue to support our customers. With Coinbase’s backing, we will enhance our infrastructure platform and make it even easier to participate in decentralized networks and build applications that connect to blockchain data.
Substrate ecosystem update 006
Apr 10 2021eth2 update 013
Apr 9 2021Why are people excited about NFTs?
Apr 8 2021Celo governance proposal 0024: Q&A with Protocol Specialist Viktor Bunin
Apr 1 2021Who’s who in eth2: Jacek Sieka from Nimbus
Mar 28 2021Guide to Crypto.org Chain
Mar 26 2021eth2 update 012
Mar 18 2021Bison Trails announces support for Mina
Mar 18 2021An introduction to Protocol Specialists at Bison Trails
Mar 15 2021Meet the Herd: Engineering Manager Arthur Burkart
Mar 12 2021Bison Trails newsletter 012
Mar 9 2021View more →