Welcome to Who’s who in eth2, presented by Elias Simos, Protocol Specialist at Bison Trails. In this series, Elias interviews key contributors at eth2’s four client teams—Prysmatic Labs, Sigma Prime, Nimbus, and Teku—to explore their involvement in eth2, visions for the future, and insider perspectives on what eth2 means for the world.

In this third post, Elias interviews Paul on how the Sigma Prime band got together, the impact of security-mindedness, how competition between clients is analogous to race car driving, and more reflections on eth2’s journey.

Let’s start with a little bit of background–what was your personal journey to get where you are today, and what’s the backstory of how the Sigma Prime band came together?

I was born in Australia, and have lived here for most of my life. I guess I was always interested in computers. I lived in a smallish city and since I was a kid I spent a lot of time just playing with my dad's computers and so on.

While in high school, I met Adrian and Luke, who later became my co-founders at Sigma Prime (two out of four), and together we spent hours on end fidgeting with electronics and computers. You know—from putting together hardware, to getting up into the school's network in ways that were “not so legal” (laughs).

Coming out of school, I didn’t get into computer science straight away. Instead, I spent a good amount of time working in audio-visual. Eventually, I got pulled over to a department that was heavily oriented towards programming: specifically the automation of large audio-visual systems like in auditoriums, or boardrooms via conferencing systems, and stuff like that.

As I was on that track, I ended up falling down the proverbial rabbit hole that is crypto. And it turned out that my friends, Adrian Manning, Luke Anderson and Mehdi Zerouali, were also on a similar path around the same time.

Given our backgrounds and interest in programming, Ethereum became our landing pad. Initially we started working together on an Ethereum mining project, and after some time a few people approached us and asked if we would do some consulting for them. Starting a company felt pretty natural. And that’s how Sigma Prime was born!

How did you land on the name?

Sigma Prime is the symbol used in the Ethereum yellow paper to represent the change in the world’s state. We thought it was pretty cool, meaningful—and we also got a free logo with it! Initially we ran with it “for fun” for maybe six months before we started getting a bit more formally into a bunch of smart contract auditing work; this would have been around October, 2016. That’s when we incorporated Sigma Prime.

Then I think we chopped through for about maybe six months until the infamous 2017 ICO craze, which brought in a lot of consulting and auditing work for us. That became the core breadwinner for Sigma Prime.

After spending a solid year dedicated to building up the security side of the business, we decided to continue our software development work again. And we also felt very much drawn to the whole proof of stake idea, mainly because of its environmental properties. So, our involvement in eth2 started back in early 2018. I initially started working with Danny (Ryan) in an open source way—Sigma Prime spare time.

Let’s talk a little more about your early days in smart contract auditing; how did that define you and Sigma Prime?

Sure! Back in 2017, when the auditing business started really picking up, Solidity was still in its pretty early days. I can't remember which language revision it was in, but it was pretty low. To give you some context, I think before 2017 there really weren’t any smart contracts out there. It was basically just the DAO and a handful of chicken lotto games that people were just gambling on.

The ICO era brought about a lot of change. Instead of either small scrappy contracts or really big fancy ones, you had these in-between contracts that were sufficiently complex to be risky, and these contracts held increasingly more money. Given how new everything was, there weren’t really any established security services as it was quite a unique skill to understand Solidity and the execution environment working inside of Ethereum.

We just picked it up kind of naturally. Mehdi was working for one of the big consulting firms as an Information Security professional, and Adrian and I were hands-on in terms of Solidity development. Luke was a lecturer in Information Security at the University of Sydney. We got our start with a local company that wanted us to review some of their contracts. We did it for an absurdly low price given today's rates, just to get our heads around it.

Our team’s background in Information Security gave us a big advantage, as far as bringing best practices to the game goes. We can’t just “declare things to be safe.” We have to articulately communicate the risks. So, over time, project by project, we developed a bunch of assets—testing suites, report formats, and so on. Then we picked up some bigger clients. Actually, Haven (the precursor to Synthetix) was one of our early big customers, and received a lot of really good feedback from both our clients and from the rest of the community. Gradually, as the industry expanded, we realized that the scope for doing this kind of work was much larger than just working with people in Australia.

From that point, we started to get quite serious, started expanding the team and so on. The good thing with the auditing business is that it is one of the few places in this industry that you can run a kind of a typical, very sustainable business model of professional services, with a large market to serve and not a lot of competition. This allowed us to bootstrap with our own capital and then to start doing more interesting things.

Given your level of knowledge around Solidity and Ethereum, I imagine you could have easily strayed towards building smart contract applications. What prompted you to hone in on the auditing game and then the core-Layer 1 approach in eth2?

It was a passion for the foundation of it all, if anything! One of the founding goals of Sigma Prime was to work with the Ethereum Foundation. Back in 2017 we watched Vitalik Buterin and Vlad Zamfir and all the researchers presenting ideas on sharding and proof of stake; I think we all really looked up to them, and were really inspired to be part of that group of core developers working on Ethereum.

We were all mining ETH back in the day and making a steady amount of income from it, so when we started Sigma Prime none of us was particularly concerned about really trying to put bread on the table. That gave us the ability to do the things that really interested us.

Shifting gears now—let’s zoom in on Sigma Prime’s eth2 journey. Talk me through how you went from a passion to developing the second most adopted eth2 client at present: Lighthouse.

If I recall correctly, the story starts with me spending some time out of town and just poking around the internet looking for things to do. I came across the eth2 sharding manager contract or something; it was something like the deposit contract for the old version of eth2. This would have been in early 2018, before the pivot to the whole eth2 beacon chain. Before I knew it, I had started contributing to the repo—picking up issues, testing frameworks, and trying to find bugs and drive optimizations. That’s how I first got connected with Danny Ryan,and we got along really well! Soon after I was going on holiday in Japan and he happened to be there at the same time—a lucky strike.

We spent some time together, and he got to know that I wasn’t a single developer but instead had this company that was interested in devoting more resources to the eth2 development effort. I think that’s the moment when the idea of us building a client started taking shape.

In racing, you don't win by overtaking the person in front of you. You win by driving the track faster than them.

Soon after, I started researching and we eventually made a grant proposal to the Ethereum Foundation. After a couple of back and forths—the spec was changing very frequently at the time—I started laying down the first few lines of code for Lighthouse in Rust. “Why Rust?” you’ll ask. Well, I hadn't worked with Rust a lot before, and I thought it would be a good language to get some more experience on. Also, it’s so “strict” that it would make it easier to find the logic bugs in the early spec.

At that time I managed to put together a basic state transition implementation. Then we went back to the EF and asked if they would be interested in helping us develop this into a more formal client implementation, as there was no other team at the time developing one on Rust. And they said “Yes!”

Aside from implementing the spec in Rust, what were some of the other key design decisions that you made along the way?

Since the beginning, given our backgrounds, we were always really keen to make the client very security-focused. We wanted Lighthouse to be—in a way—an audit of the specification as well as an implementation. We were committed to ensuring that we tested our own code way beyond what the Ethereum Foundation required.

We also really wanted Lighthouse to be performant. While when you say “We want to be safe and fast,” it kind of sounds like you shouldn't really have both of those things, I think they do go hand in hand. Put simply, when you are able to fend off attacks on the network and reject invalid things quickly, you gain on performance while also raising the bar on security!

These kinds of decisions meant that we always took a little bit longer to deliver things. Our client didn’t always come with the latest and greatest features that people wanted.

What are some of the most pronounced difficulties that you faced along the way while developing Lighthouse?

From a technical perspective, I think the greatest difficulty we had was that the specification changed quite a lot. We ended up throwing away quite a bit of code, which is understandable in the end. It's a typical problem really. The way that we work in Ethereum is in this very decentralized and open way, and it has its costs. Just like decision making in a democracy is more tedious than in a dictatorship. But, ultimately, I think it works well.

Besides that, given the rate at which the spec was changing, there were often gaps that we would then have to go and fill without a blueprint to develop on. That involved a lot of trial and error, which at times was hard to deal with, especially as different teams filled those gaps in different ways. We would then have to find a way to agree and interoperate. Danny Ryan’s role was extremely helpful in that aspect. He was the person who would make the decision when all we needed was someone to make a decision, which was great.

Now, from a management and business perspective, the most challenging aspect of things was the viability of the whole endeavor. Working off of grants is fine, but you always end up wondering “At what point will the grants stop coming?” That was a real concern, and we used to do a lot of thinking about how to diversify our revenue.

Finally, I guess the competition between clients was a bit stressful too. You're always trying to catch up with your peers. In that regard, I think you have to master the mental game. Like in racing, you don't win by overtaking the person in front of you. You win by driving the track faster than them.

Overall, I think the security-mindedness of Sigma Prime and our auditing capabilities brought a lot to the table in the eth2 development process. Now, do users think of that when they are thinking about which client they will use? I'm not so sure, but I know this capability we brought with us has had a very positive impact in the spec.

I’d love to zoom in more on the competitive dynamic between client teams. How did that develop over time?

It was hard initially, but got a lot better over time. Being based out of Australia when most of your peers are working out of CET or EST/PST can be taxing. To give you an example, the eth2 sync calls often took place at midnight in Australia. I’m an early bird, so staying up that late really affects my sleeping schedule. But after a while it got better.

And remember, the issues that client implementation teams need to work on are generally pretty difficult and lengthy ones. That is to say that we're not all like teams working in one business bouncing questions off of each other quickly. Our work sprints were long and independent, and leveraging async tools like Github and so on really helped.

Plus, I think that for the work that we do, the asynchronous flow gives you a nice time-buffer in order to really think things through and be deliberate about what you communicate. Most—if not all—of the people involved in eth2 at a core level are really intelligent and reasonable people, and once a “best idea” surfaces, everybody accepts it.

Ultimately, the offsite we did in Canada back in 2019 was a real turning point. That was the moment when we were finally able to put a face to the names we had been working with for a while, and really got to know them as people. After breaking bread and staying together for a while it was much easier to deal with with people online.

What were some of the most interesting or surprising findings that you unearthed while working on Lighthouse?

Bugs—lots of bugs! An interesting point to note here is that the security team of Sigma Prime runs a “fuzzer.” This is a piece of software that sprays random inputs to a program and it attempts to make it crash—that's a basic fuzzer. And then a “differential fuzzer” is a more advanced version that sprays the same random inputs at multiple programs, and then surveys the outputs for discrepancies. So we ended up deploying that on the different client implementations, and I think it has found maybe 30 or 40 critical bugs on clients.

Overall, I think the security-mindedness of Sigma Prime and our auditing capabilities brought a lot to the table in the eth2 development process. Now, do users think of that when they are thinking about which client they will use? I'm not so sure, but I know this capability we brought with us has had a very positive impact in the spec.

Another stream of interesting findings came from our work optimizing Lighthouse. Early on the specification was written in a way that made the software easy to read, but made it very slow to execute runtime as the network scaled. To give you an example, we understood that it would take 2 seconds to run eth2 state transitions for 10 validators, 20 seconds for 20 validators, and something like 20 years for 5000 validators. So, in our work we uncovered ways to improve on these issues that in turn made their way into the spec.

We have always shared our findings with our peers. There’s no optimization that we “kept for ourselves,” so to speak. In fact, frequently we’ll comment our code in a way that makes it easier for other clients to learn from, or write test sectors for other clients to implement.

What does success look like for Sigma Prime?

Oh, this is a good question. In fact, we had to define this for one of the grant questions. When first faced with this question, I remember thinking “If a hundred people will run our client we'll feel like we did a good job.” Suffice to say, we’ve done more than a good job by that standard.

Looking ahead, success for us will be doing our part in keeping Phase 0 stable and in getting Phase 1 out the door. We’re looking to add more monitoring capabilities on Lighthouse to understand what's happening even better, and when things aren't happening when they really should be. We’ll also be working to maximize validator effectiveness and profitability, while maintaining the high level of robustness and security that we have so far.

At the same time, we’re starting to work on how Phase 1 is going to operate, and exploring the unsolved engineering challenges around it. That's one part of success. And then the other part is figuring out how to deploy the Phase 1 logic onto Lighthouse without disrupting the smooth operation of its Phase 0 parts. If we do this successfully, I’ll be very happy!

I think maintaining Lighthouse as one of the leading clients and carving out our core capabilities in developing secure, stable, and performant software continues to be our main challenge, and what we’ll continue to be optimizing towards—always in a way that’s benevolent to the protocol.

Ultimately, we want to continue being able to maintain Lighthouse at a high standard, maintain a solid base of employment for the people that work at Sigma Prime, keep providing them with a great place to work at, and then continuing to output a product that is really useful for people. To me personally, even if it turns out that, say, two years from now, for whatever reason, Lighthouse fails because we did a terrible job at it, but eth2 has taken off, I will still feel satisfied. Even if the market has completely switched to using another client, knowing that we have made significant contributions to the development effort of eth2 and getting it to succeed is really what I feel success in this endeavor is all about.

What is your favorite detail or anecdote about another client that is not Lighthouse?

I have a ton of respect for all of the work that Nimbus does. The way that Nimbus has had to work on their compiler and the Nim language as they go, and the way they have come to things in a different manner than everyone else, is something I admire. Sometimes it has gone their way, perhaps others it hasn’t, but they've always thought about things differently. And by that, they tend to come to the table with ideas that expand the breadth of the hive-mind, which is really interesting and a big value add for the collective effort.

Interview by Elias Simos

About Bison Trails

Pioneering Blockchain Infrastructure®

Bison Trails is a blockchain infrastructure platform-as-a-service (PaaS) company based in New York City. We built a platform for anyone who wants to participate in 22 new chains effortlessly.

We also make it easy for anyone building Web 3.0 applications to connect to blockchain data from 30 protocols with Query & Transact (QT). Our goal is for the entire blockchain ecosystem to flourish by providing robust infrastructure for the pioneers of tomorrow.

In January, 2021, we announced Bison Trails joined Coinbase to accelerate our mission to provide easy-to-use blockchain infrastructure, now as a standalone product line. The Bison Trails platform will continue to support our customers. With Coinbase’s backing, we will enhance our infrastructure platform and make it even easier to participate in decentralized networks and build applications that connect to blockchain data.

THIS DOCUMENT IS FOR INFORMATIONAL PURPOSES ONLY. PLEASE DO NOT CONSTRUE ANY SUCH INFORMATION OR OTHER MATERIAL CONTAINED IN THIS DOCUMENT AS LEGAL, TAX, INVESTMENT, FINANCIAL, OR OTHER ADVICE. THIS DOCUMENT AND THE INFORMATION CONTAINED HEREIN IS NOT A RECOMMENDATION OR ENDORSEMENT OF ANY DIGITAL ASSET, PROTOCOL, NETWORK OR PROJECT. HOWEVER, BISON TRAILS (INCLUDING ITS AFFILIATES AND/OR EMPLOYEES) MAY HAVE, OR MAY IN THE FUTURE HAVE, A SIGNIFICANT FINANCIAL INTEREST IN, AND MAY RECEIVE COMPENSATION FOR SERVICES RELATED TO, ONE OR MORE OF THE DIGITAL ASSETS, PROTOCOLS, NETWORKS, ENTITIES, PROJECTS AND/OR VENTURES DISCUSSED HEREIN.

THE RISK OF LOSS IN CRYPTOCURRENCY, INCLUDING STAKING, CAN BE SUBSTANTIAL AND NOTHING HEREIN IS INTENDED TO BE A GUARANTEE AGAINST THE POSSIBILITY OF LOSS. THIS DOCUMENT AND THE CONTENT CONTAINED HEREIN ARE BASED ON INFORMATION WHICH IS BELIEVED TO BE RELIABLE AND HAS BEEN OBTAINED FROM SOURCES BELIEVED TO BE RELIABLE BUT BISON TRAILS MAKES NO REPRESENTATION OR WARRANTY, EXPRESS OR IMPLIED, AS TO THE FAIRNESS, ACCURACY, ADEQUACY, REASONABLENESS OR COMPLETENESS OF SUCH INFORMATION.

ANY USE OF BISON TRAILS’ SERVICES MAY BE CONTINGENT ON COMPLETION OF BISON TRAILS’ ONBOARDING PROCESS, INCLUDING ENTRANCE INTO APPLICABLE LEGAL DOCUMENTATION AND WILL BE, AT ALL TIMES, SUBJECT TO AND GOVERNED BY BISON TRAILS’ POLICIES, INCLUDING WITHOUT LIMITATION, ITS TERMS OF SERVICE AND PRIVACY POLICY, AS MAY BE AMENDED FROM TIME TO TIME.