Welcome to Who’s who in eth2, presented by Elias Simos, Protocol Specialist at Bison Trails. In this series, Elias interviews key contributors at eth2’s four client teams—Prysmatic Labs, Sigma Prime, Nimbus, and Teku—to explore their involvement in eth2, visions for the future, and insider perspectives on what eth2 means for the world.
In this fourth post, Elias interviews Jacek Sieka, Chief of Research at Status, on how the Nimbus team is squeezing an elephant (Ethereum), into a little hole (mobile devices), what elements are at play when using Nim as programming language, how the open environment of eth2 development is different than a traditional work setting, and why he’s excited about solving for scalability.
Sure! I come from a finance background. I worked in the high frequency trading world for a number of years in the 2010’s. Before that I was involved in the P2P file-sharing open-source community, which bore a lot of similarities in terms of ethos and ways of working to the current paradigm of open-source blockchain development.
While I was working in high-frequency trading I also wrote a compiler for this language called Nim, and then one day, Zahary Karadjov, who is one of the original designers of the Nim language and now leading the Nimbus client, called me up and said, “Hey, I’m working on this project building P2P stuff, compilers and so on. Why don’t you join us?” And so I did. I currently lead up research at Status–the developer of the Nimbus eth2 client.
Status launched one of these wildly successful ICOs back in 2017. The main idea behind it was to expose Ethereum to a much larger audience by abstracting away the complexity of the available interfaces, and bringing it to mobile. That was the starting point for us, and has been the benchmark of our mission ever since.
Status was built with GETH at its core—a wonderful Ethereum client, and what I consider to be a true guardian of stability for the ecosystem. At the same time, though, GETH is pretty hard to adapt to your own specific use case. GETH normally sits on a big server and here we came wanting to use it on a phone; those two worlds don't quite mesh all the way.
So we started hacking away on Nimbus as an eth1 client to solve that GETH issue, trying to bridge that gap, but, over time, we gravitated more towards eth2 because that’s where we felt we could drive the most impact. We did not, however, abandon the eth1 effort. Instead we’ve continued building both clients with our north star being that of lightness and embeddability in smaller devices. At the same time, in the early Status days, we developed a great collaboration with the Ethereum Foundation. Originally we were contributing to GETH as heavy users of the client, both in research and in code, and slowly but surely this collaboration morphed into a more formal commitment for us to develop an eth2 client.
It's about 15 people I'd say, give or take. It depends how you look at it. There's a bunch of people on the team building stuff that is auxiliary from the perspective of an eth2 client.
Come to think of it, most of the team at Nimbus came from really interesting backgrounds. The vast majority have applied themselves in really hard computer science and software engineering problems in the past. That might be computational stuff, security, cryptography compilers, game development—which obviously is a very performance intensive field—and so on. It's a pretty cool team to be working with!
I think we've vacuumed up quite a lot of competent Nim developers, but it's a small language maintained by a small community. Honestly though, the language itself is not that important; most of us have worked in other languages as well. Once you know a few languages picking up the next one is much easier.
The people that we’ve hired and we’re looking for are really strong, formal language “all rounders.” They usually excel in languages like C, C++, maybe some Rust, maybe some Python, maybe some more esoteric languages like Haskell. But, at the core, they all really like coding and applying themselves to difficult computer science and software engineering problems.
It's a special ilk of developers that learn these exotic languages, and they are definitely harder to find. There are already few Nim developers around, and those with Nim and blockchain experience are even fewer.
I’d say that 50% of what language you end up choosing is guided by utility, and the other 50% by religion, or call it taste. At the end of the day languages are tools. Some do the job a little bit better, some a little worse.
For us in particular it's important to be as efficient as possible, because we're trying to squeeze an elephant (Ethereum), into a little hole (mobile devices). That automatically excludes a bunch of languages. For example, it’s very hard to write memory-efficient Java. For similar reasons, you might not choose Python to develop a client. I think the PyEthereum (Trinity) team will attest to that. So that really narrows down your choices. You want something that’s fast, memory efficient, and we were interested in having it be fairly expressive as well.
I expect the new Raspberry Pi 4 (4GB RAM option, external SSD) to handle an Eth2 validator node without breaking a sweat. That's $100 of hardware running at 10 Watts to support a 32 ETH node (currently ~$10K stake).— Justin Ðrake (@drakefjustin) June 24, 2019
Nim provides this script-like experience for when you want to experiment with things, but at the same time you get very good performance out of it. And then when you need to reach for more advanced features, or, on the other end of the spectrum, when you need to go really low-level–almost hardware-level code—it’s all available to you with Nim.
Another determining factor was licensing. We wanted to build with components that were free of licencing restrictions so that we were free to experiment across all dimensions. Say that we need to deploy something special to save some bandwidth, or move some computations from one device to another. With Nim it becomes much easier to create a “clean room” implementation of that. And once you start, it’s really hard to stop!
The people that have worked on this project (Nimbus) hold this amazing amount of knowledge on Ethereum, simply because they've worked through every single layer of it. And from a security point of view, nothing un-audited makes it into the Nimbus codebase. And when I say unaudited, I mean we literally have written most of the code from scratch.
On the flipside, however, this adds to the workload—as you can imagine.
It's a bit of both, actually. First of all Nim is a small community, so naturally the base for some of the features that we needed was not available in existing libraries. However, for other features that are really core to how Nimbus works, we actively decided to write them from scratch even if there was material available to work with out there. This was the only way we could achieve the low footprint we were aiming for from the start. That is to say, that even if the library exists, it might be optimized for something completely different than our objective function.
Oftentimes, when you use a third party library you probably find that it has a thousand dependencies, and then regularly you hear about some upgrade with some modules sneaking through that have been adversarially hijacked. And then it becomes really difficult to audit that much code.
So one thing that I think is important to remember is that just because it's optimized to be minimally resource intensive, it doesn't mean that you can't run it on a server. It means that when you do run it on a server, it is consuming a lot less resources.
For example, where you might need a large instance with 8GB of RAM for a client, if you can get away with 2GB that's better because that leaves the other 6GB free to work on something else. So, from an ROI standpoint, the lower your hardware costs, the better.
Indeed! Think about it this way: what we do typically is use a single core from a processor for Nimbus to run on. Now, if you have a two core or a four core machine, you can run your GETH instance, for example, on the other cores and still maintain the same amount of performance in terms of end-user functionality, which in this case is validating on eth2. So you can isolate Nimbus without risking that the other services affect your eth2 performance.
The other cool thing that's happening is that at Status we also develop these private communications protocols. In the Status app we have the messenger, and we're just now starting to integrate Nimbus into both the app and the desktop client as well. So you can imagine that a lot of the things that we've developed for the eth2 client could wind up in other products as well.
For example, there's this nice secure communication protocol we developed called WAKU, which in part is based on the same libp2p infrastructure that Nimbus uses. In the future we'll be putting all those things together—the private communications protocol, the validator, the mobile app—which will give us this really homogeneous experience.
I view the early 2000’s era as the “burst” of the internet into the mainstream, and that brought about these international open-source software collaborations. Back then we were all working towards a shared goal. The sense of purpose was palpable. And the Ethereum development world is very similar today.
Well, the thing is that when you take it upon yourself to develop all the parts from the ground up, you end up writing a lot of code, and that takes time. For example, you might end up having to write a HTTP server, or JSON parser, or whatever. These things take time. I think we lost quite a bit of time doing things like this that weren’t directly related to eth2, but that we had to build to make sure that the client was ready to be deployed in the adversarial and hostile environment that is a large distributed network with a ton of monetary value on it.
Nevertheless I'm pretty happy with the results, because now we know everything it takes to write an eth2 client, but it’s been a long journey for sure.
A little bit more than half the team has been there since early 2018 and the Ethereum sharding workshop in Taipei. We've added people here and there, but given how we operate and the profiles of people we are after, recruiting has been a bit of a challenge. It's a special ilk of developers that learn these exotic languages, and they are definitely harder to find. There are already few Nim developers around, and those with Nim and blockchain experience are even fewer.
I think what was super interesting is that it was all done remotely. Imagine all of the Status team is remote, there are no offices, and so on. Now, when you add the eth2 development community into that, it adds a whole other layer of “remoteness.” But, contrary to what one might expect initially, it ended up working great.
To be honest, I’m not sure if this was down to the people involved or the way I am myself cut out and perceive things. Probably a bit of both. Way back when, in the early 2000’s, I was pretty heavily involved in the open-source file sharing community. Coming from Sweden and being interested in computers, it was kind of hard to avoid back then with Pirate Bay raising their flag and all these battles over copyright.
It was a really cool time back then, because that's really when these core peer-to-peer technologies were developed, Merkle Trees became popular, etc. A lot of the underlying building blocks for blockchains were sort of conceived back then.
And then it went dormant for a little bit. There wasn't that much development, and then all of the sudden blockchains and these incentive systems appeared. It was kind of a rebirth of all of these technologies that were previously being used in a particular way and could now be reassembled into something new.
I view the early 2000’s era as the “burst” of the internet into the mainstream, and that brought about these international open-source software collaborations. Back then we were all working towards a shared goal. The sense of purpose was palpable. And the Ethereum development world is very similar today. Even if people come from different backgrounds and with different dispositions, there is a certain shared ethos that glues it all together.
I think the most interesting part about collaborating like this is that all of the coercive tools (promotions, demotions, politics, etc.) that you have available when you're working in a more traditional set up—like when you're the boss, or when you lead a normal team—are out the window.
In this open environment, you can’t really force anybody to do anything. That's uniquely interesting and challenging at the same time. It's interesting because it really brings the people that you want to be working with together at the same table. But it's harder, because you can no longer point the finger and tell somebody “you do this.” And everybody knows that, and so you have to make room for different opinions and try to accommodate solutions, inviting people to the table rather than forcing them to sit there.
I wouldn't say it's the only community where I’ve felt that. Like I mentioned, the file sharing communities of the early 2000’s were very similar in that respect, at least the ones I participated in. I think a lot of the core internet infrastructure was developed this way too, as collaborations between different universities and enthusiasts.
There are actually so many projects out there that are really interesting and vibrant from that perspective. Take Blender, for example. There's a huge open-source community around it. Obviously they don't have this economic drive like you have in Ethereum, but even without it, the community thrives.
I think there's lots of thriving, open-source, collaborative communities out there. And then which one you pick is maybe a function of your background and your interests.
I think what's cool about Nimbus is that we also have an eth1 client in the works. And what that means is that we'll be able to have this unified experience/product that offers both eth1 and eth2.
Solving for scalability! I know that this has become a word that everybody just repeats, but there’s a lot of things that we want to do with Ethereum and just can’t in its current form. It’s great to bring Ethereum to the people, but when people have to pay $50 to send a transaction through it’s just bad user experience.
It's certainly exciting to see Phase 0 up and running, but, on the other hand, Phase 0 doesn’t do anything practical. I am looking forward to getting the next phases—in whatever order they come—up and running so we can see more practical stuff being done on Ethereum. And then the other challenge, of course, is to fold eth1 into eth2. That’s going to be a really interesting piece of work, both to deploy it and then seeing people use it thereafter.
I think what's cool about Nimbus is that we also have an eth1 client in the works. And what that means is that we'll be able to have this unified experience/product that offers both eth1 and eth2. I think that’s when the work that we did in building everything from the ground up will really shine. It will be super easy to use.
To give you an example, we have this mode in Nimbus where you run the validator client and the beacon node in a single process and there's good reasons to do that.
The major one is simplicity. The simpler it is, the harder it is for it to break. So far on the eth2 mainnet we’ve seen about 20 or so slashings, most of which took place because people were running two validator instances at the same time. It wasn’t really malicious attacks on the network, or anything of the type. It was a simple user error, and we’re talking about very competent people that operate those validators.
I think simplicity in the set-up and interfaces solves a lot of this. You know, you put your key in there, and then you launch a single binary, and it just works. That's our goal. It shouldn't be more complicated than that.
I say up to a point. There are diminishing returns to it all; like two is better than one, three is better than two, four is better than three, but how far can you really extend that logic? If we had a hundred clients, would that have a noticeable impact? Would that be the best use of resources? Probably not.
Of course we should have some diversity. Especially for a system that holds as much value as eth2 does; it would be ludicrous not to have it. But it comes with costs, like all systems built with redundancy do.
Looking back at my HFT days, for example, all the banks and exchanges and so on have redundant systems for everything really—from software to hardware, to power and phone lines, to computers—everything! But there are diminishing returns at some point.
Every system is as strong as its weakest link, essentially. The reason why we have many validators voting is that it makes the network hard to capture, from that point of view. Then there’s technical risk, of course.
If there was a single client team the network would be beholden to the ends of that team. Now everything is negotiated, and, because several people have to agree on something, the decisions tend to be more sane overall.
I would say so. But that’s hard to quantify. Because you don’t know that you need it until something bad happens. Perhaps the “worst” outcome of all is that no attack happens, and everybody goes, “Oh, well, you know, maybe this wasn't needed after all.”
But it is exactly because there are many client teams implementing the same protocol that the holes for attackers to exploit are found, and mended before attackers are able to exploit them.
Good question! I don’t think I can point to something specific, though I will say that it's been super interesting to see how everyone has carved out a little bit of a niche for themselves, and how we've been able to take those learnings and integrate them across the board.
In the earliest days we collaborated a lot with Sigma Prime; they have some really fantastic engineers there from whom we learned a lot. At the same time Prysm was working a lot more with users, and we were able to learn a lot from them as well, as in what validator operators actually are asking for, etc. Then the team behind Teku brought some serious cryptography expertise to the table, and an overall maturity to the project that wouldn't have been there otherwise.
So, no matter where you look, it's been really great to be able to collaborate with all the other teams. At the end of the day I think every client has sort of grown from this coalition and diversity of skills that have been made available to the eth2 effort.
Interview by Elias Simos
Bison Trails is a blockchain infrastructure platform-as-a-service (PaaS) company based in New York City. We built a platform for anyone who wants to participate in 22 new chains effortlessly.
We also make it easy for anyone building Web 3.0 applications to connect to blockchain data from 30 protocols with Query & Transact (QT). Our goal is for the entire blockchain ecosystem to flourish by providing robust infrastructure for the pioneers of tomorrow.
In January, 2021, we announced Bison Trails joined Coinbase to accelerate our mission to provide easy-to-use blockchain infrastructure, now as a standalone product line. The Bison Trails platform will continue to support our customers. With Coinbase’s backing, we will enhance our infrastructure platform and make it even easier to participate in decentralized networks and build applications that connect to blockchain data.
Substrate ecosystem update 006Apr 10 2021
eth2 update 013Apr 9 2021
Why are people excited about NFTs?Apr 8 2021
Celo governance proposal 0024: Q&A with Protocol Specialist Viktor BuninApr 1 2021
Guide to Crypto.org ChainMar 26 2021
Who’s who in eth2: Paul Hauner from Sigma PrimeMar 18 2021
eth2 update 012Mar 18 2021
Bison Trails announces support for MinaMar 18 2021
An introduction to Protocol Specialists at Bison TrailsMar 15 2021
Meet the Herd: Engineering Manager Arthur BurkartMar 12 2021
Bison Trails newsletter 012Mar 9 2021
View more →