I wanna to be your minority: Geth as a minority Client

Yeah, so a couple of years back I was experimenting with bringing Geth on the Gnosis Chain and this is a summary of this experience and also I wanted to share some learning from that. Yeah, so differences between the Gnosis Chain and Mainnet, there's a 5 second block time which is inherited, so the Gnosis Chain used to be a chain called XDAI which has a token that is a stablecoin and yeah, the block time was 5 seconds and the idea behind this whole thing, I think, that was even before I was part of Ethereum, I think, was to get some traffic, to offload some traffic into some kind of side chain, do the transactions without going to, you know, without competing for the main pool on Mainnet, that was the time of CryptoKitty and yeah, they went for a shorter block time and this was, this has been inherited since and so we find ourselves with this block time. It's quite interesting for me because as Tony was saying just before, we want to reduce the slot time, so 5 seconds, 6 seconds, it's not too different, so that's quite an interesting experiment as far as I'm concerned. When you look, and I will go into this later, but when you look at the size of the state, you find it to be a lot smaller, which makes sense, it's a side chain that is meant to offload some of the traffic, so it's not going to have the exact same size. What's the consequence of the block time, however, is that the history is, in terms of block, is about twice as large as what it is on Mainnet.

It was using this consensus called Aura, I'm not going to get too much, too deep into this, but it came from Parity and it's effectively a proof of authority consensus algorithm, but they tried to make it a bit more decentralized by using some kind of voting DAO thing, so yeah, that's the status, eventually they got rid of it. One of the interesting parts, or maybe infuriating, depending on how you look at it, it relies a lot on system contracts, so the consensus will call a contract, an on-chain contract, so you can tweak the way the consensus is working this way. In terms of history, the forks were a bit different, some people just decided not to use some EIPs during the forks, so they were activated later because they wanted to look more like Mainnet, but the original designers seemed to have decided there were some EIPs they didn't want on their chain, so it's a bit of a patchwork of history, and the work has been done by the Gnosis people to try to bring the chain closer to what Mainnet is, but yeah, historically, it's not quite true. You've got the concept of service transactions, which are transactions, for example, for the bridge that brings some tokens from Mainnet into the Gnosis chain, so you need to be able to mint and you don't pay for this, so that's another interesting difference. And then the other difference is that, for example, the other major difference is that the EL uses a stablecoin, which is XDAI, and the CL uses the GNO token.

And then, yeah, I added a rant about incompatibility of the block format, but that's a detail. So if you dump the two states, like the Gnosis state and the Mainnet state, you see some interesting information, like, for example, if you look at the code, there's about 10% as much code on the Gnosis chain as the Mainnet, but if you look at the accounts, there's about 40 times more accounts on Mainnet as there is on Gnosis, that makes sense, right? Once again, it was meant to offload some traffic from the Mainnet, so not move everything. But the interesting part is that the storage nodes, however, are about half of Mainnet. That means there's a lot more content per account on the Gnosis chain as there is on Mainnet.

And honestly, I don't have a definitive answer of why that is. It could be that it's mostly designed to do ERC trading or other things, so that would explain why there's a lot more contracts. But it seems that there are actually relatively fewer simple, able-to-die accounts on the Gnosis chain. And then you have the account and storage snapshot, which is pretty much the same information. The other interesting part is that the bodies, the total size of bodies is about one-third of what it is on Mainnet, but there are way more blocks, and that makes sense because the block time is much smaller, so you have less time to execute.

If I'm not mistaken, the gas limit, the target gas limit currently on Gnosis chain is 17 million, so about half of what we want on Mainnet. And yeah, so you can see there are more headers, but the blocks themselves are smaller, and yeah, that makes sense. The last interesting thing is the receipt. The size of the receipt on Gnosis chain and Mainnet is about the same. And I'm actually quite curious as to find out why that is.

I don't know for sure. There's definitely an explanation that I was talking about system contracts before. Those system contracts emit logs. I assume it's related, but it would be a further investigation. It would be interesting to figure out why.

Yeah, I don't have a lot of time, so I'm going to skip this one. But okay, I'm just going to summarize it very quickly. Let's say when Gnosis recovered that chain from, I think they were called PoE Networks, it was pretty much, you know, there was a lot of legacy going on. They still have parity. Nevermind was just started, but they didn't have SnapSync.

They were not serving SnapSync. They were using a protocol that was about to get discontinued in Geth. And, like I said, the Gnosis team has tried to bring the chain, like to modernize the chain, to bring it closer. So they replicated parity. They removed Aura by doing a merge.

And yeah, like, nevermind implemented SnapSync. So things started to look a lot more like Mainnet. Yeah, so the first attempt was to try to create something, the first attempt at bringing Geth on this chain was trying to get something like history expiry. And the idea is that you would have nevermind following the chain and nevermind had implemented Aura. I didn't want to.

And so the idea is that we'd get some kind of bootstrap version of Geth that would just import a dumped state from nevermind and try to pretend that this was the proper sync. And so the merge would be skipped, the whole Aura history would be skipped. And then this instance would only exist, like, would be unique. And we would spin up other Geth nodes that would SnapSync from it. For many reasons, that didn't really work out.

One of them is, yeah, I had to do a lot of guesswork. It wasn't really clear that some forks were activated. And when they were, at least it wasn't clear to me. And yeah, I came also to the realization that a lot of the Aura stuff still remains behind. So I think at this point, it made sense to just re-implement Aura.

And Aragon had done it in the meantime. They were brave enough to read the Rust code from Parity. So good on them. And I took it and I adapted it to Geth. So as it turns out, that was the other discovery, like the code of Aragon has drifted, especially in this part, has drifted significantly from Geth.

So there was some adaptation work to do, but it was manageable in the end. And yeah, like this implementation of Aura didn't have to be good enough to run a future Aura network, if anybody wants that. It was just meant to be good enough that Geth could full sync up to the merge block and move on and ignore Aura for the end times. So full sync, when I did that full sync, it took about a month. I ran into all the peculiarities.

So this is when I went from guesswork into actually figuring out what the problem, like block per block. So yeah, like for example, the most famous example is that there's a code limit that was activated at a given fork, like 24 kilobytes. People want to change that to, I think 48 is the last number. But for GnosisChain, it was not activated at the same time. As mainnet, so they activated it in Shanghai.

So this is one of those examples. But now we have Geth on Gnosis mainnet chugging along and so far it's working fine. I'd say the biggest lesson of this whole story for me is that system contracts are a bad idea. They're all the rage right now. And I think it's a mistake.

So that's some controversial take right here. I mean, I could list all the reasons, but the biggest one is that most clients, I accept Geth at this point, do not trace system contracts. And so if you are trying to develop a client for a chain that uses, whose consensus relies heavily on that, it's really hard to figure out what's going wrong. And it's also harder to debug, like the system contracts are harder to debug because you can't put a breakpoint in the system contract. You have to put a breakpoint at the client level and somehow you have like a lot of overhead to figure out what the contract is doing.

Your behavior is pretty opaque and you probably heard of the Petra testnet fork debacle. I put, so this is actually the fixed version of this, of the code that caused the debacle to begin with. But they were expecting, like the code was expecting a, I'm looking for the word, a message to come from the contract. And because Sepolia is a peculiar chain, they deployed their own, like when I say they, I mean, ETH kind of ops, they deployed their own contract. They didn't really tell anyone how it was working.

So no one was expecting this. This is not a dunk on the ETH kind of ops at all. I'm just saying this is bound to happen because to people doing DevOps, it looks like it's just delegated to the execution layer, like the consensus part is delegated to the execution. And as long as the execution works, you should expect it to work. Yeah, but then the message didn't travel.

And therefore, like client devs had no idea that this was coming. Some, if I remember, it's Aragon who did it right. But most other clients were just expecting this pattern. They were not expecting a different pattern. And this is how Sepolia got in trouble during the fork.

So I think it's pretty clear to me, my opinion that we should have, we should have, we should not have system contracts. Everything should be deployed as actual native client code. Yeah, that's my conclusion from this at least. Now I'm going to go over the list of changes that could be interested to Forgnosis chain. There's a protocol level and there's what VF could do.

At the protocol level, I think the biggest gain Forgnosis chain would be EIP-44, namely history expiry. There's a lot of blocks. And all those blocks are almost empty, but together they take a lot of space. They take a lot of space, especially for the sink, because the full sink, you have to go over every one of them and yes, recompute the state route at every block. So that's why it takes a long time.

You could do something like, for example, Aragon and REST, they do something where you just execute the transactions and they compute the route at the end. We could do something a bit similar in GIS, maybe compute the route every 10 blocks, every 128 blocks. That would be interesting to check, to do. So that could be my next experiment on Forgnosis chain, because once again, that's where the benefit is going to be the greatest. Okay, Tony talked about delayed execution.

I also forgot to mention about EPVS. So sorry for this. But the idea is because the block time is so short, it might be interesting for offering some extra capacity to the chain to implement that on Forgnosis chain as well. So that would be my two favorite topics to implement if I was the one deciding what goes into Forgnosis chain. I mentioned the bloatnet.

This is an effort that I work on for the EF. The goal is to really create, like make a fork of any network and try to really increase the size of the state. I think that would be also interesting to probe two things. The first one is for EtherMainnet, because we are close to a five second, of a six second slot with a five second slot. We could try to use the Forgnosis chain, or at least a shadow fork of it, to figure out how the mainnet would react in case the block size, sorry, the slot time is reduced.

But it's also interesting, once again, to study what capacity is really available to a network with a five second block time. Otherwise, yeah, I think EF, it's pretty clear that EF needs to be able to enable, to have a better granularity in its capacity to enable or to turn on and off EIP. So that would be something great if GES could do this. Yeah, so not really a lot to say, except GC-GES could also bring something back into GES. Honestly, yeah, I guess I'll just skip this one.

It's not very interesting to most of you. Yeah, last slide. So what is GC-GES today? You can find it at this address. It's more or less synced with GES Master.

I need to release it, rebase it on the last release, but it's not gonna be a big problem. It's just one release behind, and it will be fixed this week. It's the interesting part, and what I named this presentation after, is that GES on this network, on Gnosis, is actually a minority client. And it's quite interesting to look at its behavior, because, for example, when you start GES, Gnosis chain GES, it takes forever to find a peer. For some reason, like NetherMind and Arrigan nodes are extremely reluctant in accepting Gnosis, sorry, GC-GES connection.

So it happens over time, but it takes some time. So I'm not sure why that is, but this is definitely the next thing that needs to be investigated, because it looks like GES, being on the main chain, being the top client in terms of usage, never had this problem, in the sense that it can very easily find a GES node. Not so on the Gnosis chain, so something to dig. Yeah, and very few people have installed it, from what I can tell. So, yeah, if you're running a Gnosis validator, please run GC-GES and tell me if there are problems.

And the last thing, you know, the Gnosis chain, I'd say the marketing was they want to be a canary chain to support Ethereum and test some things. And I think recently they delivered this, because Holiskey failed, like the PICTRA4 for Holiskey failed, the PICTRA4 for Sepolia failed. But the PICTRA4 for Shiado, which is the testnet for Gnosis chain, went without a hitch. So, yeah, and that includes GES. I mean, GES was almost ready then.

It managed to go through the fork afterwards. So I was a lot more confident about the PICTRA4 on mainnet and on Gnosis chain as well. And it turns out I shouldn't have been. There was a consensus bug that showed up in GES right after the fork, but it was thanks to Nethermind and Eragon, we fixed it and it was just a configuration issue. Yeah, that's pretty much it.

Yeah, I think we have time for questions. Okay, one here. Hi, Guillaume. Hi. I was just wondering if you have a rough idea of how many nodes are running GES on Gnosis currently, and perhaps more importantly, how many validators?

Because a lot of validators are running Nethermind, I think. Right. I don't know how many exactly. I know of one guy who's running it. So he's the one reporting.

No, that's not me. I mean, I'm also running it, but another guy who writes me when there's an issue. So that's you. Luca? Yeah, awesome.

I'll give you a hug afterwards. So as far as I know, that's you. And thanks for that. Yeah, I was going to... I talked to the Dapnode guys.

I'm going to try to have it released as an option inside Dapnode. So since the Dapnode has a huge usage base in Gnosis chain, I hope more people will be as brave as you. But yeah, so far, as far as I know, it's you. Tomas? And sorry for docking you, by the way.

Oh, yeah.