# Shadow Network Simulations

- Speakers: [Daniel Knopik](https://streameth.org/speakers/daniel-knopik)
- Channel: [Devcon 7 SEA](https://streameth.org/devcon_7_sea)
- Date: 2024-11-13
- Duration: 15:36
- Watch: https://streameth.org/watch/67347cd49dbb7a90e16882fa

## Description

In my EPF project, I implemented Ethshadow, a configuration generator for simulating Ethereum networks using Shadow, and used it to research improvements to the current state of PeerDAS and to estimate the effects of IDONTWANT on node bandwidth. In this presentation, I will present my findings and make a case for testing using Ethshadow.

## About the speakers

### Daniel Knopik

Fellow in the fifth cohort of the Ethereum Protocol Fellowship.


## Transcript

Hi everyone, thank you. My name is Daniel Knopik and I worked on shadow network simulations. The usual clicker problems. Thanks. Yeah. So, in my opinion, network simulations are awesome because they allow testing changes without rolling out to public deaf nets or even test nets which is lower takes synchronization and time and Small local deafness are possible but with a software called shadow We can actually run huge simulations with thousands of nodes. And also with actual clients, like not any code written dedicated to the simulations. So I kind of had a three-phase project. First, I wanted to prepare a tool that allowed me to easily set up these shadow simulations with theorem clients. That is easier said than done because shadow can be kind of finicky sometimes. set up these shadow simulations with theorem clients. That is easier said than done because shadow can be kind of finicky sometimes, so it took a while. Yeah, in the second phase, I want to run experiments specifically on PeerDust and IDon'tWant, also known as GossipSub1.2. And in the third phase, I thought maybe if I saw in the experiments that this is actually a useful tool, I wanted to polish it and actually did, and thus, EthShadow was born. So to sum it up, there are like two main artifacts from my project, the experiment results and ETH Shadow itself. In this presentation, I will kind of focus on my results from PeerDust, and afterwards, also show you slowly or quickly ETH Shadow. Right. So, PeerDust. I won't have time to explain it as fully. So I'm just going to assume there's some awareness of what that is. Um, few points. It's an update to scale blobs and the basic principle is that we want to split the blobs into parts, which we call columns with every note taking custody of some of those columns. And there are some network changes required for this to make sure that the nodes properly distribute the columns across the network, and also so-called super nodes can reconstruct the blobs if they have enough columns. And right now there's like 128 additional subnets in gossip sub which all need to be covered by when a node wants to be proposed, right? We need to get every column out so yeah. Okay so I decided to run some simulations to help the research and development effort. Keep in mind that all results that I present here are like a few months old at this point, so the situation might be better at this point. So my simulation setup involved 1,000 nodes with Reth, Lighthouse, and Lighthouse validator client on each, with 4,000 validators, and in each experiment I simulated 45 minutes of simulated time. For a large simulation like this we want to run on some server that is sufficiently large that the simulation is fast enough for us. And I kind of experimented around a lot to make sure to get like, get nice performance going and kind of settled on a certain instance type you can see it there if you want to run your own simulations with the size and my simulations took a round of four hours per like 45 minutes of simulated time and so the cost of simulation came around at $10. Right. But if you want, like, also run smaller simulations, you can also do this on your laptop. Like, you don't need such a beefy server. This is just, like, my setup for the peer-to-peer simulations. So first I just tried running it on the current implementation in Lighthouse, and the simulated networks quickly fell apart because they lost sync and cannot really sync back up again. And that was kind of similar what was seen on DevNet at the time across basically all clients. So, I thought about how do I measure performance, and I had several metrics, but right here on DevNet at the time across basically all clients. So I thought about how do I measure performance and I had several metrics but right here I will focus on what I call score which is how many slots can over 66% of the network stay in sync. And with the unmodified client with the default configuration, zero. Like immediately after we posted blobs to the network, we lost sync. But I also noticed that if I designate every node as a super node, so custodying all the columns, not just eight or four, then the network stayed stable until the end. So I had the suspicion that something is wrong with the distribution of columns when we just don't broadcast all the columns at all times to everyone. And also that might be something with supernode reconstruction of the blobs might be wrong. Yeah. So, and yeah, the investigation showed that the problem is that if a node could not send all the data out, it would not retry that at all, causing some columns to get lost and custodying nodes for those columns would just not deem the data is available and would just accept the block and the sync and they could not catch up I guess the super nodes were not enough for that. So how do we fix this? I did some simulations where I varied the specs and I did a lot of different variations. I will focus on three here. We could increase the number of peers because if you have more peers, the likelihood that we actually cover all the columns when proposing a block with our peers is higher. So yeah, the base spec or the base configuration in lighthouse looks for 100 peers 150 peers scored 10 with the metric I just mentioned 200 Already 56 and 300 survived the whole simulation so we could see that confirming our theory this improves the situation Next up I increase the number of columns covered per each peer, which should have the same effect, right, because the total number of covered columns also increases with changing this. Increasing from 4 to 8, score 10, and 16 already survived the whole simulation. Finally, there should be an emoji here. The, you know, the diagonal face. All right. Increasing the number of supernodes, which should reconstruct the full blobs, like all columns, if they have half or more. Increasing to 10, scored 0 to 25 of 1,000, right? Scored 2, and 75 also 2. increasing to 10 scored 0 to 25 of thousand right let's go to and 75 also 2 and at that point I didn't test any further because I Want to kind of keep it in? Somewhat realistic scape and I'm not sure how many super notes there will be but we shouldn't in my opinion set the assumption too high all right but we shouldn't, in my opinion, set the assumption too high. All right. If you want to know more about that, there are my weekly reports where I go into a bit more detail. So let's move on to the state of each shadow, like the tool I developed. Because seeing these simulations, I thought, hey, this is kind of useful. I have an iterative approach and can evaluate different configurations. So I spent some weeks in the end to make it available for everyone. And the source is available on GitHub at Ethereum slash ethshadow. Right now there's like first class support for Geth and Lighthouse and there's some experimental support for Reth and Teku. There is some documentation that for Reth and Tiku. There is some documentation that has to be written. For basic use cases, there's good documentation already, so you can check it out. And I think in some cases, usability can be improved. The hard part is that Shadow needs a lot of work for all clients to work in shadow. The reason for that is the technical. Basically we need support from our Linux system calls in shadow. Um, and this is a lot of effort. Unfortunately I, um, kind of won't have much time. So I hope that in the coming Weeks a month I can at least help on the side of it to maintain this But I hope to also get the attention from from the core deaths to help me or help us all Hopefully soon have its shadow for all the clients Okay, there are some things to be said first of all, thanks to my mentorsria Manning from Sigma Prime and Pup from Ethereum Foundation Research and also thanks for to João Oliveira and Jimmy Chen from Sigma Prime who supported me with some of their time and thanks to Anton Lachatiereff from ConsenSys, he spearheaded the technical support in the last couple of weeks. Thanks to EPF organization, to Josh and Mario, thanks to the Ethereum Foundation for hosting the EPF, and thank you to all the fellows, it was really pleasant and great to work with you all. And finally, I'm going to plug my talk tomorrow, or our talk tomorrow, simulating an Ethereum network scale, which will go more into detail on how you can run the simulation yourself. So maybe see some of you there on stage one at 10 past 1 p.m. Thank you for your attention. All right, any questions for Daniel? Hi, just a couple questions. One is when you're making all of these changes to do these simulations, are you actually like updating the client code or is it kind of like network configurations? Yeah, this is like the great advantage to change these or to test these changes, I actually can change just the client. I don't need to develop anything just for that. I changed the actual client, but for the things I showed you, I think I only needed to vary the configuration a bit. But I also did some changes that actually tried to fix these underlying problems in Lighthouse itself. Cool. And then, I guess my other question, so are you starting simulations from, like, a genesis state, or is it, like, a fork of mainnet, and can you do forks of, like, existing networks? Well, I start from a genesis state. My tool supports this. It just does this for us, which is nice. And forking mainnet, I mean, I'm aware that there are shadow forks, they are unrelated to the name of the simulation tool. I haven't tried it. The problem is that the genesis state is quite large, and this might actually be really hard to simulate, like, on a single node. We have all the simulation running on a single server, and that might be hard to have multiple clients in parallel working on that. So I guess it's not really feasible. So I was wondering, once the simulation completes, can we see, like, consolidated metrics on the node or something, like logs, something like that? Very good question. This is how I evaluated those simulations. I kind of forgot to mention that here. The simulation, it is really easy to add a Prometheus node in the simulation configuration, and that automatically gets configured to pull the metrics from all the Lighthouse clients. In the end, we have one huge Prometheus database, which allows us to just pull up some Grafana dashboards or any other analysis. Also, we have all the logs, so we can also look into those if there are any specific nodes that seem to be acting strange or something. Yeah. In fact, you can see every lock of every node. You can see the STDR, but you can see the STD of that other. And in fact, you can see every that the node produced. Right. You can also run a node on a data directory that has been generated by the simulation afterwards. It's a bit weird because the simulation time always takes place in the year 2000, so it's really old data to the node if you start it right now. But you can run that node, and you can attach a block explorer or the explorer, for example, by the panda ops team to it. It's just a bit wonky because of the timing issue. Do you have to like convert the matrix or something like that? So like because the simulation time is less but the time it takes to simulate something is longer time. Do you have to do some conversion to get the matrix or something like that? Yeah, basically the simulation, Shadow starts the simulation time always at the 1st of January in the year 2000. So every time you do a Prometheus query or look into the block explorer, you have to keep in mind that you have to act as if it were the year 2000. Yeah. Any final questions? There are some up there. Oh, we got one more. How is it different from kurtosis? Okay. In a nutshell, kurtosis, you cannot run networks of this scale on a single server with kurtosis because kurtosis is not capable of like pretending to the processes that time is running faster or slower than it actually is basically like having a separated simulation time from real time yeah i would say that is the main difference, that you can scale way higher with Shadow. All right, thanks, Daniel.