Shadow Network Simulations

[Music] hi everyone thank you um my name is D kopic and I worked on Shadow nworting relations yeah um the usual clickup problems thanks um yeah so in my opinion Network donations are awesome because they allow testing changes without rolling out to public death Nets or even test Nets which is slow and takes synchronization and time and um small local definites are possible but with um a software called shadow we can actually run huge simulations with thousands of notes and also with actual clients like not any um like code written dedicated to the um simulations so I kind of had a three-phase project first I wanted to prepare a tool that allowed me to easily set up these Shadow simulations with etherum clients um that is easier said than done because Shadow can be kind of finicky sometimes so it it took a while um yeah in the second phase um I want to run experiments uh specifically on pias and I don't want also known as gossip sub 1.2 and in the third phase I thought maybe if I saw in the experiments that this is actually a useful tool um I wanted to polish it and actually did uh and S thus uh e Shadow was born um so to sum it up are like two main artifacts from my project um the experiment results and E Shadow itself in this presentation I will kind of focus on on my results from pias and afterwards uh also show you slowly um or quickly um e Shadow right so pias um I won't have time to explain pias fully so I'm just going to assume there's uh some awareness of what that is um few points it's a update to scale blobs and the basic principle is that we want to split the blobs into Parts which we call columns with every note taking custody of some of those columns and there are some Network CH changes required for this to make sure that the nodes um properly um distribute uh The Columns across the network and also so-called supernes can reconstruct the blobs if they have enough columns and right now there's like 128 additional subnets um in gosip sub uh which all need to be uh covered by when a note wants to be proposed right we need to get every column out so yeah okay so I decided to run some simulations to help the research and development effort um keep in mind that all results that I present here are like a few months old at this point so the situation might be uh better at this point so my simulation setup involved 1,000 notes with r Lighthouse and Lighthouse Val on each uh with 4,000 validators and in each experiment I simulated 45 minutes of simulated time um for a large simulation like this we want to run on some server that is sufficiently large though the simulation is fast enough for us and I kind of experimented around a lot to make sure to get like get nice performance going and kind of settled on a certain instance type um you can see it there if you want to run your own simulations with this size and my simulations took a round of 4 hours per like 45 minutes of simulated time and so the the cost of per simulation came around at uh $10 right um but if you want like also run smaller simulations we you can also do this on your laptop like you don't need such a PV server this is just like my setup for the P simulations um so first I just tried running it on the current implementation in Lighthouse and the simulated networks quickly fell apart because they lost sync and cannot really sync back up again um and that was kind of similar what was seen on Def Nets at the time uh across basically all clients um so I thought about how do I measure performance and I had several metrics but right here I will focus on what I call score uh which is how many slots can um over 66% of the network uh stay in sync and with the unmodified uh client with the default configuration zero like immediately after we posted blobs to the Network um we lost sync um but I also noticed that if I designate every node as super no so coding all the columns not just uh eight uh or four um then um the network stayed stable until the end so I had like the suspicion that something is wrong with um the distribution of uh columns when we just you know don't broadcast all the columns at all time to everyone and also that might be something with super node reconstruction of the blobs might be wrong yeah um so and yeah the investigation showed that the problem is that if a node could not send all the data out it would not retry that at all causing some columns to get lost and custody nodes uh for those columns would just not deem the datas available and would just not accept the block and lose sync and they could not catch up I guess the supernes were not uh not enough for that so how do we fix this I did some simulations where I varied the specs and um I did a lot of different um variations I will focus on three here uh we could increase the number of peers because if we have more Pi the likelihood that we actually cover all the columns uh when proposing a block um with our peers uh is higher so yeah the base back or the base configuration in Lighthouse uh looks for 100 Pi 150 Pierce scored 10 with the metric I just mentioned 200 uh already uh 56 and 300 Sur the whole simulation so we could see that confirming our Theory this improves the situation next up I increased the number of columns covered per each Pier which should have the same effect right because the total number of cover columns also increases with uh changing this um increasing from 4 to 8 score 10 and 16 already so we have the whole simulation finally uh there should be Anem Emoji here the the you know the the diagonal face um right uh increasing the number of supernotes which should reconstruct the um full blobs like all columns if they have half or more um increasing to 10 scored zero to 25 of thousand uh right scored two and 75 also too and at that point I didn't test any further because I want to kind of keep it in a somewhat realistic scape and I'm not sure how many supernes there will be but we shouldn't in my opinion set the Assumption too high all right um if you want to know more about that um there are my weekly reports where I got into a bit more detail so let's uh move on to the state of each Shadow like the tool I developed um because seeing these simulations um I thought this is hey it's kind of useful I have like an iterative approach and can evaluate different configurations so I yeah spent some weeks in the end to make it available for everyone and the source is available on GitHub at ethereum Shadow uh right now there's like first class support for GU and lighthous house and there is some experimental uh support for re and tiu um there is some documentation left to be written um for basic use cases there's like good uh documentation already so you can check it out um and I think in some cases usability can be improved um but the hard part is that shadow needs a lot of work for all clients to work in Shadow the reason for that is are technical basically we need support for more Linux system calls in shadow um and this is a lot of effort unfortunately I um kind of won't have much uh time so I hope that in the coming uh weeks and month I can at least help on the side a bit to maintain this but I hope to also get the attention from from the C devs to help me uh or help us all um hopefully soon have each for all the clients okay uh there are some thanks to be said uh first of all thanks to my mentors Aran Manning from Sigma Prime and pup from etherum Foundation research and also thanks for to J oliviera and Jimmy Chen from Sigma Prime who supported me with some of their time and thanks to Anon nurv from consensus uh he uh spare headed the tech support in a couple last couple of weeks uh thanks to EPF organization uh to Josh and uh Mario thanks to etherum Foundation for hosting the EPF and uh thank you for to all the fellows it was really Pleasant uh and great to work with you all and finally I'm going to pluck my talk tomorrow or our talk tomorrow um simulating an ethereum Network scale which will go more into detail on how you can run the simulation yourself so maybe see some of you there on stage one at 10 1 p.m. thank you for your attention all right any questions for Daniel um hi just a couple questions one is um when you're making all of these changes to do these simulations are you actually like updating the client code or is it kind of like Network configurations um yeah this is like the great advantage to change these or to test these changes I actually can change just the client I don't need to develop anything just for that I changed the actual client but for like the things I showed you I think I only needed to varry the configuration a bit right but I also did some changes that actually tried to fix these underlying problems uh in L itself cool yeah and then I guess my other question um so are you starting simulations from like a Genesis state or is it like a fork of main net and can you do Forks of like existing networks well um I I start from a j St my tool supports this it it just does this for us uh which is nice and forking main net I mean I'm aware that they are Shadow Forks they are unrelated to the name of uh the simulation tool um I haven't tried it uh the problem is that the Genesis state is quite large and this might actually be really hard to simulate like on a single uh node right we have all the simulation running on a single server and that might be hard to have multiple clients in parallel uh working on that so I guess it's not really feasible so I was wondering like once the simulation completes can we see like consolidated metrics on the Node or something logs something like that very good question uh this is how I evaluated uh those simulations um I kind of forgot to mention that here um this simulation um you it is really really easy to add a Prometheus note in the simulation configuration and that automatically gets configured to pull uh the metrics from all the lighthouse clients and in the end we have like one huge Prometheus database which allows us to just pull up some grafana uh dashboards or any other analysis also we have all the locks so we can also look into those if there are any like specific noes that seem to be acting strange or something yeah right um You can also like run a note on a data directory that has been generated by the simulation afterwards it's a bit weird because uh the simulation time always takes place in the year 2000 so it's really old data to the note if you started right now but you can run that Noe and you can attach like a block Explorer or the Explorer for example by the e p Ops Team to it it's just a bit wonky because of the timing issue yeah basically the simulation Shadow starts the simulation time always at the 1 of January in the year 2000 so every time you do a Prometheus query or um yeah look into the block explore you have to keep in mind that you have to act as if it were the year 2000 yeah any final questions there are some up there oh we got one more uh how is it different from curtosis okay in a nutshell uh kosis um you you cannot run networks at of this scale on a single server with kosis because kosis is not capable of like pretending to the processes that time is running faster or slower than it actually is basically like um having a separated simulation time from real time um yeah I would say that is the main difference that you can scale way higher with uh shadow all right thanks Daniel [Applause]