# Debugging the Ethereum Merge with Parallel Universes | Devcon Bogotá

- Channel: [Devcon](https://streameth.org/devcon)
- Date: 2023-10-07
- Duration: 25:17
- Watch: https://streameth.org/watch/yt-M3R1IvlMExI
- YouTube: https://www.youtube.com/watch?v=M3R1IvlMExI

## Description

Visit the https://archive.devcon.org/ to gain access to the entire library of Devcon talks with the ease of filtering, playlists, personalized suggestions, decentralized access on Swarm, IPFS and more.
https://archive.devcon.org/archive/watch/6/debugging-the-ethereum-merge-with-parallel-universes/

The Ethereum merge is a high-stakes event for the entire industry, and requires several large, stateful, distributed software systems to behave flawlessly in order for it to succeed. Unfortunately, it’s very difficult to develop distributed stateful systems like a blockchain with high confidence. Many of the worst bugs in these systems don’t show up in the happy case, but require particular network, hardware, or timing conditions in order to manifest.

Speaker(s): David Searle, Will Wilson
Skill level: Beginner
Track: Layer 1 Protocol
Keywords: Merge,testing,fuzzing

Follow us: https://twitter.com/efdevcon, https://twitter.com/ethereum
Learn more about devcon: https://www.devcon.org/
Learn more about ethereum: https://ethereum.org/ 

Devcon is the Ethereum conference for developers, researchers, thinkers, and makers. 
Devcon 6 was held in Bogotá, Colombia on Oct 11 - 14, 2022.
Devcon is organized and presented by the Ethereum Foundation, with the support of our sponsors. To find out more, please visit https://ethereum.foundation/

## Transcript

foreign [Music] if you're in the wrong room this room is talking about testing the merge with parallel universes my name is David Searle and I'm the head of Amir for a small startup that's been working with the ethereum foundation for just over a year now who likes Bugs okay I brought some bugs along with me today so you're going to have to be like who would like a bug because I've got some bugs that actually I'd like to give to various members of the audience anybody you'd like one okay here we go just after a little you're gonna catching so little bugs we've been spending the last year kind of really stress testing for my boat so you see this is why you should stay at the front I've got three more to go there we go see so the it talks about testing ethereum and is is basically looking at using a pretty sophisticated and unique testing methodology which is based from a company called antithesis which I belong to um the the company is involved in using deterministic simulation techniques to allow us to basically explore a whole host of iterations of of complex distributive technology that's running in a simulated environment so let me just get my my little Clicker I brought a few uh Stars along to the show um obviously I've just explained who I am um little did you know that uh doctor strange and Cher are also going to participate in this this presentation um I've got about 25 years worth of experiences in the tech industry um for the last two and a bit years now I've been working for this business antithesis and they've um you know basically made me into somebody of that there's a bug Hunter buyer by trade um I never thought testing could be fun but there's nothing like hitting a seg fault and actually getting a real kick out of it and that's what I do on a day-to-day basis this is my house in the UK you can tell from my my accent um and as you can see I've mostly got a few gpus I've got to sell so if you're interested please let me know um no joke but some the the day of the merge came round and we've been doing a lot of work with with all the various clients um all that hard arduous testing making sure that we call every last bug um what was going to happen was it really going to be smooth sailing yeah I think we saw this morning uh which obviously is great news um but um yeah I was wondering if we were going to see this you have to press play on the uh the video there is sound as well but uh um I couldn't believe this when I saw this but uh this is real yeah it's pretty bad eh um not my fun Garden again so um yeah testing is really important and this is where you know we talk about it but the merge was a significant event that's why we had so so much celebration across the world about kind of what was really going on uh when when the modes took place um so congratulations to anybody that was involved in the merge you know phenomenal job and uh yeah I think we can you know hold our heads up high there so I'll say testing is hard and you only got to use Cura to basically you know ask a question you know what you know what makes software testing difficult yeah it's a bit of a high level question you know apologies for that um but you know these two really um kind of uh sort of struck me here like the first one you know this is the basically fundamentally this is an impossible requirement because absence of evidence is not the evidence of absence and so often we run these simulations in parallel universes hitting you know hitting a whole host of code and it's clear and you're like well is that good who knows like you know we we can only really understand how much coverage we've got and it may be maybe good maybe bad who knows so that's that's a phenomenal kind of you know kind of you know requirement the the second one is then about kind of understanding like testers have to think about all those possible scenarios where issues may arise and ensure that they are handled by the code and you know that's just not it's just like how do you how do you approach that with a distributed you know kind of architecture like we have with ethereum um we've got a whole host of different clients using different architectures we've got the execution layout we've got the consensus layer it's just it's amazing to see it actually working and and but the complexity of that and the is just is just phenomenal um if we then look at kind of what ethereum has done to approach this task and this is you know this is not just looking specific antithesis we are a part of the equation um and I think we're very much of a complementary is is that ethereum is using unit testing test Nets we've got a list of Shadow forks and test net mergers occurred um some technologies that are being used called Hive and ketosis um that do similar um heaters to the equation we then have antithesis that is then doing this deterministic piece which allows us to to Really you know get into the detail of of all the different iterations um that are possible and then we we have a host of different fuzzing technologies that the ethereum are using um to to really try and isolate and kind of hammer areas that are of concern so you know throughout the entire year you know a huge amount of testing has been done and on top of that they've also used static analysis to just you know help with doing code Audits and making sure that we catch things before the merge was to take place I'd step back a little bit in terms of kind of the testing as hard statement you know it is a phenomenal kind of like you know obstacle to try and get over like how how do we find every iteration and how do we find those hard to reach bugs that you know may not be common ground but if they do occur will be catastrophic um and so if we just look at the right hand side this is a representation of a distribution Network so on the right we're going to we're going to call this node we're going to call this node um you know using Lighthouse and Geth so the lighthouse and gether running on this node they're running on an operating system you know leveraging CPU um we've got a file system we've got different processes happening and then inside those processes we've got different threads being used as well so just in its own sort of node there's a lot of complexity about what's going on and we can we can test the nodes we can test maybe a combination of Lighthouse and guests working together making sure that we're kind of you know pushing the boundaries but if we then think about how we then have a network and we have a whole host of different nodes running together and may have other underlying pieces like databases other pieces kind of actually on on top of that um the the complexity of all those different communication channels operating just means that the search space for Hidden Away Little Gems I call them gems but you know they're bugs basically those those they can hide away for a long time and you may never find that search when you're looking at testing you may never find that that particular condition where the network was slow and there's some there's a bit of thread pausing going on in a certain node and you know the the the the number of things that have to happen could could have a bug manifest and so this is really where where we step in because you know we then we then have you know the complete picture and this is what we build inside our simulation um we have the ability to kind of run the entire collection of consensus clients and execution clients in in a simulation run it and and understand exactly kind of every way in which we can actually kind of see outcomes um that's pretty intense you know how do you do that and fundamentally how do you do that in a deterministic fashion if we find a certain situation where Lighthouse crashes well can we represent that again can we replay it can we yes the first thing you do right can we reproduce it and often bugs are not reproducible they often you know they might try something different they might move around what like how do we how do we debug this and and this is again where we step in so what do we do we leveraged the ability to Auto generate networks inside the simulation environment and stand up all the containers that were necessary to bring up ethereum um we then were able to then use Network faults and other types of faults to just inject chaos into that running environment um obviously it's great to see the software handle that it's designed to handle these types of things so you know under duress we see things going wrong um and we we basically use fuzzing technology to to basically hit the entire system um not just individual pieces of software but the entire system is fuzzed and that allows us so basically kind of deterministically replay the complete orchestration of a situation not just one particular application but we know that in this combination different clients under these Network conditions something goes wrong um and so we use strategies inside the system to allow us to seek rare events and so when you when I start looking at the numbers and kind of what we've been doing over the last year you know we hear a huge amount of code edges like you know a huge amount and so how do we how do we find and allow ourselves to to seek out those edge cases that you know if they do happen again it can be can be pretty catastrophic um but you know get to them and and uncover them to help the client teams um debug and fix so we have we have the ability to have all this wrapped up in in a a tool set um that is available to ourselves and we share it to the rest of the client teams and the F and that's been really really useful I think if we look at it we have we have the the individual testing going on which is fantastic we continue to promote that and say keep keep going with that we have the test Nets and Shadow Forks again I think you know we can all say that they've been tremendously successful and useful and then we have our parallel universes we have the ability to run not just one simulation but we're running simulations literally every day that are generating and using you know exploring the various search spaces that exist across any number of different branches inside the uh the repositories so what does this mean for us well there's a little Little Help from Doctor Strange here hopefully you've seen it because it's not playing here we go this is what I look like when I'm doing the work [Music] for it in time to view alternate Futures to see all the possible outcomes of the coming conflict how many did you see 14 million 605. how many we win [Music] one so so for us winning is finding these edge cases right now your winning isn't like oh all tests passed yay um we want to find those those really intricate kind of piece you know combinations and iterations where we've covered 14 million different scenarios and we've got you know maybe not more than one but we at least have one you know we can actually hold our hat on so that's that's an example of what we're doing and we think that kind of represents you know pretty pretty good sort of analogy um if we move forward we've seen it again here we are this is kind of what it looks like in our world um you know we're looking for bugs um this this is an example of an output of just one run so we we ran this um 13 hours worth of water wall clock time which is if you look at the wall and there's a clock on it um it will last 13 hours um but uh it actually allows us to to exhaust 536 hours worth of testing um and you can see here we are we're talking about an enormous amount of edges being seen here um the branches are kind of how we get decision points we we get like an if then else on a kind of statement inside the code very simple example but just just take it um we can basically Branch off at that point and exhaust both Avenues and actually see what happens in both situations underneath those and you know under those test conditions um that's an example of a branch and obviously the code The Edge is seen is we've got the instrumentation happening so we can see what kind of functions and and pieces of code are actually being executed pretty pretty insane you know that's one run I've got 180 000 edges that we're seeing across across the entire network it's a busy Network um and obviously with all the different clients happening that's uh there's a lot going on so we see ourselves as a complimentary piece we you know we we've we've been looking at this um as a you know a great example of a project where we can basically work alongside all the other pieces that the EF are kind of using and different client teams are using um so we're just another layer in in the growing list of of of strategies being being used and I think that's sort of Testament to the approach that EF have used um to to Really hold on to this resilience you know making sure that we take advantage of every way we can we can make sure that you know nothing untoward happens in the future um you know more more and more upgrades coming I'm sure so you know we've only got 4844 around the corner I'll say around the corner we'll see um but uh it's certainly something to look at in terms of um you know making sure that uh we're involved at every step of the way so this is a very simple sort of uh illustration of what we've been doing we've been building all of the clients um giving them in in place and actually establishing Genesis so we actually have been using this at Genesis block and then moving it forward towards the merge in our in our world we would have the merge kind of positioned um we obviously start fault injection we've established a chain and then we start fault injection um to get to a point where we've got a whole host of different faults you can see them here labeled below so anything from partitioning um that's a great one obviously to encourage forking of the chains um we've then also got you know delays happening drops of packets um the nodes themselves um we can stop you know we do we do we put this thing through its faces we stop things we pause things we kill things we bring them back up I suppose in some respects it's quite realistic you know things do get rebooted and violators come back up again we we have all that kind of like happening inside the simulation and then from that we then have you know any number of threads being paused released there's there's yeah it's it's busy um every type of this sort of configuration of faults that we put into a simulation is complete again completely deterministic so if we know that this ended up with a segful we know we can literally put exactly the same into the into the environment it's basically creating a pure function of the entire system and we know we can get the exact same output which I think is a tremendous piece of technology which we have under the hood what the numbers like well yeah you know in one year we've we've conducted 31 years worth of 24 7 non-stop testing um that that has processed and and explored over 50 million edges of code um you know out of that and you can see there you know out of that we've got 45 validated errors some of which were painted for but 30 of the 33 of them were logged bugs um that were pretty catastrophic um things that really would bring either an actual note down so panics seg faults um we'd have you know nil pointer exceptions we'd have there's there's one I'm going to bring up in a minute with a little little example but um things that you don't want to have like living in your code base and and I think that's you know we would forward those through to the different client teams and they've been able to eradicate them and uh and obviously through the successful merge um it's been great to see our efforts uh put to put to the test as he wants an example or I'll give you I'll give you some um so here's here's an output of our of our sort of one of the runs this would come through from an email and hopefully you can see it's Biven our chart you've seen the side of the select the stuff on the right hand side but down here you can see a whole host of assertions that we've got running across across the entire network um this this is actually flagged up by fail and they've got a fail on a seg fault this is this is you know something that is is absolutely real zigzag V illegal storage access attempt to read from nil question mark I guess I'm really interested ones you get like you please report this oh please repeat please fix this um so this is just an example here and we're like What do we do now how do we how do we take it to the next step because it's great saying well you've got a problem um this you can see here is actually for the node that's running Nimbus and Geth so we know there's an issue there so okay fine have you got any other occurrences on any of the combinations we do um so okay that's cool um well let's let's look at an example uh of an actual kind of log entry that's coming through so this is this is a unified log the serialized activity that's occurring across the entire simulated environment bit of mouthful so you see you can see here on the third column the blue column here again this is just showing us all the different combinations so we've got Cloud we've got prism client Aragon we've got um there's nimbers never mind nimbuspessu all kind of serialized and you can see that we've got a concept of time running through so this blown up piece is basically the stack trace of showing us kind of what is happening and Nimbus is having some issue it's receiving some Json trying to figure out not doing too well for it so because it basically crashes the entire node um what you know so we go okay this is cool this is this is where I kind of go yeah you know there's got something something of Interest here um another example here this is just from from Prism um again I don't love sort of seeing panics but like you know it does it does show that there's something like of value here that we're really stressing um the the entire environment so again invalid memory address yeah okay cool what do we do now you know do we descend it to the client team well yeah you can and they might go I I wrote that code I can see what's going on here um well she had to do it didn't she yeah if she could turn back time well the great news about deterministic environments is you can turn back time right so we're going to basically jump into the actual environment and actually say okay right how when does this manifest like is it okay it crashes was it a second ago was it five seconds ago what happened in the execution pass of all the different Opera operating nodes with all the fall injection going on or or was it um yeah what where where do we where do we see this like actual kind of thing manifest and so we we want to look back so many seconds we may want to turn on car packet capture pack capture isn't by default just because the amount of data but it isn't turned on by default we want to be quite selective about kind of where we see packet capture happening and and then we can look at the data see if again we can rerun it um again obviously determinism you're stressing a little bit of kind of will it still be deterministic if I suddenly turn on packaged at a moment in time let's have to look at that but but what we've seen is actually yeah you generally if you've seen something you know manifest then we can we can actively kind of kind of turn on things to help kind of debug and work out what's going on there's some really interesting stuff coming through next year which I can't share but really cool so this is what we can do today this is what we've done with the with the uh is basically look at this mountain of data like you think of all the stuff that's going on all the different scenarios branches Paths of execution that have happened we can we have all these available in a big massive data set so we can we can look at this kind of analysis and look at the x-axis which is your time versus the probability of the bug occurring on the y-axis now the top right is where the bug is seen we can then look at the common routes that are happening across all the data and try and then bring it together to see what is happening you know where where does the execution occur where suddenly the probability of this actually becoming a bug yeah happen and and lo and behold we have a huge jump here from literally 0.05 of kind of the the kind of bug occurring to over 50 this jump here in the middle we can start to go okay right we know how many seconds to go back now we can actually Replay that simulation and turn back time I won't sing it um and and and see what's going on there because otherwise you're looking for a needle and a haystack like how do you do that and that's really been really valuable um in in our in our efforts so what's next well merger's great love it we're doing some Pokemon testing on some stable branches which isn't you know we've got you know things in a fairly good order there um but we know ethereum's not standing still we've got obviously EIP 4844 just around the corner I say again um thanks sharding withdrawals you know a whole host of new capability which is you know brand new products brand new code brand new testing um there's other pieces in there around you know actually the one that's interesting is using things like malicious clients you know what about if I brought up an environment that has a client that is you know not doing what it should be doing um you know can that cause issues you know how does that kind of part of the of the code for the rest of the the network handle handle that client and does it does it operate in a different manner so other pieces here you merge clothes clean up you know there's changes to you know kind of established pieces of code that are in in place now all those changes introduce you know probably good and it could appear uh you know Downstream so pretty cool um we're working actively with the clients um and open to really you know kind of broadening that relationship further so that's the end of that q a um any questions yeah um you showed us uh a slide with the probability of finding a bug how did you calculate that probability which model you were using or how complex is you know the calculations and that stuff that's a good question so the just because the amounts of data we've got available we we know every kind of path execution that's occurring that the outcome doesn't occur with a bug and we have then obviously all the different outcomes that do translate to the bug happening um and so where the ability to you know do some simple calculation we can see that you know the path the path that's sort of trodden and worn and you know this but this if this is if we see this happening then we can calculate the bug occurring there are obviously other branches down that that well-trodden path where my virgin's gonna go somewhere else and the bug doesn't manifest so we that's that's how we at every point in that graph we can we can calculate if you know if it does contribute or not contribute to the bug how do you kind of isolate unique bugs so you know it's like the same bug that's a good question and so we we see a huge amount of bugs that do manifest and show us like you know duplication like you know like the one you saw with nimbuson and Geth that we can we can run that literary day in there and then it will you know with the right commit code which is in history but we would we would be able to see that and we'd see we we have counts on that occurrence so we can see very quickly this isn't just like an edge case that's just happened on one sort of combined client set it's it's like it's a problem across the board and so um we have all of that wrapped up into our reporting you know what do you think about applying formal verification in this huge environment is it um do you think um possibility or I'm usually not the best person to answer that question but bring it bring it to us at the end because I've got some people that can answer us that question for you okay well please great chance of us keep keep you know keep the conversation going and uh I hope you enjoyed the talk