Quest for the Best Tests, a Retrospective on #TestingTheMerge by Parithosh Jayanthi

foreign [Music] about the quests for the best tests today we're going to look at all the different testing tools that we used and I'm going to be talking about the bugs that we didn't find on mainnet if you were here for two talks before mine Mario spoke about the bugs that we did find so I'm gonna do the contract um the first part let's talk about what we're actually doing with testing why is the merge complicated to begin with well we have roughly 20 different client combinations and regressions sneak in very easily you might see a regression in one of the client combinations but none of the other ones and that's very tricky to pinpoint the specs for an active development there were quite a few of the early test Nets where we kind of didn't pin the spec version so someone was implementing something on a different commit so of course the test net broke so we had to actually figure out how to treat modifying specs along with modifying test Nets along with 20 different combinations all changing at the same time um communicating and debugging it's great that we have a decentralized environment it's horrible if I have to wait for the Australians to wake up for anything um it does take quite a bit of effort and quite a bit of planning on our side we had to kind of schedule around people picking up their kids from school where to schedule around people waking up in Australia we have to schedule around Americans a lot of different things and figuring out how to do all of this in a reliable Manner and on a timeline was crazy um the last one was debug knowledge we were really surprised the type of debugging you need to do for CLS and years are totally different we had to see how to bring all of that competence in one place and how to actually figure out when something goes wrong and they did go wrong a lot in the beginning um and the nice part is if you figure this out once we figure it out for future test Nets so happy that that worked out what could possibly go wrong so the merge has two parts the consensus layer and the execution layer and they communicate via this thing called the engine API so if you mess up the consensus layer you're going to have a network that can't agree on anything if you mess up the execution layer you have a network that can't do anything so those are the two high level problems that we're in trying to ensure that never happen um just to enumerate a bit from the regular testing world what sort of tests even exist so we have unit tests and it is in our decentralized uh world with all the client teams the client teams take care of this themselves we don't have to do anything no coordination work from our side uh this makes sure that there's no regression but these are localized regressions something that they might have seen on their client based on how they've built their client we have integration tests part of them are done by client teams so for example the Nimbus team spins up a local Nimbus Network to make sure that their stuff can always talk to each other then on a high level we do some interrupt tests so these might be devnets or whatever they are then we have system tests and this is where external coordinators come in so I'm going to say the EF was part of the external coordinator for the testing efforts so these tests end-to-end functionality we spin up a test net we run transactions we withdraw we set up faucets we look at explorers work and so on and then we have production tests this is something you guys might have seen with shadowfox so production tests sort of work on a prod like environment and I'll go into what shadow Fox are later on but on a high level we inherit all the complexity of mainnet and this also includes public test Nets so if you guys remember Kiln or kinsuki that falls under this bucket so you have everyone all over the place testing their things layer tools are deploying over there we have random D5 protocols deploying over there and these find issues that only happen on real world workloads you probably can't simulate them any other way so the second part I'm going to talk about what different tools we had and just give you a brief overview and just to tell you how this is going to go once you have an overview what sort of testing tools we have I'm going to talk about what we actually did with them and what we didn't find so the first one uh starting at a high level we have spec tests it's the great thing about the consensus layer they have an executable spec they have specifications with a ton of tests client teams can then import these specs and make sure that they can test them in their local CI this means whenever they're making a release at least you know that they're coherent to the spec uh this is largely a sanity check it's not meant to find any massive bug but it ensures that there's no regressions happening we currently have the spec tests I think running every night on on a new CI machine the second one is hype tests you might have seen this being referenced a couple of times um so Hive tests run with the simulator and they essentially start up the clients and then run the tests against a predefined interface these are a couple hundred tests these take anywhere between I think a day or two days to run everything and a brief example of how this is is it sets up a tiny instance of another mine node it sends it two terminal blocks and asserts how the transition happens this is a lot of awesome work by Mario he should be somewhere in the crowd shout out to him and follow him on Twitter we found a lot of edge cases in this and once we do find an edge case it's always in this so that we make sure in in future updates whatever it doesn't happen if people check the website of course then we have this thing called kurtosis it's an external tool that we're working with and kurtosis obfuscates all the complexity with setting up a test net you don't have to worry about how Genesis works you don't have to worry what format that never mind needs its Genesis file in nothing you just Define it in a yaml file you say I want a five node network with this Docker image and that's what happens for you um we time this actually nightly and they think we've gone through the merge at least a couple hundred times this year um a lot of them with issues but a lot of them without which is great to see we use this also to rapidly iterate ideas so once there's a new spec version or if you want to try out a new testing tool we throw it in there first we also had map boost integrated into kurtosis so we could test it out and do a lot of cool things there um a general view of kurtosis is it it checks the happy case if you can't figure out the happy case there's no point checking the rest of it so it just starts up a test net make sure everything is fine if it's fine it shows you green on your CI the next one is sync tests um there's no point on a network if you have nodes that can't sync up to the network so what we do is we spin up nodes I think at a week's notice right now um they sync up to the head of the chain and then they assert whatever we specify over there so if you notice over here you can say is execution healthy is consensus healthy are both synced are they both reaching head so you can kind of Define what side a type of syncing you want to do here um and as you can see on the right there are a ton of different options you can do and a ton of Networks and the cool thing is you can also assert bad cases you can say start your El stop once it reaches head then start your CL and then build weird scenarios there um shout out to Sam for building this he was also running it and sending a summary I think he even presented it once on Orca devs um so we could make sure that at least when we're making releases and when we merge that we can sync the network and this is the meaty one we had test Nets and shadowfox so a shadow fork and testnet help us coordinate all client teams in one place and we check compatibility largely we take whatever assumptions we have in the spec and we assert if those assumptions are true on a large level what we're doing is we're taking the Genesis configuration of any one network and then we modify a couple of values here and there and what happens when those modified values are hit is we split away from the main network but we continue staying sync connected on the gossip Network so we're importing all the transactions and we have the old load but it's on our side we it's it's just a side Fork it runs parallel to the main Network and no one cares about it except for us because we can find dozens and dozens of bugs there this allows us to stress all of our assumptions and we've done a lot of them I have a summary on how many we actually did in the end um I sort of look at this as a release test so it's kind of one of the last things we would do on any of the future Forks that we have it's kind of when we're almost figuring out are we ready to go ahead with this are we ready to move forward with with committing to this Fork is there any unknowns that we don't know yet and then we have fuzzles and in external organization called antithesis so antithesis is a deterministic hypervisor that allows us to perform Network splits packet loss all sorts of really weird edge cases we don't necessarily expect the network to be put in these edge cases but if they are we know that the clients can handle it on the right side you could you can't even actually see it but each of those is a 256 thread machine we have three of them all running fuzzles it was insane we had way more uh I've never seen that much compute in my life we actually had the IBM data center go out and buy more CPUs because we bought up all of them um yeah various fuzzles various teams running out all of them uh super cool we found a ton of bugs referred to marius's talk if you want to know what we actually found there and we hope that some of those bugs change the spec to make it a lot more stable or sometimes than implementation issue so to give you a brief idea on the testing life cycle let me tell you it's never that clean but that's what we want to hope to achieve so you have the client releases happening and once the client releases happen they go into the integration tests so we make sure that testnets can be run that integration tests work fine and once that's done we move on to test Nets and then we move on to stress tests and we push whatever we find on these stress tests onto specifications we do regression tests and then we do fuzzing and hopefully whatever we find in the last couple of stages go back into client releases so that's a kind of a nice way to look at a full life cycle of how testing would work we're hoping to adopt the same life cycle for future test Nets maybe we moved around a tool here or there but that's the general idea and the end game what did we actually do um it was really hard to find a graph that actually fit all the test Nets we had that's how many um so we had we started off in April of 2021 with ryanism and since then we've had four public test Nets that anyone could permissionlessly join into we had six devnets meant for all the client teams we had um five Gully Shadow Fox and in the end 13 midnight Shadow Fox and after all of this we had three test Netflix and only then we hit mainnet so the whole number of just testing hours in the merge is insane I'm quite sure if we added it all up it would be at least a couple ten thousand hours put into this um so what didn't we find this is a really interesting part because even though we have all of these Cool Tools there's still always going to be something we didn't find this 99 participation 98 participation or great blocks being produced awesome but I want to know what we still didn't find so first one um we had in-memory databases that were too low to process main net blocks it just so happened that we did too good a job of deciding which machines run our test Nets which means we didn't have any resource constraints and we kind of missed the people running nodes on 8 gig ram machines or people running on 16 gig ram machines we didn't account for that or we didn't account for older versions of ram being used whatever it is um another one that we missed is non-optimal block production um we were super focused on making sure that we didn't see any zero transaction blocks on the network we didn't compare that to what the optimal blocks on the network could be so it is an optimization problem of course and it is an awful that on mainnet we have some reduced load but it is something we missed and something we probably should look into in the future um the next one there was a really specific way in which the terminal blocks could arrive that broke Netherland and it broke Netherland by causing missing receipts and we only noticed that issue when there are deposits being made so we completely missed this and for what we still yet to figure out why but this is mainly an issue for lodestar never mind and no other combination or at least we didn't get the uh Memo from other breaking combinations so lots of another mind figure that out on day one of the merge and it I think it was passed like a day later um another one that we didn't think of all of the Shadow Fox didn't have any nodes syncing up to the network after the shadowfuck was done which means they weren't serving sync data and all they were doing were keeping up with the chain which is great but constantly thinking on minet also adds load on the machine so we want accurately simulating that so potentially on future shadowfox we're gonna have to add a bunch of syncing nodes that start up later on to see what could happen there and the last one that we didn't find was a failover beacon note scenario a lot of people are really obsessed with making sure they don't miss a single attestation which means they have multiple Beacon notes set up and that's something we just never did so a lot of the requests were just being sent to the primary Beacon node and not the backup so when the failover happens the backup wasn't ready um I think this has also been patched right now but it was a really tricky thing and we should try more failover backup scenarios in the future I guess um one cool thing about why all of this wasn't found most of them are optimization bugs um as you can see merge still went fine but we could there's still some room for improvement and it's a really hard trade-off to make between do we want to spend more depth time in fixing these optimizations or did we want an earlier much and that's kind of the tricky thing do we want earlier fox or do we want completely buckless free free um free Fox no idea but we're definitely going to be adding resource constraint machines to the mix next time we're thinking of just getting a bunch of Raspberry Pi's and doing some Shadow Fox on there see what happens there and another reason some of the bugs appeared but they didn't appear on the shadow Fox is last minute commits um The Last Shadow Focus I think a week before the March and the last releases also happened a few days before the March so yeah it's a hard trade-off um if you want to join the testing efforts please join Mario Vega his email address is here send in your information there and I think I supposed to okay I was supposed to have one more slide but I will just talk about that other slide over here um so we have a bunch of testing tools that we've completely open sourced if you're someone who wants to run that's nothing kubernetes if you're someone who wants to just run kubernetes nodes if you want to set up your own networks we have easy ways to set up Genesis if you want to um yeah use the same stuff we use for shadowfox all of it is completely open source quite a few organizations are actually reusing our um our stuff so shout out to them contribute back whenever you can you can find most of the tools on either on the ethereum GitHub repo or on the eat Panda Ops GitHub repo one of the two and yeah that's about it that's most of what we did and did not find on the merge thank you