# Lessons Learned From Phase 0 Testnets by Preston Van Loon (Devcon5)

- Channel: [Devcon](https://streameth.org/devcon)
- Date: 2020-10-09
- Duration: 17:25
- Topics: blockchain, smart contracts, distributed ledger, ethereum
- Watch: https://streameth.org/watch/yt-ljpM2cgTLMM
- YouTube: https://www.youtube.com/watch?v=ljpM2cgTLMM

## Description

Visit the https://archive.devcon.org/ to gain access to the entire library of Devcon talks with the ease of filtering, playlists, personalized suggestions, decentralized access on IPFS and more.
https://archive.devcon.org/archive/watch/5/lessons-learned-from-phase-0-testnets

Preston Van Loon discusses lessons learned from Phase 0 testnets.

Speaker(s): Preston Van Loon
Skill level: Intermediate
Track: Developer Infrastructure
Keywords: phase 0, testnet, technical

Follow us: https://twitter.com/efdevcon, https://twitter.com/ethereum
Learn more about devcon: https://www.devcon.org/
Learn more about ethereum: https://ethereum.org/ 

Devcon is the Ethereum conference for developers, researchers, thinkers, and makers. 
Devcon 5 was held in Osaka, Japan on Oct 8 - 11, 2019.
Devcon is organized and presented by the Ethereum Foundation, with the support of our sponsors. To find out more, please visit https://ethereum.foundation/

## Transcript

everybody thanks for coming to the e2 client summit I know there's been a lot of e to talks so maybe you're sick of it but it's really special to be here this is my first Def Con didn't didn't go last year I really wanted to all the rest of prismatic steam went so it's truly amazing opportunity to be here so what I want to talk about is our test net we've we've been running one for some time now and we've actually learned a lot so let's dive into it first of all who who are prismatic labs or who is prismatic labs were a self-organized group of just dedicated bakhtin engineers people who really wanted to be a part of scaling etherium and sort of making this reach global adoption capacities so there are six of us today most of us are here so come say hi if you see us if you don't know us already we'd love to chat what's interesting about this group is that we all formed really organically online nobody knew anybody before any of this and we were just inspired by Vitalik wiki article about sharding and wanted to start making moves on that because we felt that you know at the peak of 2017 when things were really like maxing out that we needed something soon to keep this momentum going so we were one of the first teams to come together to start building sharding and then the proof of stake at Casper all these things to merge together so it's been pretty remarkable journey so far one of our biggest achievements is to run an actual test net this has been we started doing this back in April of this year so we've been running it for I think six months now what what this comprised what's this comprised of is several beacon chain nodes running in different environments different areas of the world it's open to anybody so anyone with a computer can join it's typically runs for several weeks or several months depending on if we run into some fatal bug or if there's a new spec update where we have to rewrite everything which has happened many times it's also the full into an experience there's nothing really mocked out here when we're talking about this test net there are real deposits coming from a real eath one chain and the POA corley network we've actually had hundreds of people join we at one point accidentally merged with the ipfs network so we learned not to use default the p2p parameters because that's where the p2p does and we were wondering why do we have 500 nodes connected to us all of a sudden so that was a kind of cool cool thing to learn but but most of all it's just been really really fun and it's actually really easy to get them all so if you wanted to be a validator to see what it's like to stake with prison client we have this test net website which is super super straightforward so so when we're talking about test nets I wanted to ask fatale ik what is the best test set of all time and I didn't get a chance to ask him yet so this is what I think he's gonna say is he's gonna say Prismatic labs best test Network and the entire planet and the entire universe so I challenged someone here to find him and ask him and see if he he actually says this but Vitalik has run it as far as we know and he's really enjoyed it as so many other people so the first lesson that we learned is that participation has to be very easy you have users coming from all types of technical backgrounds or a non-technical background people interested and they want to check it out but they're really intimidated by adding a lot of tooling to the system just to build the project and running into these like weird errors where you you you know it really depends on your local environment so what we've done is built this test net website that within six easy steps you can be validating on real be contained and actually seeing rewards balances change and and even penalties if you go offline so the approach we took was to use docker images and this is typically pretty easy for users to get on on board it still requires some installation on your machine and some you know command line inputs but overall you don't have to spend 10 to 15 minutes building the project as some of you may have seen from the Interop event if you tried to build prison for the first time it takes a really long time so downloading these images which are 15 20 megabytes each just seems to be pretty straightforward approach haven't had too many problems we also made it so that within the website you could get some test ether from the Gourley Network we have a faucet built in here an interface so that you can just paste in your deposit data key and then with meta mask or Portus you can send that transaction to the eath one chain and wait for that to be included into a block and then go through you know go through the voting period and and all these are the things and eventually you do get activated as a validator and you start receiving assignments from you beacon chain it actually has like a really cool like progress bar so you can sort of they see something's happening when nothing's happening so it's it's been really cool and and this has been our biggest value add for users and and one of the biggest things that we started off with learning because before we got here it was just running it on your own computer running one client and then running two clients on the same computer then on different computers and all of this was was you know crawl before you can walk and then now we're starting to run so this has been a my favorite lesson so far just to make onboarding easier probably the biggest lesson though is that when we're running we're running multiple instances of the beacon chain in the cloud or whatever and those are really hard to be monitoring is he scene from like screenshots from the inter prevent when you see seven terminals on one screen it could be really hard to see that there's an error or even to catch that one of those nodes was not even talking to any of the other six which did happen so having good monitoring in places is absolutely critical to understanding when errors are happening in your system and how to find the root cause of those errors so let's take a look at a few a strategy as we do for monitoring things from from the very beginning we have built out metrics collection and monitoring through these metrics so here you can see a test network that's been running for two hundred and twenty nine thousand slots it's starting to look a little weird because these these graphs should be pretty you know flat and straight line especially the you know the CPU churn and on these kind of things so here we might say like oh if there hasn't been a new block in thirty seconds and will fire an alarm and will alert to that make your phone ring or whatever and then when you are woken up in the middle of the night and you see something like this you have some clues of where to start looking so to digest this for you guys you can see on the bottom right that there's chained reorganization imbalances so nobody was agreeing on anything we had you know hundreds of thousands of go routines happening at the same time so these four concurrent processes and that's leading to the CPU maxing out and then this nice like up and to the right line is starting to fall apart and this was like a nightmare scenario so having these resources is super super helpful but now you know like what's what's kind of going wrong like how do you dive into some very specific details like maybe block production is taking longer than expected and you want to identify the route cause of these things so what we do is we do a practice called code instrumentation so will annotate pieces of the code that we think are expensive or really interesting to track so here you can see the entire workflow for a block proposal so starting at the top we have the validator client which is a separate process from the beacon chain sending the block to the beacon chain and then something along here is taking 236 milliseconds and you know we kind of expected that to maybe be 50 milliseconds typically so we can dive into each part and see why it took so long this one took a long time because it was processing a lot of slots so we might say was there a long period of skip slots or can we cache this somehow so that we don't have to do this work redundantly where can we find ways to improve and make things faster and some of these things like you know from your intuition and you can write tests for and kind of write you know benchmarks to see like okay I know this is a place that could be bad and I can make it better but other times you won't know it until it happens in the real world so this has been super helpful for us speaking of benchmarking we do profiling so we'll take when we see that there are a hundred thousand go routines or something's taking a long time using a lot of CPU will take maybe like a 30 second snapshot of the CPU profile and then look at this this is called a flame graph because it looks kind of like a flame and you can dive into the most expensive parts the things that took the longest and go and then reproduce this in a local test and be able to iterate on that and make it a little bit better this has been super helpful for us another key thing that we've done is this is kind of recent is to do log ingestion so when we're running multiple processes and trying to understand errors and we have evidence we've users are filing bugs that the signatures are not verifying we can query the the central repository of logs and see on what frequencies are happening and did we really solve the problem so here we can see that these should be zero because this is an error that we don't want to have but we can see it at like specific times and maybe correlate that with some other interesting data that will help us identify the root cause of this because you know typically it's easy to make a problem go away but to really understand what was the actual problem is how you're going to get the best solution lastly on monitoring we do something called canary analysis so we are always so when the test net is running which is almost all the time we're pushing code out as frequently as possible with and without catastrophic ly breaking it so what we'll do is we'll run the new the new binary that we've just got the new image that we've just created we'll run it alongside the existing the existing binary and start them at the same time and then see if there are any major regressions so in this case we see that memory usage is up by 56% and that is kind of concerning so this canary report got a score of 77 percent which based on your tolerance you may want to fail immediately or not let this go through maybe you want to have some manual approval before pushing it out our you know success or auto approved criteria is 90 percent so anything that's almost almost perfect will go through anything about 50 percent we can look at and anything that's below 50 percent will just stop immediately and and we'll take a look at it later but that's been really great from keeping us from tearing down our tests out on accident okay and this is something that that we that we really learned pretty recently so our team is is very distributed across the world when nobody lives in the same city it's it's uncommon that we're all in the same time zone so actually being all here in Def Con we're on the same time zone so this this monster is coming out a night is even more scarier because what's actually happening is like when all of us in North America go to sleep the test net starts to fall apart and then the people over here and in Asia are stuck with it and like kind of on their own by themselves or maybe they didn't have the same knowledge that we had learned that day and we go to sleep we try to pass off things but it's always like when you when everything is sounds really good and really nice and when you least expect it things start to fall apart so that's been super interesting when we landed here our for Def Con day one we were right all in the same time zone and when we woke up in the morning the test and it had been stuck for two hours and no one was around to see it and we couldn't fix it in time before it was just unrecoverable amount of time since finality so only ended up just restarting it but that was something something that we've learned is that if you can put somebody so you're always awake 24/7 or the the actual solution we did for this was to have the alerting that goes to your phone you know pass through your do not disturb wake you up in the middle of night and make sure that like when when no finalities happened in 100 ebox that someone is there looking at that because that's a really really bad thing to happen so lastly like why did we do a test net so early like I said we've been doing one since April and this was before we even had a networking spec so we kind of made one up to get our clients to talk to each other and we knew it wasn't gonna be final we knew we're gonna have to rewrite everything in that regards but why did we do this we actually learned a lot of things besides those those few lessons so we're able to find race conditions which you can check for with race detection but it's not always perfect one of the biggest things is it's long long lasting memory leaks so things that grow over many days will have binary that runs for maybe a week and we'll see that the memory use is just slowly slowly slowly going up when you're running this on your laptop you don't ever see it we don't never see it in a unit test so these kind of things are super helpful UX problems users come to us and say hey I don't understand why you won't let me validate like what's going on and you know that's been great to just make the client better for everyone for usability so we can help understand what users are frustrated with or what they want and then we even actually run into aggregation issues so some recent thing that we learned is that you can't overlap aggregated at two stations that are already aggregated so we were seeing we were trying to aggregate those and they were failing but silently failing for some reason and then we were getting this is not very consistent graph of finality so that was something that we only realized by running the test Network for a few weeks and then lastly testing gaps things we can do to make it better so like the canary analysis and and other other like long-lasting tooling to make the code safer and to be able to have releases more frequently and just to make it a lot more secure so that's it for me thanks everybody [Applause] you