# Monitoring an Ethereum infrastructure by Péter Szilágyi (Devcon5)

- Speakers: [Péter Szilágyi](https://streameth.org/speakers/peter-szilagyi)
- Channel: [Devcon](https://streameth.org/devcon)
- Date: 2020-10-09
- Duration: 21:42
- Topics: blockchain, smart contracts, distributed ledger, ethereum
- Watch: https://streameth.org/watch/yt-cOBab8IJMYI
- YouTube: https://www.youtube.com/watch?v=cOBab8IJMYI

## Description

Visit the https://archive.devcon.org/ to gain access to the entire library of Devcon talks with the ease of filtering, playlists, personalized suggestions, decentralized access on IPFS and more.
https://archive.devcon.org/archive/watch/5/monitoring-an-ethereum-infrastructure

Have you ever wondered what goes on under the hood of an Ethereum node? Frankly, most people don't care. If you are, however, running production systems with multiple nodes across geographical locations, knowing what your nodes do internally is more important than you might think.It is a general fallacy to believe that a software either works, or does not. In reality, there is a whole spectrum in between. A software almost never works perfectly, rather there are quirks that may or may not affect your use case. These quirks may manifest only above a certain load or compound as time passes. The only way to build out a robust infrastructure, is to accept that your nodes are quirky. Instead of getting surprised when they break - having no idea what caused it - you must always be aware of buildups and correlations to external stimuli. This talk will present the suite of metrics that Geth exports for monitoring, highlighting why these have been added and why they might be important to you. By correlatinh your own load and use case with Geth's internals, you'll be able to gauge issues before they ever get strained enough to cause catastrophic meltdowns.

Speaker(s): Péter Szilágyi
Skill level: Beginner
Track: Developer Infrastructure
Keywords: infrastructure, technical

Follow us: https://twitter.com/efdevcon, https://twitter.com/ethereum
Learn more about devcon: https://www.devcon.org/
Learn more about ethereum: https://ethereum.org/ 

Devcon is the Ethereum conference for developers, researchers, thinkers, and makers. 
Devcon 5 was held in Osaka, Japan on Oct 8 - 11, 2019.
Devcon is organized and presented by the Ethereum Foundation, with the support of our sponsors. To find out more, please visit https://ethereum.foundation/

## Transcript

hey everyone I figured that starter minutes longer I have one more minutes yes so every now and again somebody sends me a message on messenger or wherever and if message kind of goes along how do i D bug death and I always I have a follow-up question what do you mean debug guests just submit an issue and it turns out that they actually don't have an issue rather they are they want to know how they can know if gas is actually working or if gas is working properly or how healthy it is and then usually what we refer people to is well just look at the logs how hard can that be okay I'm being a bit ironic here but the problem with looking at the logs is that we either arrive in this situation where death is just silent and there are no logs and you're just waiting and waiting or maybe there's a log every 15 seconds but you still have no idea whether that's good or bad or we have the more ingenious people who raise the log level and they still have no idea what's going on so the problem is that logs are really really useful if you know specifically something that you're looking for but if you just want to get a general overview of what your node is doing then logs are perfectly not useful so what can we do instead well first things first people mostly are interested in whether they know this working or not and this is not such a simple question to answer because it's not a binary of course if they know is not working we see it that it is not working but if it's working then that's an entire spectrum it might be working very well it might be working just barely but it's still working now the issue is that if it's working very well then everything is fine if it's just working barely then you might have an issue at 3:00 a.m. in the morning and you want to be able to tell how healthy is your node now it's not suppose that we start with latest release it kind of works but a lot of him flood of things can actually influence your node in some one direction or the other for example every time we do a bug-fix release hopefully it improves the situation and of course if you run it on or powerful hardware that also papers over any issues that we might have with flaky algorithms or suboptimal things Network an activity of other externalities like workload again influency opera up your operation allottee or however it is pronounced sorry essentially of course if you put if you run a girly node on your notebook then it will perform really really well whereas if you run may not then you might run into troubles these are kind of natural now the thing is when you start running a production infrastructure then just knowing that you know it works is not really enough you want to know how well it works because it's kind of a game of trade-offs if you are yes you can put the most powerful virtual machine available on Amazon underneath it and it will probably run perfectly but it will cost you a lot or you can put a Raspberry Pi underneath it and it won't run so that's how do we found them in find the middle ground and the answer is actually metrics and monitoring now from the very very abstract perspective metrics and monitoring means measure everything that you can and visualize it and this is not something new in the theorem ecosystem I mean the stats page was available since forever and everybody knows its usefulness however once you you want to measure something more it's really hard to figure out okay what exactly do you measure I mean a software system there are gazillions of things that it does so what is important and what is not important and to answer that you actually need to have specific questions what kind of questions would you like the answer to and for example us in the gasps theme we usually have three questions one of them is how do the nose behave across the globe the second is that if some node looks weird then what is it doing actually and last but not least if you found an anomaly and we fixed it then it is really important for us to see that okay the old version did something weird does the new version fix it yes and does the new version really not ruin anything else yes or no so if we dive into actual things actual examples the caffeine is kind of running eight boot notes globally across various continents now first question that we want to answer is are these boot notes healthy and how what is the cost now whether the blue tones are healthy or not for us it's really simple ordering are they in sync with the network or not so we simply just visualize how many blocks behind there are from the chain head and immediately if it some Boonton falls behind that we know that something is wrong but this just gives us the binary thing we know that is healthy or not but we don't know what's the cost or how healthy it is now if we if in order to find out actually how healthy it is we actually need to also chart out the resource consumption of it and as a computing system you have four major resources CPU memory disk and network and of course you can split this cup a bit and network up a bit in input/output etc but only know if we chart all of our eight boot nodes across these four matrix types then immediately we can see anomalies for example on the memory chart we can see that the yellow boot node is actually using two gigabyte more memory than the rest or CPU is a similar thing and immediately you have something to take care of and the other interesting aspect of it is that if you know your notes have 13 gigabyte of sorry 32 gigabytes of memory and you are at 8 then you don't care about it you're really far away from crashing however once you start reaching your limits that's when things will start going wrong so you can just looking at this dashboard you can immediately see how close you are to failure cool now just to dive into one of these examples DevOps team a couple of weeks ago told us that hey you're kind of using too much bandwidth I mean it costs too much they care about the money not the bandwidth and so we looked at our charts and yes indeed the boot nodes are actually pushing out five six megabytes per second of data and we're wondering okay what does the boot not do I mean this chart tells us how much it costs our resource wise but we don't know what it's actually doing with those resources so those are actually our intermediate level monitoring questions let's try to figure out what a theorem node is doing with its disk resources or other resources now in case of not working what can we do well first up we have on previously we've measured the total bandwidth usage but we would like to also match the individual map bandwidth usage of individual protocols since the Bouton's are running aetherium and light client protocols those two are the ones we want to run but we can go even further down and actually measure the network usage of each individual Network packet I mean packet type within those protocols and if you chart these we get these really nice fancy spikes and immediately we have three things that are really really obviously strange for example in the top two charts those are the input an upload and download speeds of the theorem protocol and you can immediately see that the light blue thing is causing quite a lot of traffic and I'm not sure whether it's visible or not but I will tell you the white blue has the label of transaction propagation so just by looking at these charts you can immediately see that there's something wrong with transaction propagation it's taking about 1.1 megabyte per second traffic and maybe this is enough for you maybe it is not enough for you to decide what's next step but in our case yes we are kind of aware of how the transaction propagation works and this means that we actually need to roll out a new aetherium sub protocol or I mean a new version of the theorem protocol with an alternative way to propagate transactions because otherwise there's no way to get this down but we immediately know what the fault is and we have an idea of how to fix it now the other anomalies from these charts are that the boot nodes are uploading a significant amount of data and our guess again is that since these are the boot nodes everybody's trying to synchronize from them so if you want to stop people from synchronizing them well of course we can nuke the boot nodes half way offline so that they refuse to give you the data but that's not nice alternatively we need maybe a better discovery protocol so that you can find beers faster better peers etc again just we just looked at what the node is consuming its network bandwidth on and we can immediately make really nice guesses now there are four resource families why we can do a similar exercise for CPU usage now CPU search if somebody were to ask you what does the theorem node use CPU for the no-brainer answer is block processing and this was our false assumption for many years - and about maybe half a year ago somewhere in January we decided that okay something's not right so let's try to split that block processing up so we saw that okay block processing takes a hundred milliseconds but what's inside that and actually figure that well when you run a transaction you there is execution components we also load data from disk and we also write it to disk so let's and we also do some hashing so let's try to actually break it up and meter all these components individually and then the resulting chart was a bit I don't know a bit surprising it turns out that if you run a full node transaction execution I mean actually CPU running and computing stuff that's 25% max of the block processing so 75% of block processing is shuffling data around and this is an extremely important thing to know because all of a sudden you realize that optimizing the EVM is not that important so optimizing the database makes a lot more sense than that even caring about how much one on up code costs or the other and in the case of the boot knows this gets even worse since everybody is constantly hammering us with requests in the case of boot nodes the ileum actual execution is 10% of the block processing time so that's a that's kind of scary but again we have a brilliant idea of where we can optimize so just a single chart just some information we can immediately see how to proceed now other interesting facts that these charts allowed us to see is that transaction validation or propagation I mean the you know networks throughput is I don't know something like 20 transactions so you would expect that shuffling 20 transactions per second around is no big deal except when you chart it out and turns out that the boot node is receiving 11,000 transactions per second most of them are duplicates half of them are invalid half of them are or some portion big portion of them are underpriced that are get rejected but it's really strange and when you look at that number you all of a sudden realize that wait so the transaction pool needs to be really optimized for this throughput so it's not the throughput of the blockchain rather the throughput of the junk data that's coming in that's defining how much processing we're going to do and of course we have two more other big consumers for the CPU one of them are RPC calls and which we didn't chart yet so that's something that we want to do in the future and network handshake requests which might seem like why does that even take CPU well if you run the Raspberry Pi to murder one core from erase verify just doing the cryptographic handshakes in the network so again we need a new discovery protocol to fix it and a big third one third category is disk now in the case of this it is extremely hard to measure because kind of fun you run a program you have all these layers on top of one another operating systems containers libraries and they all like to be smart and all like to cache and this kind of means that there's really hard to measure the thing that you actually want to measure we could measure how much data we're pushing the disk but that depends on how much RAM the operating system has on how aggressively it caches we could check how much it could ask the operating system to tell it to us but that also depends on if you run it in darker darker also start screwing around with all kinds of caches now if you were after we figured out that we cannot reliably measure we decided okay screw it we're going to measure how much data we are pushing into the database well it turns out that that's completely useless too because leveldb has all these background processes around compactions and fancy storage models so in the end we actually needed to patch level DB and ship all our metrics into level DB so that we can get an accurate measurement and we're actually really happy for the author of go level d before allowing him allowing us to keep patching his database so this one I will probably not go into but the idea is that after you found an actual issue so after you found that specific metric that is really out of place then you need to find out why it is out of space of this place here I gave you some ideas so those are good ideas R&R enough but if you don't know if you can if you don't have any ideas then the only thing you can do is actually to just try to look into the details of the algorithms internally and try to map out what the algorithms are doing and and try to have better guesses and for example this is maybe one third of the light servers charge that Kari put together and there are lots and lots of these just to figure out individual tiny bugs but I won't go into these because that's completely out of scope here however let's suppose we did manage to find what all we did manage we have an idea of what what the anomaly what was the cause of the anomaly and we have a poor request to fix it so the next thing is we just open a pull request merge it in and everything is fixed yeah no wrong the issue is that there are simply so many moving components in a theorem note that the fact that we fixed one thing might actually break others so the lifetime of these performance or these anomaly fixing or request kind of look like people open a PR and then we actually run a one week long benchmark to see what does it do now this is the perfect case we see something like this maybe once a year these are the miracles of development when this was actually a pull request created by Garry disabling some internal data shuffling within level DB and what it here actually the green lines were the master and the yellow lines were Garry's experimental PR essentially by just swapping a few variables he actually managed to cut down the disk IO by an order of magnitude so that was I mean you don't get these kinds of charts at all these are just you don't simply don't believe them when everything is perfectly better than previously what you get a lot more often however is these kinds of charts this was from a previous monitoring system that we had where somebody opened the pull request at on Windows Windows really chokes on folders that have many files in them so let's change level DB to use larger files I mean it seems like a pretty good idea why would you use two megabyte files if you can if you can use 100 megabyte files so he implemented the PR submitted it and actually also submitted benchmark results that yes the number of files go down the number of use file descriptors goes down the PR got two reviews everybody approved it almost got merged in and then we said that he I okay do you know what let's benchmark this because it's touching scary stuff and then you go you start to look at the middle benchmarks and you realize that the disk rights simply exploded so it blew up by two orders of magnitude and again it was a perfectly good PR just it didn't take into consideration some weird internal thing of leveldb and these are the PRS that can really really bite your performance where everything that you do is perfectly logical except the result and of course these are still a good case because this PR you can at least close and then you get the really nasty ones and these are generally this is the case that we see every time when so the charts that you see here that peak that blue peak that's actually the Shanghai das attack so it I just closed it on when we were processing doing a full thing I'm processing the Shanghai denial of service attacks and that's the memory usage so previously that's the one eighth release family of Gath every time you reach the Shanghai attacks we had this huge big huge memory peak simply we had various caches in they use a lot of memory you essentially your machine needed 13 gigs of ram to be able to process those blocks and of course that's a lot so we decided to replace that 13 gigs offer I mean that caching algorithm with something completely different and yeah it was complete success it completely wiped out the the peak and also even the general data usage sorry general memory usage of Gath went down except the disk usage went up by 30% and now you have this huge problem and question what do you do I mean 30% extra disk i/o is horrible ten extra gigabytes of memory usage is equally horrible and you need to make trade-offs you need to make really hard decisions that which is better and then in the end we went with the extra memory usage is better simply because if you use a lot of money if you want to use more memory than available you crash I mean it's a heart failure whereas if you use more disk than you would like then things get slower but they still function so in the end this is why the actually the won 9-0 release uses more disk than the won 8 whatever the last one but all in all we think it was worth it however sins can be fixed it now we were much better so so this thing was fixed already cool now that is kind of how the gas team pushes out these are how we monitor stuff benchmark beers and push them out now if you would like to repeat a similar thing in your infrastructure you kind of need to decide how you want to monitor things you either pay for monitoring other service via data that we did that for two years but dated August actually has limitations it didn't allow us to push all the metrics that we had in so in the end we switch to our own node we just ran graph hana and if you actually decide to run your own on-premise instances then you need to decide also on a database to push your data in either you go with Prometheus or influx DB they say these are kind of whatever you prefer the reason I explicitly mention them is that monitoring infrastructure boils down to these four things so you don't care about anything else in the world and from our part guests can actually push data either it can we can export data into the data doc format we can push data into gravano and Prometheus too so whatever rocks your boat we can do it and just just for the sake of completeness I also linked in our own dashboards so we exported them we use graph on an influx levy but you are welcome to use them integrate them do whatever you want with them so all our configurations are published there they will probably also learn the repo actually cool now what are kind of the lessons that we learned while creating these charts they were that you must measure as low as you can meaning that every abstraction is transforms your data everybody tries to be smart and those usually ruin your matrix now you always must measure your worst case numbers I get it that the operating system is really smart and makes things more optimal for you but if you assume that you run out of memory then your worst case numbers will do will be the ones actually hitting you so we always be aware of what's the worst that can happen and measure everything that can you can afford as I shown you previously you can gain a lot of information with if you have a lots of tiny detailed matrix but eventually it will become too expensive for example Martin did a really awesome measurements where he actually measured the cost of each individual opcode it's really insightful but obviously you cannot run it in live production environment still if you have the numbers you can debug an issue if you don't have the numbers you will may get gather the numbers the next time you reproduce that issue so there's no way to fix it without numbers yeah and that's about it thank you very much [Applause] you