# Lodestar: Metrics-Driven Development by Cayman Nava | Devcon Bogotá

- Speakers: Cayman Nava
- Channel: [Devcon](https://streameth.org/devcon)
- Date: 2023-10-07
- Duration: 20:15
- Watch: https://streameth.org/watch/yt-49_qQDbLjGU
- YouTube: https://www.youtube.com/watch?v=49_qQDbLjGU

## Description

Visit the https://archive.devcon.org/ to gain access to the entire library of Devcon talks with the ease of filtering, playlists, personalized suggestions, decentralized access on Swarm, IPFS and more.
https://archive.devcon.org/archive/watch/6/lodestar-metrics-driven-development/

Lodestar is the newest production-ready consensus client, written entirely in TypeScript.

How did we do it? How did we go from buggy, slow, and scared to agile, confident chads?
Metrics, metrics, metrics!

We will introduce the concept of metrics-driven development in the context of blockchain node development and give fun examples where metrics saved our hide, time and time again.

Speaker(s): Cayman Nava
Skill level: Intermediate
Track: Layer 1 Protocol
Keywords: metrics,consensus,software practices

Follow us: https://twitter.com/efdevcon, https://twitter.com/ethereum
Learn more about devcon: https://www.devcon.org/
Learn more about ethereum: https://ethereum.org/ 

Devcon is the Ethereum conference for developers, researchers, thinkers, and makers. 
Devcon 6 was held in Bogotá, Colombia on Oct 11 - 14, 2022.
Devcon is organized and presented by the Ethereum Foundation, with the support of our sponsors. To find out more, please visit https://ethereum.foundation/

## Transcript

foreign [Music] really just our story and how we've developed lodestar over the past three years so about us my name is Cayman I work at chainsafe it is a great company we're hiring we basically build blockchains we have a whole bunch of layer one projects lodestar is one of them but if you are interested in getting into core development we're a really great place great culture so lodestar is it's one of these projects at chain safe we we're um we're basically an eth2 or ethereum consensus ecosystem written entirely in typescript um we really take the open source ethos to heart we have all of our meetings they're public everything we do is public and we really encourage anyone who wants to contribute we we try to help people out and get get people involved a lot of our team members actually started started their work on the project through their open source contributions and it was kind of like a open source to contract to hire kind of a situation so we really we really like that that kind of ethos but we're talking about today is specifically our consensus client and how we got from like prototyping typical kind of code to something that's like production ready something that you can really rely on something that doesn't just blow up all the time so before I kind of step into our story I just want to kind of do a very high level of what metrics are uh this is I'm not going to go too deep if if you really want to go no more you can like read the docs about these things so um metrics in our case we're using Prometheus Prometheus basically gives you the the tools to build time series data for specific elements that you want to track in your code the kind of three main things that you're tracking is either counter a gauge or a histogram uh we can go we'll go into kind of some examples of all of these things but basically what you want to know about metrics are you have some value something in your code that you're wanting to track and you want and Prometheus will kind of query the query your software over time and you can get a sense over time of of how things are changing how things are are moving um then you use grafana to visualize these metrics so um a picture is like a thousand words so it's basically the difference between like looking at a bunch of logs and then looking at like pretty graphs that can show you give you a little bit more insight give you something at a higher level than just squinting so our story um basically we we started um with really bad uh with not very much infrastructure not very much a discipline and how we tackled um metrics or tackled like the data that we we track and eventually we kind of like over time and like through a lot of pain and and suffering we kind of realized that metrics are the way to go metrics give you uh insights that you'll never get just by looking at logs so really metrics aggregate a bunch of information and give you um the tools you need to make better decisions about how you release your product what things to focus on like what are priorities and they're also just really pretty so they're cool so this is a picture from the early days of lodestar from the interop walk-in in Muskoka in 2019 here's our message yeah we're stuck syncing and some P2P issues uh we didn't really know what what was going on like why things were breaking we were looking at logs actually on the other side of the screen here you you know we look happy but it's like uh oh things are breaking here's some uh um some funny issues that we uh had you can see priority low adding Prometheus monitoring we didn't even know we were doing uh um yeah out of memory om out of memory it was just like plaguing a lot of our development throughout 2019 into 2020 uh lodestar was like very Proto very much like a prototype and we ran into the issues time and time again like we would have the same thing happen multiple times and we weren't like like regressions that were happening and we weren't really like learning our lesson kind of the turning point was you know starting to take some time to to Really uh which takes the time to like build out these metrics and build out these dashboards so we had a contributor I think I can save a new world maybe with peace um and so throughout 2021 we started going really hardcore into adding a bunch of just out of a bunch of metrics um and taking time to think through what questions that we want to answer what pieces of code are dark where is are you know where are we spending time in the in in our software where are we where is a bunch of memory being spent we really kind of uncovering all of the complexity of this running software I mean blockchains are really complicated software there's a whole bunch going on and um unless you're really tracking tracking key pieces you're you're kind of going blind and now we're finally kind of at a point where um we for any new feature that we're adding we ask for metrics so it'd be very important to track the retry Behavior meaning add metrics for so that brings me to metrics driven development um really the the kind of the key the key thing that we've learned is that every large feature should be documented with metrics so if you are adding a case of a retry mechanism you're adding some new feature you need to make sure that it's actually working and the only way that you can or maybe not the only way but a really the really great way of doing that is being able to show that visually being able to show okay our cache is now bounded let's see let's see that in the metrics let's see how it's How that cache is growing over time or let's see how many times the retry mechanism is actually being activated um those are the sorts of things that you really only look confined by through through these graphical methods through these metrics um you're going to have a really hard time looking at the logs for that the other great the other thing about metrics driven development is that any deployed software that you use you need to be monitoring it very carefully for regressions so look here this is the process Heap increasing over time this is a comment from Lion for the last two days the leak is clearly visible an impact of 75 megabytes a day yeah so that's going to be a problem if your blockchain is if it's just increasing at 75 megabytes a day continuously you're going to run out of memory and these are this we were able to catch this this issue before we actually before we actually cut a release so um it makes your release process a lot at least release process is a lot better here's another here's another example of a regression uh we deployed deployed a version of our software and we saw the cache size grow it's like oh is that is that is that a bug is that a is is it okay well we only actually caught it because we're tracking these things uh you probably aren't even going to have a log that's going to show you your cash sizes so unless you you actually measure this you're going to be blind the other great thing about the about dashboards and about metrics is that it lets you correlate different different pieces of the code to find out where the problem is so here we have different you can you can see a bunch of different graphs kind of all at the same time and see um okay something is affecting the event the event Loop lag I can see at the same time the active number of handles is growing the number of requests is growing so you can kind of um you know kind of solve the mystery in a in a in a way that you're not going to be able to do in the logs another great strategy uh is um is using different versions if you're running software so running um like a staging version of your software against a production version or against several recent versions and being able to overlay data on top of on top of one another and being able to see okay well it looks like our beta version is is using a whole bunch more memory than our unstable version so you know being able to kind of compare and contrast um it gives you the tools to do this and so just just kind of like drilling down to a few a few few tips um these are very very practical kind of tips don't abuse the histograms so I said there's three different types of metrics um you just wouldn't doubt you use the simplest tool possible histograms are really only used if you're wanting to see the kind of the distribution of of uh of something so like maybe like looking at like request timing maybe you'll want to see like all right certain requests are happening very very quickly but then others are happening kind of longer um so the classic mistake with histograms is um is uh using a label that's unbounded so if you had a metric that was tracking something per peer ID on the network well you might have thousands of peers with and if you each new each new peer is going to be using a new peer ID it's going to blow the metrics up and it's going to affect the performance of Prometheus and you're not going to be able to run the queries that you want to run oh so how do you know which metrics to add so I showed some of these pretty graphs and everything um but like you're in your in your software how do you know what you want what do you want to track so really what you want to do is think about questions that you couldn't ask otherwise so a really good example is the size of your internal caches another thing you can think about is like if this feature or if this part of the code would degrade or explode in the in the kind of bad case that that's a that's a prime candidate for for adding metrics around it um and then really just like keep asking questions and then it's like if you have the data to be able to answer the question you're good but otherwise keep adding metrics until you can until you can answer those questions so some examples that we we've kind of come across how often are our streams being reset how many peers do we have these are kinds of questions that that the metrics can help and so I'm going to show you just a live demo of or a live uh example of of our dashboards and we can kind of run through some things and if anyone has any questions about it while I'm going through it we can we can do that too okay so this is this is a our dashboard for our Fleet of of uh Beacon nodes and validators um this is kind of the high level dashboard right now we're looking at an unstable node we can go to like stable a stable version of lodestar so this is running uh version 1.1 or on the girly Network we're synced it looks like it got restarted about a day ago some interesting things I guess you know peer peer count is looking good all right you know there's nothing everything looks stable there we track different types of peers inbound outbound and uh for us we really care about um the number of peers that we're going to be receiving messages from kind of continuously those are our gossip uh peers so the number of mesh piers on our core topics staying about six or seven it looks like we're connected to a lot of Lighthouse nodes and a lot of prism nodes and there's another load star out there let's drill into something a little a little more detailed here so one thing one one area that we really struggled with and where we added a lot of metrics is in our gossip sub-implementation we found through adding a few metrics that we weren't getting blocks on time and it turned out that we we didn't have enough peers who were sending us these um who are setting us blocks and then we started asking questions all right well why are they not sending us blocks it turned out it was because our peer scoring was too low all right well why why was the score too low uh well it was it turned out it was because there was a specific part of the scoring that was that was too loans so we kind of you know you start asking questions and you can build up um you built you start building the answers to that okay so what you're seeing here is kind of some of what I was talking about um we have breaking out the score into different components um that different parameters that are going into your kind of aggregated score for peers um what we were seeing before is you know you'd see a graph where the values are like going down and down and down over time or if they're like very negative numbers another really useful metric that we were that we check a lot is um things related to the VM so memory CPU um and disk storage so looks like our memory is stable that's great um GC is only taking roughly seven and a half percent this is things are looking good yeah I think that's enough of a enough of a demo here so um at this point um open to questions and we're also hiring so if any of this looks cool or interesting and you know typescript hello and thank you for the talk um so I see that you have a ton of metrics and I believe that most of them are useful but um do you have some kind of an alerting system for you to be able to know when a metric has gone wrong because I think it's just unfeasible to go through each of them like daily or something like that you I think you need some automated way to to alert you that something is going wrong yeah absolutely so that was something I didn't touch on but um it definitely feeds in um so we we use uh pagerduty you can you can basically set up um thresholds for certain metrics that trigger um either slack notifications or Discord notifications or in the case of a critical kind of issue it'll call you or text you yeah we do have that set up I would definitely recommend that if you are in running something in production uh thank you um again there are lots of metrics do you did you guys um develop over time um a way to or you know kind of a mental model or framework to decide how to where to put them whenever you have new um you know graph or metric instead of or do you just like decide to put it somewhere and then go back and like you know to make it easy to to read or to uh browse the dashboard I don't know if that makes sense this is the question about organizing the yeah exactly yeah so um I think probably have to grow it organically what we did is we we started with One dashboard That Grew and grew and grew and eventually it became kind of unsustainable to have everything just piled together so we we started breaking it apart um the other thing that the other kind of issue that we had initially is we would create a bunch of metrics but we wouldn't add them to the dashboard so there was no actual like visual cue for for a lot of it and so it kind of just went un um unnoticed and so one thing we would we would definitely recommend would be like if you add a metric also add it to the dashboard at the same time thank you uh the question about storage so again too much metrics or what does it affect the storage um well so some of it kind of depends on your architecture um of how you're deploying this so um what we do is we actually have one Prometheus and grafana server servicing all our entire fleet of of Beacon nodes so you know we have a bunch of different machines and then we have one one machine for these uh right to to uh things so it it doesn't cut into the to the uh storage we retain for a month quick question uh is are the metrics relevant for like other evm chains besides ethereum mainnet um or is it just like specific to ethereum mainnet so yeah a good question um so I'd say like I don't know if it's a half and half but like half the metrics are like more like for like monitoring uh the chain something like a like a block Explorer kind of kind of thing um but I would say like a lot of the metrics are specific to the implementation so and those are the ones that are I think probably more helpful um just the internal caches and like timing timings of things that we're really really wanting to get right hi uh I wanted to know if you think these metrics are useful for other clients and not just the typescript one and if so would you maybe consider like you know putting them in the public domain so like anybody can see them and apply that to new clients they might be considering building right I know that um other clients do have metrics and I I think like the ones that are more user focused uh those are the ones that look more like a block Explorer and maybe with some like validator like showing your balance ticking up over time and I think those are those are public and I and there is an effort to standardize metrics across the consensus clients so I think some of that's happening um but again to the to the previous question a lot of the metrics will always be implementation specific in terms of roles and you know in team members who generally looks at which metrics and which graphs um just kind of to get an idea of uh you know which metrics and graphs are useful to which team members or depending on like the role or the the type of work that they do you're asking if if only certain members contribute to these are more like the uh looking at the the graphs and the browsing the the metrics themselves like you know for example there are lots of metrics so not everybody looks at everything so you know it's more um what you know what team members look at look at what kinds of graphs or right so so for any of these larger feature pull requests we have the the person who is authoring or kind of owning that that feature they're responsible for um making the case that that that it's correct and then it works as intended and so they're responsible for looking at the metrics and building building that case and then as far as like um our release process we have um a checklist of basic things that we look at um before we before we actually release we have a testing period so we cut a cut a Beta release we deploy it to our Fleet um and we watch it for a few days and then after a few days we um gather you know check do the checklist and any other kind of ad hoc things that were were noticing then okay so in web 2 there's a standard of how to consume logs metrics and traces and it's done by the cloud native uh Foundation to make the same pipeline that you have here more standard in a way that's consumed by other tools as well uh did you take a look at that I recommend it if not the the standardization effort is to to build out some kind of Baseline of functionality like you can have one one Prometheus and grafana running and then whichever client you want plugged in here you kind of have some guarantees like uh it's not breaking and like it's it's kind of working so yeah that's exactly what open Telemetry does so take a look at it