Securing Grandine's Performance

[Music] all right let's hear it for zarra BMA hi guys so we worked on consensus client performance profiling um both of us at the simar interest starting off with grandine and branching out a bit as you do during a work on a project um there's a couple of repos listed there two of them are on GitHub because that's where everybody else is and I myself prefer G laap so I threw that in um I'm going to hand it to Bulma who's GNA take the first bit okay hello hello okay sorry um my name is Messi and I'm here to make a pres presentation Although our projects shifted a little bit um the initial goal was to work on security and testing but due to some um uh I underestimated some things and so that was I had to focus on um performance analysis a comparative analysis between the cons client oh okay so okay sorry um so the initial goal was to do a security and reliability of grinding and then there was a shift in Focus which transitioned to to profiling multiple clients due to technical challenges which I faced oh okay sorry so um first of all I used so many performance security um tools the ones that stood out to me was a flame graph and then it implementing a timing metrix on grinding this is example of what a flame graph looks like uh although it's not quite readable because you have to like click in and then out to really view but okay so looking at this Stacks um grinding is actually performing very well from the stock overview although there's a drop down from the Tokyo run time which is at 61.79 per. which had a minus 35% drop down so the key observation here is there is a high Trend utilization on 97% in system Trends and initial setup significant drop down to 62 in Tokyo R time operations and then a consistent per performance across to tax management also multi trained worker execution showing similar patterns in different narratives is work okay sorry so um this is the 35% looking at the system trade level which is 97% then too one time which is which is 62% and then the which identifies a 35% drop down so the there is the the cliff there is kind of huge and then the high performance areas are trade initialization system level operations and then Co Trend management which gred pretty handled very well and then performed very well in that aspect so areas of Investigation okay a disclaimer this is a research project is still I will call it the research because so many things can change and then I might be wrong in some cases so areas of Investigation for me is a Tokyo TI sheding which is at 62% and then the sign corron time overhead and the tax profiling efficiency and um talking about the timing metrics which I had to implement for now I okay I was Sal suggested that I have to do a comparative analysis between um um grinding and then Lighthouse so for now I only have the data on grinding and I'm yet to produce the data for lious to properly do a comparative analysis but this is basically how the data looks like and I'm hoping to use this to do more investigation on um formal to do more analysis on formal verification and U foren creating a targeted foren and what's it called primtive analysis sorry and then hold on so the reason why I have to use use tools like the frame graph and the time um the timing metrics was to help me understand more about the consen client because I really underestimated what it is to force and build um a a forer for a conscious client because coming from a security a smart contous security background I thought it was the same perspective but I was proven wrong and the complexity was overwhelming but using this analysis and using this data I think I'll be able to Pro to pre produce and to come up with a strategy in in order to have a for to build a fora which has to do with um making it a targeted fora and also for formal verification because I understood that um there is no much um um um um work on formal verification on ethereum cons client thank you I can hand it over to you so small premise to the second part so as I mentioned we initially started working on grandine it was a slight shift in Focus transitioned to doing this profiling of multiple clients at least do what I focused on uh devop emphasis I decided to go there since I have no experience doing any sort of devops work and I figured having a couple of months seemed like a lot of time here that's a good thing to pick up so that was fun uh build my own server from secondhand part first first time doing that running proxmox um pretty interesting experience uh learned a lot challenges Hardware constraints I thought I had sufficient amount but turned out that maybe the minimal required specs listed are not entirely enough to run all the notes on a single uh server at least not the one that I decided to assemble um and regarding the outcome yeah well there's limited data uh the results are modest but valuable insight into running ethereum notes was acquired so let me tell you a bit about the setup that I used so I listed on the left side for you guys the consensus clients that are there and the versions of those that I used two of them don't have a version specified and that's because I didn't manage to run run them so they were not included I would like to include them at a later stage when I have the capacity to do so in terms of Hardware uh I have dedicated uh eight cores to each of these um Ram 32 GB and the dis space was two terabytes and VM Drive which is sufficient for all these clients to not be limited there in the setup I make sure that each of them running on a virtual machine is also is limited to this basically so there's no way that if one of the VMS isn't using all of its Ram that one of the others could uh use it because it's available this is normally something that Pro Mo supports but that of course doesn't allow for a proper comparative analysis um then the services that I also got to work with and got to learn about and nethermind of course I need an execution client and a whole bunch of other services that anybody who's done anything in devops probably knows so Prometheus obviously for metrics um I'm not going to mention them all but I will just say that these were a lot of tools that were new to me um quite interesting to see how many services you actually need to run such a such a node and a in a useful way so that you actually can monitor what's going on then I'm going to show a couple figures on some of the data it's a bit older data but uh so be it so this is the CPU usage that I measured over the course of About a Week a little longer uh for four of the consensus clients that I mentioned and now the question obviously is what does this mean and I don't have a crystal clear answer to you I do notice some things which you'll probably also notice straight away which is that grandine seems on the average to have the lowest CPU usage but there is somewhere a spike in the middle um I see that in a data figure in the next figure I'll show as well and honestly I'm curious what happened there but I simply don't know I probably need more data from the metrics alone it's not so clear it could be various things it might be something Network related could be something client specific simply don't have an answer for you there here I looked at the memory usage of the different clients um also you see the same spike in the same period but it's for a different client so yeah there's definitely something going on there and and I would say that my suspicion is that it has something to do with my server at that point since I see this weird pattern but like I said I have no straight answers for you and then the last figure I will show I think this is actually one of the things that is highly relevant I only show grandine and Lighthouse here because the initial Focus was more or less on grandine and well for comparison also taking into consideration Lighthouse so the number of pairs that you have is obviously very important uh and you can see that there are actually quite large spikes to the downside I have the limit set at 100 so that's why it looks kind of artificially kept there um but yeah if you lose a lot of pairs that's obviously not a good sign and well this seems to be fairly maybe not stable enough um or not as stable as you might want it so in terms of Outlook what I would like to do is refine and scale this node management so I now have almost automated setup for the provisioning of my VMS I would still need to switch to nyos or Taos to have it fairly immutable and item potent then I want to integrate e Docker this is something that I simply didn't have enough time to properly look into so so far what I've been using is sge uh well I started with simple scripts and running local binaries and eventually went on to use Docker compose but yeah I would like to move to kubernetes and I would like to also explore e Docker because I think it hoverers some of the things that I was lacking or missing in sge and then I only very recently that that is to say yesterday learned about the secret shared validator um so I'm going to look into this this is something in the direction that I was thinking I want to use this setup I want to use this setup myself with an agentic system because there I have a background um so I think it might actually partly at least overlap with this secret shared validator setup because I was thinking in that same direction and lastly I want to optimize the data queries because the idea of this agentic system would be to integrate those metrics so that the agent can proactively take the necessary measures for example manager node if you see that your peers are getting too low maybe you need to well do something to um to fix that um yeah that is all I have for you so then I only have a thank you slight left um obviously Mario and Josh it was amazing thanks so much for uh well all the time and effort that you guys spent and granting us this opportunity to be part of the EPF um yeah I think that's it so if there are any questions [Applause] then yeah uh hey thank you for an amazing talk uh I was wondering if the notes that you ran were validators or just notes no also validators also validators okay so it might be uh nice to see the difference between like a for each client what's different if they're a validator and if they're not yeah I agree there's lots of experiments you could do um really really lots more uh I would have liked to have done more of them but it's as BMA already said it's quite overwhelming because even when I thought I had all the pieces and sort of working and then you need to learn promql okay and then you learn a bit of promql and then you see all the metrics that are there for example Gra which I also used a little bit at somebody I just decided okay just cannot do it so yeah I agree there's lots more um and those those things would be interesting to look at you amazing thank you any other questions Mario um yeah um yeah I was wondering because you mentioned the setup with brm and then you want to automatize it setup with NEX it's really cool uh I'm just did you didn't really have in slides like some more specs about it like for example uh what are what virtualization are you using with Pro Mars like uh KVM or lxc or yeah KVM is what I'm using but I could explore maybe Alternatives I haven't explored any alternatives there so you think it's worth exploring uh just the Linux containers are good for servers as well uh and I was wondering like because proxo supports both and you just yeah you didn't have the details there so I was wondering like which one no this is true no this uh I use it's the lightweight version right so for Linux that's a that's a preferred option I would say cool thank you so much here all right thank you guys [Applause]