Tracing Integration in Lighthouse by Sayan | Devcon SEA

[Music] Lighthouse another fellow contributing to Lighthouse consensus client uh this time uh uh integrating the logging framework tracing into Lighthouse so thank you so much for being here yeah go ahead hello everyone uh so first of all let me say this that uh my project Isn't that cool uh it's kind of like um tedious and uh large size chore uh like you guys will uh get to witness many cool projects so I'll will just keep my presentation short and finish this uh quickly okay so okay um I'm Shion as you can see already um and I have been uh trying to integrate tracing uh into Lighthouse so first of all a little bit of background for uh those who are not familiar with the rust ecosystem so uh tracing uh what's tracing it's a cool uh Asing friendly logging framework and uh but why did we want to migrate Lighthouse from SLO to tracing uh because SLO is kind of lame and we need to pass a logger everywhere uh while using SLO slog is not uh acing friendly at all and it makes uh exporting Lighthouse crates kind of annoying and pretty much painful and uh on the other hand we have uh tracing uh uh which provides additional uh contextual info that means uh we get a better debugging experience we have a modern to logging tools and it makes uh easier for external code consumers to export LH crates um and most of the dependencies in the lighthouse uh already uses tracing so we might as well uh do the shift and it's yeah it's quite Asing friendly as well yeah just just uh way better than slog in my opinion okay so what have I done so far um so uh there's the pr uh if you want to go check it out you can just uh go to the lighthouse repo uh and in the pr section you can uh sort it uh by the most commented PRS and it's currently uh the most commented PR over uh in the lighthouse I'm not sure if it's a good thing or bad thing not not a great thing I guess but yeah um so so far I have migrated like the pr isn't merged yet but I have migrated the whole Lighthouse codebase from stock to tracing uh and I have uh almost messed with almost every file there was and I uh reimplemented the custom uh formatter uh and I implemented the SS logging component and also replaced the slogger which was used for um the file logging stuff and uh converted it to like shifted it to uh rolling appender yeah so let's move on to the next stuff um yeah so this is uh what the logs look like right now uh and we try to make sure the format is exactly almost exactly the same as we had before so yeah and next up uh let's uh discuss some challenges um uh okay so one of the challenges that we faced was to uh keep the format same uh because many uh external and internal infra depends on that uh in U depends on the Lo lock format to be the same I just got to know that siren kind of depends uh on the lock format uh so yeah uh we just uh that thing is uh before adding a custom tracing uh for uh for in layer and this one's what we have now yeah so we just just just try to make sure there isn't much difference um yeah and other thing was um tracing uh doesn't have a crit macro inbuilt so we had to implement uh implement it on our own so here's what the crit looks like um and this was uh the trickiest part uh so the previous implementation of s logging was kind of complex at least for me uh to understand uh and shout out to uh Ed for helping me understand that code so basically what it was doing was they were using slog as sync uh it collects all the logs into a single task and then the task could uh then print and write uh two files asynchronously uh and the HTTP API wasn't in a separate task so uh it couldn't uh see any of the logs so they had to uh modify the uh SL uh SLO ising last thing to create a channel and then uh HTTP API could just get uh get this Channel and uh then the slot Tas could just uh dump all the logs into that channel and for some reason slog uh Records didn't allow sending the data uh through the channel so they had to make the Asing record thing if you check it out it's uh it's like uh complex I say that because I didn't understand what was going on so no point in making comments okay so with uh tracing it was super easy to implement um uh we just had to uh set up a tracing subscriber and uh it was almost done and this is what uh the logging looks like right now and uh we had to just remove the whole logging part from before and reimplemented uh reimplement on our own okay and other thing was uh span versus uh slogs o macro uh so both of them are kind of used uh to get additional contextual info although they work kind of differently um and uh thanks to aan for helping me out out of this one because uh the first thing I did was just I replaced all of the SLO macros wherever they uh being used to span which uh like it was functional but wasn't U uh working uh as it should have uh so here's uh the P both of those PR are from Ean um and he managed to uh add the spans in the uh slasher service and E RPC service so uh and I I'll also have to add a lot of spans it's on the road map next but yeah uh they are just add so that the we can get the service uh to show up in the logs which is currently uh many of the service aren't uh showing up on the logs so yeah some uh just random challenges that we faced or uh we tried to break it up uh in multiple PRS but it it would have broken the S logging so we had to do that whole thing in uh one single PR and it has been quite a bit of back and forth uh and uh last thing was sever uh skill issues because I was really really bad at rust and uh development in general so yeah uh so next steps um adding more spans as I've said that a bunch of Serv serves aren't showing up in the log so uh we'll need to add a bunch of more uh spans and there are still uh lot of feedback fix iterations uh left to do and we'll need to do more QA and testing to make sure we don't break much stuff and other thing was yeah we need to add uh some Flags to enable or disable additional context and overall overall experience uh was great uh the APF last 5 months were uh like uh uh went uh pretty well uh working with the lighthouse team was great thanks uh to you guys for being here and attending the session and I got a bit better at rest I think it's debatable so and loved the office hours uh got a lot of Alpha from there and finally uh huge thanks to the lighthouse team especially uh age atan and mac and special thanks to atan and Mac because they are dealing with that mess of a PR like uh there all the comments and all the back and forth thanks for being so patient with me thanks to H for helping me out with the S logging stuff and uh thanks to Josh and Mario uh for conducting the whole uh EPF and uh just to just thanks guys you you guys are the OG's so yeah that's all thanks for sitting through my weird stuff whatever that was yeah uh thank you very much San for your kind words but please don't be so humble I think you still did a great job it was a learning experience so guys I mean I hope that it was productive right come on like he helped you right so don't be so humble it was great yeah and uh now we are running ahead of time we have like 10 of time for questions so please like yeah give give him the business give him some questions comments anyone yeah so are you using or tracing uh traces between instances or separate client and server are you setting remote parent IDs for traces for analyzing them in for example grafana Tempo or other not yet not yet not yet okay I haven't set up yet and uh what are you using for analyzing logs uh I'm not sure if I'm using anything uh there's a thing called yer or something I am not using that yet so there's a lot of testing and keyway and only file appender yeah file appender we have logging stuff working uh normal logs are showing up yeah got it so any plans for adding metric layer like very cool Library yeah there's uh so there's already the metrix layer that uh was already being used for the Le PTP and dis V5 but it's not functional yet because I messed up a lot of stuff so once that's done uh I'll make it functional it's if you see the the code there's the metrix added in the subscribers stuff but it's not functional yet did you see any uh memory improvements by removing the was like a global log sorry object did you see any memory improvements between I hav bran you haven't haven't checked um any comments or or uh questions yeah how do you choose which memes to put into your presentation I don't know I uh most of the memes uh like I added uh last night so whatever it's OC memes it's huh you are the you are the OC right OC the origal Creator yeah yeah yeah yeah that's great um yeah we still have time if any anything anybody yeah yeah could you that was a great talk could you could you share more about the the um the fellowship application process and how that how you came to that opportunity oh okay so I guess these two guys are the best guys to talk about this but um I had some friends who were fellows so I got to know about the fellowship from them and uh I did some contributions in the execution spec tests and like some of the clients and before that I was doing doing some security research stuff I don't know if that helped or not and uh once the application opened up I did it and uh I when the study group was starting out I just uh like did a small typo fix I'm not sure if that's the reason because I yeah yeah that's why we hired you man I mean PR that no no yeah yeah just to elaborate on it a bit because uh thank you so much for for asking for that uh we are uh going to do the next cohort next year uh now we have kind of a uh a regular schedule where we organize these cohorts June to November so it's five months starting in June with applications opening in May uh but one important thing to mention here that this program is fully open and fully permissionless uh it's like I believe that it became kind of the place to come to start contributing to core etherium but still it's more of a like word of the mouth than uh general knowledge so like please spread a word and uh uh invite other people who uh who might be interested um we are also doing the study group which is more like um uh just St learning process uh it's a there's a Wiki that we collaborate to create uh EPF Wiki all information is there about the study group and so on uh next year we're going to be iterating on it improving it adding more materials and it's something for people to get ready for the fellowship or just like those who are not ready yet who don't have the developer background ethereum background they can take uh like month or two uh of the study group with it for two months uh they can they can study those parts they are interested in and then it's much easier to start uh with the fellowship yeah uh and again like it's fully open so even those people who are not uh submitting applications or don't get a stien can still still work so a bunch of the people that you are going to hear today we are not paid for this we're just dedicated to ethereum we're so excited to to work on this we get them the trip to Bangkok to to award them but still like it's just p edication of many people that's what I love about this community yeah