Distributed Validator Tech (DVT): What is it, how to get involved | Max Sharewood (March 2024)

hello everybody no microphones so I'm going to try and yell at you for the next 20 minutes um yeah this is just a talk on distributed validators uh what they are how to run one um my name is Max Sherwood so I'm a content and Communications manager at obal um on the side I run H2 nodes which is a validator company that I co-founded two years ago um so I'm just going to talk about what is distributed validator technology um give some examples of distributed validators in the wild we're going to talk about the impact of DVS on ethereum and how to get involved uh but first I want to know a bit about you guys so I know this is probably one of the most technical Meetup audiences that exist so who's running a validator or an ethereum node today ah only one guy okay all right um that's good what is a DV uh what is a distributed validator so first let's talk about kind of ethereum staking today um so if you're a solo Staker you know you need 32 eth you're probably going to run one machine either at home or in a data center um yeah you're probably going to look like this obviously uh if you're a professional node operator um typically you'll run maybe up to a thousand validators per machine and you'll have many machines so if you have 5,000 validators you're running maybe you'll have five machines plus two or three backups um and Floren will tell you a bit more about how they do things uh after my talk um and if you're a liquid staking protocol and LSP you're probably manually onboarding node operators um and you know uh not requiring them to give a bond or maybe giving a little bit of a bond uh and you're giving them the the validator keys so that's kind of where we're at today it's all about keys on machines so the problems uh Key Management is pretty risky so even for the pros uh we've seen slashing take place at Bitcoin s at Rock logic at launch nodes uh the latter two were Lio node operators or are Li node operators and those were Lio keys that got slashed uh Stones another lot of note operator they had to rotate their entire set of keys so basically they had to like shut down all their validators and start new ones because they thought they might have compromised those keys uh and this quote here at the bottom is like what everybody says if you're getting ready to run validators they say you know you should risk downtime you shouldn't try to build any backup systems because the risk is that let's say your primary goes down you move the key to the secondary the secondary is running that's fine you've avoided downtime maybe you were only down for like five minutes but now your primary comes back up again you know obviously you didn't have control over the reason that it went down so maybe you don't have control over the fact that it's coming back up now you're running the same key in two places and you're at a very high risk of getting slashed uh because you're double signing you're you're testing twice um all those things you should not be doing so pretty much everyone just says the downtime rewards the downtime penalties aren't that bad so just be down and and you know don't try and build something more comp licated and this is this is true for the pros as well I mean Floren will will tell you more but um you know this is this is pretty much how it works today right so why do we have downtime maybe your internet service provider has a problem if you're running from home uh actually today I had a couple small outages which was annoying um or your data center might go down uh there might be different reasons for that it's not supposed to happen but it can happen it does happen uh you might have a problem with your machine so you might fill up a disc uh something like that you know the Chain's growing all the time if you're not monitoring your your disc the disc might get full and then you you go down um or there might be you know an ethereum upgrade let's say you forgot to upgrade your your node and you know you there was a big upgrade a big fork that went through and now you have problems uh or there could be a bug with one of the clients that you're running right um so you know there's a lot of reasons why things might not stay online 247 and even the pros struggle with this so if you go on the Lio forums and you just Google incident uh these these uh these come up these are from you know recently until 3 years ago so uh you know they've kind of always happened uh these are the professional node operators um and these are just downtime incidents and they're basically uh telling Lio hey look we're sorry you know here's why we were down here's what we did about it to get back up online uh and in most cases they actually reimburse the Lio da for the amount of rewards that they missed and the amount of penalties that they incude uh during that time uh I don't think Li makes people do that but it's just kind of something that people started doing to be professional uh because obviously as a node operator you're not supposed to have downtime everybody gets slashed sometimes um so this is from rated so this is like entities with more than 5,000 validator keys that they're running there's been 205 validators that have been slashed ever from entities like this so these are the pros uh and at the bottom you've got solo stakers and you've got 150 different validators that have been slashed by solo stakers this is on rated. network if you don't know it uh it's a really cool website so pretty much you know whether you're a pro node operator or a solo Staker um slashing risk is real and and people do get slashed so this is where DVS come in uh distributed validators so think about like a multisig right where you need so multisig is a wallet where you need multiple other you know wallets to sign off on transactions it's just like that for a validator so you're taking a validator key you're Distributing it to uh you know in this case as four key shards um and those it's going on to four different nodes and those four nodes are running one validator together uh in this example so it's an ethereum validator or validators run on multiple nodes uh this is a big uh selling point here is that it only requires a threshold hold of nodes to be online so for example if you've got four nodes running a validator and one goes offline the validator stays online so it actually only needs more than 2third so if you've got a six node distributed validator cluster two nodes can go offline the validator stays online if you've got a four node one can go offline Etc um and the nodes can be run by different parties right so if you're a professional uh business you know you could run four or six or 10 nodes by yourself and and uh you know do it by yourself but if you're a solo Staker um you know and you don't really want to make sure you're online all the time maybe you want to go on vacations and not have to worry about being at home all the time to fix stuff you know you get a couple friends together and you say hey let's run a cluster together and then you know if you go on vacation for two weeks and your Note is down the whole time it's fine as long as they're still online right uh I mentioned that we're dealing with key shares instead of entire keys so whereas before if you have your entire key running on two machines that's how you get slashed right now you know with a DV you're only dealing with a key share so if you put that key share somewhere else it's you know less risky uh basically the entire cluster would have to be running duplicated for for slashing to happen so it's it's a lot less risky on that front so great success um let's talk a bit more about the architecture of a node running DVS so Karen is what obal is calling the uh the middleware client that that they make um so it sits between the validator and the consensus client so an ethereum node uh runs on a execution consensus and validator client and Karen is a middleware so it actually just fits in between the consensus and the validator client um the other clients run like normal there's a tiny tiny bit more of Hardware requirement but it's it's very minimal so pretty much it's you know it's a very small difference to a normal node setup um I know that this is a very technical Meetup usually so this is copied from our documentation where you can read more but it's basically a goang based HTTP middleware it's intercepting and proxying API traffic um blah blah blah blah blah and it's communicating with the other uh Caren clients in the cluster and uh the validator client doesn't even know that it's part of a distributed validator it just thinks it's a normal validator if you go on the beacon chain website you click on a distributed validator you wouldn't be able to tell it's a distributed validator it looks exactly the same uh as a normal validator um and at the bottom like I mentioned or I haven't mentioned this actually Karen does not have access to the private key so the private key is still held with the validator client so this is different to you know for example our biggest competitor SSV they built a DV solution that replaces the validator client we haven't done that so the validator client is still responsible for for signing it's still responsible for the key um so you know we're not we're not handling those those key shares so if you take four for example of those nodes you create what's called a cluster um like I mentioned before professional node operators can run their own multi-node clusters or you know for example if you're a solo Staker let's say you don't want to put in 32 eth for your own validator you know you get four friends together everybody pitches in 8 eth and you can also kind of bring down the the requirement uh for the bond um and you can you know Squad stake like that so that's quite nice so um hopefully that made sense I know that was pretty quick uh we'll stop for questions uh in a bit does anyone have any like immediate questions that wasn't clear okay so let's talk about DVS in the wild um we're kind of in this like mainnet beta phase right now so there are Community clusters going online on on on mainnet today uh one of the first ones was run by super Fizz um you know he put out a call on Twitter and said B basically like who wants to be part of my cluster let's run one of these together um I think that was a 10 person cluster where they ran 3.2 eth per person um so pretty nice and uh yeah that's that's live on Main net today uh Team staking is kind of an initiative that was started by Aragon you might know them from kind of the Dow space so they're going to get ready to run a DV with their employees and everyone's going to pitch in a bit of e uh maybe they'll even start running some of the Aragon da treasury in the future that's to be decided that's their decision not ours um but that's kind of cool you know if you think about teams coming together and and running ethereum staking together uh or local communities you know we have we have some initiatives going on in that space uh ether fi which you know from kind of the reaking space um they had something called operation solo Staker where they were um basically letting anyone apply on the website and they were taking people and putting them into DV clusters um they had solo stakers from 35 countries and six continents uh as part of this initiative one of them was actually running in Kenya on a starlink uh satellite connection so obviously that's something where maybe you wouldn't feel comfortable doing it uh you know in a vanilla setup but in a DV setup where you know the whole validator doesn't go down if you go offline maybe you're willing to take that risk um and yeah this did not have any bonding requirements so you didn't need to bring your own eth U they gave you the eth they obviously screened the applicants and stuff so um you know they had pretty high confidence in these people and last but not least Lido simple DVT so Lio has done a lot of testing of oal DVT and SSV DVT um this week actually they're finally launching the first cohort of distributed validators on mainnet uh they're going to have about 140 new node operators when this is done so they're going to like four times the size of their of their node operator set they're only giving a little tiny bit of stake to this module for now uh we'll see how it grows in the future they're kind of you know getting comfortable with this on Main net uh but this is this is important because this is the first time that you know Solo stakers or Community stakers have been able to get stake from Lio and and run as part of Lio um you know without being kind of handpicked by the the curated set which is the way that it's worked until now so that's pretty big deal for obal and a pretty big deal for Lio obviously you know we know that that uh Li is a big part of ethereum so decentralizing Lio is is good for ethereum so let's talk more about the impact of distributed validators on ethereum um so yeah Lio becoming permissionless slowly is um you know probably the the immediate thing we can point to and say distributed validators are are helping to decentralize you know the largest liquid staking protocol on ethereum um there's also new liquid staking protocols that are going to be competing with Lio that are built natively on distributed validator technology so this is Diva staking or ether fi like I mentioned before um ether fi is still kind of in an early stage where most of the stake is in a curated set that's similar to Lio but they're going to start onboarding uh distributed validator clusters very soon uh and like I mentioned before if you're a solo Staker you know it reduces the barrier to entry where you don't need 32 eth anymore uh you can find people to to share that 32 e bond with so this is kind of one idea that I've been playing around with um I'll see how it goes over today but basically like client diversity has been a big topic I'm sure you've all heard about you know this idea that if if the majority of ethereum is running on one version of a client and there's a problem with that client all those validators have a problem so potentially all of ethereum has a problem uh there was an issue with nethermind I think a couple weeks or months back that didn't cause any kind of finality problems on the main chain I think it did bring down about a fifth of validators uh things get a bit spicy when a third or more of validators go offline um but we have seen issues like that in the past where you know the chain has failed to come to penality so client diversity is super important and basically I'm making the case here that like if you imagine that let's say we have five consensus clients right and four execution clients the best case scenario is that stake is divided evenly amongst those right um so what does that mean if we have five that means 20% of stake is with each if we have four that means 25% of stake is with each and if you think about like the worst case scenario which is that there's some kind of mass slashing event as a result of a bug on one of these clients again it's something that never should happen but it's something that you know ethereum does have correlated penalties for slashing so if you're getting slashed by yourself you're going to lose one eth per validator uh but if you're getting slashed at the same time as a bunch of other people you're going to lose a lot more than that so if you're getting slashed at the same time as 25% of validators you're going to lose 25 eth per validator and that's actually almost 20% of all staked eth in that scenario so that's pretty cataclysmic right uh if there's 20% of validators getting slashed then there's 21 eth per validator being lost which is about 133% of stake eth so if you think about you know this being evenly distributed it's still not really a solution in the Doomsday scenario that we have like a correlated slashing event as a result of a bug in one of these clients so pretty much and and this is from uh liquid collectives blog um but you can see it's pretty aggressive so this is like 20% of the network getting flashed for more than 20 eth per validator so DVS can build in client diversity to a single validator right so you have in this example four nodes each node running a unique set of clients uh so if one of those nodes goes offline as a result of a bug in one of those clients the validator does not go offline so the validator is basically protected from you know issues with a single client if there is a degree of diversity built in to that to that cluster uh same thing goes for Geographic diversity right if you run all four of these nodes in the same data center the data center goes offline your validator goes offline but if those are spread out across different data centers uh you've ALS or you know at different people's homes uh you've kind of removed those single points of failure so you can really build in client diversity and and resilience into a single validator or a set of validators in a way that you haven't been able to do before so if you imagine if the entire ethereum network was run on distributed validators that were properly uh you know Diversified within those clusters then ideally you know if there was a issue with a client like no validators would go offline at all so that's kind of why I'm arguing that DVT is the the end game for client diversity that's kind of my personal take that I'm playing with um so yeah that's that's kind of an interesting one obviously Karen the middleware client if everyone's running Karen and there's a problem with Karen everyone has a problem that's why actually this is the nethermind logo here so we're working with nether mind already to build a second uh DV client implementation ideally there's four or more um so yeah DV client diversity coming soon uh but right now it's just Karen so hopefully that kind of got you a bit interested maybe you've thought in the past about running a validator but you've realized that it's too much hassle or you know you don't want to put 32 eth into it so maybe hopefully I got you interested how can you get involved uh just generally speaking the steps to run a DV is uh you're going to find other people to run with you're going to exchange addresses with them uh you're going to select one person who's going to be responsible for creating kind of the cluster configuration um we have a a repo with uh some Docker stuff in it where you can easily generate an enr so that's kind of the identity of your node and you share that with the person that you said is going to create the cluster um so he gets the enrs and the addresses from all the people in the cluster and then he can use our our Launchpad to create the cluster configuration he just pastes all the addresses in there um Everybody signs the cluster configuration says they're happy with it uh everyone runs a dkg together the dkg makes sure that the entire key never exists in one place so you can take an existing validator and split the key but you can also generate the key shares um with the dkg so that the entire key actually never exists in one place uh that's the safest way to do it uh and then you sync your nodes you deposit to the validator like normal and you're up and running uh I mentioned we do have a a launch P it's actually built by the same guy that was helping to build the original staking Launchpad so that's pretty cool Chris battenfield um and yeah it's pretty easy to use so one more reason to get involved is we're starting to give away these things called technique credentials basically this is like saying that you've run as part of a DV you've achieved good performance and you can take that to you know LI or other sing pools or even other people that you know might be uncomfortable with running uh with you if they don't know you um and this is kind of a you know just being able to say look I've done this before I I know what I'm doing I've achieved good performance uh on testet or or on Main net blah blah blah um yeah that's the talk uh hopefully I've left some time for questions uh thanks very [Applause] much yeah how many of these notes do you recommend running it must be like something like you need three out of five or four yeah so the threshold is like more than 2/3 need to be online so that makes it nice for clusters of four seven and 10 you can run anywhere between like three and like I don't know what the upper limit is three we don't recommend um you can find it in our docs like there are certain numbers that go well but it it could be any number between three and I don't know what the upper limit is uh but yeah I I think we recommend like four seven and 10 yeah back there so what's the benefit of huge distribute with clusters in comparison to say five yeah I mean huge clusters like the more nodes there are I guess the more like traffic there needs to be so we've done testing at scale with um like three four and six node clusters and the four and the six node clusters do pretty well we haven't done a lot of testing with bigger clusters to be honest um I don't think you should go out and run like you know 20 node clusters or something like that um so yeah I think the sweet spot's somewhere between like 5 and 12 right now but we need to do more testing yeah anybody else yeah yeah I'm just curious are you aware of any estim approximate probably of the real number of validators I mean uni entities that are running distributed validators or yeah yeah I mean not the organization like 500 validators but the real number enties in total on ethereum or the ones that are running oh yeah that's hard to say I think there's like 4,000 nodes on ethereum they're not all necessarily staked so they're not all necessarily validators there's like almost a million valid validators but like you said you know there could be people running like 10,000 validators so it's kind of impossible to know like how many businesses or people are running validators on ethereum um but it's probably less than 4,000 because that's the number of nodes good question though it's Al also totally a side question there but what's like the percent of Revenue right now when you to run a stier in general to be profitable on ethereum or no just like not well I guess it will always be profitable with I what what's the amount is it like I just if you st 20 302v like what's what's the expected return rate is it 10% 5% is it more um yeah the apy on ethereum right now is like 3 and a half% per year uh if you're solo staking you might only get like one to three block proposals per year you can be lucky and get more than that and get more apy or you can be unlucky and get less and get less apy uh apy depends if you're running me boost or not um so it's good to be part of like a staking pool where you can like Smooth out the the you know block rewards um that's that is one problem with DVS is like if everyone's putting in like less and less eth per person then they're earning less and less so if you went out and bought like a $2,000 computer and you're only running staking on like like we said earlier like 3.2 eth then it's going to take longer to pay that off as if you were running than if you were running like 32 e by yourself so Lio is actually aware of this uh sorry if you have epilepsy right now um I don't know if I can zoom in here but Lio is actually going to give us or US but give the people running on simple DVT 8% of the staking reward so the way that they used to do it was they used to take 10% of the staking reward and split it 50/50 with the node operator so now they're taking 10% and splitting it 8020 with the node operator because they are aware that like each validator is not not being run by one person but it's now being run by like seven people so they are trying to make it a bit more profitable because yeah there's too many people running in a cluster then obviously each person makes less money per validator right uh I wanted to ask you you probably know about what what are the flashing events like what's most commonly because I mean the return on E now is like below 4% so uh if it's a big corporation who's running a lot of nodes and for them like how is like say a node goes down for a day or they like double sign it what would be kind of a regular like how much would be the slashing yeah so the slashing events in the past they look bad but like you said these businesses are so big that it doesn't really impact their bottom line as long as it's not one of these correlated slashing events so I think some of the ones that I gave examples of they were like 20 validators got slashed for one eth per validator so it was like 20 e in total which you know is quite a lot of money for you or me but if you're running like 5,000 validators doesn't really make a big dent in your bottom line the real problem is like everyone can always say like oh you're you're a business that got slashed right so it's probably like more impactful to your kind of reputation than it is to your actual Revenue um but yeah I you can go on you can go on rated and and look at all the the previous slashing events that have happened um which is kind of interesting I don't know how much time we have here but um definitely like have a look at rated. Network if you if you aren't aware of it because it's pretty cool so this is like slashings over time so I guess this is a week a weekly no is this it's always hard to tell yeah this is monthly so so yeah this month there's been four validators slashed last month there were eight eight 106 so that was probably one of the big ones but you can actually see okay so Bitcoin sweets was 99 stake.

us991 launch noes 20 Rock logic one but these are usually like one eth per validator getting slashed um so yeah I mean it would suck to get 99 eth slashed but if you're running like 15,000 validators you probably make that much in because comp are kind of not very friendly for solo sers because if you're running for CH and you get slashed I know like then um like probably any kind of slashing will kind of like make your Revenue like go to NE yeah yeah also when you get slashed like the validator is like forcibly exited so you stop earning rewards you have to go into the exit Q um so you're going to miss out on some period where you could have been earning rewards as well so it's just a bad time would not recommend um so apart from slash did you have any comparison between running four or six different note than going offline for I don't know few hours few days yeah so if you are going offline like basically the way it works is you're losing money at the same roughly the same rate as you would have been making money so let's say you're online for a week then you're offline for a week now you're basically back to zero so you'd have to be online for another week to get back to where you were before so it's not that brutal like if you're online all year and then you're offline like a few days it's not really going to make an impact in your bottom line to be honest which is why everybody says just be offline don't even try and build anything that's going to risk getting slashed because that's that's way worse right um yeah that's that's kind of the tldr but the next Talk's also slash uh staking related so I don't want to you know there there will be another chance for staking Q&A yeah so you mentioned one of the biggest um hurdles for solo for soloers or or people would like to Sol steak is the to eat um but the other I guess the other hurdle are the technical requirements and kind of fears around that and do you see any um any future in which the the technical requirements or the software stack is going to get a little more user friendly in this BBT envir less less less technical knowledge yeah so the um I've lost my slides here but the the uh the team staking thing that Aragon is doing doing is interesting cuz they're actually using dap nodes um so they're just ordering these little Intel Nook PCS and sending them to all their employees and then they're going to be using the obal package like in the DAP node store uh to run DVS so they're not going to touch any CLI they're just going to do everything in a UI um we'll see how it goes actually it's happening next week um but uh yeah I think we're slowly getting there um it's good if you kind of have someone looking over over your shoulder that is a bit more technical if you don't have any technical friends like you might struggle even with some of these UI based tools um sterium is also quite good but requires you to have a machine that you can like SSH into so you probably need to know like how to make an account at a bare metal provider and uh do some basic stuff with IP addresses um if you can do c stuff at a basic level you can probably use something like eth Docker which like installs all the clients and just gets them all running you just do like Docker compose up basically and click enter a few times so I'm kind of at that level where I'm like technical enough to like use eth doer and stereum and I kind of know what I'm doing but if I had to set up like each client individually it would probably take me like two days of like Googling um but yeah we're we're getting there slowly but it's it's still it's yeah it's still a little bit of a technical exercise yeah I guess in a couple of years it'll look different and and I guess the success of YouTube is also going to depend on on the visibility yeah yeah I mean there's also like white label nodes like all nodes for example is super popular they'll kind of handle all this stuff other than just like the basics of of getting the validator running so there are kind of these halfway Solutions and yeah our job should be to like make sure that DV DVS are integrated into those yeah all right I think I'll leave it there yeah cheers