# Protecting Validators From Client Bugs Using Vero

- Channel: [Dappcon](https://streameth.org/dappcon)
- Date: 2025-06-18
- Duration: 38:09
- Watch: https://streameth.org/watch/6852e1b190bd41297bac763f

## Transcript

Hi everyone. Thank you for coming today. Hi Guillaume. Yeah, so my name is Luka, for those of you who do not know me. I am the CTO at Serenata, where we specialize on staking Ethereum. And today we will be looking into how to protect validators from client bugs using a software that we developed and open sourced at Serenata, which is called Vero. So there will be an active part to this workshop as well in a few minutes. In order to actively participate, you're going to need a Linux machine with ideally Docker installed and about four gigs of RAM. Less should work as well, but four gigs would be recommended. So let's dive a bit into the background behind Vero, why it exists, what it is, and so on. The reason why Vero exists has a lot to do with something called client diversity. It's an idea in the Ethereum space where Ethereum doesn't want to rely on a single client implementation in order to keep its blockchain going. So the idea is we should have multiple clients that all track the same blockchain. And that way, Ethereum is much safer from these client bugs, because if one client has a bug, we don't want the whole blockchain to stop going. So that's a quick intro to the idea behind client diversity. It's much more important to the whole system than many people think or are aware of. There's a good article by Dan Kratz-Feist on his blog or his website, which goes into the importance of client diversity. So yeah, that's the recommended read. So currently, thanks to this whole idea and making it a priority for Ethereum, we have six available consensus layer clients and five clients on the execution layer, which you can choose from, and you can basically choose any of those. Ideally, that would mean that each of those clients would be run by a proportionate number of the network. So in case of the consensus layer clients, we'd be looking at about 70% of the network on each consensus layer client. And in terms of the execution layer clients, we'd ideally see about 20% of the network on each execution layer client. So that's the theoretical and ideal scenario. The reality is a bit different. On the consensus layer, we see some clients are being used much more than others, especially Prism and Lighthouse, which we see on the top here, are used by a lot of people, probably a bit too many. This picture is not completely up to date, like Lodestar is shown as 0.1 to 0%, but it's higher than that, as well as Grandin. But it's like a roughly accurate picture of things. And then on the execution layer side, it's a lot more uncertain. All those orange blocks in the middle that you see, those are areas of uncertainty. So we don't know which clients some people are running. But this situation on the execution layer side has improved a lot since about a year ago. So we are currently in a much better space in terms of client diversity than we were that one year ago, especially Nethermind has made a big rise in terms of adoption. So now that we're in this state where all the clients are likely used by less than two-thirds of the network, are we out of the woods? Are we good now? Can we stop worrying about client diversity, right? That's the question. And I'm here to say that we really are not quite out of the woods yet because of multiple factors. One of them is the client usage data. Like I showed on the consensus layer side, it's not 100% accurate. There were some changes in the most recent network upgrade, which made measuring this less accurate. Then there's also the fact if a client is used by more than 33% of the network, so one-third, it can still be dangerous in some edge cases. Then there's also the possibility of two clients sharing a very similar bug like we saw recently on the testnet. So if multiple clients share the same bug, they can still get over 66%. And then one more thing is kind of under-discussed, which are failover mechanisms. Many node operators run multiple client combinations in a kind of failover mode where if one of their clients goes down, they switch to the other one. And this could do the wrong thing, actually, because if you run these two clients and one starts following a different chain, you don't necessarily always want to switch over to another client as soon as something's up with the first one, unless you know what's actually going wrong. So that's the failover mechanisms. As a node operator, especially as a professional one with like a responsibility for other people's money, you want to avoid these client bugs as much as you can. But the hard truth is we cannot completely avoid client bugs, right? These are complex pieces of software. They're going to end up containing some bugs at some time. Ethereum is constantly evolving, like clients are optimizing, so there's always some room for bugs to be introduced. But we can reduce the impact of these bugs. And the simplest way to do that is just by pointing some of – like by using multiple clients, right, and pointing some of your validators at one client combination, some of your validators at another, and so on. So in this case that we have on the slide here, we have three client combinations, and we basically split up our validators among these three combinations. The pros of this approach is that if everybody were to do this, we would be in a very good space. So it's good for a network if you do this. It's a simple approach. It's relatively easy to set up, and it still reduces the impact of client bugs. But in this case, if, say, one of these client combinations does have a bug, still like a third of your validators would go offline. So there is some kind of impact, but it's not huge. It's pretty good. Then there's a couple of more sophisticated things you can do to protect validators against client bugs. One of them is using Vouch, which is a specialized validator client. It's a multi-node validator client, which means that it can talk to multiple clients at the same time. So in this case, it talks to like three consensus layer clients and three execution layer clients at the same time and combines their output. So if one of them like has a bug, goes offline, it can keep going using the other two. This software is used by a lot of professional node operators. It's been around, so it's been battle-tested. It's a really solid piece of software. It's highly configurable. You can do all sorts of things with it. It's open source, also nice. The only cons I can think of is that it requires its own specialized key manager. So that key manager which contains the keys, it's not compatible with other software. So yeah, basically Vouch requires Dirk in the same way, the other way around. And the other con is actually the same as the probe that is used by many professional node operators. The reason I mention it as a con is because all of those professional node operators use it. A lot of Ethereum validators use it. And if Vouch were to contain a bug, you would be offline with many other validators, which would, well, it could just cost you some money by being offline at the same time as they are. And you're basically forced to wait for a fixed Vouch release. You can't just switch to a different client or something. Then there's the other approach which is called DVT, which stands for Distributed Validator Technology. One of the implementations is from SSV. And with DVT, you basically run a single validator out of multiple places. You kind of split it up. And yeah, so in all those places, you can use a different client combination. And again, you can this way protect validators against a single client bug. So, moving on to the next DVT implementation, which is from Obel. It's called Karen. The high level idea is the same. So, you again split up the validator, can use multiple client combinations in each place. In this case, Obel's Karen software has a business source license, and it has a bit of a nonstandard middleware approach. So, again, not interchangeable with anything else. So, all these advanced approaches to avoiding client bugs have something in common, and that is that they're kind of their own way of doing things. They kind of lock you into their own ecosystem, and it's hard to switch to this advanced solution. It's also hard to switch back from it. So, at this point, the question arises, can we like get to the same results? Can we avoid client bugs without all the complexity? So, yeah, that's when I thought we should write a new validator client with some design goals. One of them was that it should be able to talk to multiple at the same time in order to avoid these client bugs. The other thing I wanted to do was keep it simple, which is always good when it comes to software security. Obviously, this is a validator client. It's closed to private keys. So, you want it to be secure. I want it to be observable. So, I want us to have a good idea of what's happening, when it's happening, why we're having issues, to make it easy to investigate any issues. I also wanted it to be compatible with all client implementations, and one important thing for me was making it compatible with Web Designer, which makes it then much more compatible with the rest of the ecosystem. So, that's how we came up with Vero. It succeeded in terms of all of these design goals. It's a multi-node validator client. It's simple. It has less than 4,000 lines of source code. It's secure. It never touches validator keys directly. It's highly observable, and it's compatible with all the client implementations, as well as Web 3.0 Designer. So, when it comes to a high-level overview, where does Vero come in? So, this is an example of, let's say, a more sophisticated setup with multiple clients. We have the execution layer client, the CL client, the validator client, and then a remote signer, which is connected to its slashing protection database. So, Vero is basically the same thing as all of those validator clients in the middle. You can just swap it out easily for any of those in this kind of setup. So, when it comes to Vero being multi-node, that can mean multiple different things, but in the context of Vero, it basically means that it talks to all of these client implementations at the same time, and it follows the chain based on their combined output. So, if we say you have four client combinations, and one of them has a bug, goes offline, Vero is still able to follow the client combinations agree on. In terms of observability, it has readable logs that you can follow. There's dashboards for Grafana that you can simply install. The left one is just a high-level overview of what Vero is doing, but it also offers you the ability to dive deeper into at which point in the slots are certain actions taken and so on. Vero also supports exporting traces, so you can really dive deep into, this is a blog proposal, so in case you miss a blog proposal, you can really tell at which point it actually went wrong. One novel feature that Vero has and that has only recently been started to introduce in other clients is slashing detection. Vero listens to slashing events happening on the entire network. If it detects any slashing event for a validator that is managed by Vero, it automatically shuts down its validator duties. That way, it prevents shorter validators from being slashed. This is what it looks like on the dashboard. This is under normal conditions. No slashing detected. It's performing duties. As soon as any of its validators are slashed, it stops all activity. With the new validator client, some people are worried about its performance. Vero currently powers thousands of validators, so we've made sure that it performs well. We're currently among the top on Ethereum in terms of validator effectiveness as well. If you're worried about Vero's performance, you don't necessarily need to be. In terms of risks, I've talked about the good things about Vero, but maybe there's some bad things too, right? I tried to be objective here. There's a few possible vectors that you could be looking at, like slashing, but Vero uses the battle-tested slashing protection that Web 3.0 uses, so that shouldn't be an issue. There's also the slashing detection feature that I mentioned, so it's kind of better in some ways in terms of slashing protection. In terms of validator key compromise, Vero never has access to validator keys, so you don't have to worry about Vero leaking those. In terms of bugs, obviously, every piece of software can have bugs, but this is a small code base. It has high test coverage. It's very simple, like I mentioned, so that should hopefully help us avoid bugs. There's no lock-in. You can very easily switch to a different validator client at any time. And in terms of downtime, yeah, I mean, it's possible that Vero has a bug which leads to downtime, but in that case, you can just switch to a different validator client very easily. One more question, is Vero for professionals only? It's not. You do get most of the benefits if you run it with multiple beacon nodes and so on, but you can also just run it with one beacon node, one client combination, so easy enough. Nixa wants to ask a question. Mihika, could you pass the microphone? Thank you. You said you could use it with one machine and one client combination. Why would you use Vero for one client combination? Is there an advantage there? If you like Vero's metrics, for example, they're more detailed, I think, than other validator clients for the flashing detection feature, which some don't have yet, so there are reasons to use Vero even with a single client combination. Okay. Thank you. Any other questions so far? Okay. So, yeah, there's a lot of good things about Vero, not that many bad things, so I would encourage you to try it out. In terms of what's next, we're working on supporting the next network upgrades and also considering adding doppelganger protection. The end goal is basically for people to start running Vero in these kinds of setups where they use a lot of different clients around Vero, and that way they have a lot of different clients' view of the chain that they can use to make an informed decision on which chain to follow. All right. Let's move on to the active part of the workshop. So, has anybody in here set up a machine that they can use? So, yes. Yeah. What would you be on? You're going to join the active part? Okay. It requires Docker, so just follow me then. Okay. So, look in case you want to follow it, this is a website which lists all the commands that we're going to be using so we don't have to manually transcribe. This is what it looks like. So, you just go to workshop.serenity.io and you can just copy-paste the commands. So, we're going to get started with this. Oh, right. One more thing. This is a high-level overview of the setup we're going to be using today. So, we're going to be using three execution layer clients, three CL clients, and Vero with the remote signer. But to set all of this up today would be too much work, so we're going to start off by only setting up Vero. And I've already set up the rest of the infrastructure for us. So, it's time to begin. The first command that we're going to run is this one. It basically starts Vero and points it at the beacon nodes that I've prepared. So, yeah, let's do that. We're going to copy the first command and let's paste it in here. And what this does is it runs the validator client on the Hudi network. It uses this remote signer and then it uses these three beacon node URLs. The last thing is just setting the fee recipient, which is required for Vero to start. Okay, almost. I'm going to need sudo here. So, it's going to pull a Vero image. So, all right, and we are basically live. You can see the first slot ticked off here, and we already published some attestation here. It's as simple as that, if you already have the beacon nodes running. So, yeah, you just need the attestation of the beacon nodes, a remote signer, and that's it. How are you doing, Luca? Yeah, it's working? Perfect. Okay. All right. So, that's not the end of the workshop. This was the simplest way to get Vero running, right? If you only need this single piece of this architecture, then it's pretty simple. We're going to go one step further than that today. I've got a question before we go. Okay. So, now we have run the same command, and so what are you using to not flash us? Just remote signers, a flashing protection database, or something else? Yeah. So, these are the locks we've seen, and that's what I wanted to mention. We're all using the same validity keys, because we're all using the same remote signer, and that remote signer has its own flashing protection database, exactly. So, even though we're running the same validators, we're not going to get flashed, because there is a flashing protection database in place. So, yeah, we could get into a point where we try to attest to different things, but because of that flashing protection, one of us would fail to do that. All right. You can now stop the process. It'll take a few seconds to stop, because Vero has this shutdown mechanism where it waits for any pending duties to be finished. All right. So, in this next part, we're going to go one step further. Besides Vero, we're also going to spin up the remote signer and its flashing protection database. So, the next commands to run are basically, we're going to be using e-Docker to do all of this, to make things easier. So, we're going to clone the e-Docker repository and, yeah, make some changes to it as we go along. So, we're going to run git clone. This one. Changing to the directory. We're going to make a copy of its default environment file. And then we're going to make some changes to this file. I'm going to be using Nano here, but you can use any text editor. And the changes that we're going to make are the following ones. So, we're going to change the value of the compose file. So, it's this one. And we're going to need to set the fee recipient value. The network should already be set to hoodie, so we don't need to change that. We need to enable web3 signer, which is right here. True. And then there's one more thing. We need to change the CL node setting. So, that's the one that points to these different beacon nodes that are already running. So, that one's all the way down. Where is it? Here it is. So, we're going to remove the original value and paste the new value. And I think that's it. So, we set these. We're going to save the file. How are you doing, Luca? All set? Okay. So, now that we have changed the environment value, we can bring a Docker up using this command. This will take a second as it downloads some Docker images again. So, we're working towards this. Setting this up. Veril and web3 signer. How do client updates work? Do they get pulled automatically, or do you have to wait for Veril to incorporate them or anything? They work automatically. So, e-Docker automatically pulls the latest version. So, as long as Veril tags something as the latest version, then e-Docker will automatically take that. Mm hmm. Still waiting for some images. You done, Luca? Also waiting? Okay. So, if anyone happens to be watching online, maybe you can send me a message if you want to actively participate because you're going to need a special thing soon. So, you can DM me at serenata underscore Luca, L-U-C-A. Still pulling here. So, it could be specific to what e-Docker does. It likes to build its own images. Like, even if there's an official image, e-Docker builds its own version sometimes. I think it has sometimes changes permissions and stuff like that. So, yeah, as you can see right now, it's using the web3 signer base image, and then it's going to, like, perform some other things on top of it. Right. Okay. Now I have to use a mic. Why? Yeah, like, it's the web3 signer itself that's taking, like, that's downloading and rebuilding all the layers? Or is it just that the network is slow? The network is a bit slow here, I'd say, yeah. That's... Okay. Okay. Yeah, but also, every spider is being built, I see it in my logs, so... Okay. Yeah, but... And the signer itself is very big, big e-Docker image, if I remember correctly. Okay. Understood. A lot of these e-Docker images are a bit bloated. Yeah, that's fine. But it looks like we're almost there, so... Did you manage to build it, Luka? Yeah, it's exporting, so I'm pretty much done. Okay, looks like I'm done, too. So, e-Docker is spinning up all of these services. This looks like it's done. So, at this point, we ran ECD up, and now we can run ECD logs to check the logs from the validator service. So, if we do this, we can see that we have a lot of logs. We can see Vero starting up. So, these are all the... It's a little hard to read. Let me just make it a little smaller. Hopefully, this is still big enough. So, yeah, it's started up, and the only thing we can see is that we can see some warnings in the logs, which say that there are no active or pending validators. So, it's actually not performing any validator duties. Like, the validator client is running, but it's not doing anything because it doesn't have any validators attached to it. So, that means it's now time to... We already did all this. So, yeah, we're seeing these messages warning us that duties are not being updated because there are no active validators. So, we're going to be adding some keys now. We're going to be using these magic numbers here to import keys, and, yeah, that will... Like, we will import validator keys into the Ethelker setup, and then Vero will perform the duties that those validators should be performing. So... That'll look somewhat like this. We're gonna enter a magic number here. I'll get one over here. This is your password. So, this magic number will allow you to download some validator keys. So, if it downloaded successfully, you should see, like, this kind of output. If you see an error like this with a 404 status code, then it didn't download successfully. Did you manage? Okay, perfect. So, now we have a zip file that contains all these keys. We're gonna extract the keys into the correct directory for Ethelker using this command here. So, you should see the zip files being inflated, and next we're gonna say if the keys import, and this will import all of the keys that are located in the directory that we just extracted them to. So, this will take a minute. And at this point, Ethelker is asking us these IO warnings. Yeah, if you're gonna do this, like, for real, you should read all these warnings. But yeah, it's basically saying, it's warning you, if you've used these keys before, make sure that you turn them off and that you're using a slash and protection database and so on. So, we understand these warnings. We say yes. In this case, all of the validator keys have the same password. So, we're gonna say yes, and we're gonna enter the password, which is just workshop. So, Ethelker is gonna be importing all of these validator keys now. We're at this step. So, at this point, all of the validator keys have been imported. We're gonna run ET. I think it's restart. Yeah, restart validator. That will make Vero refresh its validator keys. And if we now run the locks command again, oh, then we're gonna see Vero's logs, and we're gonna see that those warnings are no longer there. It's not warning us about anything anymore. It's actually updating duties now, and you can see already some attestation has been published here. So, it seems to be working for me. How about the other Luca also? All good. Perfect. So, yeah, that's the logs. We can also dive into metrics, but it's a bit more complicated, and I don't wanna keep you too long before the panel with Vitalik. So, I'm gonna show you how it's done, but we don't need to necessarily go into the details because it also uses some other Docker images, so we'd be just, like, waiting for a while. So, the way it works is you add Grafana to your East Docker Compose file. You adjust the share IP in order not to share Grafana with the rest of the world, and then you just run ET up again. It will pull some Docker images, and at some point, it'll be done, and then you can open a local host at port 3000 in your browser, and you'll have a Grafana spun up, so you just log into that. The default credentials are admin, admin, and then you click on dashboards, and you'll find a Vero detail dashboard. You select which instance you wanna follow. You adjust the time frame, and you basically get these nice charts that show you how Vero is doing. Yeah, for anyone interested in traces, you can, like, talk to me, and I can show you, like, what those traces look like. It's not possible to set it up as part of eDocker right now, so. Okay. Do we have any more questions? I didn't get the part with the traces, so if you can't get them from Docker, where do you get them from? eDocker doesn't have any, like, way, any endpoint where you can send traces to. Right, but, so, clearly, you can extract them, because you can show them yourself, so where, how do you extract them? You just add an environment variable to Vero, you, like, that contains the endpoint, where it should send traces to. And, yeah, we use Tempo for that, but, yeah, Tempo is not part of the eDocker stack right now, so. All right, that's all for today. If you want to, like, learn more or keep in touch, these are some contact details and locations where you can find more information. Thank you for coming, and I also saw that Ladislaus is watching online, so, hi.