"It's 10pm, do you know where your mnemonic is?" | Devcon Bogotá

foreign [Music] hello again so today we have the first panel of the day here in the tropical Sun City scenario so I'm very happy to introduce you to Adrian Sutton ocean kind Paul honer and Vasily chapovalov [Applause] yeah they are going to talk about it's 10 pm do you know where your mnemonic is foreign thank you very much everyone as some of you that can count might notice we're one short and a panelist but uh I think he'll join us um momentarily so I think we'll more or less kind of kick off and do a bit of intros and stuff beforehand um oh there's the man himself visily uh okay um probably with slides sweet so today we have a panel on validator harmonics and their and their security uh it's been almost two years since the beacon chain launched and there's quite a lot of people that haven't touched those private keys in quite a long time um over the course of the next 50 minutes we're going to look back at you know what's been done on the key security side for the last while we're gonna see what things are looking like right now in terms of validator key security and then we're going to look a bit forward towards withdrawals which I'm sure lots of people want to know about but before I get to that I would like to do a few introductions uh will this gonna work for me probably not okay I know it is um oh yeah now it's connected three so briefly I'll introduce myself my name is ocean kind I'm co-founder at oboe Labs we build distributed validator technology which is the ability to run one validator across multiple machines by multiple entities um and yeah that's my little quick intro um I'd like to introduce Paul next hello uh my name is Paul co-founder of Sigma Prime I've been working on uh Lighthouse which is a ethereum consensus implementation since 2018. um yeah I've done a lot of the initial development of it and now I do a lot of kind of overseeing and reviewing and still some development I'll pass it on to Adrian hi I'm Adrian Sutton I'm with consensus and I started out on the basic team doing the execution layer side before there was an idea of a consensus layer side I spent about a year and a half pretty much full time on Basu and then switched over to techu because I heard that Beacon chain was the new hotness and teku is a consensus client so I've spent the last oh I don't know three to four years uh helping build that and bring the merge to fruition it's been a lot of fun been working with people like Paul and a lot of fun there um so yeah and varsity hey um I'm Russell Tech lead at Lido also which is probably more relevant to to this panel I used to be CTO in P2P validator uh so I know anything to do about managing keys for validation so wait uh awesome so we're gonna like test that and figure out some of the good bad and ugly about managing uh private Keys um so yeah so the first thing I want to do is look back a little about how it started and figure out you know some of the basics about what is a mnemonic and stuff so I thought Paul would you explain to us what a mnemonic is oh yeah sure so I'm not sure I can get a textbook definition but a mnemonic is a series of words a phrase um say 20 12 to 24 variable length I think it's I think 24 is a standard in in F2 so a bunch of words easy to remember supposedly um and then from those words you can you can generate any number of uh validated keys so there'd be signing keys that would throw a keys so the premise is that um once you generate this um string of words uh then you write those string of words down on a piece of paper uh and then um you use those words to generate your validator keys and you put those validator keys on uh on some Hardware on on your hot machines um and then you only keep a copy of the mnemonic um on paper um so that yeah so that in the future if those keys were to be like if those boxes exploded disappeared you could use that mnemonic to generate exactly the same Keys again three thank you very much and I wanted to dig in a little bit uh you mentioned withdrawal keys and hotkeys I was wondering Adrian could you tell me the difference between them and do they have to have the same seed phrase uh yeah so the the hotkeys that we talk about are your validator keys and they're the ones that you will have used and seen so far um in dealing with ethereum and if you're running a validator so they're the ones that are on your node and your your validator client has to actually load them into memory any kind of secret key that's loaded into memory and hot on a machine is not as secure as something that's locked up in a safe so they're at a higher risk of being stolen you still do your best to to secure them and secure the machine obviously um but we wanted to make sure that there's a a big difference between those hotkeys that that run your valdern sign things and the keys that actually control the money and and whether you can re withdraw it so from your mnemonic by default you would generate a validator key using One path and as Paul said you can generate multiple keys so from a different path you get a separate validator key and it's that validator key that you will use once withdrawals are implemented to set up withdrawals and be able to to actually recover your money again do they have to come from the same mnemonic no they don't you don't even have to use a mnemonic you can just generate the keys directly without going through the mnemonic process but typically it's a lot easier to back up one mnemonic and not lose it and etch it into steel or whatever it is and store it in a safe well then to have multiple keys that are you know being managed and able to be lost because if you lose your withdrawal key that's it it's game over you're not getting your 32 weeks back yeah that's kind of one of the things we're going to get to a bit further on so as you mentioned in the seed phrase thing um a lot of people that are homesteaker would have used the E2 staking CLI that set up you know one 24 word mnemonic that had both your withdrawal keys and your hotkeys um it's unfortunate we were supposed to have one more panelist Jim McDonald and he's done a lot of work in this specific problem but uh the reason that this thing is titled is 10 pm do you know where your mnemonic is is because some unfortunate people have had these mnemonics compromised over the last two years and I was wondering if you guys wanted to kind of explain what uh Jim's proposal is for for how we kind of straightened this out but yeah what's what is the problem if somebody has your mnemonic well so basically if you used these the the same amount to generate the value data key and withdrawal key and withdraw credentials um which is most homesteakers users but it's um much less often happening for uh big operations where it's usually different Keys more owned by different people so if it's the same the same mnemonic it was used to like what is going to happen uh is that now the um the malicious person who compromised the key they can um like you're in a race to who can slash the protocol like Clash Royale data first which is if your small validator is not like really a bigger problem you stop validation you get very small fine uh so let's selection penalty and it's not that involved but withdrawal key compromised means that they can um when when the validator exits and withdrawals implemented in the human protocol upgrades allows to either to get back to execution layer they you're in a race to understand who gets the money and if you're almost 10K you probably lose the race versus like a craft crafty audio sorry so we will uh follow up a bit more about that later on the talk but I wanted to ask a bit about light over silly and specifically the withdrawal key setup you guys use the Genesis so not the most recent one could you tell me a bit about like how Lido originally set up with withdrawal keys uh yeah a long story short the initial I think about like 15 or 20 percent of steak that Lido has has been deposited on a threshold uh signature uh on uh zero x0 type if I'm being exact for for people who knows basically uh sharded credentials that are shared between uh 11 people uh very well known in the industry who when the time comes when draw was implemented in rotation of his raw credentials implemented they can issue a common to rotated though to the Lido smart contractors so yeah that was like we recorded it for half a year and uh uh we we've had a couple audits on it so like it was like hard to make uh because at the time there were no smart contracts or like even execution like a credentials option for uh for deposits hey thank you for teaming me up for my next question which is to talk a little bit about where things went wrong with the ox0 ones and there was like basically well there's your hotkeys and your withdrawal keys and there's like two different ones I wanted to touch on where there were problems the first one is stake.us and the hotkey issue does anyone explain what happened it's quite a while ago it was only two two years um there was 175 validated slashed because they were running the hotkeys in two different places [Music] quite quite simple to understand so stakehold is a big sophisticated or not operator uh they they run uh like I think uh maybe billions maybe hundreds of millions of stake a lot of them and they have sophisticated setups so like one of the the feature they used in in multiple protocols was they uh had implemented something like like in a something that in traditional world would be called like web application firewall so basically it's an implemented software that ran between the node and the network and it should have cached caught the the double sine events and they implemented for S5 understand for multiple blockchains including them and put out put it up uh for for the Indian validation uh setup and ran multiple uh versions of the notes uh with like with the same private key that we're making blocks in parallel and this was usually caught by the uh the system um I'm not like 100 sure that because otherwise it could be it could have been that they accidentally deployed it like uh to the two things in Parable not not like at the as the usual setup but anyway they did uh they did have uh an automated system for liveness that at times could have run multiple validators with the same private key which is uh usually lead to slashable offense in in protocols including them and they had a system to uh that should have caught the the problems which is like usually people don't do that like right um in in not operator setup because uh liveness is not as important as safety uh slashing for for lack of for so for for missing a few blocks uh much much much lesser than uh slashing liquid penalties for for double sign it's like not worth it usually um so um that system didn't work and uh they for about like 40 minutes or like about an hour they had issued blocks in parallel and they were like really lucky that terrarium system is designed like a design because like they had I think about 70 million dollars taked or something like that at this time uh and they only slashed uh like 15 valid data so like or 27 I don't like really like rookie numbers right so um and that's the case because uh not a lot of validators had uh had issued the invalid block solid stations because um it's not uh like you you don't run the with your stake on a single note like in many other systems otherwise they would have like five percent slash or like 15 slash which is like much much higher and uh uh only after that they uh uh monitoring and the Learning System had triggers so like they're they're in this uh with this problem for a long time so there had been some problems in there as well I think initially described the setup is sophisticated I think that was a very kind way to refer to it I would I would call it dangerously over complicated but yeah I agree I think that these setups where you're running multiple instances of the same key in any place I think you you really like Wesley said you're just prioritizing liveness over um safety and it's it's I think it's really bad business it's not it's not good economically um it's really bad for your reputation uh and it's bad for the network as well so I really couldn't advise against it any Stronger Yeah probably the interesting thing particularly pre-merge I think is probably a little less likely post merge now but pre-merge you could run validator keys in two places and get away with it for a surprising amount of time until you proposed a block or until your node was just slightly out of sync or you just got that little bit unlucky because both validators if you're staying in sync and are testing perfectly they'll create the exact same attestation and that's not slashable so it's incredibly dangerous and you think you're okay for quite a while until one day you wake up and you've lost all your money um well you know you don't lose it all but just if you don't mind actually because of how it didn't work so yeah yeah I know but particularly you know it's with not withdrawals not implemented it really must be frustrating to have been slashed early on in the chain and still be waiting to get your money back out yeah for sure and then as you mentioned slashing you don't lose all your money but as you touched on earlier if there's problems with withdrawal Keys you can be like really in trouble there was one other very famous issue about two years ago as well by stakehound does anyone remember what happened there with their withdrawal keys I had to go look this up myself beforehand yeah you might have to give me the background I remember stories and not names no I I also know exactly what happened yeah I read all the pass Motors so yeah uh yeah so uh the thing is uh stakehold together with fire blocks uh has designed the system not unlike the one that lighter designed with threshold signature used for holding Keys except it was like much smaller I think they only had like three or four signatures out of which two two shots was in possession of stakehunt and two shots were positioned five blocks and um nobody knows what exactly happened because like the the parties have like different interpretation of what happened but what we end like what they end up in situation when the two shares That stakehold was supposed to hold were not hold by stakehold and that could have happened like we don't know why that happened like maybe it was like problem on the software side maybe those problems take-hand side on Communications on like around backups system like that bullet and lawsuits I think right now so um I'm not sure about that but the end result is uh all that stake about 72 million by or something like that um I didn't check if it's operating still or not like if the I think it's operating still because uh like there can be a miracle uh I don't know uh someone always find transaction rewards now yeah yeah some some kind of like chance that you find like an old mnemonic in the old uh um jacket or something right I don't know but uh it's essentially unvisorable right now this is how you wind up with your CEO sifting through the dump looking for quite possibly or yeah going through like annual bit of Hardware they can find um but yeah thank you very much visily for giving those descriptions and preventing me from having to do it but yeah I think we don't necessarily know you know what happened but what we do know is that if anything happens to your withdrawal key odds are very high that there's nothing you can do um so with that in mind Genesis happened in December 2020 and about six months later so there was a change to withdrawal credentials they introduced the idea of um what we know is 0x01 type withdrawal credentials and does anyone explain what they are yeah actually know that one that's good um so the zero x01 withdrawal credentials were introduced because we realized uh that the way we were going to merge and and move forward with ethereum too uh was different to what we first expected so we set up these zero zero zero zero Keys as BLS Keys matching what validators have um partly with the expectation of all ethereum addresses becoming the ls keys and partly just didn't really know what was going to be so we went with the tech we had um with the plans for the merge keeping the execution layer and keeping the traditional ethereum addresses uh we introduced ox01 addresses which are just an ethereum public address so it's a whole bunch of zeros and then the 20 bytes of of a normal ethereum address and the guarantee is that when withdrawals are implemented and your money comes off the beacon chain it will be credited to that address the catch with it is a is a minor one but important depending on how you write your contracts if you you can use any ethereum address um it can be a contract but the evm will not execute when funds are added to that so if you're expecting log events to come out they won't you won't have that opportunity um you need to write the contract in a way that it can kind of just handle being called one day and they're being more money in its account than it previously saw and Dole that out and however it wants from there sweet and then Lido when this came out um very quickly adopted it could you tell me a bit about you know what that process was like the risk of writing a contract for something that doesn't exist yet uh how did that kind of process go and the kind of migration you might say but there's no migration but the swap out uh yeah so it wasn't too hot actually like uh it's like we have done a stop that is governance controlled and that stop will be like sometimes like in the future when this is also implemented it will be upgraded and when the ethereum oscifies uh enough um it will be made unupgradable um so the RC files as well um so there is like nothing uh complicated about this like just a smart contract that that is a stuff that does nothing except being upgraded implementing withdrawals handling withdrawals due to the fact that like you can't know when uh like you're not getting triggered when uh either comes in is like is harder uh the you know per second protocol is pretty complex and we when you add the liquidity like liquids taken protocol uh complexness on top of it it's um it's pretty hard research to to do it just right but we'll manage so sweet sweet yeah I you make it sound very easy but I'm not so sure that uh writing like a proxy smart contract holding I don't know 10 billion dollars is super easy is that about right number wise like it's not harder than uh writing a contract that's called zero like right it's just like the stakes are higher yeah if you get it wrong it can be a contract that has zero very quickly see at some point it will just become dead inside like when you walk the Easter day so I used to be a CTO for a validator like I had like 12 incidents in a year so yeah um see there's one other withdrawal credential that isn't necessarily in the spec yet and that's 0x03 does anyone want to talk about what that uh withdrawal credential is I don't know what that is I think I think you're probably keen on this but the oxo3 is designed to be a forced withdrawal so at the moment um you actually wind up needing both your hotkey and your withdrawal key effectively to to get through the full withdrawal process because while your validator is active your 32 Heath is remains locked up you need to exit your validator and then it can go through the withdrawal process and that's where the withdrawal key comes in if however you're staking so you're not staking yourself you've asked someone else to stake on your behalf and you've given them your hotkeys knowing that they can't steal your eth because you've got the withdrawal keys not a bad setup you have one little problem in that you can't force them to exit your validator and actually get your eth back and so the oxo3 credentials are intended um to be a way of the withdrawal credentials being able to force exit a validator effectively I don't know if you know the details of how that was planned so like there is no more details because like it's older drafts but I think that zero x x zero three equivalentials are completely unnecessary they're just a feature that can be added to 0x1 um like the same way handled this much the same way as deposits to the container and like just like consensus layer processing messages from execution layer and it's not even too hot it's just like needs to be designed and Implement test that we are actually working on right now on design uh I don't think it will it can make uh it into Shanghai I'm like that's probably unrealistic but like the next Hunter after Shanghai is like doable I think yeah it the the interesting trade-off as a client Devon and designing some of these protocols when we're talking about withdrawal processes is trying to be fair with who controls which key and therefore what response what rights they have um so we kind of set up this idea that the money will go to whoever has the withdrawal key um and the signing key is in control of actual signing and exit and as every time we go and change that you wind up with a whole bunch of people in quite unique situations that were built around that particular split of responsibilities and when you change the split you've got to kind of make sure that you don't break things for other people or introduce insecurities because of this particular way they've set things up I personally don't know anyone who built around the that like assumption so like who is actually relying on the fact that validators can exit and withdrawals cannot I I don't know yeah I think that's a fairly reasonable one that you know the base assumption is that you you are you know it's your money so it's your validator and you own all the keys um when you start to move away from that into um staking providers and so on you are moving out of the I guess ideal case um you know that there is a role for staking providers I don't mean that as in you know this horrible person on my right or anything um but you know it does it does the onus is almost on people doing that to to help with the security and manage it rather than on the protocol itself because it's based around that one key one validator type setup um so yeah I think I think it's quite reasonable to make a lot of these changes and and we can kind of push those boundaries a lot of the the actual expectations haven't been really formally and strictly set down either um we've kind of given a an expectation in many cases but knowing that we didn't have actually a plan for withdrawals when we set most of them we knew they were going to have to be changes and so there are kind of some really core things maybe that we've got to keep that you know like the money goes to the withdrawal key not the signing key it will never be the signing key that gets the money um but beyond that yeah there's there's a lot of wiggle room we do have I I had a similar concern I did mention my background but I also like previously ran quite a large validator and ran maybe 10 000 plus validators and uh at one point when they were discussing this you know new zero x01 withdrawal type and you know that would be the way that ultimately like money will come out if you are on the ox01 you'll need to like swap eventually before you can withdraw there was a moment where that was going to be the hotkey that gets to pick where the new ox011 was and I was quite concerned because all of a sudden that meant I was in charge of billions of dollars for a preview a small amount of time and then they were like okay no let's make the withdrawal key make that change and I think that uh definitely for the better but on that one I do have a question for the two client devs when it comes to like I don't say keeping people happy but whenever like people have requests of how to build staking are you always focused on the homesteaker are you always focused on Enterprise liquid staking pool has often come into conflict like who are you keeping happy and who are you disappointing Paul I think Adrian expressed a little bit of that sentence with his previous one um uh it's it's different for everyone I think I can't speak for all the client devs I can't even speak for my entire team I can speak for myself um I like to prioritize the home Staker I think I think that's kind of the base instance if you know what I mean where kind of all other features are kind of supersets of that um and I also think it sticks with the ethereum ethos of decentralization as well um but so I would say that making decisions that put the home Staker in an uncomfortable position is probably almost always out of the question if it's if it's to benefit the institutional Staker uh but I would say the other scenario where if we need to benefit the homesteaker and it's gonna it's gonna cause troubles for the institutional Staker I would probably be more open to doing that just because I have more resources there's fewer of them um so ideally we always want to support everyone we don't want to just like you know do things to like to disadvantage institutional snakes just because we can but my personal preference is for the for the little guy yeah yeah I think a part of that and almost probably underlying that um is that it tends to be ethereum that comes first and and those core principles so I has talked this morning was really good in terms of talking about some of those principles behind ethereum and and decentralization is a key part of that and so that's why I tend to agree with the homesaker tends to come first they don't have someone advocating for them and they're the most decentralized form of validation um so yeah it does mean that that there are sometimes those hard decisions that a lot of people say you know it'd be really nice to stake with less eth or it'll be really nice to whatever it is and you look at it and go but that's really going to hurt the protocol like it's going to cause too much system requirements and hurt decentralization or it's whatever it is it's going to reduce security or so on um enable more capture of different things those kind of principles mean that we won't always make anyone happy um because it's actually about making ethereum work well and work well for the long term yeah to add to that from my experience uh the only one feature that uh was uh like for big operators uh and not for homesteakers was uh actually an advanced key management practices and like we talked about with Sigma Prime about that I think uh in before before it didn't made it to become chained like we'll start and that was like pretty unambiguous feature like I I don't remember like even one case where this interest was in in opposite right the if homesteakers can run the node like operate big operators can run the node and all we want is like good practice of key key management and for that the only thing that is needed is a good standard of uh remote signer so like remote sinus can be written up uh like coded up independently and that's the only feature that I can think of that is like not usually used by homesteakers but almost always used by uh big operators so that's it I think yeah absolutely not most of the decisions we have to make are good for everyone and it works out pretty well the the external signer is a really interesting one because it wound up with a niche it's been used by a number of systems including damp node to make it easy to swap clients so you put your keys into the external signer and it holds them and your slashing protection and then you can change your validator client and Beacon note all you like without ever moving your keys so homesteakers have wound up adopting this technology that was really first built for for the big staking providers and that's kind of cool yeah the one thing you're describing in terms of where the protocol was changed to favor the lit the homesteakers versus the like Enterprises was probably with Mev boost and blinded Beacon blocks um this one you know is one that we're probably seeing a lot of people kind of giving out about now but I think a lot of people don't consider the counterfactual which was if these were plain text blocks Mev boost and might not be like given to homesteakers and the fear was that if they were excluded from Mev um it would like impact their reward and would massively harm them so you know we now have this world where people are giving out about like you know centralized relay or censoring and that was kind of the trade-off of going for the blinded blocks and stuff but yeah it's definitely a tricky one and I I sometimes am grateful that I can just ask tricky questions don't have to actually answer them most of the time um so the next thing I wanted to do a little was look a bit forward on kind of where we're going next in terms of withdrawals um so we talked a lot about like the like withdrawal credentials but we haven't really talked much about how it's going to happen you might say and I'm going to try and not ask like when withdrawals but can I ask um how do withdrawal spec has changed over the last six months we started with kind of a pull-based system and now we're at like push to Adrian Paulie do you want to like talk about why they swapped uh yeah I didn't follow it a whole lot um whilst whilst in in the previous uh pull method but I do know how the push method works now um and I think it is a lot simpler so the method that's going to happen now will not going to happen but it looks like it's going to happen um is a system where the beacon chain uh automatically without user input will scan through the validators and look for validators that are fully withdrawable withdrawn and then it'll just pull their eighth out and then put it on the the execution chain it'll also look for validators who have excess balance above 32f and then just pull that excess balance out and put on the execution chain so this is um kind of so it's it's an automatic process where uh you don't need to go and withdraw F you just kind of end up with money in the smart contract which I think is really really nice I think it's really good for users because um it means they're signing less weird messages I know anyone who runs a validator could probably um empathize but the idea of having to go and touch the keys on my validator client or do something with my validator client apart from just leaving it alone it's just terrifying and it's horrible so I think it's really good to be able to just like have it happen automatically um yeah there's gonna be I don't know if I so this there's going to be a complication though and it kind of it goes around these um two types of um withdrawal Keys we have so it's been mentioned there's a zero X zero zero Keys which is a BLS with throttle key um and then which was the like Adrian said we did that in the early days because we didn't know what we're going to do uh and then there's the 0x01 where we um don't have a BLS key we just have an ethereum address um so if you're on a 0x01 ethereum address then this automatic withdrawal system is gonna is just gonna work for you it's just like when the hard Fork happens you're gonna start getting some some fresh eth um but if you're with the BLS Keys what you're going to have to do is go and dig up your withdrawal keys from wherever you put it um and then you're going to need to use some sort of tooling um to sign a message um to say okay I'm now moving away from this BLS key that I'm proving that I have and I'm switching over to an ethereum address 0x0 one type and then the automatic withdrawals can kick in so yeah I think it's a shame that we have to do this we have to make I would say most users um go and sign this message um but I think it's unavoidable but then after that it'll all kind of happen automatically so knowing that that is very much in the pipeline and looking like it's going to happen where I know Lighthouse deciding to implement it I think techu is lots of clients are keen on it um I would say if you're creating a new validator today I would be using 0x01 I would be using an F address I think it's going to make your life a lot easier and I think just managing I think managing ethereum addresses is much easier than managing bail SKS just because we've already got tresors and everyone's familiar with it so yeah one really important point is that once you have switched to an ox01 key so once you've got an ethereum address whether you started with it or whether you switch through this new system that's it you can't change it again um and there's concerns around that because people want to change it but generally the reason you'd want to change it is because you've gone and lost the key to that or you typed in the wrong address or something don't make that mistake it's like transferring money to that address right it's gone there is no way to get it back if you don't control the key because as we talked about before it's the withdrawal key that has to own the money so if you tell us that ox000 the null address owns your eth well now it does and you're not going to get it back and there's really very little we can do about that well there's nothing we can do about that for sure I think um we're going to see a lot of work and there's actually a talk by a test and just briefly beforehand where it's now that we can have like solidity contract addresses you can program your like change of ownership in there and not like at with some cryptography or any sort of like multi-sig or anything too weird yeah and that's absolutely a big part of the reason that we've gone with such a simple approach to withdrawals we don't need to do anything fancy on the consensus layer side because you've now got the full power of the evm um to to work with things and and have upgradable contracts or split contracts or all kinds of things there um that let you have this Behavior so think hard about what address you said is your withdrawal address I would say for most homesteakers it should be something like from your trace or you know your Hardware wallet that that is very safe and very secure and simple keep it simple you don't need to do anything fancy and then you know staking providers will use something that is a contract and probably upgradeable so they've got some control over it and then there's a range of people in the Middle where you you think about what your needs are and possibly use a very simple contract that could be upgraded but could also just kind of hold the funds and send them to you for sure yeah I think um the you've touched on one of the things that will show up over the next few years and this is the evm having more control over the consensus layer because as I pointed out right now you write an address and money will show up but like no code is going to execute and that's about it um how do you think that's going to change and what is the evm I don't know consensus layer API going to look like in three years time I'm really hoping it doesn't ever get a say um I'm really hoping this stays a one-way thing from the consensus layer pushing to the ethereum layer or to the to the execution layer um because it's dramatically simpler to get the evm to call out to the consensus layer is a big deal and a really big challenge we've kind of done it with deposits and it's the worst code in any any consensus client is tracking those deposits and we want to simplify it now we've merged um but there isn't a generic system for passing messages from the evm back to the kazat census layer and it's a big deal to try and design it yeah definite rates but reads hopefully uh I hope that the consensus layer can start to push some information about itself into the evm so that you can you know do proof to Beacon State routes but I'm with Adrian um yeah I wouldn't get going the other way is it would be very painful for sure yeah definitely don't want to go the other way but as you said being able to read things about the consensus layer I think is going to become very important because one of the things that we look at with all of the liquid staking pools is for the last two years these were two totally separate chains and there was like no information passed even still post merge there is still more or less no information being swapped but all of these pools more or less have oracles to say what's happening over in the consensus layer side and I want to take the chance maybe to talk to facility a little bit about how does exits work for something like Lido because you guys are you know a liquid staking token and you're soon going to be able to offer Redemption for the first time but the Redemption queue is very bandwidth constrained so how does that work for Lido trying to allow people to exchange their St eth for re-leet at some point you know in the near future what's that going to look like yeah so um we haven't settled on the final design yet so what I'm talking about is the walk-in uh walk-in progress solution right um uh it's almost ready though for like presentation and voting in for photo Etc um but the gist of it the best solution we can do is to reflect the withdrawal queue in on the execution layer we like try like looked at multiple different approaches like calls uh auctions for uh for the like for the place in the queue uh some kind of buffers for his roles etc etc and it all um I think is like strictly inferior to allowing people to place the like request of withdrawal and we put them in a queue on the beacon chain and this all happens and they get there they withdrawals uh when they time in the queue passes so uh that's probably going to be it's pretty like it's pretty complicated uh because like there are multiple asynchronous processes happening like the withdrawal request the Oracle reports about what happens on the beacon chain like you said right we like the needs information what's happening in Beacon chain and we can't get in real time it's uh it's just impossible right so um Oracle requests lessons and slashing side like we haven't had any uh you know lifetime Lido but like if it happens it's like 35 days process at least and somewhere in the middle it has an uncertain amount of slashing penalty which no one knows like until it happens how much it is going to be because it's like depends on the amount of Slash from that like accumulated up to this point so when you combine all the asynchronous process and you need to like give people exactly the amount the uh like they they need to get uh it's becoming pretty complex protocol um button fire weather is just the same queue as uh on on consensus layer reflected execution layer as like basically vouchers for for the place in the queue three um awesome yeah that makes a lot of sense and I think you're not going to be the only one that's going to have a lot of pain with these withdrawal cues very soon because everyone just thinks withdrawals no one thinks withdrawal q and um that actually leads me a bit to the like balance skimming and this is another thing that gym are like fifth panelists is very like uh in favor of which is if you don't have any way to take the money above 32 ether that's accumulated all of the big um liquid staking protocols will just start to churn their validators take off the you know three either off the top and like set up a new one and the queues will be like perpetually full because you know why don't you just like go and like kind of skim and restake um can anyone tell me a bit about what you know skimming is going to look like because it's also changed a bit in the last few weeks yeah it's a it's a very similar process to what Paul described in terms of finding of scanning through the validators and just it it magically happens it's pushed from the consensus layer the the key thing is that there will be a limit to the number of validators we scan each Epoch to look for whether you've got money over that 32e that could be pushed to the execution layer the reason for that is that we've got to fit it into blocks and it's like another ethereum transaction that goes into a block but we're not charging gas for it it just happens and so we kind of have to have a rate limiting on it to make sure that we're not effectively just blowing up our block limit and and execution clients are able to process these these blocks in a reasonable amount of time so I think it's something like every two months we get around the validator set but I don't think the numbers are at all set so um yeah it's a those numbers in particular are going to go through some research and and seeing what the impact is on execution clients of this many or that many deposits um but it means that fairly regularly and it'll be in in that order of you know low number of months I think that we're aiming for generally fairly regularly whatever you've got over the 32 eth gets swept to the execution layer so you start getting your rewards on top of the transaction rewards you're getting when you propose a block now gonna say I think um I think it's really good because the idea of a validator so a validator would say 33f exiting taking an F putting 32 back in is really bad for consensus clients because that extra validator slot um they they cost us we have to store that validator forever it's more in Ram um yeah it's really bad so so the ability to like for people to take their earnings without creating a new validator index is is really good for the protocol yeah there were certainly a lot of people Jim I think leading the charge being very vocal if you've got to have partial withdrawals before you know at the same time as withdrawals and you know as as core devs we try not to promise things until we're sure but I think all of us knew that was absolutely right and there was no way that we could could manage the impact on the chain of these churning validators if we didn't have partial withdrawals there is also an economic effect aspect of it uh if there were no scheming of rewards then um the B corporate like the the big State Controllers like protocols exchanges whales Etc they would get a lot of leg up against small stakers because they can afford to compound and small stickers can't like it's it's very hard to them so um yeah it's like it's very right move to uh make it like skimming rewards a thing uh the the like the the slight drawback of this is working against the protocols uh that likes taking protocols because you can't really understand uh if the money you're getting on the result credentials comes from Rewards or comes from unstaking and you need to know that uh because like rewards you have to reduce view to stake us and unstaken is like something like someone was drawing something gone wrong or like me slashing happened like you you need to uh um but it's anyway much better than the alternative yeah I'll also Echo the it makes much more sense from the compounding thing but someone trying to write like solidity to keep track of something you know I don't know was this a very bad slashing or was this a skim so it is a bit tricky like that um I have just one more question before we go to the audience for a few so have a think of anything you want to ask the the panel here but um when the capella hard 4 comes in and we have this ox0 to ox1 upgrade how messy do you think it's going to be is it going to be over in two blocks and it's going to be 10 people is it going to be 50 000 validators I I suspect I mean probably the majority of validators on the network uh uh BLS credentials sort of more need to switch I don't think there'll be a huge rush um I think there'll be you know the initial burst and very likely what will happen is you will submit it and operation pools in clients will actually be full and it'll probably get dropped from the network and it just won't make it on train at all and you'll need to submit it again so we'll kind of have to work through the the usability of that because if there is a big rush it's more than we'll keep in the memory pool basically um but I think generally it will be over time because people have got to get their withdrawal keys out it's something you should take your time and do safely and not rush in the first instant um and and once it's done it's over so it's it's fairly fairly one off yeah I think we'll have to think a lot about the tooling to easy people as well um I know the staking I was talking to Carl from The EF Karl beak um and he was saying that it's likely that the staking deposit CLI tool will have the tooling to allow you to do it um and I think it'll be a little bit easier because the staking deposit CLI now kind of like makes a bunch file for you and then you need to put potato at those um and that's kind of annoying in my opinion and I think what now that the apis have evolved hopefully what we can do with the staking deposit CLA will like you can just input your mnemonic and then tell it where a beacon node is or even a like a public node and then just have it publish it out to it like that so it'll be a little bit scary because you have to touch some crypto stuff but I'm hoping that it'll it'll be okay and I think that I think I'm Keen I know Adrian would be Keen as well to try and make it nice and smooth for people yeah and and I would expect there to be an offline option of that as well so you can generate a file of you know your your switch command the operation that you can then you know put on a USB stick take over to a hot thing and then send to a public Beacon node yeah you could probably upload it to the file to a web page or something like that yeah very likely and it's already pre-signed so it can't be changed it's not a it's not a risky thing it's just like you something you move up yeah my estimation is about 200 000 validators will like rotate as soon as humanly possible like safely possible uh the reason is like I know about Lido we have about 50 000 and every extra second is an extra risk uh on this on this credential we don't want to do that like uh we won't overload the network of course but like as soon as humanly possible we do it um it's it's very important and uh about I think like 150 200 Southern of operators are on BLS credentials uh that are managed by exchanges and this exchange it they want to uh like to pay the rewards as soon as possible again and like convert it to liquid stake and derivatives for like custodial like coinbase doesn't CBS so um that's going to happen also very fast like the uh the protocols and custodian that aggregate users take they will and haven't rotated they will rotate like they will rotate as soon as possible because uh basically BLS 0x uh zero zero credentials are as useful as a potato uh the only thing you can do with them is like exit or rotate and like that's just not right yeah no I'm aligned there and I do think everyone will go and upgrade at like a relatively leisurely Pace um but the likes of the the lighters on the old ones and like I will block demons and they're on Oxo keys but the withdrawal keys are totally separate to the like hotkeys so I don't think they're as big a concern but I unfortunately do think and they've been people have been posting on forums like admitting to this that they have their hotkey and they're like validating key on the same mnemonic and they left that on their like validator machine or something similar and that's been compromised and the problem is is you know I think there's at least 10 if not a bit more Jim has probably good data on it but um when capella goes live for those first couple blocks there will be an awful amount of Mev on the table for you know there'll be two people that can