# Ethereum Core Devs Meeting #134 [2022-3-18]

- Channel: [Devcon](https://streameth.org/devcon)
- Date: 2022-10-09
- Duration: 1:33:31
- Topics: blockchain, smart contracts, distributed ledger, ethereum
- Watch: https://streameth.org/watch/yt-Lbsjw-lzMIw
- YouTube: https://www.youtube.com/watch?v=Lbsjw-lzMIw

## Description

https://www.youtube.com/redirect?event=video_description&redir_token=QUFFLUhqbklNWm9uc0lySDFMMTdOU2UwRHVuV1oycEFPZ3xBQ3Jtc0trelF3cUZrUWwtN1ZpQlktemE0bFpaeHUwazdWZEFSQlQ4OUxYX3NsUDVBYjZDYjlFcy04X2xvWW5QcDYxS0tKemhxYkx3NjlVaFg2a1J3Y0R0dEpnR3N4bEdmc0NycExha3l5OHJnRnFPZjFrcVViOA&q=https%3A%2F%2Fgithub.com%2Fethereum%2Fpm%2Fissues%2F492

## Transcript

good morning everyone welcome to awkwardev's number 134. um a couple things on the agenda today i think by far the biggest one is um killed and and kind of going through what what happened um making sure we we figure out uh next steps from here um then uh we have some updates on uh some shanghai proposals so um uh alex has some updates on on beacon chain withdrawals there's also someone else i'm sorry i'm blanking on their name uh who left a comment and wanted to discuss something about partial withdrawals um and then uh their proto had a bunch of updates about uh eip 4844 and then if we still have time at the end i uh i had a proposal about um how we can harmonize uh the core eip process and the executable specs uh that are being worked on um so to kick us off um yeah perry i'll i'll point to you but maybe some somebody else did a greater position but someone kind of wanted to walk through high level what happened through the kiln merge um and then we can probably hear from like the different client themes kind of specifically about what went down on their side sure um i can give a high level overview so we had the killer test net launch last uh the proof-of-work portion of it launched last week on wednesday and we had the proof-of-stake beacon chain launched last week on friday the merge itself happened on tuesday a bit earlier than expected we had to delay the merge once by using the terminal override uh terminal total difficulty override flag and that was unexpected but that exercise seemed to have worked perfectly all the clients respected it and we noticed no no weird behavior from anyone however once the merge merge transition actually happened a few clients did have issues with block proposal and or syncing um i let the individual client teams go into detail later on um since then the network seems stable i think there are still one or two clients that have some issues um but they should they all seem to be minor and should be fixed relatively soon got it thanks um one thing uh i'm curious about the studies as well as the the block explorers i know there were some issues uh with the blocking spurs like shortly around the merge and they kind of were were lagging for for a while and can you just give us a quick update if you know what happened there yeah um so we we run a fourth version of the beacon chain block explorer and one of the payloads exactly the execution payload was set to hint instead of bigint um this wasn't an issue in the previous ones because well the base fee by gas didn't wasn't too high um but apparently in this the number of transactions one too high and in this one it was high so it just overflowed um i think it's pretty much the same thing that tripped up prism that tripped up the explorer but it's fixed now and the explorer just takes a long time to sync so i think we're still stuck about uh we're lagging by a d or something okay great i apologize to everyone on the live stream there was a uh obs issue and only my voice was audible so i'll recap what perry said in 30 seconds um and uh we we also have the zoom transcript which we can we can add to the notes to get the full version um but basically what kiln launched under proof of work last wednesday uh the proof of stake vegan chain went live last friday uh we there was too much mining on the network uh so we had to use the td override feature in clients which worked well and allowed us to delay the actual merge on the network which still happened tuesday um there are a couple issues on the merge around block production and sinking and we'll dive into uh into uh the specific ones from from clients right after uh but the network was still was still uh finalizing and is still doing so today and there were some issues also with the block explorers uh some integer overflow issues and um these have been fixed but the block explorers are still lagging importing all the data anyone else have just like general comments on kiln before we kind of dive into the the specific client issues okay go ahead will we be discussing will we be discussing perhaps after the client issues whether we want to do another dev test net before public test nets given that failure or not yes i think actually we uh we might can maybe do that now like i know maris perry and i and others have talked about uh shadow forking gordy next week basically um so i i think the short answer is yes yeah yeah i think given the um given that kiln at least as we believe the specs will be is feature complete and we've kind of solicited a lot of the community to jump in on it um i'd like to just kind of keep that as quote the public test net right now and continue to ask people to do full you know application deploys and things like that um i would say certainly we should do um another uh test net you know call another transition um and and maybe that just call it a devnet um and have an end of life and i'd also i'm an advocate for um you know shadow fork and gourley and or spolia every few days for the next few months um you know if we can if we can automate that in a way that kind of just shows us the latest builds continue to work and and block production is happening and all that kind of stuff that's going to i think a lot of the concerns right and yeah i i feel i'll be more confident in this in about five ten minutes but um as i understand it right now none of the issues we found on kiln are like spec issues they all seem to be client implementation issues so to me that says like you know we don't need like a different public test net which runs like an updated version of the merge specs we need as i understand it like clients to obviously fix the issues that they found um and we obviously want to test that on on devnet's um but yeah unless i think there's like a a significant change to the spec it seems like we can just keep killing as is and obviously have clients issue new releases which will work on the network um and then there's a question in the chat about mev boost i don't think it's being used uh yet no and we've reached out uh to the flashbots teams this week to chat with them about that um cool i guess yeah to dive into the client stuff a bit more uh i'll start with the you know first one that comes to mind was that there was a prism guess uh kind of incompatibility around uh the encoding of values like the base sheet i don't know if anyone from prism is on i'm here terence yes yeah karen so you want to give us a quick quick overview of what would happen yeah sounds good thanks for having me here so um high level summary execution layer uses big indian consistently used little indian so when we we have this costume implementation for protobuf so whenever we marshal and uh and the our martial specific data field we have to be careful to basically reverse the buy code and we missed that for the big speed per gas field and unfortunately um we didn't catch them for the previous test net because um because the basically progress was quite low and there wasn't actually people are reporting only the videos are broken so should we fix that first all right uh so we are sorry can you say something yeah uh in the chat people are saying the audio yeah it's okay yeah i don't know sorry yeah everything's turned on i'll just i'll upload the zoom recording after uh okay sounds good yeah yeah yeah okay okay i'm gonna keep going then so basically the previous testnet was unable to catch that because the base feature gas was quite low so yeah i'm happy to um i'm happy to yeah i was quite happy to to realize this bug so as the corrective action i posted a um postmodern on twitter i'm sure most of y'all have seen it but um high level summary for that is that we will be up our testing infrastructure so right now we are working on our differential fusser for all the api endpoint meaning that we will uh aiming to be hundred percent compliant with all the marshall and our marshall and for our end-to-end testing we are also adding um the transaction generator and uh so we can make sure to send all the exotic transactions to make sure the base vapor gas does not remain low and stuff so yeah that's the high level summary thanks um and we're marius i understand there was no issue on the get side right it was just the guest combination but like death was working right is that exactly right yes and like we we saw this bug because uh guest prism didn't didn't create any blocks and we only noticed it because uh gas prism is such a large majority of the network and that's why we only saw gas prism but it it's probably it was probably also on gas uh it was also and guess ah visual and uh guess in the mind so it has nothing to do with got it prism piece of prism out of mind yeah yeah that makes sense okay thanks thanks for sharing any any questions for people on that yeah i guess in not just the prism ci we need to make sure that we have sufficient transaction activity in you know kurtosis nightly builds and some of the other testing we have as well also i will say i was looking at the base fee on gourley um and i don't believe it's always above 255 so um we might consider when we do shadow fork to make sure that there's sufficient activity on there as well yes marius would your transaction fuzzer work on gory if you have enough east sure sure unfortunately someone stole all of my gurley leaves send it back if you're listening to to this call and you you send it back then i can i can test so uh one one thing that uh like one thing that i wanted to talk about was uh that i think we lacked insights a bit into uh which clients were like uh missing the slots and which clients were like uh not proposing blocks or not the testing and um i think we we need to up our game there a bit so that we can like uh see the bad client the odd one out way quicker than we than we did just uh on on tuesday yes agreed i know there was an idea um i think it was perry's idea of like maybe we build um like devnets which have each combination of client as like a super majority client so that if we those issues that show up kind of they they show up uh and the network stops finalizing um but i don't know if that would catch um if that would catch the bulk of it is that is that a suggestion for the nightly builds period yeah exactly so one variation of the nightly build is just to have all clients together and the second variation is to just have a combination of every client as a super majority um and we can just do this parallelly nightly so if a super majority client fails it's a lot louder whereas if if it's just a minority client we might not even notice an issue yeah i mean if that's a tractable thing to do at this point i would say yeah definitely anything like that that we can do to kind of continuously get a view into things working is really good at this point i was gonna say for the beacon block explorer you can tie the proposer into like a name so you if we look at if we look at a printer for example if you see the proposal he actually says prison uh white house blah blah blah so we probably can do that for kim as well yeah i'll look into that one as well um and one other nice thing that jim from a testing vouch worked on updating eth2 so you can now process a block or an epoch as soon as it comes out and it will list out all the proposal indexes that have failed or missing attestations or sync committee participation so we no longer have to wait for the explorer to update the explorer will always be a bit slower but since this tool isn't actively indexing you have to call it it'll be a lot faster and should be easier to debug stuff nice um i think there were also some issues on on kiln between basu and taku if if that's right uh yeah does anyone from either team once want to give an update yeah i can uh i can speak to that there is a uh a case that we weren't expecting where uh the terminal block was finalized and our logic to check that we were descending from a valid terminal block wasn't considering that the block itself was being finalized was the terminal block so there were some cases where basically just sit sit there at ttd um so we've got that uh we've got a pr for that and uh also we had some issues with uh backwards sync which we have a matching backwards sync pr that is emerging today so we should have uh 22.1.3 snapshot which is what we are gonna recommend for using for kiln um but yeah that's the issue that we were having and encountered and there was a couple of reports of basically sitting at ttd for that reason got it and so the fact that it was basically deku was there was nothing on the takuan right it was it was just a coincidence the other side correct okay got it yes um and then i think nethermine also had had an issue or two is that correct yeah uh so the issue for another night was that some notes after the transition uh require restart to make it work so after the restart they are working fine we are still investigating it and to make sure that it was fixed we need to experiment a bit with transition uh we found a few potential places but it's still investigated uh one good news is that marius runs differential father between geth and nevermind so fazer is sending random requests to execution clients and comparing hats and seems that both clients behave in the same way awesome and i know ergon as well i think you you uh you knew that aragon would probably have some issues during the transition but still run it anyway you want to give a quick update on like yeah what you learned from this and and where you're at right now so uh there was one issue in aragon um actually related to endianness as well that was fixed but so we were incorrectly sending invalid block hash for a valid block but i think taco was incorrectly ignoring our incorrect invalid block hash and it was keeping like resending the same block um though we sent invalid block cash also because erigon was quite late to the party um i personally would like more time to we are still uh refactoring our sync code uh and uh giving that kiln like a quite a few issues uh we are discovered during kiln though maybe not like theoretically blocking but still i i would suggest not to rush the merge take take it slowly spend more time on testing more like match transitions and things like that yeah and uh i personally would like to spend more time understanding how sync works on the consensus layer side the difference between optimistic non-optimistic sync performance implications uh think about the test because i'm worried like what we were discussing in the chat really really recently is that what a kind of the performance uh implications of like consensus layer sending us when you have to sync something and then consensus layer sends a blog and just have every block to the execution layer is it okay is it not okay maybe it is okay but i personally would like to spend some more time thinking about it and also testing it great um yeah thanks thanks for sharing are there other client teams i think those were all the ones that kind of had issues um on on kyon specifically but did i did i miss anyone okay i guess not um and so i guess yes so in terms of next steps from here like obviously i think it's clear to everyone that like we need more testing infrastructure and and things like like running uh shadow forks of gordy and and rerunning through the transition uh couple more times um i think in terms of like you know rushing the merge or not and timelines um we we probably have another month or so before we need to make a call about whether we want to move to test nuts or whether uh you know whether we're not ready for that um so i i i feel like that the next step is probably to spend obviously the next two weeks and then possibly the next four weeks um improving the testing infrastructure finding these these these issues um and and you know growing confidence in our in our implementations and then we can probably make a call about you know do we feel comfortable moving this through the test nets or not and um and if not then i think at that point it's like we might have to to discuss potentially pushing back the difficulty bomb um but yeah i i do think we probably still have like one month basically until until we we have to make that call um at a high level i think that the bomb is gonna start being uh you know noticed uh early june around mid-june mid-june 10th in july we probably have like 14 15 second block times which is high but not unmanageable and late july early august assuming like things are the same you probably are looking at like 17 or more and that starts to be you know much much greater delays um and just considering the time it takes to generally go from test that's the main net um yeah i i think we basically have like about a month where or where we can kind of grow our confidence in these implementations and then have to make a call um and yeah i think you know the the more iteration cycles we get of testing of shadow forking in that period uh the better um and um so one one like one thing that i'm that i'm thinking about is like i don't think we we like i think we uncover a lot of bugs and um but we don't like we don't really notice them we don't really recognize them so um like there was uh this uh uh sink aggregation attestation thingy whatever where like that that was like at like 60 or something and then after two weeks someone decided to look into it and it turned out that nethermine was uh prison was just not sending sending some something and i think that is like that is the bigger issue like i think we do uh trigger a lot of bugs and i'm i'm pretty sure that we triggered the the prism um the prism base fee encoding bug at least five times already uh but we but we never recognized it as as such so i think we should spend more time uh building infrastructure to recognize these bugs and i would like really urge um all the client teams that if they see something funny on or something interesting on all on a test net or wherever then they should reach out to the clients that they think are affected and then and uh like uh really look into it instead of just saying okay there was that was pretty funny but if i restart my client then it's away so it's like a non-issue and i think we do that way too often i think we could probably come up with something like key 10 10 or so key indicators that a test that's healthy right um and it's not it's not just finality finality is good uh you know that's what you hope to see always on main net even if there's some sort of issue with a client here or there but there's other things like no one's looking at the the sync aggregate thing because no one's really running uh like clients and so it doesn't really matter and just kind of falls into the wayside but that's an indicator that something's not right right so there's that there's the number of blocks per epoch it should be 32 almost all the time there's the number you know there's the the finality there's um the the amount of the percentage of attestation is actually making on et cetera and so i would say most of our most of our monitoring and most of our kind of like integration testing should be looking at a number of these things rather than just um finality which i think we rely a bit too much on the two-thirds metric there we should obviously can let a lot of air through i think we should also think more about test cases for hive because mario vega is doing great work with this test and i think we should all uh try to add more test cases here yeah i mean i'm definitely the mind that at this point if there's one or a half resource from each team that can work on testing devops and other things that we would be in a much better place come four weeks from now yes agreed and i know that um i don't think he's on the call but frederick from the ef has been trying to gather uh people from different teams that uh to coordinate all of that so um yeah hopefully hopefully like we we can just have more people from each team but also kind of be a bit more proactive in sharing the stuff that's being worked on so that everybody's aware of what everybody else is working on oh frederick is here yeah yeah exactly yeah i sent that out [Music] so and i've sent a message to the client teams as well about it so hopefully that will get moving thanks and um maybe just to touch back on on one thing that uh andrew brought up the uh the thing around like syncing and and box sending uh the blocks of the execution layer i'm curious check generally how people feel about that like is that something that can realistically be changed before it emerges that's something that like we might want to improve shortly after um so it's something that can be lever like the execution layer can already leverage the information do whatever it wants here the the consensus layer if it's syncing and sending you bulk blocks means that it is not at the head and it literally doesn't have a good piece of information for you to to decide how to like do reverse sync in any sophisticated methods from the head and so it's walking its way forward and that's that's just the case and so you can either execute as it does that or because you know the current time in the world you know that the consensus layer is not quite actually at the head you can wait until the consensus layer gets to the head and then do whatever sync techniques that you like to do at that point um but there's kind of a bit of a chicken egg problem here you can you can lock step sync with them or you can wait until they get to the head and then do whatever sync method you want um and there's there's sufficient information to be able to make either of those decisions that makes sense does anyone have anything to talk about a lot more in some of the channels and stuff uh if people want to discuss different design decisions sweet yeah i think yeah it's been discussed uh a bunch so we can we can keep kind of the conversation there um was there anything else that people wanted to discuss about kiln or the merger just like testing specifically i would like to quickly discuss safe uh unsafe and finalized tags um so um yeah there is a pr uh opened into the execution apis repo that just adds finalized block tag to the uh theorem jason rpc um and on the last call we roughly decided not to have safe or i was a bit uncertain about that and uh one of the suggestions uh one of the one of the things that we may do for safe block tag is to um is to use justified justified block for it as a stop gap until we get uh safe rule um implemented um like uh in in its like full proposal that's made by the democrat and aditya so this uh using justified block is pretty cheap from cl standpoint so it's just responding with just sending these justified block cache to el and it also um brings a safe block closer to the head than the finalized one and it's a truly safe block it's not going to be reworked assuming that there is the honest majority and the synchronicity um also yeah but yeah it's it's still uh not that close to the head um like as we previously discussed as safe could be so that's just the proposal and just curious what people do think about it think that's good i think that that that that is the block that we that we should declare as safe and uh yeah i think it's nice too because it gives uh the exchanges and stuff a chance to begin using this and the algorithm can improve not good are you here can you chime in on how you feel about justified being the safe currently i mean there's definitely no downside to using that for now i think um yeah uh yeah i mean as an update on that is this like it's unfortunately it turned out that it is much harder than we thought to define a safe head with lmd so yeah i'm so optimistic that we can do it but but yeah it's definitely a good idea to have this intermediate solution i would say there's no downside the downside is that we can't point latest at that because too far behind which yeah of course but it's it's safe right that justified so like in the in the safe hat so the framework the justified one is definitely safe is there a reason we don't have a justified tag like if that's is that a useful thing beyond the safe tag like can we imagine execution or people will talk in the rpc wanting to know the justice light block it might it might be if we had safe justified and finalized where safe and justified are just kind of equivalent at this point that does give an additional granularity of uh progressive confirmation in a sense um you know not finalized but now the assumption on this break this changing is much higher than just safe um but you're also kind of just giving users more choice which may or may not be good so yeah i'm not opposed to exposing justified you can definitely think of use cases where it's nice i feel like if there's use cases then exposing it seems like the right thing to do and we can always just in the docs make it clear that hey you should probably just use safe if you if you don't if you don't know what you're doing use safe but we also offer justified and finalized or whatever like i feel like we can solve the problem of too many choices be a good documentation my only issue with that is once we have a safe adjuster then like safe is less safe than justified i think that's pretty weird it's already it's already weird that safe is less safe it sounds like you're being swallowed by a storm yeah you told me i could you told me i could go for a walk but we took we got the point we got the point the semantics of like safe versus finalized justified is weird that safe is less safe than those two things um you could just call it unsafe that wouldn't i would not say that it's like less safe it's it's safe on the finalized and safe yeah i mean finalized justified and safe all all three are safe on the different assumptions that's how it should be better i'm open to other terms for the safe if people have them um i'm also fine with safe confirmed is a good alternative but it doesn't convey the same meaning i think confirmed is very dangerous because it has the proof-of-work connotation and so people might confuse that with uh like literally finalized or they'll ask when is the block confirmed six times yeah exactly some yeah like that yeah um okay so um finalized justified um safe unsafe latest safe to justify it later still unsafe do we want to cap this all is there yeah is there a reason to have unsafe and not just keep laid this i could find i think i think they're the value of trying to over time rename latest unsafe helps inform new developers into the ecosystem that they should not be using this thing especially once we have better safes something more close to head than justified and so i think getting that name change now is the the best opportunity to get the name change so we're introducing a bunch of other terms and maybe in you know three to five years we can finally deprecate latest um but i do think there's value in making it very clear that you should not be using this unless you really know what you're doing okay so i'll just probably submit the pr or update the existing one and we can proceed with that okay and to recap you will so finalized and then uh justified safe and that's it yeah yeah later i guess inside of the consensus layer spec is where we'd probably map we just do the safe algorithm is returning justified for now and then we can update that algorithm in the future yeah and uh yeah it will be probably some time difficult to um explain the difference between finalized and justified for end users i'm not sure if it's gonna be that much useful but anyway having heavenly granularity is always good so okay let's have this all also like a minor question um is what should el respond with uh when once it gets finalized requests before the merge and i think yeah and finalized and justified and saved before the merge i think it should respond with error which is which will allow to avoid uh any uh bugs or um unexpected things happening if if your request is finalized before the merchant got something i mean in full other than error so um yeah i think error is preferable option unless someone has any other opinion i think the genesis block is more likely to lead to weird errors um i just i would worry that someone has some sort of setup where they're trying to switch from confirmations to finalized and then all of a sudden they go from thinking something 10 blocks ago was equivalently finalized to nothing ever in the chain ever being finalized and i worry about edge cases there i think an error is safer the other option would be for execution clients to have um some sort of cli flag they can pass to say how far back do i want pre-merge finalize to be so you can just define finalize to mean latest minus 10 or whatever you can always do that but maybe it should be part of the spec it seems a bit yeah yeah there's software that's deciding if something's finalized she can maybe use that rather than putting it inside of the client okay so it seems like there's no real objections against an error and it just harmonizes the behavior across everything is it possible for jason rpc clients to find out when the merge is going to happen or when it has happened they can um [Music] they can ask for the difficulty or pre-brand out and if that exceeds 64 bits then the merges have been uh that value that's i think i'm thinking of developers who want to um be merge ready and you want to deploy your app before the merge happens and you want your app to smoothly transition to pre-merge behavior to post-merge behavior with regards to finalized uh justified say fiat i'm trying to think like do we have a good story to tell them or a good narrative for how they should build their apps like what should they do you know should they be checking difficulty equal zero is that the right thing or can we give them like an actual is is proof of stake versus is proof of work query that they can run and maybe the difficulty thing is perfectly fine i don't know um they can use this error and response to finalize finalized requests because uh the merge yeah yeah yeah yeah i know it's like no code flow on errors yeah yeah i agree i agree with that uh but um difficulty zero would mean that the transition is has just started and is in progress i mean the first block was difficult to do and the merge is finished it's considered as finished once uh this first this transition block is finalized right right so they can't actually use finalize until sometimes sufficiently after the merge which is long after difficulty has switched to rando right and i mean some time being like 12 minutes or something okay anything else i'm jason rpc sorry i had myself muted and i had some closing comments um i feel like we should give users a a more clear way to like a a reasonable way to find out when it's say it's i want to use word save when it's a reasonable time to switch to using finalized um that doesn't involve them like recording things that are erroring and then having to set up error handling that alters their code flow like maybe we can give them a simple json rcpc method or something they can query that says is now the time like is the merge fully complete or um is finalized available yet or something along those lines just adapt developers don't have to put in these horrible hacks just to build good apps around them yeah there might be a good uh blog post to yeah we get some of this stuff in like yeah about these new tags and also talk about some of the ways they can use them and and maybe some of the logic you can use to kind of assess what the merge happened and that kind of stuff yeah and we can definitely organize calls like with application developers and and walking through it and also just get their concerns about specific flows right if we don't provide and like this graceful method call that will return that emergency has happened they will have to rely on errors before the merge and yeah absence of errors don't get finalized block as the signal that the merge has happened i mean that from json rfc it will not be possible to um i mean you can look at the block header right you can look at the block header and that's what yeah cause some stuff will be zero after the merge um that's that's during the transition uh so you'll you'll know you're in the transition but you won't know the merge is complete and that's when you don't want to switch over your strategy and your app until after the merge is complete and so right now there's no way for other than just like trying things and getting errors and then like catching the error and changing your behavior based on getting an error there's no way for app developer to build something that changes behavior once the merge is complete yeah i'll go i'll continue this offline or we yeah or we can create a smart contract which just has the difficulty up code in it and then people can just query that smart contract okay same problem that'll return it doesn't work so you want to know when the first block was finalized and you cannot query for that um yeah like we can implement this this is like a five line change i just don't really like the notion of having a new adjacent rpc call uh for like 12 minutes or that that is important for like i don't know maybe maybe like three weeks and then uh you know once we really want to use it uh no one need needs that anymore we can use it again when we change the consensus engine again yeah i'd prefer writing some pseudo code to show how people can decide this from the uh and then libraries can write a function if they want you know what through js yeah well we can take this position yeah we can take the software yeah okay um next up i guess yeah before we move to the next thing anything else on the merge itself or kill or testing okay uh next up uh alex has an update about uh beacon chain withdrawals and we also have uh someone else i'm sorry i'm i'm searching on the zoom screens but there's literally too many people um we had someone from the lyto team who had a proposal for partial withdrawals as well so maybe alex if you want to go first kind of give an update on on on on what you've been working on and then uh we can have uh yeah we're gonna have the partial requirements yeah sorry about that um yeah alex do you want to give a quick update and then we'll go to archon sure yeah so last time we talked about this uh essentially i think there was a lot of uh sort of demand for something to organize all different threads uh so that turned into a meta spec i'm just gonna share my screen quickly and we'll just run through it it's pretty short um wait a second sorry right oh i see can you guys see this yes we can okay so yeah i'm not going to go through this in detail if you want to read it it's here but essentially it just like has some pros of like how the drawls flow will go and then links to specifications at a high level the consensus layer uh essentially schedules when the trial should happen and then it puts them into this queue and then the consensus layer is also in charge of again dequeueing withdrawals into execution blocks uh there's a specification for how that works uh at the existence layer here there's a pr for the modifications to the engine api because then essentially again in some way the consensus layer dequeues these withdrawals they shove through the engine api to the execution layer and then what i want to talk about today is essentially two different options here there's like two different eips one of them we discussed last time in terms of having a new transaction type to represent the withdrawal there's another option that is essentially some sort of system level operation that's far more involved but essentially it's just saying okay rather than have a new 27 18 uh style transaction type we have this new type of thing called an operation and that's where the withdrawals live and the reason we want to do that is basically to firewall off uh you know mixing withdrawals from user level transactions uh and there's probably some like safety benefits there but uh and here's the catch is that one thing we'd really like to have is some sort of logging so when a withdrawal happens it'd be really nice if there's some way to just watch the execution layer and know that the withdrawal you know has actually occurred [Music] and the point here is that if we go with uh 4863 the previous cip that we talked about where it's new transaction type then uh you know we can reuse all the existing sort of events infrastructure logging and evm that's great if we go this other route 48.95 which is maybe in some sense you know cleaner more elegant uh we basically have to recreate all of the receipts infrastructure and uh that basically is a lot of work so i essentially want input on this call does anyone have any preferences on either route have you had time to look at these eips do you have any feedback i have a quick question is login actually very important so it doesn't have to be um i think it is like pretty nice ux uh but yeah i mean there's probably other ways to figure out that your withdrawal is processed as a validator and yeah it's really just a question of like what what kind of facilities do we want to provide validators right i have a comment on that uh sorry oh i was gonna say what's the argument against having some way to log this you know it's the arguments and to not have them yes right right like it's complex if we go i guess the system operations route the so yeah i mean i can just click through so basically like to give you a sketch like it's literally just having like uh is that important okay sorry it seems connecting very weird today that's okay anyway like basically it's just like adding a whole new field to the block and so because of that we can't necessarily directly reuse uh you know the receipts infrastructure that we already have so we'd have to like duplicate all of that and that's where it starts to get like quite hairy right right and spec says that in case of withdrawal operation it must never fail it's like unconditional one so you may basically use um like this withdrawal operations as logs or a kind of vlogs yeah i would i i'm definitely agree that the blogging is like we should either we go to some contract and no logs or we do something else with logs i am definitely not a fan of trying to get logging in or get receipts in with the system operations stuff right so does anyone go ahead well first of all i just wanted to say to mikhail i think the spec does not say that it must unconditionally succeed um i think it originally this text so if you had defined your your es1 contract recipient which is something which would never ever accept anything um then you would never be able to withdraw because no matter how much gas you specified you can still play but i'm kind of so this is this version this is just like a balance update right so that's kind of so both of these options are they both centered around the um yeah the gases the one where we just do it yeah these are both push good okay that's one of the questions we did analysis of existing 0x01 deployed contracts none of them rely on code execution and we've spoken with some of the deployers that had logs in there and said they don't need i think that the system approach is much cleaner and it's a very important operation so we want to do it in a reliable fashion and the transaction is just abusing the notion of transaction because it's not really a transaction it's just a balance update but it's not like it's not a balanced transfer like if we want some logs on the avm side things like that then it should be crafted specifically for this operation because uh it's yeah i wouldn't have used the notion of transaction for this which is fine but then i'd probably suggest going with this option to route so we build out a new operation thing uh but then yeah just drop the logs because i think it's going to be way too much to like have a whole new like receipts try half tooling testing etc to cover all that but then again like do we need access to this from the evm side because if we don't like if the observability will be there you can observe it by the looking at the header and the withdrawals as was mentioned they cannot fail right so if you see the withdrawals in in in the header then you you you see all of them but the question is whether you need this observability from the evm side as an evm op code right and i think i think part of where this came from is we looked at the existing zero uh sort of eth1 credentials that had been deployed and the only real thing anyone was doing was logging uh but yeah like danny said we've talked i think to some of the bigger players and it's been fun rocketpool's really the only one doing the logging and they said they hadn't actually expected code execution but just kind of put that in there as the best coding practice so uh you know they're not even necessarily expecting the block so um so as for my five so yeah and i'm sorry i'm not really up to speed on all the details but um i agree that uh pretty much transaction is abusing transacting the constable transactions however block body is now a list of uncles it's going to be empty this bank will say after the merge and it's a list of transactions and there is a lot of code out there that um purchase bot is based on this and youth protocol which have the capacity to request these pieces of information from another peer if you want to add a third aspect to block body i think that might be a lot of work that um a lot of code that needs to be rewritten i would like to heard if felix of peter have any thoughts about this i don't know if they're on the call though i don't see them on the call so the the two options for where to put these system transactions in the block i think right now are just a pen to the end or take over uncles previously peter had argued pretty strongly against taking over uncles and he strongly prefers um appending new things to the end and just throwing away like accepting the cost of the extra bytes will there be a lot of withdrawals can we simply put them in the header rather than the block body we can a fixed amount per slot and they will be so we could decide on a number you know it affects kind of some of the ux here uh but the exit queue is already bound to approximately four ish per slot um so after you clear out you know maybe some large amount of withdrawals at the beginning the bound doesn't need to be much more than that and the and the consensus layer will have a bound on what how much putting in here because there will be you know a maximum cost of this operation on the system um and that number can be turned depending [Music] so it seems and please correct me if i'm wrong here we it seems like we have rough consensus around the system level operations approach and it's like a question of how rather than like if um does that generally make sense yeah and no logging right and no logging and i think so i alex i don't know i i think the ether scan people had said they were fine with like either option as well like i think the one thing we want to make sure if there's no login is yeah that folks who monitor this stuff are still able to to access it and expose it right and if you're going block by block like the withdrawal will be there and we know that it succeeds so yeah i think the one use case you don't really get is like i'm a validator i turn on my node and i just want to i have i know my withdrawal index and i want to ask if it happened or not you know where it happened efficiently otherwise yeah you can scan i mean and they're sequential so you can do a binary search uh to find where your where your receipt happened or actually yeah you can actually know if your receipt happened very quickly because you can look at the latest withdrawal um and if it's greater than your receipt index and it has happened so there's there's there are things there you can do without logs to probably handle those these cases you care about outside of the evm okay and scar you have your hand up yeah just briefly wanted to ask and um maybe there's an answer the document but for potentially putting the full withdrawal into the header like what's the size of a withdrawal is just the two address and the amount or is there anything else because then it's basically barely bigger than just putting the hash in there right it's an it's an address it's an amount and currently an index and what's the index used for and the execution on that side yeah it's to differentiate it's when we were going with the transaction method uh and any sort of logging and that kind of stuff it allows you to differentiate um it also would allow you to do you know a search like i just talked about uh more easily um because it's not necessary amount amount and address are not necessarily unique um given partial withdrawals yeah but then that still seems only 2x basically to exercise of just the hashtag so maybe it's just the easiest to at least put the focus [Applause] but you would have more than one per block all right i guess okay in in terms of next step does it make sense to move uh the the system level versions of 48.95 to consider for inclusion and have people kind of keep obviously looking into that and kind of figuring out the quirks around it but just so uh yeah we can make sure we're kind of all focused on the same thing sounds good to me yeah i guess yeah does anyone have an objection to that nope okay um on the on the partial withdrawals i just wanted to say yeah i have this tracking issue on the consensus layer um there are three key features here one is to fully withdraw exit validators the other is to change credentials from bls credentials to uh execution layer credentials such that you can perform withdrawals and the third is a partial withdrawal option um which i'm working on in a pr right now uh i think that this is one from the from the perspective the execution layer uh all of them just look like things being withdrawals being dequeued into the uh execution layer and so that that complexity doesn't really matter but from the perspective of uh validators and and features i think is a pretty critical feature um to not put crazy pressure on validators exiting about it for sure so it is it is and has been kind of in the the consensusly real map right and uh rtm do you want to take maybe a minute to kind of explain what your proposal was uh yeah it's kind of obsolete now like the new push-based proposal is definitely more preferable from the point of view of liquid staking protocol as well yeah i'd just like to know that partial withdrawals are crucial for us yeah and um no i think for even just for many other use cases as well they're pretty critical to the experience and yeah just for the health of the beacon chain you don't want people having to withdraw they're like 33 that's something east and then just redeposit it on the other side um yeah and we've got a second part of our proposal which is more related to the question of how to pass some intentions from execution layer to the consensus layer like when validators like to rotate its keys or do a forced exit but it's out of the scope for this call right this is like more of a consensus layer call type discussion yeah and i yeah i'm happy to uh i haven't had a chance to read the proposal but i do think that that is a very nice feature and actually protects against a couple of like weird withholding attacks that we should talk about anything else on on withdrawals and so sorry and the maximum is four or forty four is the amount number of exits per slot currently um and so the number of withdrawals per slot you would probably have in that same order uh it depends on the partial withdrawal scheme but i would say 4 to 16 are the the realm of what we do here there's no execution block in a slot do those withdrawals destinated for that slot can move to the next ones you have eight in the following not currently no okay that would put a you know unbound cost on the execution layer with respect to this if it's only four or even 16 maybe we can put them in into the header um but if if you foresee that in the future it will be more than maybe to be future proof then yeah it should be better to be in the body right yeah and we yeah i think we can take that offline as we're looking into the implementation details um anything else on withdrawals uh i could also say that there from the point of view of stake in protocol there might be a slight desire to be able to distinguish partial and full withdrawals like maybe different addresses but it seems to be too difficult to implement and not not very crucial for for us just to note right yeah i mean with the beacon root up code all of that becomes possible um it's a matter of that existing or not um okay yeah just to move on because we only have 20 minutes and at least one big topic left uh proto you had an update on eip four four eight four four never pronounced that right uh for the for the shardbob transactions um exactly hello everyone give your very quick updates so since the last awkward devs call we have worked on consensus packs execution apis we have built a meta spec to link everything together i think that relates to the proposal tim is going to share where we're trying to find a structure for this cross-layer erp process for now we'll just try and work with every specs repository out there we also have exclusion apis now and we have benchmarks as well as implementer nodes uh the benchmarks are very recent i'm not sure if anyone had the time to look at those yet but it boils down to having to batch verify these blob transactions to work around the processing time issues so this requires some minor refactoring to take multiple transactions and verify them together and then the blob spiff in a transaction can also be batched verified all in all i think it's not too bad and the same applies here to the consensus layer that can also batch verify blobs and we're working on testing with prism and working on tooling to to try and get the devnet running nice so uh if you have any tlr on how so say someone sends me a batch of 1000 block transactions every single one of them isn't valid but they're constructed to be as hard as possible to verify what ballpark we're talking about well if you receive all those transactions from the same pair you can batch verify them and then if everything is invalid you can start scoring down the pair if one of them is valid and you need that one transaction to go through then you have more of in a higher situation because once you figure out that the batch is invalid you'll have to go and bisect the the thing to find the valid uh block transactions in the end of 3d you you want to like penalize players that are giving you this bad information in the first place like this is like provably bad so you like it's objective um and it doesn't affect consensus like this is in the transaction pool like this is the step before that so i think we need to improve the the peer scoring system in the exclusion layer to help filter out things and things do go wrong i can speak for all clients yes do not do peer scoring there is no such concept the consensus layer has extensive peer scoring and it really does help avoid problems with like bad at the stations or these other high throughput things what about but yeah right so we don't even ban parents no because i mean note ids are free so there's no point to balance pierce um [Music] we do kind of have some ip limits but yeah we don't spend a lot of time scoring pairs we kick out here if it does something bad but note that these are free so i guess the point is you can get a thousand of these transactions from somebody but on the first you're gonna stop processing i mean i can turn up a thousand node ids uh connect them to a guy and send them a thousand transactions from different identities uh for free um well there is some processing power because we need to generate these transactions at some point but then i can reuse them um and they've i mean the people that i sell them to will not remember them indefinitely if i have enough of them and so i could just spam the network with them and what i'm curious about is yeah how costly it is to avoid the attack so a single batch of blobs is somewhere between 50 and 70 milliseconds to verify this bench can be very large um i'd say at like the 40 or 50 inch milliseconds is for like a single transaction like the batch versus a regular like single transaction is it's not that different uh so batching is important here where we like avoid cost overhead of adding a lot more to verify in the end like it is processing and we should just like use that information that if you're getting something that's wrong to try and ignore more of that before we get the denial of surface um well how would that look i mean no different um current transactions so like if current transactions are invalid you also ignore them and hopefully like you cannot deny you cannot toss a gaf not but just keep by keeping this these lower these smaller transactions up like it doesn't matter if you send like a thousand small transactions or like one large transaction if the verification is not does not like if the peer that sends you those is not held accountable you do need some system to give feedback to what is being received right but like the only tool that is that we can trust is that if someone sends me something then i can disconnect from him that is the only guarantee we have there is no guarantee that you can rely on a scoring or to be effective right and you're saying node ids are free but ips aren't quite as free yes um and yeah so it might be possible to ban eyepiece but that can be tricky it seems to work for eve too like the whole pair scrolling system i think it's not too difficult to emulate some of that small substance enough to say like if transaction is invalid we should consider this pair as as bad and prioritize others yeah with it i mean i guess they do have pure scoring it's just a kick you know that's binary hmm i mean this is binary too if the transaction is invalid that's all because of the pair that's sending you the transaction it's very clear that there must be anything yeah yeah i just don't really see yeah okay we don't have to go too deep into this actually on this call i'll read up on the benchmarks and specs and stuff yeah and oh sorry also yeah go ahead proto the next car the agenda has lost the metal spec links the benchmarks and these optimizations right and what i was going to say is just that i think it's at least worth considering as well that there's no necessity to have a mental support for these transactions live right at the time they could remain at so basically this would not necessarily have to be a binding constraint bringing into mainnet we could launch without support and then for one of course it will be a slow wrap up for using those and it's fine for neither in the beginning to just only because of course notes would support having that locally affected mental so like they could just run their own staking notes so they could operate or they could create an off uh like a separate network for that like basically it's fine if we only add that support for memphis later of course not ideal but so this is not necessarily like strange to the timeline right andrew you also had your hand up and i think took it down and back up so do you have any comments you wanted to make okay uh anyone else have comments thoughts on 4844 uh i do think that benchmarks from a number of or stress tests from a number of different places are probably pretty important um you know just the consensus layer decrypting one to two megabyte blocks uh you know and then passing the execution layer there's just you're going from 20 kilobyte blocks 90 as we are with the merge i don't expect things to burst the steams but i do i do think that it's not unlikely that once we get to one megabyte that like little things we didn't expect um start to operate in different ways that we are unexpected in bad ways i don't think any of this can be intractable solvably but um i think that it's good to investigate something sweet anything else okay um last on the agenda i had something so uh we discussed this uh i think on the last call uh if not on the discord after um basically uh the the the core eip process is kind of reaching its limits with uh the merge where we have a completely different process on the beacon chain than than on on mainnet um and we are starting to have proposals which uh clearly span across both so the two things we talked today are talked about today are good examples of that um so it's quite hard to like reason about like what the entire spec for something should be and and how the different parts all work together um and in parallel uh there are folks working on an executable spec for the execution layer um which aims to kind of over time uh complement or replace the yellow paper as a canonical spec for ethereum um so i had a proposal that uh yeah i put together about how we could harmonize all of this um just shared it in the chat at a very high level the idea is that uh we would keep core eips as the way to like describe changes provide the motivation the rationale uh list security considerations and also just have like a eip number that's easy to reference within the community um use these for both consensus layer and execution layer changes um but then over time basically move the implementation sections to the execution specifications rather than having them live directly in in the eip itself um and so that you know the benefit we get there is that uh a is like harmonized across the beacon chain and the execution layer b uh you can link both so if you have any ip like you can chain withdrawals uh you can just say hey here's the change to the execution spec here's the change to consensus specs and maybe even the api repositories and then see uh there's always been like this big uh concern with like that we don't have a lot of eip editors uh so we want it to be easy for them to actually review eips and one of the one of the things that's actually quite hard for them to review is when people put links in the ips because there's a bunch of dead links over time it's hard to assess the quality um so by having links out to just the different specs repo and you can have a pretty easy to enforce rule that's like you only allow links to you know these two or three repositories um and and and just blocking the ip if it has a link elsewhere um and then if you know the eip author wants to add a whole bunch of links as part of their pr to the to the specs repo then they can do that um but it's not it's not like blocked in the eip process um i know greg you had some comments um about this is greg still on the call um yes yes so yeah greg you had some comments about this i'll i'll let you share them i also put together an east magicians link for people to discuss um yeah great a lot of this we'll just need to discuss as editors we've only got about seven minutes left so i don't think we can dig very deep um there's some good ideas there but i think it's a lot more intrusion on the eip process than we want to see and in some ways it's making it harder the whole point of the executable spec is it is another client so in the usual process the clients often with the help of the eip author implement the eip and the beauty of the spec is that once that client is running and is on the main net and in consensus that client becomes the reference so i i actually don't expect that a core ip eip could be a totally complete and accurate reference when it's done the the network itself is ground truth and so having one client that we can point to and say we intend for that to be the actual reference is great um but whether we try to pull that back into the eip as a diff against a particular implementation doesn't doesn't really seem to help matters i don't think that's where the bottleneck is and i don't think the issue of reference is is really directly related that that's a different discussion we're having i disagree on that one too but enough and danny you also have your hand up yeah so just a quick follow-up on that and then i have a quick comment um on the consensus layer it isn't a full client it is actually um you know implementation of core state transition logic such in very non-optimized ways in ways that uh just expose what the logic should be rather than uh the sophistication of the logic as it will be in a client and so it can't run on a maintenance it also doesn't have networking interfaces and other things yeah so yeah and then we can build test vectors of it so i just there's there's a different there's a spectrum of what you can do with it um and i i just wanted to say that out loud and then but i will say i don't know if this helps our eip editor problem i mean it just shifts the burden to a different highly specialized group um you know on the consensus layer there's a handful of people that have the ability to review these types of specs and provide rich dynamic feedback and sometimes uh there are pr's that are open for a very long time because uh it's hard to take the time to to dig into it so i i it may be it's useful in getting more people on the table but i don't know if it like solves the eip editor problem i do think it maybe solves other types of problems though i think it makes it harder for editors i can't we have to become experts in yet another thing to be fair right yeah it's not expected that the editors would have to review the executable spec as well like i think like so like daddy said you do have different people who then review the actual spec and like they might be the same humans but you don't impose that so you don't impose basically like a technical review of the executable spec before merging the eip itself yeah so so eip editors would only be responsible for ensuring that the format is correct that like like the the tooling says the right things but the actual review of the content of the diff would be probably by you know uh core devs right so i guess i don't see how this helps i mean there's a huge overlap too like if i i would have a hard time reading a metaspec and being like okay this sounds fine i'm not going to go look at the code that rep that actually has all the logic i just take a look at the wasm spec it's entirely declarative way back in appendix there's an algorithm and they make clear the algorithm's not the spec it's an example the spec is totally declarative it's up to you how to implement it so there's it's just not clear that this language is going to be the best way to actually say this is what i want to do um so to ask someone with an idea to improve the protocol to say oh but first you have to figure out how to say it in this specialized language that may or may not be the best way to express your idea um you know a paragraph of english can turn into a page or two of code when the english was totally clear to anyone who knew how to write the code yeah martin you had your hand up oh sorry go ahead stamina i think that's a fair criticism i mean i think um one of the the it's kind of a trade-off right like a lot of things will be easier to express without having to describe the current state so that's a problem that comes up in eips a lot today is that to say how you're changing something you first have to have to define how that something works um and you know the this code if process will improve that part of it but you're right there are some changes that are going to be way easier to describe as english that you'll have to implement as python if we go down this road so yeah i know just enough python to hack a script together um or to to read fairly simple descriptions of structures and stuff but um yeah and if you do take a look at the consensus layer specs that's that is the requisite knowledge um and it doesn't use anything pythonic and it does not compose things in weird ways and is pretty much it doesn't use even like complex uh not even complex but just pythonic type construction such that you you know you have for loops you have variable assignments and you have very simple data structures um again i i i i think there's i'm not trying to make a claim one way or the other on on the best ideas yeah i wouldn't see the the eip process close out and then this this this executable spec is its own thing um and i think it's there'll have to be some way of vectoring over but trying to do both in one process i think will only make it harder i mean when i put in code for instance that code is often going to be uh go not python because i might have implemented the algorithm in geth and then i can put in code i've actually tested and translating the code to python isn't that hard for the core group that's doing it but it's hard for me um now if if we want to pull that text back in to the eip at the end of the process that's not too hard or if someone who knows that stuff wants to become a co-author that's fine too but yeah martin you were about to say something um yeah well initially i was gonna say the same thing that i said originally that doesn't need to be a full-blown you know network machine that can little minutes but then i want to say that you know i've spent quite a lot of time uh reviewing um eats and also implementing eeps and the english language is great but when you translate the english language into actual code when you that's when you find all the corner cases and things that were implicit in the english language and when you put it into code it needs to you need to make it explicit and that's when you find the quirks in the corner cases which shows that this specification was underspecified it was too vague so i think it's good if we get closer to how it looks on these 2.0 side where it is code where the author who put this down was actually forced to figure all these things out and anyone who actually implements it can can basically pass his implementation against the reference implementation uh or just transpile it into his language so i think that lowers the lowers the amount of work needed to be done at five different client implementers um [Music] since the work of you know putting it down in the code is only done once in the like translate from english to code it's only done once by the author or someone else so i think it's good to move in that direction but there's a spectrum yeah they do get implemented the translation they get implemented they get translated however whoever is writing it might find it easier to read the english might find it right but it's kind of like the way it is now whoever implements it first does one implement one interpretation and things yeah that's obviously how it must be and then someone else does another interpretation implements its slight difference because you know yeah it would be so i think it's nicer today which which is just yeah i'm just saying if if i have an eip i've completely implemented it in go or c plus plus in a client and then it's like well that's all very well and good but it's not an acceptable eip until you translate it to python and it's like but um it's expressed here and go and the go works do you want me to translate it to something else that i can't actually run and test no but i guess what martin is saying is if i understand correctly if that's not the case for like the median eip the median eip is like basically underspecified in the current format because yeah and and so this spec kind of forces you to at least fully specify it and or or realize that you can't and that you need to do some more work basically yeah yeah i'm just saying that happens in the process by the time by the time something gets accepted it will have been implemented in more than one client we will have found these cases the authors generally aren't in a position to do all of that implementing and testing anyway so the question gets more at the tail end when it's actually working how best to express that and currently slowly the eips make it into the yellow paper and the yellow paper is is the canonical spec and we can change that but trying to get the authors to write canonically up front it's going to be hard you're going to need an expert to work with the author to do that and i think that's just going to be even harder to find that person but if such a person stands up and volunteers yes we're already uh five minutes over time so we can continue this on each magician so i i shared the link it's it's in the agenda yeah and i pasted all your comments from the agenda indeed um anyone have anything else they wanted to share before we head out yeah to echo what like client said you can run the full python implementation for the consensus layer side and i think very importantly the tests that you write for that implementation for that python spec actually become the consensus tests and so when we have people build new features they also write tests and those actually become the reference tests uh whereas i think when we have many different clients and learning eips we don't always capture all of the edge cases and reference tests uh even though even when we're kind of like cross testing our implementations and stuff so it's just another another component of this process that can be useful right thanks sir yeah how to connect anything else before we wrap up okay i think yeah worth noting i think europeans your time will shift before the next awkward devs uh we are not shifting the awkward devs time so it'll be at a different local time for you if you live somewhere where daylight savings time isn't um yeah thanks everyone for coming on thanks to everybody who didn't drop off halfway through after the merge stuff um and i'll see you all in two weeks thank you thank you thanks bye everyone hi