# Hybrid PBS from CL's Perspective by Terence Tsao | Devcon Bogotá

- Speakers: Terence Tsao
- Channel: [Devcon](https://streameth.org/devcon)
- Date: 2023-10-07
- Duration: 30:23
- Watch: https://streameth.org/watch/yt-mXGA-u-sH_w
- YouTube: https://www.youtube.com/watch?v=mXGA-u-sH_w

## Description

Visit the https://archive.devcon.org/ to gain access to the entire library of Devcon talks with the ease of filtering, playlists, personalized suggestions, decentralized access on Swarm, IPFS and more.
https://archive.devcon.org/archive/watch/6/hybrid-pbs-from-cls-perspective/

The talk seeks to address the following questions:

- Why does PBS matter from CL's point of view? What happens if we completely ignore it on the protocol layer? What if we wait to release it?
- How did we implement hybrid PBS on CL clients from history to functional products?
- What do they look like in code?
- What are the gotchas and the trade-offs?
- What are the remaining questions we shall seek before moving to full PBS?

Speaker(s): Terence Tsao
Skill level: Beginner
Track: Layer 1 Protocol
Keywords: MEV,consensus layer,PBS

Follow us: https://twitter.com/efdevcon, https://twitter.com/ethereum
Learn more about devcon: https://www.devcon.org/
Learn more about ethereum: https://ethereum.org/ 

Devcon is the Ethereum conference for developers, researchers, thinkers, and makers. 
Devcon 6 was held in Bogotá, Colombia on Oct 11 - 14, 2022.
Devcon is organized and presented by the Ethereum Foundation, with the support of our sponsors. To find out more, please visit https://ethereum.foundation/

## Transcript

foreign [Music] PBS instructions layer I'm Terence I'm a core Dev and uh I saw earlier this week we are product of chain Labs so I had to change my name real quick before I get in trouble so let's get started so this talk is about consensus layer interfacing with hybrid PBS it's not so much your like Searcher Builder your typical Mev Towers so we want to understand more like what does it mean when transcends this layer interface with hybrid PBS from like from the it's it's tonality such as latency file and censorship then we'll also talk about mitigations just to reiterate what I said everything in the purple we will go over it we'll have three layers matte boost which is a relayer aggregator we have consensus layer client a solution layer client ability to the client and this is what we have been me and my team and other teams have been working on over the last few months so background why are we here right so there's options and options are nice so the first options which most people use these days is normal blog processing so as the consensus of their client you know which validators you're serving you know when validators are proposing a Blog so you will build the blog for the validator and post merge we utilize execution layer client to prepare payload so consensus of that client uses attitude on their client to prepare payload they put a payload in the blog that passes to the validator validator signs it return the blog and consensus their client broadcast the blog so here the separation concern is nice here because like it's usually a client and because this is their client they're both like very complicated piece of software but they only need to know each other through the engine API and so the separation so the separation of concern is very nice the second option which people are starting to use more and more now is that I can Outsource blood production right a consistently client if I want to participate in the Mev game I can utilize the relay Network I can hey hey relate that what can you propose a block for me and a Blog are usually more profitable and also in the background consistently client can also talk to the attitudinal kind of like hey can you make me a backup blog in the event that the relay network doesn't work I can still use it and this is kind of the Paradigm that we're heading towards and it's important to understand like what does this mean so this is today's number and hopefully you guys can see it I capture it this morning there are 55 of the network participation is using Mev blog production it is outsourcing their blog and then out of the 55 81 percent are dominated by flash boss relayers and the Seven active three layers so let's talk about the first risk latency right so when when you propose a block normally under your local setup you have your consensus layer client execution client without their client so how does this work you ask educational client to prepare a payload and then you pass the payload to the other client to sign it well their client sign the payload and then you broadcast the payload simple easy very easy to reasonable right but with Mev blood processing it's a commit and reveal approach right why is it coming in reveal because like as a relay as a builder you don't want real data client to steal your um transaction if it's if it's if it's in the clear text so you made them sign it first then return the signature and then then you give the full transaction so given it's a community approach there's a more like Steps in the middle right you first you get a header the relay Network return the header then you sign the header and then you submit the sign header and the relay Network return the full payload which is transaction in clear text so as you can see there's a there's like two more steps here right and besides that it's just like it's also on a different network right it's not local anymore so ideally like you want to relay Network to broadcast the block as fast as possible instead of giving it back to you and then you broadcast it so hopefully like most of the relayers have been like publishing blogs like by themselves instead of just passing back so for that you do save some latency which is nice so let's ask yourself right do the additional run through and latency matter because like I said before when you have your El and your CL they are both in the local setup it's lightning fast it's it is reliable because they just go through Hardware they just go through electron circuits right but if the relay network is slightly different you are actually talking to some like um infra provider gcp or AWS on some regions and um so this is I was able to capture some numbers unfortunately I haven't been home for like like the last two weeks so these numbers are slightly old but on growthy with a thousand validators they're they're not mainly validators but yeah just a thousand thousand validators but given it's worthy the network topology may be slightly different but I mean I hope we can get the picture here right here I'm not even using Mev Bulls I'm just talking to the relay directly it takes about three times slower to propose a block given the additional latencies right so what does this mean when you when there's additional latency because when you propose a Blog you shoot it how it works it's like you run a four chairs to get ahead then you build a block based on the head then you get the payload from the execution engine then you broadcast the block right and then at the four second mod which is one third of the seconds per slot the testers will vote right what is the head of the chain and if the testers did not see your block then your blog may get orphaned and that's not ideal you do not want to lose a lot right so this is what we don't want we don't want something that's taking up so much in the Middle with the gay header second header and suddenly blind block right so this is this is this to me is worrying and then let's look at some more numbers some block arrival latency differences this is actually capture on mainnet this is my home setup at home with 300 Mega bandwidth so this is a home setup as possible and we capture over 15 000 samples and map and then the MVP block took about 500 milliseconds longer right and um and what does this mean from the uh some big anticipation timeline it just means that if the block takes longer to reach to to arrive then a testers will unfortunately miss it like that right you don't if you're waiting like in the front like for that long then you would essentially eat up the time that you have so we don't want to be the lazy block we don't want to be the late block because you would get orphaned right at the top example Block C was supposed to build on Block B a and then because the blood B was laid on the bottom example of block e instead of building on block D because of proposed boost or something it was supposed to be the head but it's not the head so block e built on Block B therefore C and D got orphaned right so another set of numbers 50 of the orphan block actually came from relayer from September 17 to 27 and there's our the orphan plus law the relays they're using validator ID the entities right and that's unfortunate I mean I mean you could ask that okay well maybe it was gonna get off and one or another we don't know but still like 50 off they come from the relayers it's not just like sunshine rainbow and Butterfly right this is like like this is I mean there is risk to this right I mean like we often tell people hey we should use the relayer because it's more profitable but it also comes with risk right if using a real layer makes your blood two to three times more profitable but you get offering ten percent of the time is it worth it that's something that you have to ask yourself about and another risk I think with latency is your centralization right the whole point of Mev Booth is like to make validated decentralized because now we can extract Mev at home so it gives everyone equal access to Mev you don't want to look like this right you don't want people to start realizing oh well if if I have a better latency then I won't get orphaned so I'm gonna move by a home stating node close to the relayer so it it becomes a negative externality this is what we don't want so what's the takeaway with this right the takeaway is latest thing matters both for hybrid or even maybe for a little PBS right and latency can lead to centralization risk which we have known a long time already and it's actually really hard to optimize towards Network latency right at the client level like I don't think there's much we can do I think on the map boost side there's talks about like instead of Marshall and Marshall Json with emotional module SSD the there's something improved from there from the real perspective I hope like they have really good Network config they have a lot more peers I feel like they have a robust like infrastructure there I think that's as much as we can do because unfortunately if someone just have a slower internet connection they want to use a real layer I mean this is going to be affected right and it's important we educate these risks besides latency there's also thoughts too right so what does fault mean fault here means that sorry the stock got a little mess up there point so when you ask for the header the relay Network failed to return the header so that's the convey phase or the happens when you submit the header the relay failed to replay failed to reply the payload so that's the second fall so we'll focus on these two type of faults here right so the first so the first category of faults is just gear header false when you get header and then the relay net will fail to reply and they can be categorized as like you have a mail form header you have a consensus invalid header you have a payment evaluate header or you have a non-conforming header so I will go over them one by one so what does it mean when like a head is male form right it just means that it is syntactically invalid it has an invalid structure it has to email the signature can the consensus can can the CL client detect it yes it can right which is when you are more show it if is not the right structure then well you know it's run you also can verify the signature so this type of so this type of like false we can detect and we can mitigate so this is fine another type of fault is just consensus value header so that just means that the block hash is reality the transaction similarity but for this we unfortunately cannot validate because we cannot see the full transactions right we are we cannot calculate the block hatch ourselves so that's something that we just have to trust blindly that's unfortunate but it is what it is then you have payment invalid header so at this type of fault it just means that well the Builder was promised to pay the validated proposal some like some if but it failed to pay so consistently consensus that your client cannot detail that type of file this which we like relay we we trust relay to simulated that's why relayer is trusted for us and then there's also the non-conforming head that just means that when the validator register it basically say hey this is the guess limit I want to use but I guess the name is incorrect the time stamp is incorrect the block hatch is incorrect right can consensual client detect that yes he can so those are like what we say the commit faults so now we're jumping to reveal false so this is the set second type of false right so what are some like reveal faults so the payload could be invalid or or the payload is available and keep in mind there's no falling back for this like if you just you sign the signature ready right so at this point you can complain on Twitter you can probably like complain something else but there's not much you can do here to basically make the block as a whole header it just means that it is invalid like the block the flip Halo does not match the header can the consensus layer client validate yes it can but hey it's too late right you at that point you lost the block already there's not much you can do and it's the same with consensus in value payload you can validate the transactions now because we see everything but hey it doesn't work too late sorry then unavailable this this means that the relay Network just went to sleep when malicious decided to turn off it did not reply back the header to you so it did not fulfill its commitment right and then you know when it does that because he never replied he never received the payload and there's someone and there's still there's nothing you can do there because you already signed something your signature is out there we have this concept of falling back to a situation layer client just if gear header goes wrong you can produce real local execution client that's that's that's probably fine but if the gay payload fails then you cannot produce with your local assistant client because you don't want to double sign so the return header the first commit can fail two ways it can either fall or time out right we do prefer for better just because like you get a response right away you can start proposing the blog right away if the timeout it kind of sucks because you have to wait for them to time out you wait for them for a second then you lost a second of like your apprecious time there so let's go through some many incidents I don't want to sound like I'm like like pulling out their fault incidents but I do think it's important to go over this type of incidents so that we can learn as a community right the first type of incident so the the first incident is um September 16th flashbacks relay they fail to our Marshall deposit for the payload reply the damage we missed three blocks the second one blocks rewards relay this one is when the this one is going to relay did not validate the block and then and then they rep and then they reply it the Constance smelled it we miss Ada blocks here the third one is velociros again this is consistency value payload and the damage of that is 15 blocks was missed so this type of things do happen right first happened and and then it seems like it's mostly happened on the on the commit and then also the reveal phase so we need some mitigations like the circuit breaker and for example as a beacon client I can detect when there's a liveness failure and then when the life has fitter is determined by clients say if you miss three shots in a row or if if or the chair misses eight slots over 32 slots if that type of things trigger then we just default to local X3 engine right so this is to prevent like a dominant relay Builder go offline and it doesn't solve like the cases I mentioned before just because like those are happen maybe like 0.1 percent of the time for this this happens all that but but for this like this is a stronger defense then you have your relay monitor for example such that you can monitor relay space on performance such as behavior and such as Behavior which is like safety and liveness and performance which is latency and people can see how the relays are performing and people can figure out okay based on that do I want to connect with every day so did it just makes the information more available then we have features like beef filtering solos so as a proposal you can you can say hey I only want to use the relayer if they give me something that's over this value right so there's some like so there's some nice things you can do there so what so what is takeaways right we're still early but we need more robust relay and also we need a way to hold relays more accountable right for example a simple idea I just you monitor all the missing slots and the offering slots live and then you pull the API the relay API and then if you see hey the missing slots is coming from the relay okay I'm gonna just like shout out loud Tweety or something so that people know hey they're like they're it's the incident right there is happening live you need to turn off your relay or you need to switch local processing in terms of just like faults time out I think I prefer Gear Head defaults then that perfectly had the timeout then I prefer forget Halo false just because like with gate header files you can propose the blog just like still and then it's it's most likely fine and I do believe that relay quality will improve over time just because like it's still relatively early-ish so I mean they're still learning and we're still learning and something that will proof over time so okay last section censorship that's something that we have been talking a lot about so as of today um let's see whatever set 49 of the many blogs have some sort of effect compliance building and that's unfortunate because that's already over half right so we have to understand like who is censor right like who can censor Builder can sensor right if you're a builder you don't want to build blocks they can send like offer compliant transaction you can censor or the relation sensor as well right so so therefore it's really hard it's really hard what's the problem here problem here is the enemy boost is a neutral piece of software it doesn't care about censorship right and then the ux of just figuring out like how do we defend and censorship is still early right now it just essentially everyone chooses the relay that's non-censoring that's it right but then it's hard to figure out who is censoring at the given time there's it's like you don't know who is censoring I mean we just look at the news with the real search and that's it we need more like information there so potential Solutions right these are very instrumental ideas that I just have been thinking top of my head right you can have some active encryption such as MacBook C or this you can have some solar essential issue or censorship Oracle so for active inclusion right high level how it works it's just like a proposal you express the intent to like force transactions into the payload right and the relayer has to present those slide transactions to the Builder and then proposal will only accept those transactions if they are included or the block has been full right so for this you do require some sort of multi-proof to make sure that hey the transactions are actually included and then consistently client can do the validation so what's the downside with that right that's not that we just like there's more timing because now as a proposal at the previous slot you want you do need to send the transactions you want to send a transition transitions during the previous slot so we need to figure out what's the timing for that and now the proposals also need access to the mempool to the institutional area State there's also some latency complexity here just because like now it may take longer to propose a Blog and then yeah and then you can also have like this is like a poor man version of just like censorship like um filtering basically like like proposal you just walk into the mempool for like the top earned transactions that's that's that's based on ksv at the given time and then when the reply the relayer is also have to show a proof that hey this top and guest transactions are included unless that the guest is at its price at a given time right so we don't really trying to force transactions inside but we just want to make sure that the top end transactions are actually getting in and and then which when here we're assuming that okay if someone's in the sensor someone will probably use a higher dsv and stuff so it's not ideal because it kind of loses like this inclusion control but it's probably easier to implement then there's also like ideas like censorship Oracle you can introduce like a new actor to produce censorship kind of like relay Monitor and now as a proposal I mean boost when I receive a header I can add this Oracle this new attribute hey is this a sensor one if they reply yes then I won't use it right but then again you're putting Trust on like a new actor and that's obviously not ideal because like who because who is going to monitor the censorship Oracle and there's always that problem so there has been a lot more research going on which I'm really happy to see right we posted a research a couple weeks ago on just like how do we constrain Builders without bringing blood proposal burdens to to the proposal there's ideas like using blood prefix and stuff and I highly recommend you to take a look at that and then there's like from Barnaby I think it came like um last week which I haven't had time to read it it basically enforces that proposal commitments on chain and then I think that's very neat I definitely want to take a look at that very soon so what are the take care of ways right I think they take away with this just like it's important to figure out like who can censor and then who can feel the censorship because there are so many actors in the future there's proposals there's there is some Meb boosts and then there's also the relayer right it's important to figure out who does what and then I do think like we need to leverage the Builder API a lot more because the Builder API is probably the best thing we have today it's the best defense we use Builder API a lot more we can like provide more ways to like to basically organize defenses against censorship such as like inclusion this block prefix and then there is like a spectrum of solutions and then we're out there which we're thinking but the simplest solution typically has like more trusty assumption so I'm not sure if that's the way that we want to go so some final thoughts I think like for me I think censorship resistance should be the highest priority a size from scaling and withdrawal like what is ethereum if 50 of the transactions are censored right that's something that we have to ask ourselves about right and I do think the hybrid PBS is basically our best two bars to defend against that because it allows fast iterations less and then before we enchant into full protocol PBS then we kind of lose the step because everything will be hard for based right for high PBS you have to Builder API then you can play around with that and then you can we can figure out like what works best and what doesn't work best and then I know like people that have been working on high bpbs such as Med boost relays Builders they have been getting a lot of like bashing and stuff I I don't think it should be that way I think we should be like working together as one team and to like to basically Advance this um Advanced this um censorship thing forward so yeah definitely um shout out to other teams that have been working on that I mean they are the real heroes so yeah that's all I have today thank you so much for having me and yes thank you clearance we have some questions for you hi uh so regarding latency the numbers you showed would more or less the same as Dennis in the first day so if you have 50 of the network proposing through three layers and 50 of the orphan blocks are coming from relayers I would have taken that as a sign that latency did not matter that would be the expected yes but I do think that we can do better from the orphan point of view right because now if you look at like 20 plus kids orphaned a week if like if let's say just have temples they get orphan then it's probably better than 20 plus video orphan right it just I'm coming from more like the orphans perspective because when something orphaned then it's obviously not ideal because that transactions could be included and there may be some U.S concerns there hey Terence thanks for a super informative talk um can you say a little bit more about the three relay faults that have happened recently like how were the malformed headers or or payloads generated and like how was that mitigated how do we prevent that in the future right I would say like the relay landscape there's still a lot of work to do there because I need more testing and these bad tests and more end-to-end tests and in terms of the fault right the first fault is Flash ball and then they fail to Marshall the deposit and that point is too late because someone already submitted the header and stuff there you have a signature so there's not much you can do there and then but then they fail to our Marshalls because they did not test um the payload with a full signature embedded and the second fall and the third floor are basically this is the second incident and the third incident are basically the same and I think the plot relay just so the block Rod relay did not validate the payload which is what they're supposed to do they're supposed to make sure the payload is welded before they pass it to the proposal but they just did not validate yeah hey turns thanks for the talk uh I was curious about the out of protocol CR lists and the proofs of transaction inclusion uh have you done any Research into it is it practical like would the builders or the relayers be actually able to calculate the proofs in time yeah there's something still under research I think like Chris from flashbots does high just open the pr so I need to look at it but high level I think like how it should work is just that say you're proposing a slot n at slot M minus one you have access to the main pool and then you see some different sessions they get filtered you basically I started minus one you send those you basically present those transactions to the relayer and then relay will also present those transitions to the Builder and then the Builder will essentially build a block just include those transactions real quick and then send it back to the real layer and then like so at the end after the signature is done when you get the payload right there so they they can include like they can include like the multi-proof to basically prove that the signatures sorry to make sure that the transitions are indeed like are are basically indeed included in the payload I don't know yet that something's still under research yeah hi uh Alex made coinmetrics thank you so much for your talk really loved it just wondering if you would comment on your opinion about maybe non-public relays do they exist is this a possibility just wondering as for now I don't know any and I'm probably relay I only know the seven relay that was present in MVB boost.org and I also think that if well if there is non-public relay It's probably hard for us to know just because it's coinbase is using a relay they're extracting Mev they're not open to public I mean that's not something that we can easily find out um why do we think that proofs help in that like what he was talking about before peripheral inclusion of transactions why can't you just if the things that are not included once you see the actual payload disconnect from that relay because I mean the relay even if there is a proof could always just not release the block and order the block could be invalid or there could be all kinds of faults that make the proof kind of meaningless and what you end up doing is anyway disconnecting from the relay in either case so just yeah we can just make it simpler on ourselves and just you know you you say please include these transactions if they don't want to disconnect from them and next time that won't happen no yeah I think yeah I mean I fully agree I I think that's possible and I think there may be a better solution yeah I think yeah hey yeah um I apologies I've already missed this before but you said that uh 50 of the network is making use of Mev boost do you have um more like fine grain numbers in terms of like whether or not uh like what the numbers of institutional stickers versus like homesteakers um I just know like an institution that I work at they're security considerations it may be boost it's sort of like a centralized yeah yeah actor at the moment yeah there is a site for that I'm happy to share with you after but yeah there's a site that actually I typically track like example Lido coinbase Finance like the top three like like just like what relays are they using and stuff like that and what percentage for each layer yeah we there there is active tracking on that but I don't have the data in this slide sorry yeah but they're yeah but there is a site for that given all of this a validator that's deciding whether they should run Mev boost or use Mev or not what would be your advice to them I will say this is a hard question for me personally and then me and my team have different debates I think for now it's too early to be using this type of Technology unless you know what you're doing just because we don't have any like public infra out there for I would give you an example just like for example for the second incident right when the block Rod relay has support it took them six hours to turn off the relay which they could turn off right away but as the public validators like we don't know right I don't have the access information be like hey there's something wrong I need to I I basically need to shut up MVP boost people the best thing you can do that people can go to Twitter but I'm not sure that's like the best like media for these type of things right so I would say wait yeah that's that's my advice so you mentioned at some point that if you can't reach the relayer your execution client you'll the sorry the consensus client will reach out to your execution client and build your own block it'll default to that so that makes sense if you can't get the header right but you mentioned there are two situations one where you sign the header and return it and then you can't get you know a receipt of that is that not a slashing risk no you cannot get the right so once you sign the header you you basically pass the header to the relayer you you don't want to like you don't want to use your local blog anymore because now it's a slashing risk because now you're assigning two blocks so so you don't you don't default to building your own Block in that situation no okay I misunderstood yeah good thank you very much there is amazing talk