Agent-based modeling of Execution Tickets by Pascal Stichler | Devcon SEA

so it's the last day last session uh before the closing ceremony there's still some people around interested in agent based simulation of execution tickets so before we wrap it off we go deep again um yeah um happy to be here for everyone coming in feel free to join in also because we are more of an intimate crowd if you have questions feel free to raise your hand anytime um feel free to to yeah let me know if something is unclear you can also ask questions via the Q QR code um feel free to scan it um I try to have an eye on the Q&A as well if something comes in um otherwise I would say we start right off um yeah the general structure um is going to be half an hour of theory a bit of background on this yeah execution ticket quick uh wrap up what's execution tickets again then some research insights what did we find in the agent based simulation um and then the second part is kind of a let's say open-ended part about uh running the simulation running the code on your own machine um I provide there's some information um how you can set it up how you can run it um and I'll be around to answer questions um and then it's basically you can run it as long as you want try different settings run different settings um yeah with this I would Dive Right In um maybe just as a quick recap why do we care so much about M why is that a topic that everyone talks about um me Boost payments since um the merge 1.7 billion uh US dollar that's what um yeah has been paid out via MEF boost so that's quite a big number that's $2 million us per day on me per average so I think what concludes from that at least for for me um means we should be very deliberate how we distribute me how we go about yeah Distributing it um what incentives it creates um $2 million per day it's quite a lot of money people go a long way to go around the protocol do other stuff so it's important important to design a mechanism that really works that is bulletproof and one of the ideas to do this is execution tickets um the idea was first introduced by Justin late 2023 as a test a proposal separation with the idea of separating the beacon the consensus chain from the execution chain um thereby kind of um yeah um splitting off the consensus part trying to isolate the consensus part and and having then the execution part that could be um yeah somehow differently allocated and the idea to allocate it was is basically the execution tickets u meaning the protocol itself auctions off or sells the tickets uh which are then kind of lottery tickets to be chosen as the next execution block proposer um so as a builder for example if you want to propose a block you buy an execution ticket you get into the lot and if you're lucky you get chosen to propose the next block the idea of the randomization is to prevent multiblock me and liveness issues if we would just auction off the block for example execution auctions we have the problem that there could be multiblock me strategies where someone in advance buys a lot of blocks and then runs this m strategy covering multiple blocks in a row quick recap again how is the process um basically without going too much into detail the idea is that first the beacon chain runs um for the beacon chain um the selected proposer proposes to Beacon prop um very similar to how it works right now um ATT test us um yeah vote on it um probably some form of inclusion list will be involved as well to ensure censorship resistance and then in the second round um the execution block is proposed by the holder of The Chosen execution ticket and then um this proposer can propose the block and this is also then um from the tests validated to ensure timeliness and validity of the block so when we look at execution tickets at this mechanism how to sell the block space the execution block space that contains all the mvv what are things to focus on so in our research we came up with four guiding research questions in this process one of them was basically The Meta question what's the objective what do we want to achieve what's the goal of all of this um the second one going along is how do we measure it um if we have a goal and we can't measure it it's hard to optimize for um based on this was then the follow-up Point what are the possible mechanism Des design choices that we have what is the what's our options space what's our design space and then the last one is if we run the simulation and do this yeah as a simulation based um what are actually the outcomes um how the different configurations score on this um when yeah looking into the first part first question what are our optimization objectives um we split it into on the one side optimization parameters then price behavior of the tickets um with the most important part being the yeah three objectives that we categorized as one decentralization it's for decentralization of the um consensus layer Beacon chain which is very important but also of the execution layer for certain reasons we don't want to sell all the tickets to one ticket holder um we want to have at least a certain degree of decentralization the second point is me capture the idea of selling the tickets is to also capture the MV at protocol level so we should optimize the mechanism for a design that actually captures the m at protocol level and the last one is more coming from economic theory block producer incentive compatibility it means we have to incentivize block producers that yeah propose the block to participate in the protocol in the first place because no one can be forced to do something we have to incentivize people to join in um so we have to find a mechanism where yeah people that participate profit from um joining into the mechanism um on the price Behavior side um yeah we focused on price predictability price smoothness price accuracy and then defined couple of measurement metrics where I won't go too much into the details um there will be a paper coming out soon on it where we more go in depth how we Define the metrics but I think that goes a bit beyond the scope of Friday afternoon um then the next step was to lay out the design space um when laying out the design space we had um different parameters that we looked at um it was for one um the amount of tickets how many tickets execution tickets do we we want to sell is it a fixed amount is it a flexible amount what's the target amount of tickets um the second part is should tickets expire so when giving out a ticket is it indefinitely running or does it expire at a certain point um third part is about refundability and related to resell ability if can I give my ticket back to the protocol or not and also can I sell my ticket um we looked at it more from an um yeah theoretical perspective on a technical perspective it might be hard to restrict a secondary Market um but on a theoretical perspective uh you can also divide between allocated and unallocated tickets um but it's uh in our yeah simulations what we saw it has quite a big of an impact if a secondary Market is allowed or not and then also look ahead period currently we have 64 slots look ahead should that be changed or not and then also pretty relevant the pricing mechanism what kind of pricing mechanism do we use um in our setting we looked at four different pricing mechanisms two auction formats first price auction and second price auction and two quarted price formats um an adapted version of an amm and uh similar to EAP 1559 style mechanism um and use this in the simulation to see how the market behaves um yeah then the next step was we ran the simulations um I didn't go through all the six um yeah specified configurations that we have um just I think that goes a bit beyond the scope but basically what we did is yeah we set up a first price auction and a different other um formats configurations um where we mixed and matched together kind of the attribut that we can have and tested it some of them are pretty close to the current setup for example the justtin time second price auction slot auction um some of them as the yeah later ones are more experimental the idea was to see a bit what's the impact what's the yeah impact of each of these um parameters on the outcome of the simulation what's the impact on um our metric if we run it um yeah in theory there are more than 500 combinations to be run so we could go on for a long time running different ones um so yeah we choose a bit some of them more as yeah examples uh then trying to run a holistic um yeah simulation of all of the possible configurations um yeah in the solution then or running it um what we saw is um on the metrics um I hope you can read it but the summary what we can see is that on the decentralization decentralization metrics um it's pretty hard to get to a um good result um what we see is we have a pretty strong focus on centralization in this market however on the other metrics depending on the configurations we see that a lot of them yeah score pretty good there's some outliers um where go bit deeper into in a second um but uh there are configurations that have a very high M capture have a high price predictability and so on um yeah to go there one step deeper or actually as just mentioned um decentralization as one of the goals so decentralization to be precise here of the execution chain is pretty hard um remains hard um um we tried a lot of different configurations um however the problem that we have it's also consistent with yeah the theoretical literature that you're doing an ex an pricing so you're basically pricing your execution ticket on an expected value in the future um you expect what the value will be in a day in seven days uh which means you're kind of yeah using the mean you using the mean or your expected value um and this leads to one ticket holder one block Builder who has on average the highest intrinsic value has on average the highest uh bidding value to win most if not all of the auctions um so what happens is that kind of the specialization effects that you have right now where in certain time periods certain Builders are more specialized on high or low volatility environments this is kind of smoothed up out due to working with expected values um which leads to this strong centralization um which yeah also um in the meantime a lot of theoretical papers have shown that this most probably is the case um however what we saw in the simulation which is pretty interesting is that the secondary Market reduces this effect because what happens is that the um ticket holder the buyer with the highest average expectation buy the ticket however then sells it on the just in time secondary Market um and on the yeah just in time secondary Market there might be other builders or other yeah participants that have an higher value just for this specific slot it can be due to propietary overflow it can be due to volatility effects it can be due to different other factors um but um there we saw that actually the secondary Market leads at in the Redemption of the tickets to a lower centralization on the second main objective capturing me at a protocol level um we actually saw that uh in a lot of simulation uh we get to quite promising results um we get to Value somewhere between 70 to 90% of the theoretical me that exists um captured at a protocol level um we you can see here one exampl chart um where we see yeah there's a certain level of theoretical M part of it is captured by the ticket holders and then um quite a high part is captured at protocol level as well um on the on the pricing formats what we saw also quite interesting is that actually auction based formats worked pretty well um so and more specifically we tested Ste bit first price auction and sealed bit second price auctions um which both got similarly well results there was slight differences but overall pretty similar and also the amm sty pricing worked pretty well um however was with a bit of a lack because it always adapts kind of afterwards but adapts pretty fast uh while the 1559 style pricing um had significantly lower Meb capture I go also a bit into problems of this uh yeah in in a second um so overall summing it up on the two say main objectives decentralization is difficult me capture works well with the mechanism in different configuration different designs um yeah just a bit of a deep dive on the E IP 1559 star pricing so basically how we set it up in the simulation very similar to the yeah standard EIP 1559 pricing is that after each slot it was looked at how many tickets have been sold if a lot of tickets have been sold the price increases if uh no ticket has been sold the price decreases um which has from the mechanism the problem the inherent problem that you're only yeah looking at it after the fact and you kind of have uh fast changing demand as m me in general doesn't if you look at the data it doesn't have faces of high and low meev but there's this fast reversion to the mean and so you're pricing it too late in a way to when you want to price it which leads to oscilating pricing effect so actually the price starts to fluctuate stronger and stronger um to a point where it first overshoots then it unders shoots and then it overshoots even stronger um because it always kind of has this um yeah too high latency so this from our perspective um we tried it with the 12.5% adjustment Factor but even if we go there to 50% or 100% or lower to 5% we only found a few settings where the price stabilizes um it works but it really depends on the attributes of the ticket holders um which uh makes it not very stable which makes it pretty yeah fragile um so at least in all the implementation settings that we found um it is not a good pricing mechanism for execution tickets um so that was quite interesting because also in yeah a lot of the theoretical discussions it was frequently mentioned as one of the yeah useful pricing mechanisms but um I think from what we saw so far auction auction based pricing or kind kind of an amm style uh or adapted amm style pricing makes more sense for execution tickets um a few more findings on the other attributes I talked about fixed versus variable amount of tickets um what we saw is there it depends a lot on the mechanism as well for amm or 159 sty pricing you of course need to have a flexible amount of tickets um for other formats it could make s to have a more fixed uh amount of tickets expiring tickets um there it was actually interesting when on the agent side implementing their optimal bidding strategy if you have expiring tickets makes things a lot more complicated because you have to take into amount when does each ticket um when does each each ticket expire how many tickets are in the pool how many slots are allocated already what's the look ahead um so it makes pricing a lot more complicated without having a lot of benefits um so our preliminary conclusion was that it doesn't really make sense to have it um similar for refundability um we saw that it adds complexity without adding too much value to the mechanism design um because it's yeah depending where you set it either you create Arbitrage opportunities for Ticket buyers or it's just an in kind of inferior secondary Market um and on the secondary Market we saw that actually yeah allowing the secondary Market um helped a lot more with yeah creating um decentralization uh due to the fact that in that just in time secondary Market specialized ticket holders can buy tickets and uh so even though it has some disadvantages um I think it makes sense to plan for a design a secondary Market exists and also the second part about it is um from a technical side it's really hard to prevent a secondary Market because yeah it's decentralized it's Anonymous people can create lot of different IDs um so there's almost no way to avoid um yeah ticket holders to sell the tickets anyway or sell the slots I mean it could be that the ticket holder runs kind of a me boost auction um just uh for the slot um that he get assigned to Via execution tickets good that was um the theory part um there's a lot more to say I didn't go too much into the all of the economic properties of execution tickets but that could be covered as well um but um I wanted to leave enough um time also for the workshop part um I put in the question do we need a break but I think um I would not do a break right now only if someone says hey I need a I need a pause um otherwise I would go straight into the workshop part maybe just the quick question are there questions right now any questions on the um on the theory so far on execution tickets on our results I know you said it's out of scope but um what prevents centralization from occurring in the secondary Market um I think it happens to a certain degree but what uh we saw in our um yeah simulations um that you have for some um specific environments you have certain very you have builders that are very specialized for example on sex tax Arbitrage and high volatile environments and they might not be the most competitive Builder on average but they are competitive in certain um slots in certain environments um so what we saw is you still have centralization um but it's a bit less it's basically it's like the market now let's put it that way you still have kind of an oligopoly um but it doesn't Worth to the worse to the market that we have right now hi good summary um have has there been uh any work done on how um exclusive order flow is uh going to be uh I guess a winning Advantage for some people in the execution tickets environment and do you have thoughts on exclusive order flow in this setting yes I think um we didn't in the simulation we didn't model exclusive order flow explicitly but I think um as in the market right now it would be one of the winning factors um having exclusive autoflow gives you an advantage in bidding for the tickets it allows you to bid for yeah at a higher price on average I think you would still have the fact that um if a secondary Market is enabled and someone else also has exclusive order flow for them the pressure builds up to bid um so we did some yeah multislot analysis and what you see there is that over time prices go up and one of the theories that we haven't proven yet is um that uh for let's say you have one winning bidder who has the highest value but you have a second bidder that also has exclusive private orderflow um that needs to place their block at some point in time so there is an incentive of kind of a as long as the exclusive orderflow doesn't only go to the one monopolist um it creates a pressure for having different block Builders uh for um yeah for for row of slots um and I think that is pretty pretty pretty similar for execution tickets um that um each Builder that has exclusive autoflow needs to um buy a certain amount of tickets to ensure inclusion of their orderflow um however if you have one Builder ticket holder that has the best autoflow they will probably still dominate The Market at least to a certain degree any more questions I think we are good perfect thank you very much then um yeah I quickly go through the workshop setup um as said um it's pretty open so the general idea is I just run through it I run through hard works and then everyone can um yeah self-guided set it up um I have uh also online a setup guide um that explains a bit how everything works um yeah first step download and run the simulation so make sure that it works on your machine should be pretty simple I tested it with some people um but yeah it's first step second step is design another configuration mix and match the parameters and and then run it test test it and if you want I have set up a notion dock where you can share results you don't have to um just if people feel like they found a very good simulation or simulation configuration that they want to share with the group um on the simulation um here on the right you can see the general setup of the code I used a red cut simulation framework in Python um the high level logic is in the first step it updates the market metadata so such stuff as um slot number me the slot which is randomly generated based on historical distributions um in the Second Step um ticket holders can buy tickets um depending on what mechanism is chosen there are different ways to buy the tickets if a secondary Market is enabled um that runs in the third step and the last step is redemption of tickets where the ticket that get allocated to the slot is redeemed the ticket holder gets the me of the slot and the ticket is burned in a way um just very quickly on the file structure um there's one major um python notebook where most of the logic runs in um I've set it up in Google collab so it's pretty easy to share and run um there are a few Support classes the models uh the data models purchase functions on buying the tickets and some yeah support functions utils statistics and there's also results folder where all results are stored um they're just a quick primer if you run it on a large setting results can get pretty large because Red Cat stores a lot of very granular results um yeah other than that on the agent ticket holders they are modeled after the current uh block Builder Market you have top Builders medium and tail Builders um that get randomly allocated different um capabilities such as M capturing abilities aggressiveness uh initial funds and um their discount Factor however that's used less um yeah where can you or what can you adjust um there's a CIS params file where all the information is in um that's in the second cell if you go into the notebook um could be should be easy to find other than that the time steps how many rounds you want to simulate how many runs you want to do in parallel in the beginning I would recommend to do one run otherwise it can take a while um so their main message is sis perms is the place to adjust the parameters of the simulation if you want to go um if you want to go really deep um then uh you can also adjust the bidding mechanisms how ticket holders byy buy tickets and um yeah much more in terms of how the auction runs and so on but uh for the beginning I would just adjust the CIS perams and yeah with this basically link to the notion dooru where you can find the setup guide um uh there yeah I think the main important info is to yeah access to Google collab notebook which you can find the link in the notion doku to copy it um it's pretty easy you go to yeah file save a copy in my drive you store a copy of the notebook in your drive and then the second part is to download the rest of the code from the Google repo um it's EMA simulation on GitHub um you download the code you move the code to the Google my drive root folder that is important because the notebook needs to access the Support classes and then you should be able to run the simulation via runtime run all um hopefully pretty forward straightforward and then you can adjust and run the simulation adjust a lot more code and um I hope it makes sense and then you can yeah design your own um configurations run your own setup on execution tickets and see what results come out um I leave this year so you have the links and feel free to ask me any questions on it feel free to yeah ping me also now or afterwards also for the people in the live stream that's better to ping me afterwards um also if people run it later um my yeah contact details are in the end um so um I'll share all of this with you and um you can use I think the best is to use the notion and go through it to set it up awesome then any questions for now guys yeah we are good thank [Music] you I think that real trouble starts when they download the refer and try to install it the real trouble starts with the reer probably you should had started like good guys and solid yeah maybe next time that's a first step then everyone has half an hour to to download it and run it but um I hope yeah it it should be pretty straightforward for yes is it on yeah I think I'll turn off the microphone for now and if people have questions just let me know I come around e e e for e for e e e