Build-a-builder: Intro to building performant block builders - makemake | Phylax

Uh so yeah, that was a lovely introduction. My name is Mak. I've uh you probably some of you might know me from Twitter or as the guy who hacked to finance. Uh I've been writing code for quite some time. I've been uh before crypto I've been doing th stuff uh hardware and binary exploitation and reverse engineering.

I've been writing rust for the past three to four years and I'm currently working at file where we do appde defined security in forceby networks which is a fancy name for hack prevention. So uh in this talk I'll be covering uh the role uh of blog builders how we use them at fileex some tips and tricks for building your own blog builder. So the tips will generally be around making your blog builder more performant and not about extracting me as that's not really what we do at fileax. So enjoy to understand how transactions land on the blockchain we have to understand how the flow from sending the transaction to getting into the blockchain looks like. So we have a funny meme explaining the life cycle of a transaction on mainet.

So on unis swap you would click the swap button. This will get sent to the memp pool then relay to the builders and the builders will complete in a builder auction for uh the extracting the most me and it will finally land on mainet. Uh at file X we deal with building blocks for L2s. So this will look different from the order flow of this happening on the L1. So like keep that in mind.

Uh a key difference is that on L2s we don't have multiple proposers or a public meool. And this means that the sequencer is the one including everything and uh holding all of the transactions. Uh thankfully OP stack is architectured very similarly to Ethereum. uh we have op node which acts like the consensus layer and we have op gas or rat or whatever uh acting as the execution layer. So what we can do to get you know an external block builder is we use a tool called roller boost made by flashbot logo here uh to proxy the connection between the execution layer and consensus layer and forward the engine API calls uh to our blog builder.

So we have like pretty much what's happening here. We have an engine fork choice update which we proxy via roller boost and send it to our builder which is the sign for our builder to start building a block basically and then uh op node will request a block via engine get payload and then we have like 150 milliseconds to respond with a payload and if everything goes according to plan we have successfully built a block with an external builder. So on a file protected L2 the life cycle of the transaction looks something like this. Uh it gets sent via the RPC to the sequencer. It will then be inside of the sequencer and shared to the builder mele.

Uh the builder will build a block with the transactions inside of its meool and the transactions will be included in the cononical block on the L2 if all of them are fine and well. So on the inside builders selectively order transactions inside blocks to maximize value or perform some other task. So on the L2 we don't care about maximizing value as that's zero sum and generally negative for users. We more care about other productive use cases we have. So some of these include world queen's human prioritization uh flash blocks on unit chain and of course file hack prevention.

Um to better explain what I'm going to talk about for the rest of this talk, I'll need to introduce you to what file is and how the file credible layer works. So the credible layer basically enables DAPs to define states they don't want their protocols to be in. Think of it like in uh invariance tests in Foundry. You as the developer can write assertions which are similar to foundary tests and we make sure that transactions that invalidate set assertions never get included on chain and for example like if they are force included we have other methods of uh making sure that they don't cause cause damage. So we protect against that as well.

Uh what does an assertion look like? So here we have a basic example of an assertion that tracks for price or oracle deviation. We get the price of the token before the user begins a transaction and we define a maximum range u that the price can move in one transaction. Uh kite among you might have uh noticed that we have yeah you can see phwork pre-state and ph fork state and those uh are part of the assertion syntax. So we have a special EVM called a PHVM and we have additional pre-ompiles or cheat codes inside of it and those allow us to do things like arbitrarily forking the state uh reading storage slots of external contexts that don't have getter functions uh reading logs of your transaction uh getting like the call inputs uh and much more.

So yeah um and this all happens like during block execution like as soon as like your the regular transaction gets executed. Uh if you're interested to learn more about this very cool primitive, I'd say uh there's a QR code here. Uh you can scan it and that will lead to you to our docs where you can find the assertion book uh where you can uh see what assertions can do and write even write your own assertions and include them as part of your uh unit tests. I'll leave it up screen for but yeah. Okay.

Uh so to achieve realtime hack prevention, the number one thing we need is performance. uh the file builder cannot be a major performance downgrade from the default sequencer setup that's not running any assertions. Uh we need to be real time and we need when we do run assertions we need to be able to execute as much gas as possible so we can perform all the checks with set assertions. There's no point if we just like you know run a million gas after each transaction. Nobody's going to use that.

Uh if you don't get those two right, there's no point to this. Networks won't integrate. Developers will not want to use it because it's too limited and they cannot perform the checks they otherwise would want to perform. So opalis, which is our block builder, manages to do upwards of 2.4 gagas per second.

To put this in perspective, that's 828 full Ethereum minute blocks worth of gas for every transaction that land inside of a block. uh with a full 40 million block of just a regular user transactions, we able to execute an additional 2.4 G sort of compute and this is just like the MVP. We are just like beginning. We can squeeze more performance out of it and we are consistently working towards doing that and we know how to do it.

So now to answer the question everyone here is for what makes it so performant. Even though it would be very nice if it existed, there is no silver bullet for performance. To build performance software, you need to be constantly monitoring and profiling the code you write. When designing access methods for data structures, databases, systems, whatever, there generally exists a three-way trade-off. read, update and memory overhead.

If you choose to optimize two of the above, the third one will suffer. Um that can either be through contention or coherent cost. So for example, if we focus on read and update overhead, we will have to use a lot of memory. If we want to optimize for update and memory overhead, uh that means that reads will suffer. We will have to use locks everywhere.

And you get the point. And updating means just like writing. So read memory. Memory like how big your data structure needs to be. Um yeah, humans are generally really bad at guessing where performance is lost.

So eyeballing is generally not going to do it. Uh flame graphs tracing perf deletion profiling are some tools you have at your disposal to gauge the performance of your program. Optimization is an incredibly difficult job. Large companies have like entire teams just dedicated to like optimizing their software. You can generally get like 95% of the way there if you have like I don't know if you use like the tools from the above and you I don't know have like a beer zens and like faith and god with you.

So we could be here all day discussing like the intricacies of like how to optimize software, how to like you know optimize data lines, caches, whatever. But to make it short, I recommend reading the sled theoretical performance guide. It's an amazing TLDDR of what you need to actually focus on when you're dealing with IOBound systems and uh how you are supposed to benchmark them. So let's get into some of the things we did at Filelex to improve our performance. Uh in Opalis we basically have two modes of execution.

We have the regular transaction execution and we have the assertion execution. We execute transactions sequentially and then run the assertion EVM code in parallel. Uh we can do parallel execution for assertions quite easily because assertions do not modify any state of their own. They just execute code and we only care if they revert. If they did revert that means that the assertion has failed and we have to throw out the transaction that triggered the assertion that caus it to revert.

When designing data access structures, we kept in mind that we need extremely fast multi-threaded read only access. Uh the design we ended up landing on was the overlay DB. Uh it keeps a large inmemory cache of PVM state and it has an underlying database which in our case is the RAT database. Um it will first read a cache and if the state is not in the cache, it reads the RAT database and puts it in the cache. Um the cache itself is um a buffer is tiny LFU data structure.

What that means is that uh when data comes in it lands in a buffer and we will periodically have a job that uh removes things from the buffer puts it into the cache itself and evicts any cache that's need to be evicted and we do that when we are not building a block. So we don't have like that kind of contention when we build a block. Um the reason for just not relying on the rest and the operating system because that's also a thing. Uh when you have a lot of like disk access the kernel will at some point use unused RAM to cach your data but we don't do that because we want control over it. We want to know exactly how much state we have in RAM and we want it to be reliable.

It also gives us the flexibility of being able to like change the data structure we have quite easily. So if we for example want to store like 16 storage slots in the past, we can do that. Um okay uh we were then presented with another problem quite quickly. Uh because of the way that gives database access, we were essentially limited to immutable reference of the database. So we had to get quite creative.

Uh what we ended up doing was building the active overlay database. It is essentially a wrapper over a mutable RAM database to turn it into like a nonmutable database ref. Uh the end result was a data structure that was like 64 bytes large and did everything we needed. We got fast data access for as little complexity as we possibly could and fasting uh across multiple threads. Uh we have just today and I believe like an hour and 30 minutes ago open sourced the repository on GitHub which contains all of this code and data structures.

So you can go ahead and look at it now. I unfortunately don't have it in the slides because it was done like an hour ago. But yeah, if you go to file systems GitHub, you've able to like go into the assertion executive repo and you can investigate both of these data structures to see what makes them uh so interesting. Uh yeah, early on we got niped into building our own database and what would essentially be a blockchain client from scratch. Uh how it worked is uh we were performing an ETL pipeline from RATS MDBX database to SLED and uh why we did this I'm not quite sure uh but some block builders had this approach so okay we said let's let's see what happens uh then very quickly updated their database schema at broker pipeline completely and we very quickly realized why it was a bad idea so we have this mending repository that's like the ETL pipeline we have the shared DB data structure that was the data structure we planning to use for our database which held like an inmemory cache and like you know an FS file system database thing.

Uh and then we had like the function to do the ETL. So yeah yikes. Uh optimizing the actual EVM transaction execution is incredibly difficult. um you're not going to be optimizing the EVM execution code beyond what is already in Raven without spending literal months for like maybe a few percentage points here and there. Ravom is a very well optimized piece of software already.

What you can however do is reduce the time you spend on processing transactions. Do you need to recover the signer for example? Okay, amazing. You can do that in parallel to other jobs that you do. So I don't know while you're executing like the sequence of transaction you can like do cryptography on the transactions you have already that are synced by users uh it saves you a lot of time pipelining all of these things for us saved us around like five to 10% in performance uh old L1s don't want you to know this but the bottleneck for scalability is actually IO um only when we get to a certain point where we optimize the data structures enough uh so that IO it doesn't matter does it make sense to pursue parallelism uh and parallelizing the EVM is very difficult it is inherently a sequential piece of software executing ste transactions in parallel means that some transactions will inevitably work on conflicting data and what that happens is you have to like at worst like throw out your parallelized work and do it sequentially and that's really not ideal because you're throwing away work and you have to reexecute The block state access lists don't always work because you know a transaction before another transaction might change something and that transaction impact something else along the lines and when you are processing that transaction the state will change.

So it gets quite complicated. So we solve this very easily. There is no state. I love you. Um so if you remember the transaction flow we execute a transaction and then we have a bunch of EVM code we have to execute to verify set transactions validity assertions cannot modify state they are completely read only.

So what we do do is just throw set assertions into a thread pool and we basically call it a day. We have arrow iterator and we join the results on that and amazing everything is happy and everyone everything works. Um, a fun property of Rust is that you have multiple different allocators at once. It's kind of like a zigg in that regard. So this means you can easily use arenas and you have custom allocators for set arenas.

So arenas are pre-allocated regions of memory that are reserved for upcoming data. Data that you know you will use but that you're not you don't have you don't have now. The rationale is that allocating things in advance reduces constant like allocations of memory and freeing some memory as the memory requirement of your program grow at runtime. So you will just like you know do like allocate like instead of doing like tiny I don't know one megabyte allocations every I don't know when whenever you will allocate like one gigabyte and then you will assign that memory inside of the arena. uh when we are dealing with thread pools for example rust really likes to allocate and then free memory as soon as it goes out of scope and this can be very bad for performance uh which is why we use arenas and I have two here I have a sharded slab which is a a lock free parallel arena for memory and I have GC arena which is basically like arena but garbage collected uh and it's kind of neat So, thank you.

That's pretty much it. Thank you all for listening. I don't didn't want to like get too much into details to not bore you all. Um, if you have any specific questions, I'd be very glad to answer them. We will be open sourcing our builder quite soon.

And we will have a public devet in the coming weeks. Uh, so make sure to follow us on Twitter for updates at Files Systems. Uh, and we are also hiring. So if you are interested, let me know after the talk. Uh we have Rust and DevOps positions open.

So yes, thank you all.

Okay, let's start with questions.

Ah, there is one there.

Hello. Uh a more a general question about Flex. um where are the assertions stored? So I deploy a a protocol and I want uh to assert that my protocol never reaches this certain state. Where do I publish this assertion so that the block builder can use them?

Yeah, that's a great question. So uh the flow basically goes like this. You will um have we have like a command line tool called PCL and you will you would use that within like your foundry project to write set assertions and you use it to test them. Once you are confident that your assertions are sound you would u use PCL submit which will submit the assertion to what we call the assertion DA which is the DA layer for these assertions. uh the assertions will be stored there and when you submit a transaction on chain to activate the assertion and assign it to like a contract you will also because you also have to prove the ownership that you are the owner of that contract.

You cannot arbitrarily add the certain to any contract, right? So when you do that, the builder will index them and after a time block has passed, after finality, these assertions will become active and they will start to be enforced.

And I have question uh just so FAX is a block builder. Yes.

That will make sure that transactions that land in a block uh respect these assertions. Uh well not necessarily. So we offer like the we offer like the blog builder ourselves but it's up to the networks to run them. So we provide support. We you know do everything we can to like you know support them however but it's up to like I don't know whatever L2 I can't say specific names for obvious reasons but um but it's up to the L2 to actually run it

and to and to promote the usage of this.

Yeah. Yeah. That's also enough.

Great. Thank you. Uh, any other questions if we have

Okay, no takers. Well,

I've blown everyone away.

Yes. Well, thank you so much much