Decentralize your sequencer -- A guide for L2’s by Joe Andrews | Devcon SEA

All right. Thank you everyone for coming. I know there's a big talk next door about the Ethereum roadmap. So I appreciate you guys for coming to this one. My name's Joe.

I'm one of the co-founders of Aztec. We're building a privacy layer for Ethereum. But today I'm going to talk a bit about a general guide for L2s and how to decentralize sequencers. You can follow me on Twitter and let's dive in. So first of all, we're going to look up a bit about our journey.

Kind of the journey that Aztec's been on, what we've learned. I'm going to try and convince hopefully the L2s in the room, people who work at L2s to care. And I'll also talk about why we care. We'll talk about why we can't rely on trust me, bro. This is something kind of this is a bit of a pet peeve of mine, but we'll get to that in a second.

Um We'll just debunk the myth around forced inclusion being the same as decentralized sequencing. We'll look at the different components of a sequencer. And then, you know, decentralizing a sequencer is just the beginning. Hopefully we'll have time for questions. So back in 2023, we started kind of a process at Aztec.

You know, for the last 18 months we've been thinking about how we decentralize and launch Aztec as a decentralized L2. We're doing this all in public on our forums. You can check check out these various RFPs and the responses on on our forum. I'll share a link in a second. We've had to think about this a bit before everyone else.

You know, being a privacy focused layer two, like the regulatory climate is is slightly hostile towards us. We can't be a central point of failure for all of the private transactions that go through Aztec. So we've been thinking about this and hopefully I can share some of the learnings with you today. Our latest kind of block production designs are now live on the forum. You can scan this QR code.

Spoiler alert, like we use the random leader election to decentralize our sequencer. There's also a payload timeliness committee on the L2 to help with fast finality and kind of better UX for pre-confirmations. But ultimately we use L1 to verify everything and so trying to stay firmly as a ZK roll-up. All right. So, first of all, let's look to see if other L2s care.

I pulled some Dune dashboards last week and all time sequencer profits for kind of other L2s are north of 150 million. This is kind of a crazy stat, but you can kind of see all of the kind of top L2s, you know, they're they're printing multiple millions of dollars per year and they may say one thing publicly. You know, we're working on it. Um The next version will be decentralized. Um There's kind of like not really the right incentives to decentralize here.

And so my goal with this talk is to try and convince you that A, we need to care and maybe look at some of the options we have available to us. For a meme, Yoda says it better than me. So, yeah, too much money at the centralized sequencers they have. Um In terms of kind of like where we've got to and how much money these L2s are securing, um users trust L2s with 38 billion dollars today. And so, you know, L2Beat is is great at kind of showing us what's happening.

But it's also kind of it's just a scary number when you think about it. When I think about this, it's 38 billion dollars that doesn't live up to the vision of rollups. Um Um this doesn't inherit the censorship properties and the liveness properties of Ethereum. Maybe it gets some eventual safety properties from Ethereum, but ultimately this money just relies on trust me, bro. You know, there's a central entity in the middle of all these transactions that we all trust to sequence, tell us what the state is, um and promise inclusion.

And for me, that's it's just not okay. Um you know, this meme says it as well. There'll be lots of memes today. Um why is this not okay? Um I think that in order to have like a credibly decentralized uh L2, like we need to then decentralize the sequencer to inherit the properties of censorship resistance, liveness, and safety from Ethereum.

Um you know, eventually we get safety um when we verify a proof uh or we have the fraud proof window if it's implemented on uh like an optimistic rollup. But a lot of these transactions are happening, you know, sub-Ethereum block times, and so there's a lot of pending chain uh finality that is not safe currently. So, we need to look at how to make that safe. Some more reasons why we can't rely on trust me, bro. Um yeah, centralized sequencers, they can lie.

Um they can lie about whether your transaction will be included in a block. They can lie about the transactions that are before or after your transaction, the state of the L2. Uh and this can happen for minutes, hours, or days depending on um the state of the system. Um we've also tried this before, all right? Most people are at Devcon because we're fed up with the Web2 ecosystem.

Um and like if we wanted to just trust someone with our data, we can use our favorite like new banking app. Um we can use a database for that. So, hopefully we can do better. I think in another reason here is in 2024 that censorship is is is very real. Like, as we start to get more and more usage and adoption of crypto, um the chance of one central entity sharing the same beliefs of everyone in this room or all of the transactions on the network, it's vanishingly small.

Um and so, you know, as we start to get more interesting use cases, sequencers will start to prioritize one type of transaction over the other. Um and we we can't have that. And the last one, this has been said before, um but a liveness failure is still a failure, right? You know, if you get used to, you know, sub-second block times, and then it goes down for 5 hours, uh you can't send a transaction, you can't use your money, your funds are stuck. Um and so, this may be okay for today when we're trading, you know, meme coins, but if you're trying to get to work or use real value, um this is not okay.

Um this has been said before, but I I like this meme. Okay, so this is like a personal pet peeve of mine, and I think sometimes centralized rollups uh rely on forced inclusion to say, "We'll do it later." or to keep printing the the millions of dollars that we saw saw earlier. For me, forced inclusion is a bit like this uh this traffic jam, you know? If everything's great, and uh you're friends with a centralized sequencer, you can use the fast lane.

You can get to work on time, uh and everything's working perfectly. Um as soon as something goes wrong, uh you get stuck in this traffic jam, you know? The capacity's not there, um you wait for hours, um and really, like, you can only use this in emergencies. You may use the right-hand side to flee a country, like, run away from a storm, but you're not going to use it to get to work. Um and forced inclusions are are pretty much the same.

They can only be used for um transactions which have a long duration. Um and so, we want to make sure that every transaction has the same properties for censorship resistance and liveness. Um enforced transactions are kind of a second-class citizen. Um all they guarantee is like an eventual sequencing. Um and we need to decentralize our sequencer to ensure that these properties are the same for all transactions.

So yeah, if you leave with anything, please leave with this. Forced inclusion is eventual sequencing, not decentralized sequencing, and we can't rely on it. Okay. So a lot of why. Hopefully, you're with me now, and you want to decentralize your sequencers, so let's look at some of the core components.

There's four main components. Um we have leader election, uh DA, uh slashing and incentives, and consensus. Um we'll go into each of these um in a little bit more detail. Um but I think the the key one is is the one on the right with consensus. Um we need to verify that everything uh to the left here, these first three, uh happen correctly.

Um and how do we do that, you know? How do we verify that the correct leader was elected from a decentralized set? How do we verify that they published enough data for the next sequencer to publish a block? Um how do we verify where slashing occurs? If we do that on a one, a separate consensus network, or use economics, they all have different uh assumptions around where we inherit our security and liveness from.

So let's look at the first one, uh leader election. Um you know, if we have a white list, like a sequencing white list, and you're not on the list, you may end up like this guy, uh waiting to be picked as sequencer. And we can do better than this. So leader election is all about picking who's going to be the next sequencer. Um and there's a few different ways we can do this.

Um the simplest is a round robin. Uh we can kind of have an ordered set and just cycle through them one after the other um in a deterministic order. It works, but you know, the determinism is actually in a bit of an issue. Um you can have denial of service attacks because of this. A better model may be a random shuffle.

You know, you can shuffle uh either sequences or block proposals. Um both can work, um but they rely on some sort of random beacon to work. Uh another interesting one is kind of let anyone be a sequencer um and actually just vote on the proposals. So, you could pick the proposal which makes the most money or, you know, uh solve some sort of optimization problem. And the last one is kind of a combination of some of these.

Um you can take a random shuffle, but weight it by some factor. It could be weighted by reputation or stake. Um so, these are kind of the different options available for the first uh leader election. This does need to be verified somewhere, so it's important to kind of take that into account when choosing. Okay, so the next one is uh it is also equally important.

Um if I'm the sequencer and I'm centralized and I want to build block N on as block N plus one on block N, I've already got the data. Like, I don't need to send it to anyone. I made the last block, so I can just keep going uh kind of for all of time being this the centralized sequencer. That doesn't work if someone else is the next sequencer. So, we'll run through a little kind of chain here.

Um the purple blocks are finalized blocks. Um they're either finalized because there's a validity proof or, you know, there's a fraud proof window that's expired. To build a block on the finalized chain, um I just need the side effects of the roll up. Um I need to, you know, all the transactions that have happened uh throughout the time. I need to have a database of those uh side effects.

I need to move the state from the old side effects to the new side effects to to build the next block. So, I can build block four here. And then imagine we have like a pre-confirmation chain where the sequence is changing quite quickly, Uh but these blocks that have not been confirmed now, they've not been confirmed by a one. They're just kind of you know They've been confirmed between maybe three different sequences, but there's a lot of data there now that needs to be shared to build block six. We still have to share the roll up side effects.

You know, we have to have block five state and then we need to you know apply the state diff from the transactions in block six to get the new side effects. But there's this other type of data which usually gets swallowed in a ZK proof or is not posted on chain called transaction off data. And this is where the problem kind of starts. So for a public roll up, you know, optimism, roll ups that don't deal with privacy, ZK sync, like public ZK roll ups have this as well. The data is quite small.

So you have a TX request and a signature. It's around 400 bytes. You can post that data. You know, you could put it on Ethereum, you could send it to um you another DA layer. You can put it in lots of places.

For a privacy roll up, the data is a lot. It's a lot larger. Like the transaction accompanies a zero knowledge proof which can be you know, 32 to 64 kilobytes depending on the proving system. And so sharing that data between sequences becomes a bit of a problem which we'll go into in a second. Like if you don't have this data, you can't verify that the transactions in block five are authorized, which means you don't know if block five will happen.

So you don't know if you can build upon it. So in terms of DA, like where can this data go? You know, we can put it in core data. It's been around for a while. It's kind of the only way we can put data on Ethereum if we want to use it in a smart contract.

It's very expensive though and limited in capacity. Blobs came out last year I believe. Um we like these. They're a lot cheaper um and it's very useful for kind of posting this side effect data. Um it can also be used to post uh the transactional authorization data we talked about earlier, but only for kind of public rollups.

Um there's not enough capacity for private rollups to use this. Um payload timeliness committees, these are like an L2 specific committee like the sequencers can join together um and they can sign off on the fact that enough exists. They've seen this data to approve the block. Um you know, if they if they give a signature and you know, a majority of the committee signs off, then a sequencer can look at that signature and be like, "I don't have the data, but you know, I trust I trust this committee." It's like a mini DA lab, but it's very short-lived.

And then you can also put it on an old DA. Um so, there's lots of capacity on these, but they don't bridge back to Ethereum very quickly. Um so, it's only really usable for kind of side effect data and you can't use it for kind of data that's critical for block production because then you won't be a rollup. This is just a quick kind of table of the different comparisons. Um yeah, you can see kind of you have to pick and choose based on, you know, security uh throughput that can fit on these different DAs and uh what they can be used for.

Cool. So, the last part is you know, uh is the most important. Like we have to verify everything we just talked about. And so, usually when we think about consensus, we just think about verifying what happened. Um you know, did these transactions produce this state?

Um and that's kind of you know, that's fine. Uh but when in when we're talking about decentralizing sequencers, we have to verify a lot more than that. We also have to verify like, "How did this happen?" Um you know, if we're waiting for a validity proof, which can take 10 minutes um in some blockchains in some rollups, or it can take several hours. Um, or challenge periods in a optimistic kind of fraud proof scenario can take 7 days.

What do we do in that time to verify the the blocks that are pending actually happened? Like, we need some data. It's the data we talked about on on the last slides. And so, depending on how we can trust what happened, we may need to kind of also verify data locally as a sequencer to know what happened. And then lastly, like, we need to check who did it.

Like, you know, did the sequencer who proposed a block actually have the right to propose a block? Um, and so, we need to come to consensus on all of these three things to make sure that the blocks that are being proposed by decentralized sequencer are valid and follow the rules of the system. Um, if we don't kind of verify who did it, like, you know, someone could just keep proposing a block who's not able to, they could monopolize sequencing rights. And so, depending on where we verify these things, we inherit different censorship and different liveness and different safety properties, and they're not all equal. Um, so, I think just looking at the different options we have here, you know, the most secure is to be an a base rollup, you know, let's try and inherit everything from L1.

Um, this is great. We can, you know, defer our sequencing to the L1 proposing set. We can kind of inherit censorship and liveness from L1. Like, if Ethereum's running, you can always propose a block. It's also quite easy to do.

Um, however, this is only really possible for public rollups, and it's also quite slow. Um, so, you're limited to a block every Ethereum block. Most L2s have a lot faster finality than that, um, or sorry, faster pre-confirmations. like they have sub-second block times, so this doesn't really work. Um Another alternative is use L1 for ordering, um but for this data which doesn't fit on L1, introduce like a payload timeliness committee um for the transactional of data.

You know, have this committee that sticks around for just long enough uh to make sure that the block can be proven um and this is possible for public and private rollups. It has some pretty nice properties. Um you know, we get the censorship properties from L1 because L1's deciding who's sequencer. Um liveness properties are pretty good if the committee is still honest um and if you have a decentralized sequencer, this is actually a pretty strong assumption and we inherit safety from L1 as well. Um other ways of doing this, you know, are less less good.

Some people talk about putting, you know, a Tendermint Cosmos instance in front of the uh L2. Um you know, have an actual consensus network on the L2. This kind of has uh I guess it doesn't inherit the properties we want from L1. Um you know, if you have a separate consensus network that's deciding who's sequencer, is deciding if the data's available, um and it's deciding, you know, is the state of the pending chain correct? You only inherit the properties of that staking set uh in the short term.

Eventually, you may settle to L1 and have, you know, a velocity proof of all of this. That's very complicated to do. You have to prove the consensus of the L2 on L1 in like a mega proof. I think the the other thing that's worth saying is that like you also would have to reorg your L2 consensus network based on whether that proof succeeds or fails on L1. So, it's just messy.

Um So, where you sit in this kind of uh chart depends on, you know, as you decentralize uh what what you you to kind of give your users. Uh all right, we're running out of time. So, I think a few slides left. Um The last thing I wanted to go through is like this is just the beginning. Like you can pick where you are in this kind of map of like DA, uh leader election, uh you know, slashing, uh you know, implement all these different things.

Um but it's just the beginning. Like there's no point decentralizing your sequencer if you don't think about the entire kind of like stack. Um So, sequencing is one part. Um we got provers, right? If you're a ZK L2 and you're relying on someone to actually prove the validity of all of this happening and you have one prover, then it doesn't matter how decentralize your sequence sequencer set is because no one can actually advance the chain.

So, yeah, decentralizing proving is is a key part of this. Um and governance is the last part. Like if governance can overrule any of these two things or change the system uh and that's centralized, like there's a centralized centralized multi-sig, um it also doesn't matter if you have decentralized sequencing and proving because the results can basically be thrown away. And so, to actually fully decentralize an L2, you have to exist in this middle part. Um and so, you know, doing any one of these in isolation uh doesn't really work.

And this is kind of what I wanted to leave you guys with. Um maybe next year I'll give a talk on on the other two. Thanks for listening and uh yeah, you can look at our forum post here um if you want to follow along for some of the other designs on uh governance and decentralized proving and follow me on Twitter. And maybe scan the QR code if there's questions. Thank you.

All right. So, we're actually a little bit short on time, but we can still take some questions. Let's go from the very first one. ZKPC sequencing still costs a lot. How do we guarantee sequencers are run by different actors in different locals, even if decentralization is supported?

Yeah, so the cost of sequencing, um there's two there's two things that I think are important in the cost of sequencing. It's like the, you know, cost of the hardware and the cost of like, um the bandwidth needed to run the node. Um you know, sequencing zero-knowledge proofs or anything else, there shouldn't be any more additional like hardware um requirements than that of like an Ethereum node. There may be more bandwidth requirements just because there's more data um if the proofs are larger. Um and so, you know, you can still get geographic decentralization if you just focus on hubs like universities or areas that have good fiber connections, but the hardware is is pretty easy to to get.

All right. Um which of your users are most excited and empowered by decentralized sequencing on Aztec? Um this is a good question. There's I guess there's people who want to run sequencers, but that's a maybe a an obvious one. But, I think like the reason we're decentralizing sequencing is because it creates a neutral layer where you can have, you know, very different uh apps inter- interoperating with each other.

And so, I think if we look at the types of applications that people are building in in kind of Aztec pilots, like there's things like proof of passport, things that like you need a neutral base layer. And so, I'd say that any application that kind of is at the cutting edge of its field uh probably needs a decentralized sequencer and is excited by this. Cool. And how easy in technicality, in your opinion, that ZK sync, OP, and other move to decentralized sequencer? Yeah, this is a tough one.

So, um 2 3 years ago, we had a product called Aztec Connect, um and we built it centralized, and we said we were going to decentralize it later. Um that never happened. Um so, I think this is this is harder uh than than it looks. I think the the research is there to do it. I think it's more of an incentive question.

So, if you're in these communities, I think this has to come from users. It's not going to come always from core teams because, you know, centralized sequences make a lot of money. It is possible today, but it does require pretty serious upgrades. Awesome. And what should be decentralized first?

Sequencer, prover, or governance? Yeah, this is a this is is a tough one. My last slide kind of hints that maybe all three have to happen at once. I think in a privacy roll-up, that's definitely true. For for public roll-ups, I think you can get away with, you know, decentralizing them a bit more one at one at a time.

I've I've seen some pretty interesting prover decentralization where proofs are happening in TEEs, that's like a second proof, like a multi-prover model. Yeah, I think it can happen one at a time, but only in a public roll-up context. All right, so here we got a question actually from Token 2049. The cost is a big reason why L2s are still centralized sequencers. Decentralized L2 will increase cost and users don't like it.

And what is your opinion on this? I don't know if it will increase cost. It may increase or make the UX slightly worse, but I think it depends on the type of application. So, you know, if you think about micro-payments or fast fast finality payments, they may need a centralized like sequencer to process transactions, but those things should exist as an L3 or like a an app chain. So, I think it's more of a UX consideration than a cost consideration.

Like with decentralized sequencing, we're targeting like between 1 and 10 cents a transaction on Aztec. So, I don't the cost argument is is there. All right, I think time for one last question. There is incentive misalignment for centralized sequencers to decentralize. How do we solve this?

That was kind of the point of the talk, so uh it's on us to kind of uh talk about this and um I think push them to to do it. I think yeah, it's it's easier to not do it, um but one day something will go wrong and we we we'll wish we had done it, so I think you know, if you're in these communities, um yeah, kind of push push the push your kind of uh core devs to decentralize.