Playing Jumanji: shipping upgrades of a DAO-governed protocol - Kate Zueva | Lido

Thank you. Hey folks, I'm Kate. Um, I'm D operations lead at Lida and today I'm here to take you on a little jungle adventure. So, let's start. Um, have you ever tried shipping a code inside LA or inside any DA?

It is fun. It is Jumanji fun. So, nobody left as a map. Traps are everywhere and if you touch the wrong thing, boom, you're wrecked. So, um, here is a small notice on what LA is.

Not for bragging, just for your context. Lida is a liquid staking protocol uh running on Ethereum. Uh and it has over around 9 million ETH uh under the hood. Uh it lets you stake any amount of ETH and gives you a liquid staking token instead and you can use this liquid staking token around the DeFi. So you can think of it as a middleware that connects users with a and validators and it is governed by DAO.

What is lighter DAO? How uh how it works? So um lighter DAO or any DAO is decentralized autonomous organization when token holders uh make decisions. So there are no CEO, no board, just token holders and a bunch of frameworks or voting platforms they agreed to use. Who is in charge here?

LTO our governance token how it actually work uh how it works. So anyone literally anyone you me uh can propose something to the DAO. Usually it starts on the lighter research forums. Um even if you are a contributor to a DAO, you can't just do stuff. You need DAO approval for every decision, every upgrade that you prepared.

Uh then there are multiple states stages, different votings, different communication parts, different magic that we operation managers help facilitate. So we run the this governance process uh and make sure it runs smoothly. But I'd rather skip all that part part in the center because it deserves its own full talk or just grab me later. I love talking about the mechanics. Now let's pretend it's a black box.

So LDO holders, no anyone, anyone proposes something to the DAO, LDO holders, these anonymous, faceless, strangers on the internet, they vote, voila, a decision is made. And so now we know that final decision lays on token holders. Then they make this decision. And the whole responsibility is on token holders. But um do they read your code?

No. Do they go deep into your smart smart contracts? No. They can, but they usually don't have time to. They don't read line by line your smart smart contracts but uh they should make this decision and they trust you to bring them something solid something baked something you believe in and something then they can check that it is solid that's where the real responsibility is not in governance but in that moment when you come to the DA So propose your upgrade and say this is ready.

So um but did you catch that little big number on one of the first slides? So uh this protocol secures around 9 million ETH under the hood and when you propose an upgrade you should be you should be sure that it is safe that it is secure that it doesn't break anything. And uh so imagine you're contributing to the DAO and you okay maybe not just you but your whole dev team proposes something uh and the DAO only pushes the button. How do you feel? Confident, calm.

If yes, you're too brave for this game. We can't hire you. But uh we have these folks who are paranoid who who uh check everything and they uh are contributing to light tech. Actually light isn't a monolithic team. It's a bunch of different teams working on different parts of protocol.

All of them have uh have their own tech leads, their own road maps. Uh they own rit shows. Some of them working on staking part of the protocol and stalking. Some uh develop staking modules like simple DBT module or community staking module. Some build governance tools.

Some build offchain tool link around the protocol. Oracles, alerts, bots, staking interfaces, other user interfaces. All that stuff runs simultaneously and it is decentralized by its nature which is cool but also it is very hard. So uh how do we keep the quality bar so high? We spend a ton of energy on security.

Uh we every protocol upgrade at LA goes through procco. It's protocol security committee. Uh the name is very funny but these are senior devs who have been with the protocol since uh its early ages. Uh they step in twice or the first time during uh design review. They uh they review each and every upgrade and the second time before we launch they they review everything that is related to the upgrade.

They put their name, their reputation, their technical judgment behind every release. When Proca gives you green light, it means something. On top of that, we have audits committee. Uh they pick uh independent auditors best in the industry to review every upgrade at least once. So every upgrade big or small have uh has uh at least one audit.

Usually we run two audits. If the upgrade is big, there could be three, four, five different audits. They're independent. So, uh it's very hard work also to coordinate all that stuff. Uh so, LA has had over 70 audits during its history.

The number is very huge and the amount of work there is also huge. Um why I say that it feels like Jumanji. Uh we've been releasing uh stuff at LA for for several years, but every release felt like the first one because the teams are different. Project owners are different. Wa wa wa wa wa.

Stop. Stop. Stop. Project owners are different. Expectations are new every time.

And also when the protocol is so big, everybody wants to be sure that everything is secure, everything is safe. That is why you have multiple rounds of internal reviews, sometimes from unexpected parties, sometimes at unexpected time, wrong time. So uh also very important thing that it is a DAO that's why uh uh the release should be not only technically solid but narratively clear publicly that's why uh if you uh haven't uh released uh something technical for the DAO it's always surprise for you you are like I'm always released everything shipped everything and now boom you face DOPS managers who say did you post it on forum. Do you prepare this vault? So yeah, so you um you had to um keep in mind that there are a lot of different obstacles that keep popping up and you should be strong.

So uh especially if it is your first release, welcome to the jungle. There are holes, there are monsters, sometimes you go back to square one. I was there. It felt awful. So instead of panicking my team and I the operation managers we built a map.

Basically it is a described release flow. So not just deploy and pray. I mean we still pray but now we have a structured process that gives you to release with checkpoints all timelines weird governance steps. So now uh you come to the you come to the process and you think okay I've got it. It doesn't remove the risk but it gives you grip.

Um you can ask d ops managers why are they a part of technical release. I have an answer and the end of the day to make it happen the DAO should vote and make some decision but and we facilitate that process as you remember but to make the decision voters need some confidence some proofs some real confidence that proposals are clear and solid. This is our job as DAO operation managers to make sure that the DAO isn't guessing, it is deciding. So the proposals are clear, clean, production grade. And this becomes uh really hard when it um comes to technical proposals because um it is not the same as choosing a direction or choosing a goal for the DAO.

It is very hard to comprehend for people who are not familiar familiar maybe with technical code who don't know how this all that smart contract stuff work. That's why we put a lot of steps in the process up front to prepare the DA. Uh, of course, we didn't uh the checklist ourselves in vacuum. We interviewed everyone who has ever shipped or tried to ship a release uh in the DAO. We uh pulled in all the expertise, real stories, ugly surprises, last minute blockers.

too. So we gathered all that stuff and tried to digest it and then ordered it and now we had this process. One interesting thing you know what when we interviewed every tech lead at Lida all of them had different notion of this process. All of them released. All of them went through this path and when we asked them questions they argued like what comes first, what comes next next?

What is mandatory? What is optional? So we did only one reasonable thing. We just put everything and now we work with very long checklist. So it has seven stages.

I think that if you uh any time released something technical you pass through them. So it is not something new for everyone uh in this uh room. Uh it's a path from I have a I have an idea to everything is done. Vote is uh vote is done. Uh alerts are green.

Everything is good. Where is my tequila? So concept drafting and spec review implementation full review deployment voting and take off. Every stage has its own logic and its key artifacts uh which are both public and internal. Most of them are public actually.

So how it works as I said it is not a minimal process not a minimal checklist. It's a comprehensive very big document. But do you have to follow every step there? No. That's the that's the essence of it.

So, uh at the beginning of every project, we the operation managers sit down with the project owner and adapt the checklist to the project because everything depends on the scope, on the context, on the goals of the project. So it's not strict, it's not linear, it's not about bureaucracy. It's simply about not missing things and about giving the DA confidence that what it is voting on is good. So as I said, I think that everyone knows how to release. That's why I don't want to uh stop much on every stage.

So I just highlight different parts that I find interesting. Um this is the earliest part when ideas are born. So uh the teams write business landscape that defines the problem space the desired um outcomes the solutions for that. So it's about this high level business uh decision for the uh implementation. After that teams write white paper or ADR architectural decision record and this document I find it very interesting that it highlights uh the chosen options and explain why they are superior to alternatives.

There are also different parts like signaling stepshot to know whether the DAO even wants to start this project if if we hadn't um some approval before if it's something new or something great that that the DAO should spend a lot of funds. So uh if it wasn't uh incorporated into the previous uh budgets we also start signaling snapshot to know whether the DAO even wants to start. Yeah. The next stage and draft is drafting and spec review. Um here's one thing that we learned.

If you can think before you code, that's really great. If you have time to put it on paper, it's even better. If you have some people who will thoroughly review what you created, what you want to to uh implement uh after it is the best thing. So uh in this stage we write technical spec. It's more aimed to proca to technical contributors to auditors and also light improvement proposal.

It's semi- techchnical document aimed to uh a broadest um audience voters delegates stakeholders contributors. It helps people understand what is coming the next phase implementation itself. So here code is ready. Rigorous testing is ready. Alerts of chain tools.

Several desk death nets could be uh run and also a p um final rehearsal on public test net takes place. Um one more tricky part um we used to think about time this way. We are done when smart contracts code is done. Uh but the real the reality turned that no uh the code is done when your smart contracts code is done when your tests code is done when your alerts are ready when code of your offchain tools is also done. So it takes away way more time than smart contract development.

The trickiest part full review uh it like pingpong game but you're the ball. Uh so imagine um you have a lot of audits. Yes. So you invite independent auditors to review your code but also you want procco protocol security committee our internal committee to review your code not only code but code deploy plan voting scripts and tests. But these are senior devs as I said working on other project at the same protocol and they need to like find some time in their business schedule to shift gears to your project and focus on it and review it.

That's why it's a separate stage. So you book their time and then you wait and usually again wait. So uh that's the uh final stage uh related to coding. At this stage you get a green light on your code deploy plan scripts and you post it on the forum for the public your whole proposal summary. Right after you get green lights you deploy the contracts on my net.

And here's one thing we like. So it's deploy verification record. It's a paper we ask do our auditors. So uh the thing is is um you have your reple with your audited code and you deployed some smart contracts on the mainet and somebody should compare that the code is the same. So we ask our auditors to do it and to publish on the forum that lets our voters know that smart contracts we attached to the protocol in the vault are the same has have the same code as um auditors reviewed.

My favorite stage of course because we run the votes voting. Uh imagine you did all the work everybody reviewed it. You have green lights on everything you did and now you have to convince strangers on the internet token holders to say yes. That's why you publish a lot of stuff. So numeric param you publish proofs for for all numeric parameters for all methods you call in your vote for all.

So you uh describe every item in your launch in your launching vault to let the voters believe in it taking off. So when the votes gate gain quorum the vote is enacted all dogs communications are published and here is supervision or hyper hyper care period starts. So don't run to your tequila just yet. Just stay with your love list protocol for some time to make sure it runs smoothly. That's it.

What changed with this checklist? Uh we still ship which is already a good result. We still ship, we still stress, but now we don't get lost. So every release has a plan. Every project knows where it is and where it is going and the DAO knows what it is voting on and also it is now repeatable success even for newcomers.

That's the shift from gas to clarity from just wipes to yeah I've got this. That's the shift. So now you know that D now governed protocols are powerful but fragile. You there's a lot at stake uh figuratively and literally. You can't ask token holders to dive into your repo.

So we build trust not just by writing great code. We do write great code but also by building clear and trackable processes. ones that give every water and every stakeholder a reason to say, "Yeah, these folks know what they are doing." That's the shift. Welcome to the game.

Let's play it. Well, that's all from me. I'm kid from the operations at Lido. If you want to talk about checklist, how chaos or wireless managers don't sleep, grab me after the talk. Thank you.

We actually have a couple more minutes. So, we actually have a couple more minutes. So, if somebody wants to ask a question.

Yeah. Okay.

So, we actually have a couple more minutes. So, we have time for one question. Any questions in the room?

Yeah. One question. Can you pass the mic, please?

Hello. Uh I'm curious to understand a bit more how that's compensated though. People who write great code are expensive. Increasing the size of their process makes it more expensive. How do you guys account for that?

Is there a standard way to understand the extra hours that are taken into this process?

Um the an uh the answer is um we don't separate we don't separate the budgets. Yeah. So now we work like one team that's why the time is uh is counted together. Yeah. Uh so but um the essence not in the time but in um being safe because uh we balance we we can't balance because uh when it the protocol secures billions that's why we need uh ensure to ensure to that it is safe and we need to our token holders make sure that it is safe.

So, um we'd rather spend more time on processes and security than just on coding. Yeah, that is true. That is true that it costs much but still uh it is u it is necessity.

Thank you.

Thank you very much.

Thank you. Thank you.