Defending Self-Custody

Good afternoon, everyone. Today I want to talk about probably the most impactful event that affected self-custody this year, which is the hack on Bybit. This is not a simple exploit. It was a very targeted attack by a nation-state actor on Bybit, and I think it's something where we should take the time together as an industry to reflect on what this means for us, how we approach self-custody, what takeaways we can take from that, so I want to spend all of the talk about that, about the takeaways that we took and that the industry should also have internalized. I'm sure most of you have followed this incident and are mostly aware of what happened, but for those who are not, I just want to briefly give a summary.

So the North Korean-sponsored state actor, the Lazarus Group, was able to compromise the safe wallet front-end, and using that, was able to deploy code that was very tailored to Bybit, which was using the safe wallet front-end, and actually the code that they deployed there, what it did is, first, it prevented one of the signers of the Bybit multisig to sign at all, so completely blocked that for some reason, and second, all the other signers, when they would use the safe wallet front-end, they would see the transaction they want to do, but the front-end would forward a different payload to the ledger devices, and this was used then in order to trick the signers of the multisig to confirm this transaction, and this transaction, which was a malicious transaction, upgraded the account, so it did a delicate call, it upgraded the account to another singleton, which is a malicious smart contract, which then allowed the hackers to take full control of that account. So yeah, that's the rough summary, the outcome is very unfortunate, Bybit lost 500,000 ETH during that process, and yeah, it's been, in my view, the most impactful event on self-custody as mentioned. So let's go into the learnings there. First one is secure self-custody is a constant battle. Secure self-custody is not a solved problem, and I don't think it will be ever a completely solved problem, that's the thing with security, there is no perfect security, you can just maximize as much as possible, harden different systems, but perfect security is a myth, but that means that securing self-custody is an ongoing process, especially as tactics of attackers change, as the environment changes, as also the industry, the tooling that we use, the stacks that we deploy is changing, so it's something that we cannot, as an industry, stand still, we need to constantly evolve, respond to tactics, build up new defense systems, that's one thing, it's constant, but it's also an unfair battle, because when we look at custody in custodial solutions, the attackers need to compromise just one entity, that's the thing that we don't like about intermediaries, about custodians, that they become the central point of failure, but then the investments into security, to protect against these attacks, they're socialized across all users, in self-custody, the protections, the investments into security need to be tailored for each user, and that's, it's just an uphill battle, because on the other hand, you have actors, like the Lazarus Group, who are willing and able to invest millions of dollars just to hack one user, because if it's a big enough user, the investment actually is worth it, so that just means that the users themselves need to actually be fighting this battle against potentially much bigger entities, and this creates a very imbalanced battlefield, but that also means, actually, that we, as an industry, should give the tools to the users in order to fight this battle, I think that it's clear that different users, depending on their use cases, they need different defensive tooling, and especially users at the scale of Bybit, they just need much more hardened security, much more tooling in order to stand this battle.

Takeaway number two, even strong on-chain security can be broken through off-chain trust assumptions. As an industry, we've invested a lot into on-chain security, into audits, simulations, formal verifications, like we've really done a lot there, but users are not interacting with Solidity, they're interacting with JavaScript, interacting with, like, off-chain components, with wallets that they download from the App Store, with a lot of things that are on top of the on-chain security, and that means that we create an extended transaction supply chain that depends on a lot of components, on libraries, on relayers, on indexes, a lot of off-chain infrastructure, we're kind of building this Web 2.0 stack that sits on top of the Web 3.0 account, and that needs to be dealt with in the same way that Web 2.0 security needs to be dealt with, and that is different approaches than the on-chain security, and the key thing is that the longer this transaction supply chain gets, the more complex it gets, the harder it is to ensure that this chain doesn't break at any point, that there's no weak link in it, and the real challenge there is that when such a chain breaks, it's quite easy to point where is the weak link, but the hard part is at any point of time before something breaks to make sure that each of the links is hardened maximally, and the more complex we get building these Web 2.

0 solutions on top of the Web 3.0 stack, the harder this gets for us as an industry. So, for takeaway number three, and that's actually for me a silver lining here, is that client diversity differentiates self-custody from custodians, because after the hack, the safe order team right away took down the front end, so it investigated and remediated, hardened the security of the front end, and that required the front end to use at times of uncertainty and the work that had to be done, it wasn't possible to keep it online, and in any case in traditional finance, when the front end is down, even just for maintenance, users cannot access their assets, they're completely locked out. I think here it was good to see that during that time, there were projects on the ecosystem stepping up, offering alternatives, products that have been existing before, being able to help users do transactions during that time, there's mechanisms like CLI that we can run locally, there's like a decentralized front end on IPFS, the eternal safe, you can run the stack on your own, which also a lot of users actually do, and which also I think should be supported, and that should get easier, but that meant that during that time, in self-custody there's no need for bank run, because someone else was drawing funds from their accounts, doesn't affect anyone else, and that I think shows the power of self-custody in the end, but it also showed challenges, where even though there are a lot of solutions out there where you can interact with your safe account, some of those products, they also depended on the transaction queue, which was also taken down during that time, some projects have their own transaction queues, they have some redundancy, but that also means that we need to think client diversity for the entire stack, and we cannot just limit it to the front end, and we need to make sure that we can decentralize as many parts of the stack as possible, and I think this is relevant generally for self-custody, for different DeFi products as well, that they have access points beyond the canonical interface, it also is very relevant in the context of smart accounts and 7702, where potentially like very custom implementations may not have this redundancy, and then they become suddenly dependent on the availability of one specific front end, that's something where further investment is needed, and especially on the safe side, I think there's, yeah, certain things we took from that learning, where we need to make sure that there's redundancy across the stack. Takeaway number four, we need better tools to educate, and then education to enable users to verify transactions.

Yeah, we often say, don't trust verify, and we need this ability to, as a user of self-custodial solutions, to be really the last line of defense, and be able to verify the transaction down to like the payload, the methods that are being called, like this is really a key thing of our industry, but it's also easier said than done, and I think that's something where we need also to invest further, where in this case, Bybit verified. They verified what they saw on the front end was what they intended to do, but that's not the full verification that's needed to really have this last line of defense, because the front end told a lie, the ledger devices told the truth, and they relied on the front end there, and that's something where education is needed, but also better tunings in order to make it actually feasible for people to verify the transactions that they sign. And there's already a lot of initiatives at the back of the Bybit incidents that is working towards that, both in the safe wallet front end, where there's better ways to guide users to tooling, and better ways to verify transactions, but also the ecosystem players, ledger is pushing for clear signing, also together with MetaMask, the safe utils tool from OpenZeppelin, and also from the Ethereum community member who wrote the initial script, like these things are needed, but we need to do more there, because it's still too difficult to verify transactions, even with these toolings today, and we need to make sure that people can really have human-readable instructions that they confirm on a trusted device, or trusted wallet, and that needs, yeah, we need to do more. So, takeaway number five, we don't just need stronger security, but we need layered security. And to explain that, right now, essentially we have two layers, two high-level layers of security.

One is the security of the transaction supply chain, that goes from the front end that you use, the RPC, the SDKs, end-game packages, like everything, every line of code, every dependency in there, that is part of actually crafting, or turning the user intent into the actual payload. But the other layer is the transaction verification, and that's really this last line of defense I talked about, where the user can verify on a trusted hardware or software the payloads that they're signing. And while I think we need to harden both of these layers, I don't think this will be enough. I think we need additional layers of security that actually can ensure that even if those assumptions break, that the outcome is not catastrophic. I think that's, yeah, that can be achieved in different ways, like this can be on application, on a protocol level, where it could be like hooks in the context of modular AMMs, it could be assertions that are enforced by a sequencer, different ways that we could actually have execution guarantees on transactions, that even if the user signs something wrong, these fail-safes catch the bad outcome, so they can at least reduce the harm.

And here just a call also to the community, especially people building on SAFe, to explore guards, because they can really enable that. Guards, they're essentially a plugin that does pre- and post-transaction checks on each transaction, and that enables things like enforceable policies, whether they're provided by an application, whether they're set as custom policies by the users, but there could be policies like, my SAFe can only transfer this amount of assets to this specific other SAFe or something like that, that then every transaction, no matter what you sign, needs to fulfill this check and otherwise reverse. Guards also can enable things like co-signers, where there's external parties, maybe a decentralized network having the right to check transactions and either give a thumbs up or thumbs down, and therefore allowing more of these fail-safes to live, whether they're on-chain, whether they're off-chain, to provide these execution guarantees. Also resource locks, that's also a topic that's becoming more popular generally that in a way allows additional escrow logic before assets leave an account, but there's different ways how it can be implemented in SAFe. There's actually projects already doing that in different ways, hyper-native blockades, they have such co-signers, Jocera has their chat system, like the Zodiac team has built this already way in the past, but we need to push on that front, because this can build up a third pillar of security, which makes sure that we have fail-safes in place in case the other assumptions break.

One simple thing that the SAFe team has been building now is a delegate call guard, which essentially would just limit to what Singleton's account can be upgraded to a curated set, and this would have closed at least this specific attack vector that private experience, but the design space is much broader than that, and like it really allows to cover many different attack vectors in different ways. Now to the last one. So final takeaway for today, self-custody needs to become more cypherpunk again. I think there's always this tension in whatever we build between on the one side, simplicity, UX, mainstream adoption, and on the other side, the values of cypherpunk, which is censorship resistance, privacy, and security, and I think that's especially in recent years where we at the industry wanted to push for more mass adoption, we wanted to onboard more users, this can easily dilute the very values of what we're building, and that's something where we need to be mindful at least of trade-offs, but also actively push against and make sure that we retain these important values of our ecosystem. That's something, at least at F.

A.F.E., we want to double down on that for a new initiative, which is F.A.

F.E. Research, which really aims to solve the hard problems of self-custody around censorship resistance, security, and privacy. This will take a lot of work, they're not easy problems, but that's important that we continue to invest into that part as well, and like there's a lot of problems, it's just a subset of things that we need to cover there. For everyone that's part of SAFECON tomorrow, which I highly recommend, I think it's on the fifth floor here at the DAPCON venue, I would suggest you come by the talk at 2 p.

m. from Richard, who's actually leading this initiative, where he can tell you more about what that actually means and how that will look like. I see time is up, but maybe to end with, there's a lot of people in this room that are fighting this exact battle, and I know that it's often very hard, because it feels very much like an uphill battle, like there's constant new threat actors that we need to consider. It feels like a never-ending thing, because we've been building for a long time, we've been hardening security across the board, and so we see a lot of hacks and exploits on a monthly basis. I just want to thank everyone that's fighting this fight together with SAFE.

I think it's especially also in today's world, where we see financial censorship, we see data breaches of centralized parties, we see even intermediaries sponsoring military parades. I think it might be the most important fight that we're doing, and so I think it's crucial that we keep at it, we don't give up, and we really retain the passion for SAFE custody and also the values of our industry, and don't lose that. Yeah, that's it. Thank you very much. Do you have any practical tips for teams that want to avoid such supply chain attacks?

I mean, there's the boring answer, which is there's certain standardizations, especially in Web 2.0, SOC 2.0 compliance, MDM files, the best practice is just in Web 2.0, there's a lot of literature, training that we can do, that needs to become a must-have for the industry. That's one thing.

But maybe the more practical thing, just assume at any point of time that there's a target on your back, and that feels like paranoia, but it really is important, because we need to assume that it is, and we need to operate always under the assumption that someone at this very moment tries to infiltrate our systems, tries to infiltrate the team, be hired in a position, like somehow trying to get to our users, because they're actually not trying to get to us when we're working SAFE custody, but to our users, and we need to make sure we're operating with the interest to protect the users there, and that needs a little bit of paranoia, and like always, thinking twice about everything we do.