Defcon at Devcon: A table top experience | Devcon SEA

[Music] hey guys hope you guys can hear me okay now but uh welcome to Defcon at Devcon a tabletop experience today we're going to go through the heroing experience alog together of what happens when your protocol gets wrecked now today uh my name is Heidi and I'm joined here by Peter my colleague we both work at coinbase on the unit ZX team now let's set the stage a little bit over 620 million doar have already been stolen this year from almost 300 different projects the average time to steal funds is only 15 minutes but the average time to react is usually at least an hour now there have been about 27 or so instances which were great where white hats were able to rescue about 50 million and you too can also save millions of dollars by practicing incident response in a very safe and controlled environment today here with us so I'm going to go and hand off the discussion to Peter where he's going to be talking about the tabletop set up cool thanks Heidi so one thing that we found that really helps folks to deal with incidents is to practice practice not just you know when you have a real live one when you're freaking out or waking up at 2: in the morning but in a control safe environment where you can actually take lessons and build things um without wrecking your whole protocol so what we do at coinbase is we run tabletop exercises all the time where we think of a really scary scenario and we practice that scenario in a very controlled simulated environment so tabletops um so the goals for today one we want to practice incident response handling how many of you have participated on a live or maybe simulated incident response uh exercise not all of you that's a good thing that means you didn't have to go through the worst of the worst thing that your protocol can go through but I encourage you to take notes what we're about to do today and bring it back home and do the same thing again again and again with your project um the other thing that we want to do today is we want to educate you we want to educate you about some of the threats and exploitation tactics which are most like this I don't want to give you any spoilers but the scenario that we picked out for you today the thing that we're going to practice is something that will most likely hit your protocol and most likely be damaging enough to pose an existential risk to you um and you know like the way that we designed it is something which is very portable you know we're we're not bringing like large infrastructure or things you can literally just after you come back home schedule some time with your teams and say hey let's let's uh you know Heidi and Peter they taught us how to do those things let's run a quick thing like this is what worries me the most like we upgraded a contract or we deployed a new dap or we have some cool Cloud thing that we just sign up for let's let's practice what happens if that gets hacked participation is not only encouraged it's required so we're going to use slido we're going to give you questions you know things to discuss you can team up if you want to um but we're looking for your live responses we'll give you time so you can you know brainstorm how you would respond to different things that we're going to throw at you we're going to throw a lot at you uh suspension of disbelief you know we're practicing this is all simulated so we may give you like certain scenarios which may think like oh that would never happen to me in me real life well most likely it will but but you know suspend your disbelief sometimes uh it won't be perfect but the key is we will learn and the key thing I want you to bring home this like feeling of um if something really nasty happens to me at least I kind of played with it once so I'm going to breathe in and do what needs to be done okay so terms injects artifacts discussion so injects are things that we will provide so you know we we'll inject something some event that just happened throughout each inject you may ask us questions those are the artifacts so we can you can ask me like hey uh Heidi you know if I was to look at that EA address do I see any interesting labels on ether scan you know or if I talk to person X we can even do like a real roleplay maybe we can you can talk to one of your devs so we can we can play those games together so you can ask for more facts and we will provide those facts to you and at last discussion so this is where the slido comes in we will go back and forth based on what questions you insert through slido and we'll discuss them together okay so Logistics so we are the facilitators we will provide the injections and if we see that within your slido answers you're kind of going down the Deep Rabbit Hole like is this like something really outrageous we will you know call it out and redirect you to something more more reasonable um we'll give you a break you know it's uh 10:00 so we'll give you around 11: we'll give you a 5 minute break um and then after the exercise is complete we're going to sit down and talk through what actually happened and you know all the all the things all the lessons that how we would approach it and how we would we would love for you to what takeaways we would like for you to take go there um one more thing which is pretty cool for every inject that we provide we will also give you a case study everything that we're giving you today is based on a real life compromise so we will give you an inject something happened and then we're going to give you a project which was hacked in this exact way and then we're going to compare and contrast how this project reacted and how you reacted and then how we would react so we will learn together um before before we jump in make sure that you can scan this QR code and you can or go to sl.com or go to slido.com and just type in that meeting number and you will see that there's a drop down of various different injects if you don't feel comfortable scanning QR codes yeah we'll give you a minute or two to make sure that you're all to be clear that QR code you see on the side there for mircat we are not going to be using today so make sure you use slido the reason being is that a slido is a slightly more interactive um and B we also wanted to encourage remote participation so if there anybody if anyone's watching this live stream right now you too can join the slido link uh and uh start asking questions or making statements how's everyone looking you're able to log in all right cool okay well let's begin then so there's this really cool project out there it's called founder mode down and unsurprisingly it actually was launched on the mode L2 chain it's got over 100 million tvl right now in fact actually it's it's got almost 2 million 200 million tvl at the moment and what's interesting um about this project it is that it involves the fumo token which is the governance token for the project now what this project does is it invests in various native nent D5 projects and requires them to create a token pairing with fumo so unsurprisingly about 80% of all projects on the mode chain hold fumo or some derivative thereof and you are trying to look for to join a new project and guess what you just were hired by this project you're very excited uh one of your contacts from Devcon back in Boga in 2022 reaches out and is like hey we'd really love you to participate with us and you are psyched the team is fully remote um and it consists of only five people but you guys are ready to go and what's really nice about this project is you kind of know everyone you everyone uses their real name everyone's very chatty and conversational and you're really excited you're at your computer you just finished like day five of getting onboarded and you're reading some of the developer docs making sure you fully understand what's going on within the project when suddenly you wake up at 3:00 a.m. in the morning your phone is blowing up Telegram signal Discord X all your friends are messaging you that your project just got wrecked your team is panicking and begging you to sign off on a quick real quick protocol upgrade what do you do so if you could pull up your slido and if you have questions put those in the slido feed what would you investigate I mean and look at those questions right there that we have to kind of prompt you um but yeah you've got two minutes we'll give you two minutes time to think of questions responses and then we'll talk through them so I'm already seeing a lot of responses here and unfortunately we're not going to be able to share the slido here directly with you guys I guess I can plug my laptop in maybe we want to do that um but I'm seeing lots of questions come in and I'm sure when you're on your phones you see them as well um some people are asking and it looks like it's being quite up voted right now is what happened here I don't know Peter do you know what happened here well the key thing is that the contract had a A Fault in that contract and it appears that the funds are flowing out of those vaults someone also said I'm going to call my mom and apologize to her for getting into crypto I totally get that it seems that as as folks are investigating a little bit more it looks like the mean contract holding all the vaults was recently upgraded keep putting in your question I'm seeing start a war room my question for that person is is how would you go about doing that who would you start a war room [Music] with your team who's all asleep at 3:00 a.

m. is there pause free function in the protocol there used to be but now that the protocol is upgraded there is some unknown contract which is sitting in its place with no source code someone's already asked has the upgrade been audited the response here is uh no we don't have time for that it's 3:00 a.m. the upgrade does not to be appears to be expected in any way you can update the resume that's fine yeah we have some people rapidly trying to just get out of the space Fair let's try to share the screen I think it would be fun can can we share can we do um can we do a screen share a screen share yeah can we plug in the can we plug in our laptops again sorry sorry I think it'll be fun to see the outputs but someone asked war room and I know that I prompted them like saying well with who your team um I think uh just to provide some artifacts you reach out to seal 911 um which is a part of the security Alliance consists of various different people many of whom are actually in the room uh that are willing to help um and they're starting to look into it but you don't want to yet spin up a full-blown uh investigation quite so what does the protocol consist of so the artifact to inject is that it appears to be an unknown contract which is being called by an attacker and after every single call a new vault is being drained you're good we've got a bit of a delay but now we're going to be able to share no signal that's good okay there you go they drained the volts could you could you please repeat the last response yes so uh following the upgrade you're observing onchain a whole series of transactions and after every transaction appears to be large amount of fund are being drained from The Vault previously controlled by that contract someone also said roll back first analyze after what do you mean by roll back roll back the chain Contra roll back the contract okay but funds are already drained you're not recouping those funds there used to be an emergency pause in the original contract but since it was upgraded there's no it's unclear what functionality is in the contract how would you approach that oh let's get a mic so you need a mic we I'm actually a little bit confused about the the cause of the hack are you saying that it was the protocol was upgraded to an implementation and there's no source code correct and who performed that upgrade was it the that's a good question how would you figure that out oh okay so wasn't the team that performed that upgrade and caused it themselves might want to ask let's let's figure this out that's a good question let's let's follow this uh Chain of Thought like what what questions and who would you ask is there a CTO no there everyone the CTO yeah there are five people there's a CTO did the upgrade happen at 3:00 a.m.

and then the hacks happened right afterwards it appears that the upgrade happened shortly before the the drainage started the drainage started um so we could ask everyone on the team in the war room if anyone knows anything about the upgrade first and then check the onchain transactions to see like what kind of governance system implemented that upgrade if any if there's like a multi- we could start looking into that so interestingly enough the upgrade was performed by a private key that was used by the original deployer okay and the team members all deny perform this upgrade themselves there are only four team members on this war room call right okay so we can assume that the deployer he got compromised by hacker potentially possibly possibly okay I think yeah so you're you try to get five people on the call and so far you have four it's a It's let's say it's 4:00 a.m. you're still trying to reach out to people this other Dev is a little flaky so you don't know a little flaky Miss don't yeah right now we don't know a lot the Seal team is investigating that along with the uh the project team in the War Room at the moment I think um we're going to pause this conversation here to actually go over some real world case studies and I think since I'm sharing my screen it's gonna be pretty nice perect oh yeah okay so before we dive into next steps we just want to step back and look at just real world recent very recent real world scenario we talking about end of September of how similar protocols react to that kind of 3:00 a.m. phone call so the case study we're going to use a shzu so the sorry let me uh so the initial detection that someone reached out to the protocol saying that well how the protocol even found out there is an issue was the classic Tweet someone tweeted at them saying that there's an issue so this is at uh 8:00 p.

m. 8:30 p.m. it took the protocol just 20 something minutes to make an announcement to triage this so before that like the actual Hack That was happening so this is the kill chain the hack itself happened at 7:20 so it took a whole hour for someone out there to detect that there was an issue and reach out to the protocol um the exploit itself was deployed 1918 1921 the exploit was deployed the first transaction started happening at 7:30 p.m.

and 5 minutes later the attacker already started swapping funds so this is a typical timeline when we when you get the initial call like that every single minute counts so when we we eventually ared like oh like the right question was asked like hey like who made that upgrade where is that coming from can we upgrade the protocol again to save it like those type of things need to be in your playbook they need to be in your kind of like you known things to do ahead of time so you can shave off those minutes so those five minutes before the attacker so the two exploit transactions that were executed usually attackers don't drain everything all in the same goal so you have just a little bit of time literal minutes that you can minimize the losses so it's just a one case study um so in terms of response for for this hack in particular um an hour later someone detected it half an hour later there was an announcement and then a day later the pools were paused so someone asked like can we pause the pools right away rief I think I think it was you that question was asked and responded to almost a day later so just just a just some some things to absorb know what you need to do including mitigating actions like pausing pools or doing an emergency upgrade or something like that ahead of time now one of you guys also already mentioned this but there are some online slle out there we're already starting to look into the actual exploit trans actions and an online sluth Zack actually points out that one of the funding addresses of the attacker has an ens name with the same telegram handle as a recently hired contractor the project now what are immediate security concerns here how should we investigate this situation now and how can we respond internally or externally now that we know that there might be some Insider things going down I'm going to give you guys two minutes um to write your questions in slido um and the way you're going to go about doing that um within uh slido is actually go up to the top and just flip over to inject to and start writing uh any kind of questions you have we'll try to answer them real time but someone's asked what access permissions do the contractor had can we revoke those yes we can to an extent unfortunately mode uh to founder mode uh we let everyone set up uh their their environments using their personal laptops so we can only revoke access keys that we know about in the cloud otherwise though we can't really revoke much access what's the contractor's role someone asked um the contractor's role was as a developer um they were testing out upgrading some of the pool contracts um so that you know we could um have uh better better and more rapid transactions flow that was one of your [Music] devs has that Dev ever said anything negative about Kim Jong-un no no he hasn't but you know what we didn't apply the test of like hey can you please say that our dear fist leader is terrible no we we uh we didn't do that here in the project team unfortunately um I mean you can try to contact him and that's actually what you do you're like hey I want you to write me on slack immediately [ __ ] Kim Jong-un he does not respond though he has not by the way responded to any messages that's on telegram on X on his phone nowhere he's like Gone Gone does this contractor have access to other protocol contracts interestingly enough this contractor was involved initially with the development of founder mode he just didn't think he could commit full-time to the project is he dead I don't know someone's got a check on him so in terms of actions um so feel free to ask for artifacts like what's going on but what are your steps what what are you working on right now as an incident respond and let's flip back to those questions to kind of help guide us right what are immediate security concerns some of you guys ask like hey does this guy still have access to any kind of privileged roles yes he did um and let's say you start revoking them all immediately there's another question here of how can we respond internally and externally we'll let you guys sit with that one especially externally it's one that project teams think less about but is critical yep rotating deployer account quickly so that's one action so on the proxy contract you are trying to rotate to new keys okay that's a great idea where are the exploited funds going well currently the exploited funds are just sitting in any eoa belonging to the exploiter but that's a good question we're concerned about them moving along what piece of code does he actually inject into the protocol so when you go to Ether scale and you try to see like what is the code it just looks like a large binary blob that's that's all the data that you have like what's any like what would you do with what would you guys do with that there's no there's no nice solidity code uploaded to Ether scan attacker was um not nice enough to do that someone asked where's the onboarding guide use that guide for Access revocation I'm sorry this project just spun up a couple of weeks ago yeah sure it's taken over mode but we don't have an onboarding guide here we are the onboarding guide as the team hire Hitman okay it's a good way to start thought he was dead someone decided he need we needed to hire a hitman he's getting a little wild here have we tried decompiling that's a great idea thank you so uh you try to decompile the contract and the one function it's all kind of scrambled but the one function that stands out was the one that was used to drain volts it appears to be very simple um token transfer uh functionality that links to the attacker's address there's a proper access controls in place so only the attacker can call it and all it does it takes the token address that the attacker wants to transfer and just simply drains whatever is in the contract so it's very simple you're also seeing other you're also seeing other um I guess functions within that decompiled piece of code so when you you can also specify it looks like the token address to drain and the amount and also the token uh the token address the some kind of like owner address and also the amount so you have two functions in there which appear to be non-standards some people said uh take down any external sites where users might add funds to pools and issue a warning not to interact with any contracts that's a great idea um hopefully the team is talking about this in the war room and is issuing tweets and reaching out immediately to these websites because as you guys remember this project um sure it has a high tvl but impacts all most other projects on the mode chain so effectively we're this this whole incident could make Mo chain entirely toxic which is not what anyone wants so that's something that we haven't heard the room do and oftentimes we see projects also forgetting that they're so busy with handling the incident they forget to do the coms and it's very important to tell users tell other projects what is going on what is the impact and potentially if it's a systemic issue to give them a heads up like maybe we're dealing with yet another compiler bug right where not just your project is affected but everyone is affected or there's like a Mass fishing campaign where not only project your project was hit but others so it's very important for the sake of the ecosystem to communicate there's an issue someone else said uh pause the protocol is a safety measure tweet about it assuring user funds are safe investigation's ongoing this is exactly to Peter's Point messaging out to users up front even though you don't exactly know what's going on is critical it provides that trust for users to potentially still come back to you even if you do end up being wrecked time and time again a lot of Protocols are not exactly explicit and forthright about what exactly is going on and that destroys trust within you and your users what you might do in this sense I and it it's worth asking is what funds are not safe here are there still treasury funds and if there are treasury funds then you can say hey we have treasury funds funds are safu in one way or another we can help reestablish so that would be one way to do it another person said contact exchanges Bridges to see if we can stop the funds from being withdrawn that's a really good question um especially because we're right now the funds haven't yet moved from this one EA but once they do start moving they're going to immediately attempt to flow out of mode chain why as I just mentioned that entire chain is going to become very toxic soon mainly this attacker has withdrawn mode related tokens right and so these mode tokens aren't useful on any other chain you want to get out of there if you're an attacker as soon as humanly possible and so you're going to bridge over those funds so bridges are most as well as any centralized entity that is interacting with the specific founder mode token I think uh we're going to move on to a case study actually just because in in uh the essence of time because we have a lot of different injects to talk about here today but um well if it loads we're going to talk about a very recent case study in fact actually this slide is way out of date because it's many of you guys know um Delta Prime was not just hacked on September 15th but also just a couple of days ago again um and on September 15th though it was hacked due to a compromised admin key we don't have to go into all the different details of it but it was because the admin key was compromised and interestingly enough there are quite a number of concerns my bad there are quite a number of concerns about Delta Prime and how they were managing their contractor base in particular is you do a little bit of onchain sleuthing as many people have already done you can find that there are relations to several unsavory actors out there whove been paid directly by the treasury to do some contract testing um for the protocol and these guys have been involved in lots of other really interesting exploits in the past which date back all the way even to like the yam Finance exploit in defi summer of 2020 that links back to one of the cont contractors that these these guys hired so I'm not going to go into all the details on that one but having a chat offline um about the Delta Prime situation but various different unsavory characters have been involved there and they've been alerted of this in the past if you are a protocol and you do get alerted of this make sure you take it seriously dig in assess whether or not you think the risk is worth the potential reward of having hired this contractor yeah and uh uhoh there's another exploit just as you were thinking that you've made the comms looks like the attacker's done just holding all the funds at this point they drained the entire protocol now you get another alert users are users on the mode chain are reaching out saying that their wallets are now being drained so not the funds stored in the vulnerable protocol itself but the money is moving from their very own wallets and as you start interviewing all those users which were compromised or sorry are losing funds the only common thing is that at one point or another they interacted with the founder mode contract what could be causing these wallet drains how should we respond to protect users and what is the overall impact to the protocol ecosystem we're going to give you two minutes to ask questions yes so for those of you that didn't didn't hear this revoke approvals immediately how do you do that across a decentralized community of individuals sorry I'll go back awesome um I'm gonna start uh reading some questions here so it looks like someone also said infinite approvals can we easily compl revoke all approvals uh that would be really scary if we could I'd be concerned about there being additional attack vectors if that were the case so that is no we cannot should we try to upgrade the contract to a null address uh something like null code okay so we upgrade the contract to basically overwrite yeah attackers whatever malicious code okay that's one approach if that's possible yeah I just have a general question like before this even happens we communicate externally in this situation would you recommend preemptively for users to revoke all their approvals against the protocol as part of that almost like a standard action I mean there there multiple stages of the inci response like one one stage is the triage and then the root cause analysis so it's very critical to perform root cause analysis before jumping into like mitigations and Cs and all that to understand what happened what's affected and how big is a blast radius so you may accidentally advise users to take actions if if it's just for safety that's fine what I'm more worried about you you you force people to do actions which may be more damaging because you don't have a chance to fully understand what is the root cause I'm trying to think of a scenario so if there is a compromise and they start interacting with a token or protocol revoking approvals and by making that approval they somehow are triggering the code they shouldn't we could have a mass I hav attackers don't listen to this but point is if the approval itself is getting back doored and all the users are calling it on the compromis token now you're poning the whole ecosystem um that's a good point extreme example but the point is root cause analysis and understanding is is very important there's also like the Panic thing like what what we're seeing is was actually IM mify recently published an article about like what is the true impact of a token compromise and we usually focus on like hey pro project ta was compromised and they stole $5 million or whatever but the secondary effects are often times 10x if not more the initial impact so we have token prices which are dropping to zero the value is dropping then you have secondary ecosystem the like the people are getting wrecked if they're loaning something if they're getting liquidated so very important to be very careful what is it that we're advis to users and externally after we fully understand what what exactly happened actually on that note Peter I was just thinking about I forgot if it was in 2022 or 2023 but wasn't revoke cash overtaken uh at one point or another and so if you're tweeting out hey use revoke docash to revoke all approvals if an exploer has taken over a website like that you're dunzo right now your users will never trust you again um and that would actually be one of my bigger concerns as opposed to some you know little smart contract thing but um someone wrote on here why don't you tweet to revoke all token approvals with the fumo contracts that's one thing you could do but as Peter mentioned again under understanding the landscape is extremely important but messaging to users is also important so we don't want to forget to message to them saying that we're looking into it we're investigating um but you might not want to say hey revoke all token approvals until you've actually done analysis yeah so so far we got a couple good points if the attacker was nice enough not to change the administrative account on the proxy upgrade it to something not malicious comms revoke revoke approvals anything else that you would you folks would jump in on okay which token got uh drained ERC 20s doesn't matter oh so if it is like uh proprietary token of the fumu contract then we can just like uh remove the liquidity from all the decks and like we will tell all the centralized exchange like this is a blacklisted token so at least that hacker or the the trctor can't uh get out it fun we can do that for fumo but the problem is because there were so many pairs right you know how he talked about how the fumo governance token works you can maybe do it for fumo but all the other tokens out there are screwed right all the other users that hold those um someone asked here what wallets did they use they used all sorts of different wallets metamask coinbase wallet privy doesn't matter Unfortunately they all got wrecked so unfor this is not a situation of like a slope wallet uh here but good question and someone asked here what does a sample transaction drain looks like well they're calling to a contract and you can see here suddenly there are a whole bunch of transfers then looks like over 100 transfers take place in one transaction alone all just transferring ER c20s from whatever uh whatever person whatever EA it was that had done the approval to the particular exploiter remember when you were decompiling and there was this unusual method which was taking a token address amount and also an an I guess an owner EA address it looks like the attacker is calling just that so they were not only greedy enough to steal money from the Protocol no they came after users as well so someone has the cool idea kind of like you where it was snapshot the balance before attack and try to migrate the fumo token or fumo contract token into a new one great idea but only good for poor fumo and not everything else and unfortunately as we know fumo is an ecosystem token if the ecosystem's wrecked doesn't matter if fumo exists or not someone also asked hey can we get Waller providers to show warnings about approval revocation yes that's possible so you're in the seal 911 team uh who can hook you up with metamask and with a couple other wallet providers and they're happy to help um and they're happy to provide sort of that warning sign um but as you see on chain you see at least 10 of these transactions going down as you're trying to reach out and get contact with metamask and that's something I someone mentioned it here but we should continue pushing on the ecosystem if you recall this is the ecosystem wide token we should be in the war room talking to all the other projects right now pausing their protocols because it's not enough that we are compromised the the fumo token is compromised all these other liquidity provider Pairs and they're about to get wrecked as well they're about to get drained because someone is has these Mass amounts of uh fumo token and they will use it because they need liquidity to get out of the chain a lot of other questions uh no there's one question about did Ledger and BM package get compromised again I don't know do you want do we want to research that maybe we get a researcher in Seal 91 to take a look at it um and they find HM no I don't think this is The Ledger mpm package it doesn't look like it this does not look like that particular compromise at all that developer the fourth developer sorry the fifth developer is still not online by the way you still can't reach them yeah we're not sure dead Hitman what were the other ideas that happened to this guy food poisoning I don't know someone asked can white hats recover funds from uh from the user using this approved related method are there legal concerns with this well unfortunately the attacker was nasty enough to change the admin owner of the original compromis contract so you cannot just upgrade it as easily so white hat action is not possible but what action would you take like uh is there anything that if there's a chain wide hack like what any other creative things that you could take so we're already messaging out to wallets and asking wallets can you please please please throw up a warning sign revoke approvals we're already sending out tweets saying hey revoke approvals not using the revoke docash link or anything like that because we're too paranoid is there anything else we can do uh was that fifth employee working remote or where can you repeat that so the one guy is still missing on the call right so we can uh talk to the local police uh to their uh living area yes and of course you know this person's true name right and the previous inject we also learned about as enss handle we can do a little bit more due diligence on that front in steel 911 you also have some online SLO that is trying to figure out where he lives however our project is entirely decentralized we love that we pay everyone in fumo right we don't know exactly where this dude lives unfortunately we do have his logs though and unfortunately his logs point to him using mulvad VPN and so we're not exactly sure of his location and mulvad uh is a little notorious for not necessarily providing us the details we want um but we can do a little bit more digging again we're a startup with five people okay I'm seeing a note heart Fork the chain here we go who whoever H asked that question if could you get the mic please and let's discuss this a little bit or they might be remote or they might be remote well on a high level so we've seen I'm trying to count the number of times that full chains halted or hard Fork so we have the blast that comes to mind we have Linea we have the good old BSC when the the bridge got BSC token token Hub was hacked a few years years ago so it's not an unheard of thing to do to hold the entire chain when a major protocol is hacked so what what how would that even look like just just to just to um you know we're what we're doing this here is like we're simulating scenarios so if you are arriving at a point where you know as Heidi mentioned this is half half of tvl of a given over half okay half over half of tvl of a given chain L2 chain got compromised is it unheard of to reach out to the chain owner like provider and start a discussion to Halt it or maybe like what are What are the in between actions like would we call mode into the war room and beg them and tell them your Project's dead in the water your is effec tox I want to say Phantom had that problem right Phantom hasn't really come back from the grave since the multichain exploit it's effectively become a toxic chain right so maybe you could dangle that and say hey Mo you don't want to become another Phantom you know please come help us but is there any other precedent for that I don't know but we can certainly talk to them and maybe try to get some lawyers in the room again we're a bunch of devs we're only four people now we don't have lawyers yet but we're still talking to seal 91 one and uh trying to work that through with them so one thing to keep in mind when you even talking about like you know nuclear options this is a nuclear option it's a because it breaks the fundamental Assumption of decentralization for what is it that we do here if a single entity can decide to make Act take actions like that that's um that's uh you better you know I mean we're ethere like this is ethereum conference like we've been through this before so the question you need to answer is where and how exactly you're going to take do that so you have a variety of points you can you can uh like the linear approach or blast approach you can you know block list malicious addresses on the sequencer side you can approach it on the bridge side on the main net side you can potentially start looking for outbounds and start you know paying attention maybe block them there where else would you would you put some kind of stoppage or like what are your thoughts like go ahead maybe can get a mic or could we ask for a recap of what's worked so far because we've thrown a whole bunch of things on the wall what's what we were we actually able to accomplish in the last last 40 minutes so so far in this last scenario the vaults are drained before we had a chance to war room and get everyone in attackers as I showed you with the uh one of sample exploit attackers need 5 minutes to do their thing so attacker is currently holding all the funds in their eoa the funds are not moving yet following that internal investigation showed that it's potentially someone who has access to private Keys the deployer private keys and the last piece the attacker is currently actively draining all the users that ever interacted with this protocol and collecting those funds as well so funds are still not moving out of that eoa but we are still in the war room kind of like deciding what are we what are we doing next we've unable to we were unable to pause the contract because the core functionality of the contract changed it's now a a contract which is fully controlled by the attacker so that ship has sailed there's one other fact too that uh your fifth Dev you're unable to reach out to them right and we know that they may or may not be associated with other elicit activity let's actually take a quick pause here and let's do a quick case study I'm sure many of you um already kind of assumed some of these case studies coming up here sure okay so the case study that we're basing this on is radium Capital so radium Capital if you look at the attacker right now on chain so the initial hack was $53 million and you know pretty fascinating attack all by itself the the three key signers of the protocol were not NE let's say tricked they were tricked by substituting a transaction that was on the way to their Hardware wallets to make them sign something they did not intend to sign which is an upgrade of a contract here's what's more interesting in this not only did the attackers perform this upgrade to drain all the vaults they did add the functionality just like our attacker to also drain the users I have not seen this level of greed and also sophistication at the same time that it's not enough for you to be $53 million richer no you continue the attack and I'm I I looked at the the attacker just a few days ago they continue draining users even as we speak right now so the second someone put some money in their wallet drained so they have a script running which is just monitoring all those wallets that are approved to that contract and they continue getting themselves more and more assets so just a case study so this is again every single inject is based on something that unfortunately happened in the real world you should be prepared for so the takeaway here is just because your protocol just got compromised it doesn't mean you're done not only not only is your protocol now lost funds you have to start thinking okay what is you know we discussed like you need to do the root cause analysis to fully understand the problem you do need to quickly decompile the contract look at what other interesting functionality there may be sometimes you get surprised attackers don't put uh some kind of access controls so you can wi hat it so you can take all those funds back um that happened before or at the same time you need to immediately start warning your users and reaching out to your partner protocol saying the first wave of the attack is done we are about to get hit by the second wave which is going to Target our users as opposed to the protocol itself so just another lesson learned so the tldr of that is you shouldn't just be worried about your treasury your pools you also need to be worried about your users right your your core users are the reason why you even have your project to begin with cool so we are now at halftime basically and I think it's a good time to go ahead and take a five minute break because it's been a lot um and we're going to come back in five minutes time um to uh continue maybe we can give him like so it's 10:51 right now let's come back at 11: and and there's more there's always more there will be a resolution not necessarily happy this is not this is not Hollywood so the story does not always end with with us walking in the sunset we're going to try though e e not on uh on here okay I think we're ready to begin after our lovely break so does any want to try their hand did a quick recap of what we talked about in the last hour all the injects anyone that's okay if you don't the long and short of it is is you're running founder mode your project right it involves the fumo token and your project as well as others with the derivative token make up 80% of Supply on the mode chain it seems like your project got wrecked all of the funds were withdrawn from various different pools um for your token uh due to a malicious contract upgrade um you no longer have access so it's not like you can try to take it back um and you know that one of your project team members is not responding to any emails slacks phone calls Etc they've just fully gone Mia you also know that this particular employee or contractor they seem a little bit suspect and finally you thought the worst of it was done it wasn't because suddenly users who had approved their tokens to all the pools that were drained are suddenly getting drained themselves so you spray the sigh of relief to an extent all the pools are fully drained at this point all the users appear to have been drained sure some users can still send funds to their EAS and those can get drained again but you know you've messaged out you've done all the comms that you can you've like talked with c911 you've set up a war room you're in conversation with local law enforcement you're in conversations with various different exchanges through seal 911 you think you're kind of done with the incident you the guys think that you can wind down the War Room at this point but unfortunately this is usually when the real fund begins because as we mentioned last time the attacker was mainly just sending funds to one eoa alone but in the middle of the night they start swapping and moving funds over to various evm chains so there are a couple questions we have here can devs do something what actions can we take who can help us stop this and how do we warn everyone so I'm going to go back to slido here and we're going to pivot over to inject four where we're going to start that discussion and now some of you one of you guys actually asked earlier like how is it that we would maybe reach out to a centralized exchange to maybe stop the flow of funds um and I wanted to leave it actually to this point uh so we could talk about like how do we even reach out to centralized exchanges my advice would actually be if you are already in a war room with SE 911 they can usually get you in touch so someone WR immediately on here stop Bridges so there are two different type of bridges here there are bridges right like a canonical Bridge which usually has a 7-Day lockup period um of funds so yeah you could potentially you know stop those Bridge funds from moving over but various different third-party Bridges right um those are going to be a little bit more tricky to stop now some of you mentioned Way Way Back like hey we should already be reaching out to Bridges let's say you reach reach out to a couple of them and only one responds and is blocking the funds that's great you've only blocked though $20 million 20 million is great but as we mentioned before we have a $100 million at stake here plus someone says Blacklist the address on token contracts like usdc that's a great idea unfortunately though usdc here on the mode chain is not native meaning that uh we have we're dealing with usdc e here which is a bridged asset there is no blacklister ability same with tether but that's a great idea on chains where you can potentially have funds blacklisted like usdc and usdt that is indeed possible however I would not rely on that these the process for getting funds blacklisted is usually slow and arduous and you have to already have a contact with law enforcement established and I would say that nine times out of 10 the attacker is going to move out of those tokens immediately into tokens that cannot be blacklisted so I wouldn't even bother necessarily someone says pray pray Zach xpt finds the person behind this account well what would you look I mean it's great to rely on third parties but like what would your approach be to like to figure out like who is what is the identity like do we know who is the attacker at this point someone said uh why don't we warn the Mia contractor right remember we had that contractor that was our fifth Dev that isn't responding to us that everyone is a suspect at this point and there will be legal implications if laundering doesn't stop immediately that is possible but they're not responding to anything right they're not opening any of their messages it doesn't even look like they've been on Telegram in the last two days since this incident some say why don't we reach out to the attacker and uh have and say that uh we won't pursue any legal action if uh if they give us some of the funds back we're going to hold on that it seems like the teams back and forth as to whether or not we should indeed you know reach out to them we don't really feel comfortable we've been talking a little bit with law enforcement even though some people on seal want to do that we're just not exactly comfortable at this front uh doing that but that's an interesting idea well going back to laundry like what what are we doing to to see well one to identify where are we going where the funds moving and then potentially stopping or freezing them so we had some token freezes exchange Outreach anything else that we should be doing right now is there anyone else you maybe want to reach out to some of you guys mentioned uh Zach xbt is a person to reach out to there are a couple different uh online sluth that you might want to reach out to your team at this point is going to be extremely tired if you're up for X number of hours trying to solve this in a war room you're not going to have the ability tra track down all funds and you guys might be sitting there thinking well why do we even have to do that funds are gone well law enforcement's not going to be capable enough to do that when you reach out to them no matter if you reach out to them today or tomorrow they're not going to be capable of doing that so it's going to be in your best interest to understand where the funds are so you can best explain how to potentially get them back if indeed that's even possible there are a couple of different companies you could be reaching out to at this point um zero shadow is one of them that tracks funds and gets alerts another thing though too is you can start tracking the funds there are several different Services out there that are free like metas Luth that you can actually set up alerts on various different addresses uh so that you can see for example like with this attacker right let's say he sent funds only to one EA you could send an alert on that EA if it ever sends funds out right boom then you're going to be aware um so there are a couple different services like that that you can already just out of the box get uh without having to sign any crazy contracts or doing anything self-reliance is key like it's definitely great to reach out to third parties and ask for help but you know the reason why we're doing this exercise is to identify your gaps so if you today if you're sitting in the room and you're saying like well if I had to trace funds I don't I don't know if I'm capable of doing that well why don't you go it's a it's a free it's a free uh project that you can use today go to metas sluth it's free of charge pick up any exploit I think we have a case study coming up so you can take out the case study any of the case studies that we at we're we're discussing today and just see if you can just trace the funds like where where's the money flowing like what exchanges does it hit what project what Bridges what swaps they perform spend half an hour on that so that that 30 minutes that you're going to spend will mean the world if you have to do this under much more stressful scenarios where you have to like do this live like okay where where the money go at least you know the tools you know how to use them and just it will give you some guidance of like what who to reach out for help someone has this cute comment in here why don't you interact with an ofac tainted smart contractor wallet so that all of the addresses for founder mode get blocked so they can't drain money anymore that's cute but uh uh I don't think I don't advise anyone doing that if they are working or interacting with on uh users um or uh platforms that are based within the us or Europe generally speaking I I don't think that's a good idea it's cute though a security research company that sh remain unnamed did do this for the sake of the experiment it's not a good idea to do this let's let's stay there there's certain ethical things as a Defender that you can or cannot do so one of the things and it's well established in traditional security and applies to our side as well you can't do things like hacking back a protocol or targeting you know you know interacting with something that you're not supposed to like ofac uh addresses and so on and so on so it's a don't stay on the right side of the law actually on the hacking back aspect someone mentioned here you know why don't we get founder mode to sign with the seal Safe Harbor um yeah certainly that can happen but I mean seal Safe Harbor is great you know signing that beforehand would have been nice maybe there could have been some more prevention that could have happened uh but that's potential and for those of you that don't know what the Safe Harbor is effectively your project signs uh a a legal release so to speak saying that hey if I get hacked and in this particular instance where a white hat sees the hack going down they can actually hack those funds back and return them to this particular address that I have designated now seal safe Harper is fantastic but it works in very very specific legal instances so it's just one thing to keep in mind here and maybe now that you are thinking about you know what happens if my protocol gets compromised this will be the push that you need to sign up for something like this so if we if we go back to the case study we we covered earlier the shezmu the shzu hack right it was actually a success story because the majority of funds were returned um immed well first of all the the initial attack of returned funds for for the a bounty what's called let's call it a bounty Ransom Bounty um but something amazing happened the same protocol also had a weakness of vulnerability which was even more damaging than the original hack and a white hat explicitly white hat found it reached out that they performed a white hat attack returned all of the funds so engaging the community signing up for something like Safe Harbor ahead of time so that people don't have like this moral dilemma like do I save all that money or do I risk going to jail if for some reason I don't have permission to maybe you can short circuit this this thought process and sign up for the Safe Harbor so that if someone does want to help you they they're not afraid for their own safety and their legal repercussions of that there's some pretty funny comments in here I just read one saying SWAT the contractor's address gez like we're going wild here some people said to hire a hitman some people said check if the contractor's dead there's a lot of stuff going on here I would say the the top comment here is secure all access logs like Discord um so that we can use it for future analysis I highly recommend that um there are a couple other things you can do as well and in the essence of time uh because we do want to kind of do a m postmortem of this attack I'm going to kind of move on with a particular case study that I kind of want to talk to you guys about um with regard to laundering to help you guys think through it now generally when funds are stolen nine times out of 10 on an evm chain what they attackers will do is they use some mix of mixers Bridges coin swop services and also sexes in order to launder funds and interestingly enough tapioca Finance which got hacked what was it last month um they the attackers here instead of using a mixing service they primarily actually used bridges for opusc purposes so as many of you guys know tornado cach used to be the main stay for attackers to just mix funds and for any Anonymous develop ER to actually you know anonymize the source of funds or destination of funds or whatever right unfortunately now that it's been oaed though it's basically Persona on grata and a lot of exploiters actually tend to use it so you're effectively mixing dirty money with dirty money and you're going to be getting dirty money out and that's not great if you're an attacker so they're trying to evolve a little bit here so I'm going to go very very quickly through the general flow of a theft so usually adap will have whatever their native currency stolen so in this case it's the fumo token right that fumo token though right let's be real here it's worthless to us because in theory if mode finally reaches Z out to us right as the as the the uh as the founders of the company maybe they'll P the chain right maybe there's something we can do here right so I want to get out of fumo as fast as possible because if I dump that token it's valueless so usually those funds are bridged over to an evm chain from there they're usually mixed through a mixer and then it's one or the other they either go through a coin Swap and Bridge and usually this will happen a couple times over but eventually those funds will go into Bitcoin for whatever reason I don't know I think it's because in Bitcoin there's common spend wallets look a little bit more scary researchers decide to quit tracking through that and so because of that they send through mixers and then they go back bridge over to evm chains and they cash out at otc's and Sexes I see this happen all the damn time it's really annoying because Bitcoin basically Chas like it basically halts a lot of people from tracing funds down the line even though I would highly suggest spending some time in the Bitcoin ecosystem and you'll see how easy it is to sometimes break some of these mixers now um sometimes they'll M rinse and repeat some of this stuff to make it a little bit more complicated but this is the long and short of it now unfortunately as I mentioned before right here at the beginning part tornado cach is no longer super trust like we don't want to use tornado cash because it's full of dirty money there also aren't many privacy protocols even rail gun doesn't have the liquidity enough to off youate a lot of times attackers funds so what exactly are they doing now in tapioca instance what happened was is they stole a whole bunch of money some usdt usdc um on arbitrum and what these guys did was they used a series of different Bridges and just started bridging tether over through all of these different Bridges and chain hopping and interestingly enough some of these bridges are easier to trace through than others so what these guys are doing is I mean you use a mixing service right you use a mixing service to off youate the source and destination of funds you don't keep the money there as like a savings account right to like keep your dirty money forever that is not the point and especially if you're an attacker you don't trust any service right so all these guys are doing is trying to buy time so that you as the protocol cannot Trace through where funds are so you can't stop them so this is how they buy themselves time and eventually of course they get into Bitcoin and these attackers their new way of doing it is through wanchain new way it's actually we've seen this a couple of times and from wanchain they go to Wasabi wallet which is a mixing service and from there they go back to evm and the whole song and dance again the same thing we see with usdc right and this is just to kind of just give you guys a flavor of what you guys might see out in the wild if indeed you ever see a protocol getting wrecked or your own protocol getting wrecked okay so yeah so the name of the game is to stay just far ahead of researchers and investigators that they can move the fund safely to a centralized exchange so they can cash out so whatever aisc techniques they can come up with and it's like especially like moving through Bridges or swapping and all of that the reason why they do this is that standard analytical Analytics tool they they don't have like proper parsers in place to very quickly identify okay the funds are you have to look at the full like transaction call data figure out like where are they sending them funds and then for every single protocol you have to look up like oh it's a the chain X means they're moving money to I don't know polygon or Avalanche or whatever so it's it's massive pain in the ass to like go through the docks figure out where the money is Flowing go to the other blockchain Explorer catch it on the other side and you have to do this for every single protocol so if you have a an idea for a brand new killer analytics product is is is uh buil something that would have modules for every major protocol for every major exchange and so on that traces through those things actually there are a couple of bridges out there right now some of which that I mentioned in tapioca like um Stargate for example uses layer zero so you can go look up the transaction the bridge transaction on layer scan um dln has that as well across does not but you know what would be a really killer app is bridging all of these different scans Bridge scans effectively together to make one big I don't know cros hopping chain scan out there um yeah effectively though you as a protocol need to be on top of what's possible and what's not um attackers obviously like to you know be the first adopters for a reason um because some of these things are very difficult to understand if you only have two minutes to really Trace through to figure out where the hell funds are going especially if you're very hopeful that since these funds are in both usdc and usdt that eventually they will hit ethereum and that's when The Blacklist or function can potentially be invoked well you got to trace through that very rapidly because otherwise they're going to dump those funds and you know your one chance is gone okay yeah continue m yes so centralized entity yeah so uh what one person here said was uh centralized entities act lot like like mixers from The Outsiders perspective if you're working with a company or with Cal 911 you can potentially collaborate with these centralized entities that will help you trace the funds on further one thing I highly recommend too is when you're working with law enforcement they can also more quickly get that information from whatever the centralized entity is so that you can continue tracing them on now there are ways to break these centralized entities especially coin swap Services now for those of you that don't know what I'm talking about when I make a coin swap service it's a service there are a ton of them on best change. Ru that effectively allow you to swap like Bitcoin for eth without providing any kyc details whatsoever so it's very quick fixed float is one of them change now as you probably know change um there a bunch of them um but yeah those are the ones that um yeah that you can also reach out to as well in that in that instance yeah the time is not on your side and then the attackers when well depending how sophisticat is the attacker they they know exactly what they're doing and they're moving funds like every few minutes like boom boom boom one after another so not only do you have to know how to trace through those things you have to be fast to go through through all those different Hobs but again depends on how sophisticated is the attacker like a lot of times if we're dealing with the you know kind of like if you've seen my talk the lone wolves the you know the crypto natives who are not like fullon criminals that you know lering is their second nature they will fumble those things because they don't they don't know we're we're preparing you for a professional money launderer who's been doing this for many many years and just to prepare you for the speed of tracing that you will need to perform and one of the thing that I want to make clear here is they partition out funds into increments of 100,000 so they're not dealing with the full amount that was stolen they usually don't um a lot of times attackers are quite lazy when they launder initially through tornado cach and they just like throw it all through and then they basically just withdraw using the exact same tactics over and over and over again so it's very easy to detect what percentage of protectors would you say are professional like this and what percentage is okay oh sorry I'll ask it again what percentage of attackers would you say are professionals versus fumblers like yeah half and half for what I've seen okay so it's still makes sense 50% chance that you're going to hit a professional it also depends on the root cause yeah I would also say though in terms if we though translate this into volume the professionals are getting in ton of volume but the professionals that that conduct the attack initially the professionals who conduct the attack are not usually the launderers though in that case and the launders will screw up all the damn time like they'll mix funds through tornado and then withdraw from tornado to like an address they already used like I mean the amount of things that I've seen like that happen on both Bitcoin and ethereum is wild but yet they're not very great they're contractors usually cool and you said it depends on the type of the attack so is it going to going to be the like the nation states right if you're dealing with nation states you're dealing with Pros if you're dealing with criminal gangs you are dealing with Pros if you're dealing with like the early example I used was the Shmo there was a lot of fumbling there was like a lot of doxing there was a eoa account that was used and reused many many times you can find all sorts of data if you want to dig more hence why it was so easy to reach out to the attacker and negotiate a return cool thank you yeah I would say nation states it depends though whether or not they fumble the laundering process laundering looks very different in a lot of nation states dprk in particular fumbles laundering because they hire various different contractors out to conduct the laundering and like basically follow a flow right and sometimes the contractors screw up and you can basically see that then the manager's like no you're done here like just send me the money and then you're like yeah I see what happened here cool well uh okay speaking of dealing with attackers and and the next steps a common tactic that we're seeing that happens post the compromise is projects reaching out and talking to attackers saying hey uh you know can we uh can we negotiate and such is the case with your team as as the uh as your all of your friends and colleagues they're increasingly getting panicked and there's no way to get the funds back they start begging you to reach out to the attacker so the question for for the room is how would you do that what would your message read and what are you negotiating for what what are you hoping to get back so feel free to throw answers and some of you guys already asked that question like hey why don't we just negotiate with the attacker so this question let's negotiate I love these responses uh that's that's the swatting the the the Hitman yep we have some aggressive participants here today I do not want to get on your bad side call it aggro Dow instead some people are saying hey we're going to offer a bounty maybe five to 10% of the amount that's five Ian is it five or 10 you kind of want to be precise about those things have you guys ever taken a negotiation course yeah only one of you guys uh highly recommend um if you're a Dev and you're concerned about this ever take a negotiations course it will help you figure out exactly what you are willing to negotiate on cuz it's not just the funds that you're necessarily going to be negotiating on right um You might be negotiating on getting access back to your contracts um they could have other things that you might want the ownership of your company director role yeah you get to be the director again director inside joke which we'll reveal in a second yeah um ask them if they need a nap first yep exactly nap are important before starting negotiations again this is actual case study it's not a joke that actually happened someone was like 5.88% I have no idea where you got that from but you know seems like an interesting anchoring number to use yep um someone wrote threatening will not work for them exactly yeah so that's why you want to negotiate right um offer 5% and expect them to accept for 10 to 20% and call it a lucky day someone has definitely dealt with negotiations before this sometimes and this oftentimes actually happens uh but you don't want to necessarily lowball them either because then they'll get offended um and then they might not even respond with you well it' be great to see like full text if you want to if you want to throw it in the chat like what would you actually send on chain and then the Assumption I'm not seeing this mentioned but the assumption is normally negotiations start on chain where you put a literal asky text in a cold data and you set a transaction to the attacker with a message so bad actors usually expect that to come through sooner or later or sometimes they will reach out to you and say like hey let's uh let's talk let's talk this has not happened in this case not in this case yeah um someone actually mentioned get get a hold of the teer and reach out to law enforcement at the same time this is a really interesting one actually because usually it's we're assuming this year but usually the negotiation is hey you give me Bounty I won't reach out to law enforcement or pursue anything you do not have control of law enforcement I don't care what you say you have no control over law enforcement so don't even bother I would say negotiating on that angle well you should say because it's not eily binding you can oh sorry you should you should say because it's not legally binding you can tell them that you won't press any charges and then press as many charges as you want there's that there so interesting case study is the mango markets with the AI Eisenberg right so there was everyone familiar mango markets you know a very profitable transaction you know following that very profitable transaction there was negotiation where you know there's a little back and forth where a agreed to return the funds in exchange I forgot what the exact terms were I think they were they wanted to cover whatever the loan was that that he manipulated and not pressing charges and not pressing charges which was immediately followed a few months later by the uh manga Market's pressing charges a very detailed and interesting um I guess U case and an indictment by doj which fully understood exactly what happened so to Heidi's Point not only can law enforcement well not only can you say later on that you sign an agreement under duress so on the attacker side like this this kind of like verbal agreement that I'm not going to I'm not going to um reach out to law enforcement means nothing really um yeah I'm not sure that's I don't think it's legally enforceable in any way for sure and then on top of that law enforcement may get involved regardless of this negotiation if there was any like let's say there was a single Us customer that was harmed by this compromise they can swoop in it doesn't matter if the project wants it or the attacker did something else they can just swoop in on the case and take over it it's just a matter of finding someone interested enough try to introduce the fear we'll see how that works in a second yep so we've got Intel that may identify them interestingly enough don't forget we still have contractor or Dev number five that we still haven't heard anything from so we're assuming that Dev number five is our guy and we have a bunch of different information on him we have his ens address we have all the other addresses connected to that ens address as well as other social media media handles so we can definitely pull that uh pull on those strings a little bit more to do some analysis so we've certainly seen in those initial messages where people outline um basically like this is what we know about you you just so happen to interact with the centralized exchange and you know use that as kind of a leverage so the more data that you know this is why the investigation side is so important the more data you can collect on the attacker even if you do did not reveal it at the time of negotiation this is your leverage that you can use to get the money back in one of the case studies that we actually have coming up interestingly enough the um negotiators started the conversation with the exploiter in Chinese specifically to intimidate someone's asking should we even be negotiating Oiler was intentional in St ing information that's a great question can you elaborate on that one more can you you read the BL sorry yeah sorry so if you read the blog post um I think the founder it's it's really a really great story of but I remember like the founder said that he was very intentional in um not wanting to publish anything not like tweeting um so it's kind of like it's to make the attacker feel like hey what's going going on you know like um and it would kind of like cause him to make mistakes I suppose so so that was the angle that they were going for yeah that's a great Point that's one one possibility so would you still reach out though or would you not even bother reaching out I'm not sure to be honest um if you can have a yeah I I I don't know I guess it depends on the information that you collect about the attacker if you can kind of uh get a feel for his personality on what might work um yeah then maybe that would give you a better Direction on what approach to take well Oiler case was really interesting right because for Oiler attacker it wasn't just the the oiler that reached out it was also like the entire security Community kind of like reaching out to the to to the person um basically saying like what are the consequences like advising them to return the funds so there's a lot of back and forth that you can watch on chain to encourage the return of funds there are lots of people talking in here about offering bounties I think Andrew here has said that uh yeah tell them to well we'll give them 10% they transfer money back to us then immediately go for the kill and press charges for those other 10% sounds like I mean it this the the whole negotiation thing is tricky like if we call it for what it is this is this is the web 3 version of ransomware know you get attacked in the traditional security your machine is attacked your data is held at Ransom and then you pay a ransom here your money is is held and there's no reason for the attacker to return those assets they just for for just kind of like this warm fuzzy feeling that the project promises not to press charges that that's seems naive um but again I I think one thing to and we can jump into the case study I think but one thing to just remember when you do the negotiation think of who is the attacker if the attacker is someone that you know and is fully dxed right if it's a let's say if it's a malicious Insider then you have enough on the person to like offer kind of like hey we can soften the impact of what's going to happen by just like let's let's let's walk away from from this so no one gets hurt too much if it's a loan in it's a you know a crypto native not a criminal just just some person who is very clever very smart got overly excited and you negotiated with that person this great likelihood they they don't know how to laund their funds or they're not proficient at it and they understand understand what their capabilities are they're very good at hacking smart contracts but not they're not criminals they may not necessarily be as proficient at laundering and opening up fake kyc identities and exchanges seems like a hassle this uh 5 10 20% 20% seems to be a going right now but uh seems like good enough a pretty good life-changing amount of uh money to come out of this if you're dealing with a nation state good luck that's not going to happen if you're also dealing with a professional criminal group also good luck that's that negotiation is not going to really go anywhere so it's very important you know how we did the root cause analysis now you have to do the kind of like personality analysis like kind of get into the psychology of who are you dealing with and craft you message appropriately yeah are you dealing with a person or group of persons I know in the oiler case it worked to their advantage that there were actually three participants who were the attackers and one attacker was like I don't want to negotiate I want to keep all the funds and then there were the other guys who were like no no no no no I want to be done with this I'm out I'm out and of course that worked really in their favor right if you have a small team now I'm curious actually here we all know stories of funds being returned but for all the stories of funds being returned what do you guys think the percentages of fund of stories of funds were not returned from negotiations just curious I think it's 95 mid mid 90s so it's interesting that we're all talking about negotiation here even though it seems like many of us think that there's an over 50% chance that ain't going to happen so it's something just to keep in mind exactly um and the key like we wanted to share just a few case studies like one a success story and one not a success story so one is a success story so that's the recent socket compromise for $3.3 million that was another infinite approval one too infinite approval yeah as always and then uhan we can show the negotiation that took place so this is this is the one that Heidi mentioned where the Outreach was immediately you know they were they were they were thinking they're using the native language of the attacker based on the information they provided this is this is the translation uh look at the tone and the language used we understand you're responsible for the socket attack we would like to discuss a bounty with you and return most of the funds to affected users contact us to blog scam chat you have 12 hours so couple things to note here very neutral you're not accusing anyone like pure business transaction very calm like acknowledging the fact like this happened some people go slightly On The Other Extreme like they say good job on hey white hat good job on attacking sorry uh you're you're very good attacker like your skills are good or whatever sorry so another thing to point out is the um the blog scan sorry losing voice okay yeah the other thing that we can point out here is the block scan chat so that's one way that you can start having that discussion without the public seeing um and another thing to point out as well is the time limit sometimes um when negotiating it helps to put a Time out there to be like hey attacker you need to respond to me in 12 hours now there are pros and cons to that because as someone mentioned in the chat earlier sometimes they also need a nap and setting that 12-hour time limit it it seems kind of pointless if you're going to go over it anyways and you're still going to negotiate with them it seems like an empty threat if anything Peter are you back okay s getting cold sorry I'm getting some cold here but um so the other thing is they took this off off platform they got it on emails and finally acknowledging that the funds were returned so that's a standard flow so let's look at another example so this is the kyber swap so a few of you mentioned it already that was the the director that was a massive 48 million compromise and the negotiation there was much more forceful so reached out to law enforcement we have Footprints so very threatening the way that it worked out beyond that is it was a complete breakdown so the attacker came back and basically on the next slide demanded everything like hey I want full ownership of your company I want to be a director I want full records and everything else so this is this is the example like a lesson learned from this put your egos in check this is a business transaction it sucks the fact that you even have to negotiate for this but it is what it is so be smart be understand who you're dealing with but attackers they not only want to show that they can steal that money from you they also want to show that they're the smarter person in the room so play to their egos you know validate their skill set validate their uh control over your assets and focus on your users that you need to get the money back that's that's the real win how you get there it doesn't matter yeah I think this case is really interesting because it not just only bro it broke down but I don't know how used kyber swap is anymore after this so clearly you know they didn't relinquish full control over kyber but of course this attacker the kyber swap director has all the funds of kyber effectively right and it destroys legitimacy entirely in the protocol now um let's move on to our second inject let's say we decide to write the attacker saying hey let's negotiate maybe over an email contact us here by the way this is what um the I think the oiler team might have done the reason why you wouldn't want to post that on chain is then you get a ton of different people reaching out to that one particular email that's why we recommend you know having a chat via block scan so that not as many people can actually read your message um and therefore you're not going to get inundated by a bunch of spam so in this case let's say we reached out and we initially hadn't heard anything back uh to our message we just said hey we'd like to negotiate um and that's all we left on shame the attacker actually responds with an encrypted message this is on chain right in the call data and it links to a downloadable library on GitHub for an encrypted chat and you can see here what do you guys think this is are there any risks with this should we respond and who do we think the attacker is here and you guys can check out that GitHub repo I don't know if I would uh download anything from it though do not click the link yes do not click the link I would agree with you be very careful with downloading that software it may be malicious exactly so you might want to do you might want to reach out to some people who might be able to reverse engineer it to understand exactly what's going on there absolutely do not decrypt using the GitHub link provided this may be malicious code which may cause further attack yes do not use that do not Point like all the money is gone you guys are paranoid enough this is great or maybe we scared you enough by now um it looks like um uh oh someone has a great question any thoughts about an attacker who might actually be attending this conference like this and learning these tactics as well there might be a potential future attacker in the room absolutely we've considered this that's all I'll say but yeah that's one thing to always be paranoid about right um You may invite people to your project in fact that's one reason why with our founder mode now um we even said hey you know wait a second we all know one another we all use our real real quote unquote names someone can introduce themselves as whoever right um You don't know if that's necessarily true you want to verify all of that um and even if you do verify all of it be skeptical be skeptical in this space yeah all the money is going click link yellow yep be careful about the software so someone said why don't you click the link in a in a virtual machine right money's gone anyways who cares uh there's a lot of other things that could be happening here um so I think actually if I go to this um case study example I'm going to talk through the case study and we'll go back to the questions because this is one's really nice so of course C we know about the oiler Finance hack um Oiler oh well so it looks like my laptop went off okay so the oiler Finance attackers were I don't know what where they were in their minds but they decided to donate um 100 eth to the Ronin Bridge exploiter address why well because they wanted to make it look like they were like dprk so sorry bro funds are gone like nothing to see here nothing new um these were kids in reality and they thought that they could then get away with it because everyone would be like oh it's dprk whatever um well funny enough 3 days later that address decides to respond and that address posts this lovely message saying hey why don't you decrypt your um your address um uh with using or sorry why don't you decrypt this message using a private key belonging to this address like it's so obvious what's going to happen here by the way this attacker this is where all the funds were sitting so dprk was like sweet bro like we didn't do the attack but we'll take your money um well it's a win-win even if it was a legit message what would they do like come come visit us in North Korea oneway ticket oneway ticket or or we just take all your money so I regardless of the message of being malicious or not like this is a win for for tprk good PR good PR and by the way like when we were we were Heidi and I were watching this hack live and when we saw that those funds flowing to to a dprc to Lazarus account it's like you're not we know exactly what you're doing you're not you're not Laz we know how they operate and you're not that like no matter how much you try to misdirect we saw right through that that's how you know that's not a probe by the way like I immediately was like oh this is kids kids it could be one Lone Wolf kid or can be multiple kids but it's kids so some people are saying to analyze a software to look for Clues on who the attackers might be that's true that's absolutely worth worth your time in this case we kind of know who the attackers are in that case study but yes you might want to be be doing that and that's the key when you do this kind of analysis you start building up profiles on the attacker like what exchangers like to interact with how do they swap what what what are interesting components even we even sometimes dive into the exploit contracts themselves to look at like ha like I wonder like do they prefer to put access controls in such a way or at what point they like to put access controls or how they exactly they interact with all the different like exchangers and so on so every little bit that the attacker does on chain is a signature on them and once enough signatures are built up you will start seeing patterns involved so we that's how we classify and cluster together attackers lunding tactics exploitation tactics like all of that starts kind of creating a profile we got some great troll questions in here well now they know you're communicating send them malware instead um yeah I don't know if that's in the seal Safe Harbor yet but uh talk to them about that that's a tricky one again there's there's a line that we cannot cross so we cannot hack back like that's I guess depending what jurisdiction you're in but from where we're standing like we we we just can't do the same stuff that the the bad guys can do unfortunately or fortunately like that's that's the distinction between us yeah and a lot of you guys are saying well why don't you open up you know um why don't you open this up in some virtual environment or you know air gapped computer certainly you can do that that's that's one possibility there's one question I do want to address here is how do we know that the attacker wouldn't do anything else after the agreement right and we don't and that's why you're negotiating basically on your back foot from the get-go when you even bother to open up these negotiations right like you've already lost everything why should they even talk to you and that should you should kind of be going into it with that expectation and I think one thing that we usually find is that and it's kind of what we kind of came up on here earlier is that nine times out of 10 negotiations do not work out and they actually are a waste of your time because you could have been spending that time trying to track the funds and trying to stop the flow of funds so you can actually get them back because as we know at least in this case study we only reached out to the attacker after they started Laing all over the damn place right the funds are effectively gone and it's going to be really annoying to try to get them back and the attacker is not going to want to spend the time to do that unless we have some leverage on them be patient like we''re seeing that after the initial Outreach to the attacker it sometimes takes a day or days for them to reach back out to you maybe they realized maybe they had like a some kind of like ethical Awakening or who knows um but regardless like be patient that just because they don't reply immediately they may reply in a day or so in the Ence of time I think we should move on um this is the end of all of the index congratulations the protocol of founder mode is it still exists on chain it's unfortunately been wrecked um and there's still a lot of cleanup to do and we didn't have time to really talk through the cleanup yet but we should probably have a postmortem regarding the past six injects that have happened so the actual exploit right exfiltration of funds laundering and where we currently stand sure um so just on the high level what what is it that happened what happened was and a lot of you were digging that directions like hey like where is that other worker why has he disappeared in fact it was a dprk it worker it was a malicious Insider who performed an unauthorized upgrade they already had all the keys it's just at some point something flipped they made the upgrade and started trading all the locked funds after the initial attack they also implemented additional functionality into the contract to also Target the users hence why you you saw there was like a an alert that came shortly after the drainage that hey now all the users are being drained as well all the user wallets yep the attacker then started swapping stolen funds and bridging them everywhere until until then they were launder through a Bitcoin mixer and we didn't even get to that part because we didn't even ask where the funds were we were just like let's stop them let's stop them that is such a great I loved your guys's reaction there because just trying to stop the funds as fast as humanly possible as usually best because once they start splitting them out and partitioning them it's hell on Earth Track so the sooner you stop the flow of funds at the beginning the better it is now the team obviously then attempted to negotiate with the bad actor but instead they received a link from malware so let's have a quick postmortem what went well here what could be improved what's left for us to do and I know we only have seven minutes left here but uh let's go back to slido and go into that postmortem section and start typing out some responses things you think went well what didn't go so well to have a bit of a discussion here but congrats for staying with us for two hours through this harrowing experience yep someone's already written having private key being able to perform this upgrade shouldn't have been possible yes if you have been to any security discussion I think this Devcon one thing people say is multisig multi multisig and try to have like a multisig where it would have been four out of five individuals so in that case even if we did have that one malicious Dev that they wouldn't have been able to deploy the upgrade unless they had three other people that they had pulled the wo over their eyes of right it's hard to do when your your entire team is five people but even with five people you cannot have a single point of failure you can't have a single person that either mistakenly or like don't even think about malicious inside of just think that someone may have gotten compromised on your team a single compromised Dev box should not result in your entire protocol getting trained people say rip fumo that's a big one actually so the entirety of mode chain is basically toxic now right no one's on it anymore there are no more projects building on there basically that entire ecosystem's wrecked based on you know what our project was right it's something to really consider your position right and the risks that your protocol has on the space as well as other protocols have maybe on you and your protocol don't hire Anonymous contractors like even the contractor was actually not Anonymous right we mentioned that all five people in the very beginning all five people have their names it's just that what are your practices to verify what what kind of background checks do you actually do to verify that the person saying who they are really is that person like for smaller teams it's very hard to do like we don't have the same capability well we have to build quickly and you know we meet a person at Defcon at conference and like okay let's let's build something together we get all excited but you know either they were an intentional plant with fake Identity or it could be that someone just goes Rogue because something happened I would say because of that again you want to partition out ownership you never want one person to be the owner of for example everything in your AWS Cloud you do not want that you want there to be multiple people who have access to things right you want to partition out Keys as best as possible that's one way to this kind of stuff sure it slows you down from deploying things upgrading things but at the same time it will prevent any sort of instance like this from happening at least you'll have one check check in place I'm seeing a lot of yeah I'm seeing a lot of stuff about pre-exit like just like the top question like ability to upgrade what I'm not seeing is I guess what I'm not seeing our thoughts about what happens after the hack the incident response process the slothing and Chang tracing there's a lot of stuff like hacks can happen to anyone it's what we do after the hack happens how quickly we react to it knowing like like one thing we could point out I wish we knew exactly what levers we can pull in the protocol so someone said can we pause like that that was a perfect example like knowing what are those levers that we can pull to quickly start moving funds which may be a risk quickly that's that's the matter of are you losing just a few million dollars or you're wrecking your whole protocol so preparation in terms of incident response when the incident does happen preparation in terms of fast reaction how to start a war room with who and what are the participants and who what are the responsibilities in that war room shaving up those minutes means a completely different outcome for how the hack is going to go some people were also saying like we need to actually have done more online sleuthing and I highly agree with this a lot of times when you're in a war room the sleuthing is mainly happening on the smart contract right seeing how exactly the exploit occurred and trying to patch it at the same time though you need to also have people looking into who actually deployed this contract what were they previously doing did they leave any other interesting tidbits behind there's a lot of information you can actually glean out of that you can see if they also might have deployed other cont similar contracts on mayet or other l2s um and we could we also had the internal information too about this Dev not responding to anything but of course we had their quote unquote real name right we had their email we had all of their different handles we could have been doing a lot with that so don't let that ENT information scare you use that to your advantage and just I mean all of you guys know how to look up things on the internet I'm not concerned about that here just use that to your advantage and start digging the stats on malicious I guess the stats on funds recovery for malicious Insider actually good knowing who who actually perpetrated it and leveraging that against them to be able to return funds that's that's a reasonable approach towards things so someone said require North Kore Korean attestation prior to hiring it's funny um uh for those of you that attended the defi security Summit I think a lot of us have talked about that um the problem is is the concern here is that eventually even dprk is going to be like yeah I hate Kim jugun right like they're just going to say it um there are a lot of other ways of getting around this like even if you turn on the cameras even if you you know require real names there are lots of ways dprk will get around this right they can hire body double so to speak um they can do all sorts of stuff like have fake kyc documents prepared instead again partition out key access that's your the main way that you are going to protect yourself don't trust anyone basically even if they're your BFF online cool it looks like our time's up here today we want to do a Q&A but um hope you guys had a lot of fun and chat with us in the hallway and most [Music] importantly take take these lessons home today and start thinking what is it that you will change in your product in your protocol to respond to the things that we covered today that's that's the biggest takeaway that's your homework that's it easy