# Top Hacks since Devcon VI: what did we learn? | Devcon SEA

- Channel: [Devcon](https://streameth.org/devcon)
- Date: 2025-10-07
- Duration: 1:21:03
- Watch: https://streameth.org/watch/yt-MQjw2ffttzw
- YouTube: https://www.youtube.com/watch?v=MQjw2ffttzw

## Description

Discover the most daring blockchain hacks of '22-'24 and how to defend against them. Join Mudit Gupta, CISO of Polygon, and Matthias Egli from ChainSecurity for an analysis of tactics and vulnerabilities, and gain valuable insights to stay ahead of the game. And stay tuned for a prominent anon surprise guest!

Speaker(s): Matthias Egli, Mudit Gupta
Skill level: Intermediate
Track: Security
Keywords: Security, Hacks, Use Cases, war, room

Follow us: https://twitter.com/efdevcon, https://twitter.com/ethereum, https://warpcast.com/devcon
Learn more about devcon: https://www.devcon.org/
Learn more about ethereum: https://ethereum.org/ 

Visit the https://archive.devcon.org/ to gain access to the entire library of Devcon talks with the ease of filtering, playlists, personalized suggestions, decentralized access on Swarm, IPFS and more.

Devcon is the Ethereum conference for developers, researchers, thinkers, and makers. 
Devcon SEA was held in Bangkok, Thailand on Nov 12 - Nov 15, 2024.
Devcon is organized and presented by the Ethereum Foundation. To find out more, please visit https://ethereum.foundation/

## Transcript

[Music] okay welcome everyone to this Workshop about top hacks since last Devcon last Defcon was a long time ago actually so a lot of hacks have happened since then we'll talk about what we learn but first quick introduction about myself um I'm one of the founders at chain security uh we are one of the old Auditors in the space you can say uh responsible for a lot of audits a lot of things in the ethereum ecosystem I will skip most of that but just so you know who you're having in front of you I'm more coordinating stuff seeing a lot of the hacks of course working a lot on the responses we also have our team here who is actually doing the Audits and I'm here together with mudit I'll let you introduce yourself but many of you will know him hey folks U I'm modit I lead security and uh engineering as well now at polygon um and yeah like this is going to be an interesting one like U the aim is to go over all of the top hacks that have happened happened uh since last Devcon uh Matos and insecurity have done all of the hard work so I mean I'm here as a guest speaker sort of I I'll go over a few things a few nice things I want to talk about especially about the Vaz xak and all of the other T things Lazarus is doing which I'm going to go in more detail in my talk tomorrow but it's going to be a good teaser today uh and yeah let's continue so the other participant is of course you guys right this is a workshop so the way I think we can best learn from security incidents is by sharing that knowledge of what happened what did you learn from some of the things you have seen what are your takeaways from some of the hacks we'll go over in addition to the ones um I compiled on these so we'll try something first on this and that's one of the new web three thingies to do documents right so try this see if you can log in see if it shows top hacks it's a document we can all work on together to add um insides too it's kind of empty it should show a heading if this is not working there is a Google Docs backup to do this right um but I'll I'll give you a second to try yeah we're going to have a bunch of good learnings there and I'm going to use them later because the vazer X ha I would reverse it like I would want you guys to explain me how it happened I have no idea how it happened so let's focus on these and then we can together figure out perhaps how the V and radiant hacks happened so who has the document open and sees something like top Haws just quick raise of your hands two three managed four okay yeah some more I think we have contentus yeah yeah yeah you know like uh what is this a five out of 100 multii here so good enough okay now this is an ethereum Conference of course and ethereum has seen a lot of uh hacks because a lot of building happens actually on ethereum right but the biggest hack is not on ethereum the biggest hack we have seen since last Devcon was a compromise of DMM Bitcoins uh wallets this I need to go into it a little bit because uh the the Bitcoin world is a little bit different right it did use a two out of three multis and and this is natively supported on bitcoin right it is not known a lot about how the attackers managed to gain access so at least not to me perhaps some of you have more insights in here [Music] um all I know is at some point this Exchange in something resembling a cold wallet was um splitting up like a lot of transactions into 500 Bitcoin bunches batches you can say right but they all controlled by a single private key or a multi6 so two out of three private keys and um they were sitting there for quite some time right and it was being used for some unspent and um yeah it it was kind of active whenever they had to to re uh fall back to it but then at one point an attacker drained but really not completely they just used um I think nine unspent the nine largest ones and they had a few months to collect those signatures in this case who so yeah this this is roughly what happened right um those uh the exchange was able to secure the remaining amount roughly one hour later to me it looks like a signature attack once again not a compromise of private keys because with a compromise of private Keys we would have seen everything drained right somehow they managed to perhaps with an Insider perhaps to fishing or whatever it's very hard to say that those uh signatures were created and then were in the end used so this was by far the largest tag we have seen yeah and the it it's very similar to actually how vazer X got drained and also radiant in the methodology which we don't have 100% Clarity on we have theories on how such an attack could have happened and we definitely have protections on how to prevent these um regardless of what the method was which we'll discuss but there it's one of those brain teases you guys can while we go over other hacks try to decipher how something like this could have happened so my my initial take on what can we do to make such an attack harder right is like you did see a very large wallet holding significant funds and from a human perspective you know like at the end someone had to sign right but a signature here exposed a lot of funds so one way to reduce the risk is splitting funds but that comes with its whole own set of risks right and we would have a later hack which shows this whole set of risks there um the other thing is what was special about this haug which you are familiar with here right but which I haven't seen in other blockchains too much is they choose the destination address carefully it was an address which had the same five initial characters in the Bitcoin uh address and the last two this points that someone did compare the first five and the last two or those were shown to them as a valid address right um and and I think we'll go over this also even if it is kind of possible to compare a hash because you have a trusted source and you can compare the whole hash laziness prevents us from doing that often because we just start with the first and the last and yeah that's fine right we we kind of know this is secure now it's the right address but now to you guys so who has something to add to this first hack the only one which is not on the ethereum ecosystem yes can we have the volte pass the microphones also feel free to use the document and add stuff there right this is just collaborative comp competitive too it's just us farming our workout so we don't have to do it yes I have not so um what if to avoid this laziness we would double hash things like it is hard to brute force that you start and end with the same hex nibbles and then that the hash of that itself matches the first and last hex nibbles would that help any way or would it over complicate things I can answer this one it doesn't matter like it's same complexity um to create that collision between a hash and a hash of hash and a hash of hash of hash it's still Brute Force so it's same complexity actually I can also repeat just chout and I repeat that's perhaps faster there's no way that you organically hit those those numbers like there's almost no way that you hit a another account that has the same first four and last four it's like one and 82 trillion whatever so why doesn't it just get blocked at that stage good point so the the point here is like Why didn't they have some software in place which automatically detects that there was something which looks similar and just blocks everything which looks similar right well hindsight this is a good idea but but it's not something that's new it's not new but that's the thing like we are maybe 10 years behind from traditional security when it comes to security measures here I mean our the epitome of security here is seeing a random hash on Ledger before for signing it so we are far from where we should be uh but yes 100% like um in an Ideal World a wallet should protect you with these scenarios and should warn you hey this address is slightly different from where you send are you sure you want to send funds here um Banks do it with even names and things like that even with email uh fishing and so on like if someone sends you an email with same name as someone you know it shows you a warning like it's not that that same person and so on so yes these protections exist and our wallets should have it but our wallets are very barebones the best we have gotten so far is this tiny window on a browser uh which you have to scroll down to see details before you sign so lots of work to be there to be done there and it will happen we have made great strides but we got to remain secure while that works finishes okay question there one more and then I go to the next heck yeah so the um question here is why is this the standard that you have centralized exchanges which do hold user funds instead that is a standard that you don't and people hold their funds um so I can answer this one as well like because I'm a very practical person like in reality yes I want every one of you to do self custody and uh keep your keys secure but the reality is there was one bigger hack than this one which was just user fishing like at least around $2 billion were lost since last Devcon to just users that are self ceding being compromised and fish and so on so like and these right now our audience is very technical like anyone who's self custody is a relatively much more technical person than your average Joe so if we want this technology to have mass adoption we either need to make it much simpler to self custody or we need to make custodial Services much safer um so in any case there's lots of work needs to happen there are trade-offs but none is like objectively better than the other choice like imagine telling your grandom how to use metamask like it's just not going to happen I would prefer she uses coinbase so okay let's go over to the next thing and and if you have more to add right don't hesitate to put it in this document so the next one was already teased I'll let you cover it yeah exactly it's um it's I guess the second biggest one this was I think around $230 million or so um and the way this was happened was very similar to the last one um it all started by so vazir X had a system of a cold wallet and a hot wallet so the hot wallet is was used to uh honor all of the user withdrawals of funds and so on and the cold wallet topped up the hot wallet whenever it needed more funds there were more withdrawals so the hackers first thing they did was trigger this movement or Force vx to move funds from cold wallet to hot wallet by continuously in a loop of depositing Gala token and withdrawing Gala token so when you deposit to an exchange you get a unique address and you deposit there it doesn't go to the hot wallet directly so what this meant was the hot wallet ran out of funds um vazer still had all of the tokens but in different wallets it didn't have tokens to honor withdrawals from the hot wallet so so they decided to move funds from their cold wallet to hot wallet which is a routine activity for them they do it every week nothing suspicious although it should have been suspicious like why there are so many Gala deposits and withdrawals suddenly happening but beyond that point they started uh doing this deposit of Gala uh in from cold wallet to hot wallet uh initially the transactions kept failing for them so they involved the they were using linal as their self custody provider which has a product suit in itself is hilarious like it's basically a wrapper of unsafe on nosis safe but they were using it so uh they got them involved it started debugging and everything but then it just started working after like five failed attempts it was working um so that's fine what they didn't realize is those five failed attempts could have been just them signing malicious transactions and obviously UI rejecting them so they happily yeah press the button on Ledger press the button on Ledger press the button on Ledger all malicious transactions signed all good um but yeah at the end I won't go into details of how the signatures were collected but yes they had failed transactions which they signed and just ignored because one like the ux in our space is so broken that failures are normal like five times fail six time success perfect it's a nothing to worry about um but that's yeah we'll come to it how that's bad but yeah they just signed these bad transactions without realizing they're signing bad transactions the UI was showing them the good details the correct transaction but they were signing something bad um how that happened the UI showing them something good and them something signing something bad is an unknown with lots of theories I have my own Theory uh but what's your theory uh my theory is that it's something I would share later I don't want to poison others thoughts because I want people to post their theories uh on that thread or Channel or whatever um so there are lots of possibilities I'll give you some possibilities one possibility is that lonl was compromised their uh like uh service provider so the UI was showing something else but sending bad data for signing uh it is a plausible the another theory is uh the wallet they were signing was compromised so they instead of installing metamask they installed meta Fox I don't know and the wallet was uh changing the transaction another theory is those laptops they were using to sign were compromised themselves could have been like um they were at some point like in the same infrastructure or something so someone spread a virus on it or whatever and the laptops themselves uh changed the transactions up after the transaction so once you compromise the laptop you can modify the wallet you can modify the uh interface between Ledger and so on so you can do anything so one theory is that the laptops were compromised uh but those were three different laptops MacBooks in three different regions so yes another theory is the ledgers themselves were compromised and since it's a single company single Hardware device maybe they used the same vendor to buy these ledgers and they got compromised fake ledgers somehow another theory is um the wires they bought were compromised so this right signal left the computer wrong signal left reach the device and it just happened over longer period so people didn't realize so there bunch of theories you can theorize more uh but try to theorize what is most plausible here um but we'll ignore that come to like how do you make yourself safer so it doesn't matter what that the theory is is there there are some basic things you can do to keep yourself safe uh I have these is this is the basic stuff right this is a teaser for your talk yes it's a it's the teaser so I'll go through these very briefly number one is use Hardware wallets like don't trust software wallets or browser wallets they are much much much easier to attack than a hardware wallet the second one I have to repeat this so many times but it it's obvious like don't share your pneumonic key not to metam mass support not to coin B support not to your grandma nobody just just just never share it never type it out on a computer never copy and paste it anywhere just write it down um you can write it down on different pieces of paper split it down whatever and just keep it safe um then on just keep it safe that's all you need to do it's a teaser it's a we we're going little fast it's it's a very loaded word and I can do a whole 1 hour session on how to keep pneumonic safe but I'm assuming most of the people who are here have heard that a lot already but yeah fair point then another thing is like security is always about layers even when you have a hardware wallet if you're connecting to a Dap I would recommend you to connect it through RAB or metamask or some wallet your Ledger instead of connecting leder directly through the leder kit to the DAP uh because whatever wallet you connect to Rabbi is good for this it would decode the transaction for you and show you what actually you're signing so even if the website is compromised you can catch it at the browser level if you directly connect your ledger to the website if the website is malicious what you're seeing on Ledger is just gibberish you're you're not going to know what you're signing and you're going to uh get ricked so just keep adding layers um verify what you're signing on the browser wallet you can potentially use dedicated airgap devices these don't need to be super expensive can be like $300 $400 Chromebooks which you don't do anything on except signing transaction ctions on noses safe so you can add a firewall to it which only allows connection to the noses website and nothing else so you only open it sign it you close it there is no other thing going in and out of that don't do your regular work on it uh another basic one is ensure secure threshold for your multi6 like the biggest hacks we have had were like 3x 11 multisig 5x9 multisig uh 2x3 multisig these are um and this one was I think 3x 11 yeah yeah 3X 11 radiant and then they say they were following the best industry practices and they don't know how to how they got hacked so it's like if you following the best practices do actually follow the best practices 3x 11 is not the best practice um I would argue verify hashes on your Hardware wallets this is a big one now Ledger is working on making those more legible which is super nice but in the meantime we have some Community tools you can use uh like uh PC has made the super nice safe TX um hashes util which you can use to basically it will fetch all of the data from nosis API hash it so you actually get a hash which you can verify on Ledger like you then just compare two things now you can be lazy and ignore that but at least you have something to compare uh before this if you were just using the Ledger on nosis site nosis will show you all of the d uh like dhash things the actual data of what you're are signing and lger will show you an just an hash so you can never verify like what you are signing is actually what you are seeing uh using these tools and I think noes are adding this officially now you can actually verify that you're signing what you are seeing uh finally uh use a diverse set for your signers don't make everyone use a MacBook with Ledger with uh Chrome browser and everything use a diverse it and especially throw in mobile devices there which because they have complete completely different risk profile uh like hacking a Windows device is much different than hacking an Android device so just use a different uh set of things and do not ignore failures I know web3 ux sucks and failure happens but figure out why that failure happen especially when there are critical funds at risk do not just ignore a failure and move on I so my proposal on this is we should aim for this three star security thing we have some experience with that when really large funds are being handled by centralized entities and it really makes sense to try to force an attacker to break three systems before they get you right now we had cases where you can say like if they got into the UI of safe they got you if they got into the Lial UI they got you right if they control if they get https um certificates for those websites you will access them they get your red man in the middle attacks were possible of course these are State actors right we we do face here a different Threat Vector than we usually face this is not about you protecting your own uh Ledger or so right this is about protecting funds which um groups things are worthwhile to attack um there is a lot of things Unknown about vazir and what happened here are there things you want to add from what we said ideas we already talked about it theories what how how do you think they have been compromised um and then if you'd be the ceso of that project what would you do what would you learn earning spe out of that what would you try to do differently I think we'll look at the document later what ideas people have can we have a microphone there uh didn't they have some logging system also logging yes but you can't TR Trust trust logging and it doesn't help because it's in this scenario because it has already happened okay you logged and realized oh I did a mistake that mistake already happened so you need Beyond logging you also need active monitoring and alerting and actually taking action on those um they had logging but it didn't matter again like errors happen they said H errors are normal it's fine let's move on yeah there were some really after the fact things where you can say like it was in the locks and it was very suspicious right right the problem is the noise of locks I've been working in this myself and they are very very noisy and it is very hard to write before you know exactly what the attack is what to listen for in these noisy locks yeah I think um around about the same time there was a Microsoft threat research article on uh dprk using um Chrome zero days uh um and it could have been a zero day and um I think you've already covered that m highly likely though if you look at some of the compromises beforehand is that it it was probably spear fishing via browser extension sideloaded brow browser extension that that got them man in the browser I think it was important for wiir to not have been reading their email on the same machines where they signed transactions so I think um separate Chromebook or separate iOS uh would have been a much better um operation for them and also there was probably time for them to add delay as soon as soon as they noticed the failures of the two gala transactions they should have probably realized that they were uh being targeted potentially raise a security incident started to go through the transaction hashes and data hashes um and I think lastly they had a really high concentration of signers so if you go and look at every signed transaction for wiir X there was around about 1,700 it it was basically uh 3 to four signers so the the adversary knew exactly who to go after um whereas they should have much more entropy in their sonic Quorum yeah in reality I mean humans are the weakest link in security almost always um vazer X had proceses in place or policies in place but in practice those processes were never followed for example the first question I also asked them like why was your devices not air gabed and they're like it's our policy to have air gap devices but we installed Telegram and that on it to communicate and so they had all of the policies none of them were followed um and that actually sadly happens in 90% of the ORS um I've seen yeah on this a difference I've seen in the more traditional industry is really you have paid employees to do this thing and they really care versus they are deafs also paid employees right but they are kind of doing this because because I have to and you do it quickly so we we are all lazy right so yeah I'll go to the next hack uh we have one more question um in my view they kind of never fought deeply into their actual like system they took a ledger connected to like a four out of six uh s multisig but they gave Lial only one signature so they could they have easily just bypassed linal if they wanted any protection that may have been there could have just just didn't have need to do it yeah for sure they could have but the lonl was sort of like their signal mechanism like if a lonel is signing they would say okay it's good if they're not signing it's bad but the thing is in this scenario linal signed so one of the signatures were lials and linal firewall or the security solution was like yes perfect solution perfect transaction let's go it so that that's why I was saying like it's I have no idea why they chose lonl as their custodial semi custodial solution when they were using no safe because it ended up like weakening their security instead of strong strengthening as soon as lemel system saw okay three people have signed it signed the transaction so it became from 4 to six multis to a 3 to six three of six multis um I prepared 20 hacks no okay I'll skip over some anyway um for like okay we take one more but write your stuff otherwise in the document so was the right threshold for a safe like let's say for example if the funds are huge like couple of hundred million or like a billion dollars what's the right threshold to use and how many multi6 and how to choose that yeah for sure it always Bas it should be based on your risk um because again like if I can if I make the most strict policies in the world nobody would follow them uh security is always about risk management and it's something you learn with experience like you can create the best policies but they're useless if nobody follows them how how is it at polygon um at polygon uh we have different levels of multix so we have a threshold for under a million dollars uh of exposure we have a threshold for $1 to5 million of uh exposure of 10 to 20 and above 20 above 20 is our like we have to keep the secure kind of scenario and we almost never use the above 20 multi6 because all of the operations should happen from the multi6 with lower uh exposure U ultimately the 20 plus how so um ultimately like the ultimate multisig we have are like for example Security Council uh the protocol Council um they are uh 9 by 12 I think right now now on 9x1 something along uh I think it's 9x2 ji might know exactly but it's 9x1 or 9x2 uh for our protocol Council uh and that's our like if you want the most security this is that that's where we settled not different phones the way we enforce it is we don't enforce any common Hardware so naturally you would have a diversified set of signers but do collect like what sort of signing devices people are using so if I see okay like it's very Mac heavy or it's very um this heavy then the next signer I add on or replace would be on a different set but I've honestly not had to do it so far because just naturally it has been very Diversified do you know if they sent the laptop of the developer to a fory company to try to investigate where the issue could have come from Vaz X had Mand and uh lonel hir I don't remember someone to investigate so they are working with professionals but it it's very hard post the hack because if the hackers are state level actors like uh dprk in this case they know how to erase their traces like this is not web 3 this is web two stuff that they've been working on for 20 years they have by best guesses around 25,000 people in the lazer Lazarus or I don't know what's the right word to call it but yeah the threat actor yeah yeah just just because we can't find it after the fact makes it really not a strong signal that there wasn't one um so yeah even though I wrote this down here before like where where was it no like no fishing was involved no sign was found this does not mean that it wasn't like this right okay let's continue um a different case but still a key compromise Gala games the third biggest tck this was a very likely I'm going out out of the way here because it's not definitive but I'd say very likely an Insider um Gala was able to recover all of these tokens and um um somehow an old Minter address uh was able to Mint a lot of Gala dump them and and return them the next day uh there were some internal conflicts at the project uh key people left um and and then this happened right so coincidence or not not hard to say um what did we learn here from my perspective we had dormant access to uh very powerful functionality in a project so um likely yeah I mean this this happens all the time right you forget to withdraw access to people when they leave um you had here a a tricky case of trusted employees which then turned into people more adversarial to the company I think in general you'd see a neglect of people risk um in in this case um of course you'd also saw on the technical level that um eoas had very high power in the system so the there was no multiactor authentic a or um authorization for privileged actions or highrisk actions um yeah um there there were no no checks and balances in many on many of these um places where they could have happened yes so this another key compromise but of a different nature a common thing on Insider threats is recovery of funds right people put on pressure on these people very effectively and usually the money gets returned we will see other cases any inputs from you guys on this one I'm also interested in how some of you manage the people risk in your organizations or in organizations you know where you'd say look there was this worldwide west of web 3 what we kind of moved on but now we do things differently what works what doesn't work I know we take this very serious chain security ourselves uh polygon takes a the I don't know how you do it would be very interested in this yeah how how to manage this people risk access um for a limited amount of time for whatever the access is required but kind of don't give permanent access to security sensitive things yeah makes sense and those are like the basic and obvious one everyone should be doing unfortunately people don't do it but they should I I I'll make this conversation hot and more more interesting by throwing a curveball in our space we respect privacy a lot and we want Anon contributors so how do you manage Anon contributors and people risk that's the biggest question how do you tell this Anon is not uh dprk anyone has any thoughts on that I I can share the policy we have at polygon because we are Anon friendly but still safe I want to know if anyone else has cracked the code or is working want sorry I mean dprk can own all the anons I like they have more people than all of the blockchain developers combined so that strategy could work for first three months but then they will pick up and yeah we had one more in the back uh yeah how about just changing addresses so authorization to the smart contracts where rotate to another authorization address or you know in exchanges just move the money to another address after key people leave yeah so have having good offboarding policies right um Eric already mentioned this this is very important right when someone leaves can you really clean it and you need to know you need to keep track of every permission you give during the lifetime this is hard honestly and then you need to have efficient processes in place to quickly withdraw them too but but it's very worthwhile you need to do it anyway right um yeah for sure and the reality is like in practice from what I've seen most of The Insider threats have originated from people who are currently employed by the org not the X people do do you do background checks on employees yes so I I can give some info about like what we do as polygon in the Anon friendly way so it's not 100% Anonymous but we have two basic levels of security there one for everyone basically we use a third- party background uh company to do background checks so they would have all the actual details and everything about the person which they would delete in whatever their retention policy is I don't remember the exact number of days but uh they would get all the details they would verify they will do police background checks and uh everything but they would not share any of that with us so we will not know we will just get to know okay this person cleared it or this person has this this this issue um so that is one thing the other is for hiring Anon people we do have a few Anon people uh in polygon um we require at least one uh proof of recommendation sort of so you must be recommended by at least one person in our org who trusts you and knows you um for a long time so it basically like it's U we we trust our org to know what's best or who which Anon is good or not you should not be recommending anyone who you are not comfortable with it it's sort of like we are trusting our employees to be good it's the philosophy of private trackers if I don't know if how many of you do Torance and all but it's basically the invite tree um if you know someone good you invite them if the person who did you invite it does bad you also get kicked out so you you only share these invites very carefully and only when you really really really trust this person and this process has worked wonderfully for us we have not had a single case where an internal employee has recommended an bad Anon like it has always been very highly trusted good anons yes not to the word okay so this is for inside us there is so much more of course which can be done here but I'll keep it more technical because I'm just more familiar with the technical stuff please also add stuff to your document do we have okay one one really last one on this there was one more on top of what you just said uh if you want to uh to include more Unknown People is the use of uh session keys or Access Control roles where if you want them to do something very specific give them access to some address some function selector and the some other conditions to the call data and the payload and then they can execute the stuff it can be U time uh time gated as well or not or can be removed with refresh and um something that's been actively worked actually a lot but for this one time access it works right but but this is mostly about the people you trust for years or so you can still Implement that for those people as well yeah makes sense this also just comes down to layer of security I would do this not just for anons why I do it for everyone like it's just a additional layer of security which we should do for everyone uh but then you have to again balance like practicality versus uh policy situation but yes it's it's a good layer of security to add yes okay um I go on this is a a case I don't want to talk about too much it's um an exchange was hacked because someone got access to their Google cloud and they had all the private keys in there uh this is of course bad and we don't need to say how bad this is on on how many levels um this happens if you hear you need to segregate wallets you need to keep funds in different wallets to be secure right but you don't really know what you're doing because yeah I mean uh okay I'll quickly go through it just so you know many are doing a lot of things right right not everyone though so in this case uh like 150 million were drained over a long time by an actor uh who got access to those private Keys just one by one draining all these wallets right then the project didn't even noticed for really a long time um and um I guess they got noticed because someone told them look I can't withdraw somehow it's not working uh on next Monday right this was a traditional Saturday morning Friday attack something like that um yeah uh what did we learn um there's obviously no use in uh segregating if you lose access to all of these keys if they are stored in a central place that's kind of equivalent to not segregating wallets um relying on cloud infrastructure so first off in this case it was Google Cloud generally this is more secure than most of the things we can build in reasonable amounts of time right this this has to be said right but if you are an exchange that's your business right you can be a lot more secure and there are dedicated systems to allow you to do that Hardware wallets and I'm not talking about Ledger here right proper Hardware hsms which manage this stuff like that right there is really this is the bread and butter of normal Banks today too obviously they need to sign a lot of things digitally too right it's not only us um so yeah there's a lot you can do this is really not how you should approach it yeah I add quick like sighting tangential attack Vector here because I it just reminded me which I've seen increase now a lot is people trusting AI tools blindly when all of the AI tools are just farming all of your data for training so there are two attack vectors here one if you use the public version of it for example even if you buy the chat GPT Plus for $20 whatever you enter in there is going to be used to train their model so for whatever reason if you pass all of your if you enter any secret obviously it's gone but what people end up doing is um I I've seen like they would upload your whole they will upload their whole code base and then ask for suggestions or something so it chat GB is good at explaining code or uh doing things like that so but the problem is if your codebase had Secrets any environment files or whatever now chat gbt knows this and it's actually going add that to its model and in future maybe someone would type give me example of get a private key and boom your private key is shown to them so this is one attack Vector the second one is even if you buy an Enterprise version of the software where they claim private data and no training on your on your data and whatnot the thing is they they still log all of your data and their support and Engineers are able to look at this data for uh training for their own like Improvement purposes and so on so if um there if you read their TNC you would see there like um so if you enter any secret there any of the open AI employee support engineer can still see it and drain you I've seen this happen a lot with things like Google photos like people would take a photo of their pneumonic and upload it on Google photos usually it's very safe but in edge cases they still have access to it um so not great uh and GitHub co-pilot like how many of us added to our repo where we have in like our private secrets and environment variables and everything just look at their privacy policy one because I'm actually not sure about the privacy policy of getup co-pilot but it's definitely a risk Vector people should explore uh so can you recommend something uh so let's suppose in our system like we are a bridge and we have a private key that signs our transaction I mean like 20 transaction per second so where should how should we secure that single private key like where should we store what best practices should we have I mean like we can't have a air gap device who is just sign transaction this is a hot key just like someone which needs sign like automatically a lot right so which technology should you use to to enable that to secure that private key like the securest way or like the best practices I would say yeah for any hot key there are two things you need to secure one is the actual key itself you can throw it in any HSM or something and it's good enough for a hotkey purposes you shouldn't it shouldn't have billions of dollars on it but the bigger attack vector and the how most of the attacks happen to hotkeys is the authentication mechanism if your HSM just has a public API and uh it it has SL post data and it Returns the signature it's worthless so it's a like that is the actual mechanism you need to secure that only people uh only the right people should be able to request a signature from there only the right things should be able to request the signature from there uh you need to secure it through firewalls uh through filters through everything but that's the harder component you need to focus on yeah thanks you want to add something uh but what if like the dedicated HSM if you have like a gcp admin and you control the HSM if you compromise JCP admin like what can you do with dedicated hsms yeah that's the thing like that's the access part once the key is in HSM it's secure like but if you can access the HSM somehow and being an admin is just one of those things it is dangerous in GCB actually you can have no admins uh in AWS as well nobody has that permission um and only select products have select permissions you could have a system where nobody has uh read permission to the HSM only signing requests and so on but again then you need to ensure people are not signing the wrong things and so on so it's permissioning thing but obviously you need to ensure the people that are in there are trusted and uh so you're proposing having like a segregated like also HSM infrastructure that oh yeah like in your that's again traditional web to stuff like um in your infrastructure it's not just HSM like everything should be segregated and it should be the least uh privilege of uh like um I'm forgetting what the principle is called but everyone should only have access to things they need nothing extra uh it shouldn't be like okay I'm the CEO I I have admin access no like it it's actually the opposite in polygon like I lead security and I have less access than all almost everyone under me the simple reason is like I am the bigger Target um and I don't do the day-to-day activity that I need this access every day like maybe I'll need it once a week I'll create a ticket I'll get access for 15 minutes I'll do my thing I'll uh create a ticket remove my access so it's just yeah like access management is one of the biggest things to like do it's the hardest thing to do right uh and it's something nobody does in our uh in our space but needs to be done what but this is a good question like is there someone here say like I want a shill and HSM right basically a solution I am actually using and feeling very comfortable with you can actually use uh tees for this you will answer but just ideally something you can actually use yeah oh look I think you could use KMS or cloud hsms and and you can restrict policies do just in time do you know a good one yeah KMS is fine for holding for for holding keys I think the admin per is the problem right is so you just in time um um you know just in time permissions using something like lumos or other approval systems but to tell you the truth um maybe just use fire blocks I don't work for fire blocks um but that's a a much better solution um where it notifies multiple signers you can build quorums so that's probably a better solution here okay um okay one one more in the back and yeah we'll come to Smart contract Tex too uh so in terms of HSM capabilities you have the cloud Solutions which is obviously aws's Cloud HSM solution I think is slightly better than gcps at the moment sorry uh but I'm a I'm a dual I'm a hybrid Cloud user so I don't judge either way I just find the best solution uh AWS is cloud HSM Solutions a bit more uh segmented and can allow for sort of a bit more user granularity whereas gcp and that's within the HSM construct itself not just I am principles uh gcps HSM is built into their Cloud cams system and therefore you're a little bit more dependent on just sort of the standard identity access management within gcp and I think that's your biggest risk factor on both of these HSM Solutions um as the previous uh person mentioned like it's really less the HSM is fantastic but it's going to do what you tell it to do so unless you set up granularity within the access control mechanisms you're still going to have a similar issue and that's why typically a lot of people do move to fire blocks because they've created that sort of multiactor capability uh and authorization patterns um again AWS you can do it a little bit easier because you can dial into the HSM console and and programmatically set up some of these kind of multiactor capabilities um otherwise on Prem um Talis is a really good HSM tion if you have unlimited money um if you've minted unlimited money then you can go for IBM Z series if you really want to be fancy um but really uh the best thing to do is really making sure that your your access controls um prevent your HSM from just blindly signing anything that comes through to it uh especially if you're leveraging web application services that are specifically signing on your behalf yeah I I think this is a good thing to repeat you need to think a step further from my perspective if you think the HSM is just holding your key securely that's not enough it is really you need to think of your hsf the system which validates if the if that transaction should happen right and then then yeah some good suggestions were mentioned okay let's go on this is one message I just want to highlight right we we will and are smart contract Auditors right and so on and we we really uh try to secure the space from this perspective but key compromises are the new big thing there was a very good talk by Peter at DSS you can see the recording highlighting this fishing a type of key compromise right and this these are the big hacks of today so the big smart contract TXS the biggest one it's long time ago but Devcon was longer this is why it's in here we talked about this one a lot and you might be familiar with it right um there were great great videos made on this and I was planning to show the video here the short part which explains the hack it takes like three four minutes um do you want that okay first some any some uh advertising okay let me see okay um I don't know the sound um can you help me with the sound you cannot okay um subtitles yes let's see I would try something crazy Buton flash flash a special type of loan anyone can take as long as they the money the same transaction now that we have $30 million let's deposit it into Oiler then let's borrow 390 million start start from the hack drained six different tokens but they all work in the same way let's look at the first it starts by borrowing 30 million die through a flash loan from a a flash loan is a special type of loan anyone can take as long as they repay the money in the same transaction now that we have $30 million let's deposit it into Oiler then let's borrow 390 million and deposit it right back usually Dy has a collateral and borrow factor that leads to a maximum 75% loan to value however Oiler has a special mechanism for self- collateralized loans when your assets andb are the same token it bumps up your max loan value to 95% dividing our borrows by our collateral we get a loan to value of 93% and a health factor of 1.02 so everything is still perfectly legal on Oiler we just have a highly self- leveraged die position now here is where the vulnerability comes into play boiler has a donate function that allows users to donate their collateral to the protocol itself it was implemented because sometimes if you have a very small balance it can be cheaper to donate the dust than withdraw it yourself however this function lacks a critical Security check it doesn't require the user's Health factor to be above one afterward and this is the bug it allows a user to donate into insolvency back to our leverage position we now abuse this by donating 100 million now our health factor is 0.77 this lets us take another account and liquidate the Violator earning the maximum 20% penalty our deposits times 20% gives us a premium of $64 million finally we repay the 30 million flash loan to net a profit of 34 million however Oiler doesn't have that much money in its Reserve so we walk away with $8.9 million the hacker repeated this five more times to drain a total of $200 million news of Oilers ha spread like wildfire with 200 million gone okay okay so I think there there are more uh in-depth explanations more walkthroughs there's a lot to learn from this one but there is again there is a part which is Technical and then very interesting and there is a part which is very human right and Michael did a amazing write up here the War and Peace uh right up at the end of this year I heavily recommend it for everyone to read it is a small book in itself takes some time so I will not cover this here what's important so this hack started with a small change to the code which the small change was also audited the small change was actually to to protect against another vulnerability which was missed in the previous audits right but which was in itself much less impactful than the later hack would be and and then it was a small change right so you ordered a small change and if you ordered a small change it is hard to say this will take me three weeks while it's 10 lines of code right the problem obviously is that small changes are very hard to uh audit um and then we yeah we see this again and again um also you don't want to audit re audit a protocol which has been heavily audited too as an auditor you don't like that right because you really like to find bucks um So the lucky thing of this story in the end the funds were returned right um they they lost 200 millions they got 240 Millions this is because there's a time delay and uh uh also an insurance payout um so what did we learn here I already said it it's really hard to audit diffs but also security which which oer took very serious and took to the Extremes in the version We a lot of you might have seen when you took part in the crowd competition or as Auditors and so on um really pays off on these levels right because of the levels of security of the things they had in place they were able in the end to recover those funds and I think it is a good case of of what kind of methods you can put up to recover funds from The Human Side Right their communication was really good and this is a super super stressful scenario if you read the War and Peace you will see like there were situations where it was like blood tests and passing out on the oiler team side of things um but if you have some guidance some plan and and it will not survive like famously said right it will not survive the incident but it will help you so much to to make the playing field a little bit different to if you are well prepared then you will still get hit by the unknown but just because you're well prepared you can really go ahead and and kind of win in the end because the attacker might not be so well prepared and in this case this was a case right it was a single person who later went to prison uh who was found and um yeah so so in the end this this was one of the few success stories um smart few smart contract TXS are like that almost no key compromises are like that I'd say key compromises our main adversary today is Lazarus group yeah that's true because um I'll jump in here as well because like reality is it is very hard to stay Anonymous when uh first it's hard just when doing the hack but it's even harder when you are laundering the fund so you have done the hack it's fine then what do you do with it like you would naturally start laundering and that's when people get caught so most of the hackers actually get caught at one stage or another uh dprk also get caught but they don't care so that's why they are a bigger threat to us like you can't go arrest uh them in North Korea um with this case in particular couple of more things I want to highlight is upgradeability that's a double-edged swad like I've worked a lot on Smart contract upgradability and made The Primitives to make it happen in 201819 but and that is also why I know why it's a double HED SWAT it it is I'm seeing right now people upgrading contracts like there's no tomorrow they they add a new uh feature they upgraded like any small feature and in this case yes they for fixing a bug is but they might have just upgraded to add a donate function you can donate to protocol now $10 we'll have $10 more Revenue let's upgrade the contracts so the you have to be very careful about upgrading contracts that you don't do it just for small features either like it should be security fixes or very large features that you can't afford to launch in a new contract so upgradability needs to be taken seriously and this has happened with more bigger protocols not just Oiler another big one comes to mind is compound Finance I don't remember if it happened before last Defcon or after but that was a more hilarious case because they upgraded something people noticed the bug and they ended up minting like $250 Million worth of comp or something extra in rewards than they should have um and as soon as the upgrade happened and they started minting more people noticed oh something is wrong at that point it was maybe like 5 million more had been minted but guess what like the to upgrade it again they had a time lock of 7 days so the fix took seven more days and by that time they lost 250 260 million actually on this we we have expert in the room on the the compound like I'd be very interested in the learnings compound took after that uh if if those are fine to share there there was a good talk at DSS on this by Michael um there's so many Michaels in this space but yeah um if someone wants to add something to those learnings that would be great um Michael you need to look up from your phone yes from the compound case were were there learnings okay never mind yeah but but watch the talk it was good yeah for sure the basic learnings are be careful with upgradability able H SW okay so we go to the next we don't have that many minutes left to be honest um I mean want to skip it's another key compromise with Insider involvement potentially who knows little info little to be talked about here another one little info key compromise hot wallet drained but here we are back to a Smart contract actually one which uh is a polygon using polygon um in this case it's an oracle manipulation and those used to be the big hacks of last year not of this year anymore because we really learned um in this case how did it work so you uh you might know in general Oracle attacks they always work by having a landing platform and manipulating the prices this Landing platform depends on to um first deposit and then either get it back cheap or like to to drain the um the lending platform by pretending your assets are really really worth uh worth a lot of money right so this is also what happened here the interesting part is always how was the Oracle manipulated uh in this case surprisingly simple teller is an oracle where uh it's it's less permissioned than uh like chain link for example and the way you can uh the way they secure it is you deposit um trb then you can post prices and people should um go in and say no this price is not correct and they slash your deposit right so this naturally takes time the buck a buck we see in many Oracle attacks is you want to have the latest time you don't want to wait right so in this case also Bon DA they used that they used the latest price which the attacker just by staking 10 trb could basically arbitrary set and at this point the attack is easy right at this point it simply means uh lower the price deposit a little for a lot of the other the other value in this case b Euro and um and then increase the price and withdraw and and off you go right so this is what happened here um what did we learn here and this is really an important part if you build on stuff which is critical to your protocol you need to understand it really well and this is especially hard if you change a little right you go out you have an idea how to improve something you change it a little but you don't understand even if you read the code and try really careful to do that but even if you develop it from scratch and you depend on libraries on Services outside of what you developed you depend on them and honestly as as Auditors ourselves right it's so hard because we as an ecosystem some some are very good at documentation some aren't and in the end you need to read code of that large system you are integrating with to really understand what it's doing because the documentation doesn't give that away um I was just yesterday talking to one of the main devs um of uh the the open Zeppelin libraries or the main Dev um and an idea hopefully it happens is do an underhanded contest for those libraries we all depend on right we have an underhanded solidity contest it was so so this year was hard to find something apparently but it's probably easier to find underhanded stuff in libraries or uh these kind of systems so it's really important I'd say one of the main learnings you need to understand your stack to a degree which feels gets paranoid at times um yeah of course in this case also there were some uh neglects on reiting the uh changes later and so on so so this is also recurring theme um yeah other other learnings like what would make this easier use multile oracles and we'll have a hack where that really fails um we we can put hard caps in sometimes this works sometimes not try to do holistic audits this really helps U but but it's really expensive so I mean I get it when you when it's hard but of course in this system there was really a lot at stake um yeah and then on oracles yeah it's it's very hard you you really like I I applaud those it's a centralized often a centralized piece of infrastructure and and trying not to have that is very important but you need to understand the security caveats of those and also they should really try very hard to push the knowledge about their security caveats down to users and then this is going Beyond like I can say like just use chain link right but that's not also that means yeah you trust a multis signal right so um you need to know what you're into here yes there is a security flow in the ear C20 standard that was classified as a lack of transaction handling in 2017 in 2023 the creator of ec20 standard confirm that this is a security issue in 2023 there were 60 millions of dollars loss because of this problem on 1st November 20124 there were 90 Millions lost two or 3 days ago another Us lost 25 millions of dollars to this issue which remained unpatched since 2017 what do you think of a situation where a sec security issues that was discovered 7 years ago cost 55 millions of financial damage to ethereum users during this year and did chain security report the security issue in any of your C20 contract that you audited so far the last part you need to repeat was did you report this issue if you any is this about approvals so what what's the issue with here I mean the there are known caveats with r c20s you mean that we don't verify zero address transfers here which which attack exactly uh this is not the attack this is the violation of security principle of the fail sa defaults when a user is trying to deposit tokens uh to addresses there is no way to verify if this is a correct invocation or not and when a user is trying to deposit funds to a Smart contract this is 100% possible to verify this is a correct transaction or it must not happen for example if you are trying to deposit nft to a Smart contract which is not designed to receive nfts it will not be deposited if you are trying to deposit e or erc223 token to a contract which is not designed to handle these deposits it will be automatically rejected and with ec20 it doesn't happen as a result users lost5 millions of dollars yeah so I'll answer that first of all like it's not an issue in the standard itself standards are meant to be generic if you use web to in fact Swift for example you can use Swift to transfer money to anyone who doesn't even know how to handle it or what to do with it to a bank which is malicious to a bank which is under collateralized it it's just a way to transfer money although I do agree that there should be protection for users to not transfer to things to places which can't handle it but it needs to happen at the wallet level not at the standard level so the the the general thing is here how hard as developers as security people should we make it or how how hard should we try to prevent user errors right we can always say like these are cases where you say a careful user will not have a problem but the standard allows a user to make errors clearly the code is much simpler by allowing those it's harder to think of all the caveats this to this I'd say it is valid it is important to explain that people who use it know how this is done it's not a fault of a standard to allow this it's a fault of a standard plus the community to not educate about this problem right excuse me are you trying to say that error handling is optional and in some cases it is unnecessary to handle software errors even if it causes the loss of money for your users are you saying that what's the creator of C20 standard is calling a security issue is in fact not a security issue and uh are you trying to say that if wallets failed to solve it for seven years they will magically solve it in the next years what do you expect now again you're confusing I'm not denying it's a security issue I'm saying it's not the standard where it should be fixed any standards by definition should be very generic to uh support any use case even if you look at as a traditional Finance you can see send money to anyone it doesn't matter uh using Swift using different current company uh like uh countries have different standards ERC 20 is a very generic standard and it's not even just for money where you have lost it erc20 token could be any representation of anything it doesn't need to have any value people could use it uh you could build functionality where you intentionally send tokens to places where they are not supported to increase visibility of that token it just goes there it stays there for example so there are many use cases of erc20 standard this is a security issue that needs to be fixed not at the standard level at the wallet level or any place where the user interacts with it um and I agree with you like the wallets have not 100% fixed it yet but a lot of wallets now have added protection I think metamask has it rabby has it and so on if you try sending money to like self address or zero address it will give you a warning before moving it so it's not that we have not done anything but yes our industry as we discussed at start it's not mature enough we have not had the 50 years of uh progress that the web2 has had to iron out these issues it is an issue people are working on it we'll get there okay I'm skipping a few more things here Atomic wallet it a very interesting hack to be honest and I think some people know a lot more about this hack than I do um some stuff happened in here but somehow like a lot of people got drained they got drained at the same time using Atomic wallet but it's unknown how um this this simply is another private key compromise um yeah so um uh quick uh how how much more time do I have 2 minutes [Laughter] the classic we can open the document real quick see if there's anything um yeah so so no then when we will share the presentation afterwards it it has like of course more more things in there um a lot more hacks um yes so so we'll use it for for final final other questions um because yeah I will not explain a hack in in a minute okay then any final questions any other questions comments anything so one so of course feel free to um is there one ah very good yeah with uh air gap machines how do you manage software updates uh with What machines uh let's say you have an airgap signing machine how do you handle updates yeah for sure so there can be different levels of air gapping if you are just purely using it for signing you don't necessarily need to ever upgrade it like it's just there it works if it's working don't it's not broken you don't need to fix it like even in fact ATMs for example were running Windows XP for 10 years after it's like security timeline so if there is if it's air gapped the security vulnerabilities are not really applicable to it uh but you can also uh just Whit list certain repositories where you get updates from and just uh in the firewall and get updates from there um you don't necessarily need to disconnect the machine completely from the internet and it would be very hard to actually do signing uh when it's completely disconnected you can technically but it would more be an HSM then than an uh air gaap machine so you it's just use of firewall good connections whiteless Bas are okay everything else blocked by default I also made good experience with um live CD style systems so where where yeah you kind of uh just have it for that transaction and afterwards you you get rid of the system again just by unplugging shutting it down right and then it's it uh this works quite nice to uh not have continuous exposure okay thanks everyone thanks and you can if you have more things to um if you want to ask more stuff about other hacks um we we can't be here in the front because there's a next Workshop a very interesting one by by the red Guild I can recommend it on how not to get fished but um you can catch us outside too of course thanks folks us