# Security panel

- Channel: [ETH Belgrade Community](https://streameth.org/eth-belgrade-community)
- Date: 2024-10-07
- Duration: 54:25
- Topics: People & Blogs
- Watch: https://streameth.org/watch/yt-egyqv3cb3TQ
- YouTube: https://www.youtube.com/watch?v=egyqv3cb3TQ

## Description

Pietro Carta (ChainSecurity), 
Brad Harrison (Venus Protocol), 
Brenda Loya (Tellor), 
Bogdan Habic (Tenderly), 
moderated by Emilie Raffo (ChainSecurity)

## Transcript

and we're back and this time we have one hell of a panel with amazing experts from the industry so I'm going to let them introduce themselves starting from the moderator Emily go ahead please hi my name is Emily I'm a co-founder of chain security and the head of sales I'm non- technical so we have amazing experts and me hi my name is you hear me from ahuh sorry okay good they almost got robbed of a joke okay good so hi everyone my name is name is bden I'm one of the co-founders in CTO at tenderly uh for this panel I'm just a warm body filling a seat for someone who actually knows something about security hello I'm Petro I'm a a blockchain engineer at chain security where I do smart contract audits hi I'm Brenda I'm the CEO and one of the co-founders of tiller we're in oraco hi I'm Brad Harrison I head up Venus Labs we work on Venus protocol which is a lending protocol on about four Networks and with that being said let the panel begin this is actually a correct way to do the intro with yeah great MC I mean what are you doing next week um all right so I wanted to introduce this in a bit of a personable relatable way uh I'd like to ask you one time you had a security scare or a security incident uh that you want to share with us maybe Bugan you want to start yeah sure so so I'm going to share both because both of them are short uh I don't know if anybody here knows what re request. link is it was used a couple of years back where if someone doesn't give you their wallet address you can put money in send them the link and then they can pull funds from it or they can request funds from you from you via QR code and you can do something uh I got fished basically for tickets for a pendulum drum and Basse concert in London for some reason I thought a guy will send me a PDF with the ticket that didn't work so that's uh fishing thing that happened and then for the scare uh I was actually uh changing my hot wallets and cold wallets recently and even though I have been in the space for the past 6 years I wear an aura ring and for 2 hours my heart rate was like at 130 or 140 consistently because of a dress poisoning and everything else with that people are doing an ether scan and it's definitely not fun so that's a scare and that's a [ __ ] up on my side did you end up going to see pendulum or no no I went very s uh very sad to Fab fabric actually alone because my friends only wanted to see pendulum so I went to fabric [Laughter] Alone um so on one occasion after lunch at the office I go back to my laptop uh I start working and I see my mouse moving by itself so of course I start panicking I started opening Terminals and Googling how do I list all devices that are connected to my computer because I thought like a hacker had a version virtual device uh set up to control my interfaces um as I open the terminal I hear some laughing behind me and it was my colleague who set up a Bluetooth mouse on my screen and that was it actually uh it's actually one of my recurring nightmare to to get hacked so I wake up uh quite often actually uh U shivering uh because I I'm using my computer in my dream and think just start working bizarrely yeah yeah there's no there's not much stress at chin security um Brenda yeah so um in I started learning solidity in 2018 uh like in February and then um by m by may I started pushing some code you know GitHub and and doing deployments and so while I was doing that one of the times um I'm doing deployments so obviously I have a private key and I just just pushed my private key straight onto GitHub obviously it was a deployment uh address so it didn't have that much eth but it's still till this day it haunts me like every time I'm doing anything I first thing I look at is the G ignore is there you know and make sure that it's there and obviously we have like the EnV file now separate and stuff but back then I feel like it was a lot of hacky ways of doing things and um yeah that was that was a real nightmare so but a little bit little one uh this one that hit close to home for me was not a little one uh so Venus was one of the few major lending protocols or if only to list ter Tera as collateral uh I even hate saying the word terara still it's like when or uh when can we stop saying Tara or luna but um an important lesson for everybody to learn so uh that vulnerability uh resulted in low eight figure lossage that has since been cleared from the protocol but it stemmed from uh the way that the chain link Oracle was configured because the chain link price actually uh the way that they served it up would bottom out at a dollar uh and we all know that even a dollar was too expensive for Luna uh so it was it allowed somebody to come in and borrow a lot more value against that uh non-existing collateral um yeah Les lesson learned and uh we we've had that tattooed into our soul and memory uh permanently so we're going to talk a lot about Oracle soon but before that there's some some really interesting Dynamic here so on this side we have the security people with tenderly with Petro who's my colleague from chain security and on that side we have the protocols with Venus and Teller I mean teller is an oracle but also also a protocol and specifically Venus got uh let me tell you how many audits they got eight audits from open Zeppelin 11 from quantstamp 21 from Peck Shield 24 from CTIC two from codina one from Cantina two from pessimistic and more so this is maybe one extreme of the spectrum a very very audited protocol but I don't think you're on the W leaderboard so maybe that works and then uh we have teller that kind of uh started I think more bootstrapped and kind of found maybe creative ways to ensure their security so I would kind of like to dig a little bit uh in these topics maybe uh Brad um if you want to start maybe can you tell us a bit more about what is Venus and why do you think you needed that many audits sure so Venus is a lend and borrow protocol you can come and collateralize uh about 20 25 different types of digital assets uh to borrow against uh we you can do this on ethereum mainnet arbitrum uh BC uh and it's really basically the same uh types of mechanics if you're used to compounder a uh so it's algorithmic that interest rates fluctuate uh with supply and demand uh so yeah there's a lot of audits and part of that is a function of how uh the dev team has been shipping to Venus so with some other lending protocols you get big monolithic uh release Cycles so for example uh I don't I think a hasn't released a new version in three years or something like this uh at Venus the uh the releases are smaller uh as a way of mitigating risk and that kind of inflates the number of audits uh but generally for any shipment that goes out we' like to have uh three to five audits depending on the criticality of what that code is uh so nothing will go to main net nothing will nothing will go live unless there's redundant audits uh by multiple teams different types of audits so audits from uh uh audit agencies but also open audits like code Arena uh and this has uh helped us uh stay safe uh for the past couple years there hasn't been a major incident since 2022 we we want to keep it that way uh and I think big audit investment is is definitely part of that Brenda what would you have to say to this I wish I had his budget for starter um we obviously we've had uh several audits for teller but it's the we were operate closer to how you know we get we do big releases when we release it's like it's going to be the next version so we don't do as many a audits as you you do we we we do make sure that the the protocol is is secure but uh we also don't de deal with uh customer funds as much we can have an effect on them because we are an oracle um so a lot of our efforts go really into the game theory behind how we operate and then also having a lot of internal infrastructure and procedures to make sure that we take care of a lot of the low hanging fruit that we don't want to you know pass over to a very expensive auditor so a lot we have checklists uh that we go through when we deploy checklist that we go through when um when we are actually developing anything we we we actually have you know a lot of testing that goes into it and a lot of like little step by step instructions for every one of our developers in the past used to be just like two developers and now that we've grown we obviously want them to follow certain pattern so that we can make sure that uh there is a certain level of professionalism and security within the company before we go out and pay but I also don't have the budget that you have probably so so if we have here in the room some you know new startups what advice could you give to them you know how can you create this security level on a on a on a small budget basically yeah so I I I go back to to the checklist um one of the main things that we do within within our protocol now it's we have uh I guess you guys call it War Room whatever W room t w room Tuesdays where we actually go through and as a team try and figure out how to attack our own system and then give it a shot and then from that we either create most of the time it's just monitoring tools that we can sort of because we we can't really change the protocol at this point but we can monitor it so that we can actually jump into action if something happens when you are really a startup obviously like save up for for that first audit um it's it's going to be necessary but there are a lot of things that you can do within your developers um you know make sure that you test everything there's got to be unit testing there's got to be end to end testing there's got to be proper comments there's um proper comments there's got to be um I don't know that everybody has Economist in their teams I'm really lucky that um I'm in econ my background is in economics and one of my co-founders Nick is also an economist so a lot of the game theory we can play out ourselves and we you know that's how we came up with our economic incentives in our infrastructure and design for teller but that is something that can be very expensive if you Outsource but it's also something that you can learn um obviously once you go through the audit make sure that if you you know your toomics need to be looked at as well that's if that's a big aspect of your of your protocol um I think it goes back to just processes and procedures within the company for us even the deployment process we have like two people looking at it making sure that there's uh there's bu code checks and and after you deploy and making sure that everything functions you have testing be you know before you deploy and then once you deploy before you upgrade if you're upgradeable make sure you go go through those you know by hand and and and make sure that everything works or have some uh processes already already that as soon as you deploy you run this test straight against mayet or wherever you are deploying into um yeah thank you so on the security side do you also have some advice you'd like to share with people who are maybe building their their first startup their first protocol um so from my point of view uh I suggest first of all to save for an audit uh you don't have to get the most expensive one but don't be cheap about it it's really something necessary and secondly uh try to optimize your code Bas so that it is also easily auditable which is an advantage both for the Auditors but also I think for the developing team and in that sense I suggest you to use um um known libraries use simple patterns uh don't do difficult stuff don't do weird stuff if you are doing difficult stuff keep it contained in a single place and be aware that what you're doing is tricky uh don't distribute it over the code base um I would say yeah uh the simpler you keep it and the the less likely it is uh to be buggy I think looking at I think can you hear me yeah okay good uh I think looking at What U other large teams have done I mean the beauty of our space is that most of this stuff is open source so look at the Saar codebase look at the maple Finance codebase look at The Spar codebase why I mentioned these for example is all of them did a very interesting thing uh that others maybe didn't do uh a thing that you're building conceptually there's a huge chance it's very close to something else that being built maybe not the code itself so you can learn like ways of how to do stuff one of my favorite documents that is is the urine War Room uh or incident response doc I cannot uh remember the concrete name of the dog but they have literally a checklist of hey this is if [ __ ] heads def fan this is what we do without thinking that is literally something that you can adapt on your own from them because it's very abstracted away from how urine works up up until a certain point so how people do fuzzing how people do formal verification how people do all of these things things basically steal it from them because it's open source for a reason so others can build on top of it so I I think that's the correct way to learn yourself both internally and then the other thing is like in this room you have a couple of security people that I know for a fact that if you message them with a question they will answer they won't charge you like 300 bucks an hour they will gladly help now if you do that consecutively for 10 hours they probably will say hey this is something I charge for but yeah I think as someone just starting out maybe without the budget for these large auditing companies or whatever that's the first step you gain some trust you get the money and then you go to the Auditors who do this and their existence depends on them doing a good audit So yeah thank you can people text you for free advice no no no uh I I mean they can but it is free advice so you get what you're going to pay for tenderly is basically there's a joke in the music industry of uh drummers are musicians best friends so in that sense as well tangerly is the best friends of security people so we're not really security people but we really like talking to them and they like talking to us so I think probably everybody knows tenderly already but can you give us the super quick overview if someone doesn't know who and what tenderly is smart contract go fast no uh uh so full stack infrastructure platform uh what does that mean uh if you're developing we have developer tooling if you're deploying something or testing something we have both production and testing infrastructure and then if you get W woken up in the middle of the night it's basically monitoring stack telling you hey you should really check this out so those are like the three categories and we're used by a lot of security people in the space even though we have zero security products or like catering to the security stuff itself so thank you all right so moving on from the topic of you know audits budgets and so on uh talking about oracles right because we have people who are very interested in oracles here um and maybe we can open this topic maybe Brenda do you want to tell us a bit more what is St how is it different from other articles yeah so teller we're we're a very pragmatic team and one of the things that has always been at the top of our priorities is decentralization so teller the way that it works is anybody can become a data reporter as long as you stake on chain or token and anybody can come and raise a dispute on the validity of the data that you're providing for starting for at 10% the stake but if they win they take your whole stake so that incentivizes people to be looking into the network and keep it clean for the users the way that it works is they post a tip on chain and then um a tip is basically a payment on chain that signals that you want your data for a specific value or type of data and then anybody races to provide you the data on chain and then whoever provides it to you they get to take your payments and the team we we are um we didn't do an Ico or any of that so we we have a similar we have a Dev Shar or like ccache so we don't get to see any of that but we provide the protocol for that to happen and it's completely um permissionless so if you want some really weird data you can come and ask for it as long as you provide a full definition that the data reporters feel comfortable with providing the data on chain then they're not going to be slashed they're probably going to provide it or that they're not going to be disputed and then slashed um you don't have to wait for the dispute of your a user by the way you just re request it and they get new data if your data was disputed but everything is on Shain and everything is permissionless problem well we can go into problems with that or or how that uh so how the fact that is permissionless for you to request any type of data can bring lots of troubles to protocols and they use proper definitions or they use things with really low liquidity to try and secure things with you know really high tbls but but that's how teller Works uh that's it's just one of the Oracles in the space we we value decentralization more so we're a little bit slower um than you know if you're more centralized so so I think um Brad you you experienced some issues with oracles you kind of SW that there were some you know shortcomings and then you decided for bit of a creative solution for Venus can you tell us more so uh Venus developed an offering called the resilient Oracle which basically adds support for multiple price feed oracles uh at the same time and use them in a way that gives you fallback and sense checking and this was just us solving our own problem with the chain link Oracle uh that we experienced a couple years ago when it effectively just paused in the middle of Extreme market conditions right uh and if I learned anything from my time working in banking infrastructure it's that single point of failures are bad uh and effectively all of defi is or you know 90 at least 90% of it is is resting on one Oracle provider uh so probably not the best approach if you uh look a little deeper into the effectiveness and accuracy of chain link which uh we're only now starting to develop an empirical ill like picture of what's going on because defi is so new uh actually one recent study showed that uh of 150 million uh pricing events over the past 18 months uh chain link was over one standard deviation uh inaccurate uh to about 3% of the time so about 3% of 150 million is almost 4 million pricing events so in 18 months you're uh pretty wrong four million times that's defi right now uh and if if you don't think that's good enough well you're probably right because to have a nearly 3% margin of error uh it would be catastrophic in banking and so if we want defi to be the Global Financial infrastructure of the world and get it to the billions that everyone always talks about we need hardened infrastructure we can't can't have single points of failure uh and until then everybody's resting effectively on you know one multisig that may have too much control behind chain link so uh this was a key thing for Venus to solve H we're looking at productizing this for other protocols in case they want a turnkey way of implementing this if in case you think that maybe having uh your entire protocol rest on Shain link is also a bad idea uh but you know that's not to say that chain link of course hasn't been uh a Pioneer uh that helped us get to where we are uh but we just shouldn't put all of our weight on its shoulders interesting thank you um Petro you been doing a lot of defi audits recently you worked on Oiler you've been working on curve a lot um so you've seen a lot of oracles and a lot of kind of the pitfalls of of using them is there some advice you could share or some kind of common traps that you see that people fall in yeah uh so both for uh offchain oracles oracles that derive the value offchain such as chain link and onchain oracles Oracle that take uh their price from pools uh I think when implementing a protocol it is essential not to rely on a too strict uh definition of price for example you if you rely on a price to be uh within 0.1% correct it's probably going to not go too well for your protocol if there is very is some slack between what can happen with price manipulation and the consequen is on your protocol I think that's already a very good idea uh what I really liked for example in um Oiler is that they have a liquidation logic based on Dutch auctions uh in which if you manage to manipulate a a price to make some position liquidable The Profit that you can make is uh is U kind of proportional to the manipulation you can put in and so uh if you are just barely making the position liquidable the attacker cannot make any profit and this is a kind of strong property uh for a protocol as opposed to a protocol where I can manipulate the price by 0.1% and get a 10% profit on the position uh in general um I would say combining multiple oracles can improve uh your security um but also you should uh understand very well what can what can go wrong when uh the the price return by the Oracle is uh is not behaving as you expect so you might expect that the price will the Oracle will always track correctly the price this will rarely be the the case sometimes it will be delayed um it's important to quantify uh these aspects uh imagine a price drop by 50% in 1 hour how will your protocol react uh these are aspects that relate both to manipulation and to actual market conditions so I would say it's one of the trickiest aspect of uh of protocol development and it's some something we always take a long time looking into um so we were talking about kind of when an attack is profitable right uh when is it profitable to manipulate the cost of manipulation and then how you can make in the attack and I think when when you do an audit you kind of assume okay a certain pool size a certain but maybe at some point the parameters are going to change it's going to become cheaper to manipulate or maybe the pool is going to become bigger and then there's a bigger reward so some people say this is where monitoring comes in um but uh some people also say monitoring is great uh so that you can get an alert when something is wrong and then you can look at people emptying your pools so um buan can you tell us a bit more about kind of monitoring and how how this can actually uh prevent attacks instead of just letting you know that you're getting attacked although it's very fun at 4:00 a.m. like eyes all crunchy look at you losing your money but um yeah I think a lot of people think about monitoring from the perspective of okay I deployed something on chain now I'm going to look how it behaves um that's one thing I want to put as a first statement I need to cover three things to make a concrete statement the second thing compliance officer say uh second thing is um who here worked in web 2 before I need to make a point okay did you integrate with the stripe API or any external cool did you monitor the ACT API like did you have an agent monitoring their up time no so a [ __ ] up thing in web 3 is that you need to monitor your Upstream dependency so this is what you guys were actually talking about oracles is that you kind of have to figure out how someone else's company Works to know if your company will work because if stripe is down we're kind of okay with it like the payment didn't go through but here you have like huge liquidations and stuff like that happening uh that's the second thing the third thing is that a lot of people when talking about monitoring okay we have it before deployment after deployment but then you also have a thing of can you look into the future so because time plays a huge role in these things especially liquidations and other stuff now I can make the concrete statement uh a thing I'm trying to Advocate to most protocols is a lot of people are monitoring for when stuff happens on chain so did did this liquidation event happen did the price orical update happen there's another thing that was pioneered initially I might be mistaken but they think it is by Maple the maple Finance team is something called invariant monitoring so you can think of it in a similar way wave how I don't know you do invariant testing with Foundry the idea behind this is that as a human being it's much harder for you to figure out when something should happen when something shouldn't happen or when the absence of something happening is bad the idea is that you're much better at figuring out in a concrete point in time in this case a block number in which state your system should be a very concrete example let's say that we're building a smart contract that can do tick toe tic teac toe turn so we know for a fact that the number of started games and finished games cannot be higher or lower than the total number of games on that particular smart contract so that's our invariant this is a very plastic and simple example but this is an invariant check that you could do why did I mention all these things that is if you look at your system as a finite State machine and you're actually testing on state transitions or how the state looks you can suddenly monitor before deploying anything on chain I'm guessing that's also what you said on uh when talking about testing teller before updating running test in CI is basically monitoring when you wrap your head around it uh so that's one benefit that you get the second benefit that you get I'm going to show simulations here a bit frendly but is basically you can do you can do this with Anvil as well so I don't particularly care I just hope more people in the space do this is basically you can uh travel forward and backwards in time so okay how would have our protocol behaved uh 10,000 blocks ago or how will our protocol behave in a thousand blocks so you can suddenly do all of these checks even in production so in six blocks which positions will be liquidated let's take all of the positions that are here now go back in time 6 months would they been liquidated or not and that way when you're looking at monitoring it's much more powerful because that way you're thinking about how your protocol is going to behave you're thinking about how your system the whole state of it and then suddenly you're not just being woken up 400 a.m. and looking all of the funds being drained you can actually drain yourself in a simulated environment or as part of your cicd pipeline whatever rent over I don't have a mic I Cannot drop this but mic drop yeah um interesting so I would like to also hear a more concrete example and Brenda before the panel you told me about the Leana example and how they recreated this black Thursday can you tell us a bit more right so Leana protocol uh lives in OS chain and one of the things that are popping up and I'm sure all of you guys know everybody's building their own chain there's app chains for this app chains for that and it's making it hello it's making it quite a lot more it's making it cheaper to cost congestion in Chains because transaction cost generally when you build an app chain you're probably very likely trying to lower your transaction cost right so but anyway as we expand that's one of our main goals and what they ended up doing is um the attacker ended up sort recreating black Thursday by causing a lot of congestion and osmosis chain and only allowing basically Oracle updates are you following can I interrupt you do you guys know what black Thursday is maybe a little primer on black Thursday okay yeah so uh black Thursday so black Thursday did not was a mishap on ethereum and I'm going to try you guys can correct me if I got any of this wrong but it was a mishap and I don't even know what year and what ended up happening um gas ethereum um ethereum price increased um increased and there was a big selloff and there was a lot of congestion on the chain and then maker maker uses chain link as an oracle but they weren't able to get any updates because of the congestion and the high high gas cost on chain so even if they submitted it wasn't with enough gas to to go through so this costed eventually an update on to the Oracle that was late and actually triggered a bunch of liquidations maker has a lot of really good back stops in their system so they were they were able to freeze the system and mitigate uh they did have liquidations but not all of them them went through because they ended up freezing the system and this was a mishap it was something that just happened because of market conditions nobody expected that to happen it still happened I think it hunts us a little bit I think for for you know if you've been in the space that day was kind of like you couldn't really transact on chain and it was just sort of know what happening it was March 2020 when covid was announced something like that then the stock market dropped a lot and then that kind of yeah yeah everything followed and it just seemed like but it was mishap nothing none of this was malicious like it just happened and it just sort of triggered uh events you know the congestion high prices oracles couldn't update liquidations and we just shocked it up okay well that happen and that's it but now with app chains popping up that incident that was a mishap is actually becoming an attack vector and it was just tested um successfully on levana protocol on osmosis and what they did is they they caused the con uh the congestion on osmosis and they were only allowing Oracle updates uh whenever the attacker you know stopped attacking the chain with their transaction so it would update they would take the position the winning position and take the winnings and then they would do it uh they did it for a few times until they also I think they froze their system so but probably not remembered as a mishap by the people that got liquidated I yeah but true true true but but that was definitely not specifically malicious like it just it was a market condition whereas this was somebody set out to attack osmosis chain because it was cheap it was it was cheaper to attack it and make money off of it so as long as the incentive is there for somebody to go and attack your system they're going to do it and if transaction costs are really low then they can they can actually create this or do this purposefully and now we've seen all of these app chains popping up and this is only I in my perspective once the cat's out of the bag you can't put crypto you know you can't hide it this is going to continue to happen as long as it's uh profitable thank you do you want to add something or I remember it was called a Six Sigma liquidation uh there was similar stuff even before app chains and it was a lot of times market conditions I remember when Elon Musk tweeted that Tesla won't accept Bitcoin anymore we started lagging as tenderly on binance because people started selling off I don't know what exactly happened but but stuff like that and I think uh the more TPS chains we start that will start popping up um there isn't even so there's two problems that can happen um so one is the network itself it being congested the other one and this is us as an RPC provider we see this when you're running nodes is this guyo how it's going to behave because updating State uh starts being extremely expensive and for example now arbitrum Nova was having issues where the only way to run an arbitrum Nova node was in on a beast of a machine with a local nvme SSD by the way we're in a cloud era so saying local nvme SSD is like a very specific requirement um I could go deeper why was happening but uh I think another thing that we're going to see here is chains working on their gas prices and how uh s loads s stores and other stuff is going to behave because it's getting harder and harder to operate nodes and if we actually do want to keep the centralization and I mean even though we're an RPC provider I do hope that we're going to live in a world where I run a node on my phone um basically we need to think about this as well because it's one thing congesting the whole network but imagine your dap is connected let's say even to tenderly and we start lagging against the network and suddenly you're looking at completely stale data so not bad data but St stale data so that's another part of the equation which I think is is very important to think about completely lower down the stack outside of the smart contracts outside of the consensus layer and and other stuff um Brad I think Venus is deployed on several chains can you tell us a bit more what was your you know decision-making process on which chains you're going to launch and potentially what security implications there are well the community is making the decisions and we go where the community wants and uh where that usually tends to be where the capital is and and where uh the the action is um uh but uh we're actually we're a little bit behind in with the other lending protocols and uh the other cadences with new deployments I think part of that is uh well the community wants to go a lot of places but let's see uh how it plays out uh if there's any risks that were unforeseen or that were not anticipated that arise with l2s um and with for example any any chains that might have some congestion right because that could have maybe some spillover effects like we uh discussed with u make with maker a few years ago we don't want anything like that to happen again right so uh being faster is not always better in defi uh especially when uh the entire infrastructure is you know just a matter of years old and we're scaling it for the first time uh to you know tens of billions uh but uh yeah I think that's generally the the philosophy there thank you um one thing that happened recently that's quite interesting uh everybody's talking about the Linea incident um Petro Can you tell us a bit more about what exactly happened yeah uh so velocore is a is a protocol a defa protocol on linear U on the linear chain and on Bay which had uh received three audits but still uh there were a couple of bugs left over and chaining two uh two bugs allowed an exploiter to uh to steal 8.6 million uh e of value by the way one of the bugs was caused by some very um very low impact gas optimization so they had an unchecked block um for arithmetics which was not needed actually it was just a gap gas optimization measure but on L2 you really don't need to optimize this kind of gas so uh don't do that uh take time to make as many checks as you want um well so the the hack happened and uh the linear team was notified by uh by a a monitoring team I think it was hexon notes or something like that uh they were notified about 10 minutes after the hack happened uh they tried to get in contact with u with velore because other pools were still exploitable um they couldn't and what they decided to do was to uh stop the sequencer so it Linea is one is a ZK rollup which relies on a c centralized sequencer and prover um so uh new transactions are processed by this Central node and then are published on L1 in batches uh if the central note decided to stop uh no transaction can go uh go in so that's what they did uh they stopped it for about 1 hour and then they restarted it after um deciding to Blacklist the hacker address so he couldn't put any transactions in anymore he couldn't bridge out the remaining tokens he had act meanwhile he managed to still uh uh still uh bridge out 7 100 e uh so it it opened a big discussion because um what is the role of blockchain if everything is just centralized and controlled by uh by the sequencer if there's arbitrary power to to to perform censorship I think it's a it can be a good thing in term of security but it can also open the door to possibly some problems in term of uh uh for example the role of blockchain as infrastructure it becomes it becomes more vulnerable to political pressure and stuff like that I have an unpopular opinion about this I think what they did made sense to be completely honest now the reason why we're still not in a space where like nobody's going to come and play with us in web 3 if every Monday you see millions and millions getting hacked there is going to be a point where that happens but uh if the whole Community thinks let's say l XYZ chain we're bet we work with so many networks that I cannot use examples of real networks let's say ban chain does something extremely fishy and censors a valid transaction like a very important valid transaction nobody's going to use the chain I mean right now I even yesterday I talked the human part of this is governing of if will people go to a particular chain to use it or not once we figure out the decentralized sequencing as a thing at that point we can talk about okay let's build out censorship resistance Etc the thing that they really like is um there is one network uh that the idea is for the sequencer to be able to very transparently sensor transactions so what does this actually mean is that uh on the sequencer you can actually say for my particular smart contract I never want more than a thousand e going out of it and then the sequencer itself won't include transactions at all so the sequencer will basically simulate the transaction see that there more uh more than a th000 e is coming out of the smart contract and just not include the transaction and drop it and in that way you can put that under governance for example so the Dow of a particular protocol can actually vote and push constraints onto the sequencer of hey this is how we want the protocol to behave because we don't think that there's actually going to be this high of a move of a funds or anything like that or we want to block the attacker on the sequencer layer so you could have a very fast three-day governance proposal that would block the attacker but basically I think that's the way to move forward right now I I think they did the correct thing because it's not controversial I think that's one of because it's not controversial but once it does it it's I think this sets the stage to are will we ever decentralized in the name of security and and it brings that very up to you know and I I I guess let's talk politics a little bit like right now the US you know we're going through our election cycle and every time there's a hack on crypto no matter how small or big like they take it so that to tell everybody how unsafe it is and how crappy it is and whatever try to shut us down and every time something happens it makes it more and more difficult at least for us to operate there but whatever happens there extends to you know everybody's watching everybody's watching and if there is a choke point that can be used specifically for any government it's going to be used so if we could it's I I understand like the development cycle like we you know when we first uh launched you know yeah okay you know you you we even had a governance thing to upgrade whatever but eventually that is a choke point in an attack it can be an attack vector and it's like okay so right now it's not controversial but then in the the future if he was who decides and who determines which Fork is right then you're choosing winners and losers and who that person is or who that entity is becomes a very very big no no I completely agree I'm just saying gradual decentralization is the point this is say in this current point in time Linea return or saving funds because it's not controversial is the good way to I think that people are right now currently on May 5th are giving them too much of a tough time for what they did if they continue doing it yes but then also everybody's going to bridge out of lunia and not use it at least I hope like in that case so but they agree with you yeah so I'm soon going to ask you for a short closing statement so if you want to start thinking about that and then we're going to go for the Q&A but I just wanted to add something on this I think it's also uh it increases the responsibility of the team because now if the team can pause transactions and can hold the chain when they judge that it's needed then do they become liable for all the other transactions that could be illegal that they haven't censored and typically in the tornado cash case um this was one of the arguments put forward by the defense is that they could not hold the transactions they didn't have disc control over the protocol and then it's kind of taking this argument on the on the other side so yeah I think it's interesting as well all right that was my stalling for your closing statement who wants to start I started with the intro I'm going to go last okay everybody's looking at me so I guess I'll talk uh defi is really hard and it doesn't come down to audits uh alone right uh defi is operating at the intersection of Financial Risk and immutability risk it's like we're launching a bank something with bank type CA capabilities into space if the SAT if the satellite goes bad you're not going to go up to space to try and fix it right so uh that's why there's uh so much pressure to get it right the first time and uh that's not easy uh and people don't invest enough resources in uh in the get-go to make it right and so what happens is uh people get wrecked and uh continually year after year and the number of exploits and the number of lossage actually increased from 2022 to 2023 so we're not doing enough uh to Abate the issues uh because they're multi-dimensional and it's complicated but I think that there needs to be more at various levels of our industry uh to work the problem like Auditors and uh monitoring and all this is just like scratching the surface we have to come together we have to have security consorti we have to have some types of uh not just best practices but uh uh grading that everybody agrees on uh that doesn't come from one auditor but that every everybody can say you know around the world yes this this is a way to determine if a protocol is worth putting Capital into because otherwise people are just going to be throwing money everywhere uh recklessly getting wrecked like you know in pump. fun or what have you $80 million gone the next day uh like it's nothing uh and it's it's a lot of money uh so I hope that we we figure out new ways uh and new models uh and new Frameworks uh for advancing uh security across all these Dimensions uh because uh we should be able to say together one year that we had uh no seven figure lossage uh due to exploits that would be something H I don't know when it's going to happen but we need to get there it would be amazing um defi is really hard and earlier I heard a talk where they said you know when I and I forget who it was but they said something like well you know if if somebody uses chain link um it's sort of like a signal like that you know it's fine because everybody uses it um I I kind I disagreed with that point because not everything should be under one thing but also um just because everybody's trying to do defi really really fast it doesn't mean that it's just it's it's it's fine just just do it faster you know maker has grown to be one of the biggest players in the space and they used to at least I don't know about now cuz I I've hav't looked into the protocol for a bit but at the beginning they took an hour before they actually used their Oracle price if they can wait an hour I think if you're building something wait the hour make sure that you don't lose funds don't get hacked make sure that you grow to be that big before you know you you do anything stupid like and it's it's a matter of timing like sometimes a lot of these hacks I went back and actually looked at rck and list listed all of the hacks for 2023 flash loans were like the biggest thing that is used to do asset manipulation for the or to manipulate the Oracle and it's such a know a well-known attack that it was done like too many times at least once a month somebody got hacked in 2023 through that same attack uh recipe it's like learn from our mistakes and take it slow it's okay you can you you can still calculate who's going to win and who's going to lose and just free it and or make it slow to withdraw and that will take you a long way for being able to actually do something if you get attacked but that's it so I think Oracle are a good examples where uh code correctness which is something that Audits and formal verification can address and monitoring are really need to work uh in concert um we rely on oracles to for example create Landing markets but the safety of the Oracle um depends on the capital required for manipulation manipulation is always possible the only variable is the cost and the profit that can be derived from the manipulation so it's not uh an immutable parameter it's something that evolves over time uh and um it's often very hard to quantify and this is something that protocol need to take a a a step a step back about and um realize uh the ver security depends on on understanding this and this is a bit peculiar for every protocol uh but um yeah understanding the cost of manipulation and the consequences is something uh that's very um very important and very often overseen by especially Landing protocols which um which assume the Oracle price is correct for whatever amount pool size whatever amount of collateral while there can be slipage there can be uh liquid markets this kind of thing will influence uh the effectiveness of price manipulation just a Qui comment on that as a protocol if you don't understand or don't know the cost to hack or to attack your protocol you need to do that on a periodic basis and if you don't trust me an attacker will calculate it for you and it'll be too late so um read what others have done because you're literally probably not the first one trying to tackle an issue that you're doing one statement second statement don't use the uh space being yes young as an excuse to make toys and maybe my last words were chin security we do smart contract audits if you're looking for an audit please come to me and we've been Distributing bugs if you guys are hungry and you want some nice snacks uh we have insects for you all right so do we have questions from the audience yes over there can we give him a mic one question the only question no pressure should you should I give it to yeah airing on the side of action yeah thank you thank you for the panel so we've heard so many times about audits so excluding audits was the second security activity you would use as protocol or you would recommend to protocols I would say active transaction monitoring so you can see uh what's happening economically financially and anticipate try and anticipate something you know the lead time is minutes but having this uh gives lending protocols the opportunity to put some circuit breaker in place uh when circuit breakers are reliable enough or good enough that the protocol feels comfortable to uh Implement them don't use uh don't use testnet I think testnet should die for smart contract development they should just be for cons senses testing uh use tools I am chilling tally but there's open source variance as well use tools uh that let you actually test on production data uh keep thinking about invariance in your protocol and keep uh test like monitoring them uh on chain and also have a big bu Bounty which uh which will attract hopefully uh eyes yes a big one though internally ongoing checks within your own team nobody knows your protocol better than you and what the weaknesses of it make sure that you understand them make sure that you do anything to track them or you know um anything that you can track as a prerequisite before something B happens like an example is like for us we we monitor how much our bridges basically bridges that bridge our token why because before we before somebody goes and tries and take over our system they have to bridge so if we can catch them there then we can actually do something versus like right with versus just monitoring when somebody starts taking like 50% of the or trying to take over the network so um yeah internally thank you we don't have time for more questions right all right thank you so much guys thanks