# Reducing smart contract hacks - Panel w/ neburo, NPalinkasevic, engn33r

- Channel: [ETH Belgrade Community](https://streameth.org/eth-belgrade-community)
- Date: 2023-10-07
- Duration: 35:17
- Watch: https://streameth.org/watch/yt-1OIScyASkkQ
- YouTube: https://www.youtube.com/watch?v=1OIScyASkkQ

## Description

Scalable ideas for reducing smart contract hacks in the next 3 years

Participants:
Nebojsa Urosevic (Tenderly)
Nenad Palinkasevic (DeFi Saver)
engn33r (yAudit)

Moderator: 
Noah Jelich

## Transcript

okay so uh keeping up with the topic of security Audits and Security in general uh we will now have a panel uh talking about scalable ideas for reducing smart contract hacks in the next three years and the three years is particularly important because we should get to that sooner rather than later so today with us we have nebucha from tenderly uh nanad from uh DFA saver and again engineer this time from y Academy I mean we're all involved with multiple things uh Mr moderator will present himself and I I'm leaving it to you boys to take it away please Applause I'm a lead security auditor they're working on solidity and rust solids as well as leading a lot of the efforts in terms of standardization and best practices on security within the company uh I I have to say I am responsible for a lot of the improvements in the hacking methodology uh and that is my main sort of movement uh this year and today we will be talking about all the security stuff so uh what should we open up with um that's a good question maybe the state of the current ecosystem and what we hate the most about like current development the most movies it's the best thing like yes we started off hating and then we can move to something yeah yeah let's get that out so I mean clients can be pretty bad but uh really they're only that bad because we allow them to be right uh I mean most security auditing companies they do not really enforce the best practices right we've been talking about uh a lot about the ways to measure everything and I mentioned in my question uh and it was mentioned also by another gentleman here test coverage and such practices but from what I know most companies do not really enforce that uh for most projects they're just like checking the current state of the Affairs the current state of bugs but they do not make sure everything is going to be well in an ongoing manner so uh what do you guys think about desk coverage I think that every audit firm who's done more than three audits has seen a large variety of quality of code coming in uh I mean all of you that have met developers I'm sure you know some developers who write amazing code and some that don't write so amazing code unfortunately I do find it very interesting that some firms or some D5 protocols let's say specifically have a really high bar internally for where they get the code quality to before they do an audit to be honest if we could have everyone setting such a high bar internally to their own D5 protocols that would be amazing I think we would all enjoy the security field much better if the code we looked at was uh at that high level so developers remember to level up and I would say as for the test coverage um I mean people are trying to ship things as fast as possible and often they are like um like let's not do the test right now let's try and testing afterwards uh and because of that like most of the project I think that unlike uh trying to get to the production as fast as possible without like doing a proper set of like security um aspects and checkings in place well but if you're doing tests afterwards like do you even have requirements ahead of time like do you even know what you're building um well I would say yes I mean like you can like build things in the beginning uh and then test later maybe like for some special edge cases uh you're just like um not thinking about them at the time and just like later on like when the when the contracts are deployed um like start thinking about them oh wow that's I mean it's not good but like we live in that kind of environment right now what do you think about like test driven development uh as an approach like I'm I'm all for it but as mentioned again like hard to implementer yeah and I mean as uh engineers said like uh we have different levels of code quality coming in and that's like one of the biggest problems in the space yeah I actually think like the the auditing firms should be a lot harder on that because as we all know like from a application side when we're looking at like what auditing firms are available and where we can find an audit and so on usually it's overbooked and it's just kind of uh for the auditing firm they kind of accept everything and it's just a matter of like money and right timetable I actually think they should have like kind of ranking of like if this is better code if this is this is like a highly tested like an easily auditable code actually code is ready for an audit that should have a like a higher priority of actually like going into the audit Pipeline and for projects that ask for an audit and the auditor like assesses it's not ready yet it should be kind of pushed back a little bit I think the auditor should like tell the project like hey you first need to actually like test all the business logic and then when you when you know that the whole code is actually working correctly we can look it from the security side no wait but do you have like hard requirements then for for what's going in or do you still let it slip yeah so I mean basically we're I'm not talking from an auditor side because we're more on the application side we don't really do audit but in my opinion I I guess it's kind of uh heuristic like it's not like a hard metric but auditing firms probably should have like a kind of a ranking system at least internally of like stuff we actually think is ready for an audit and that's the stuff that's going to get priority to to actually get audited what about you know the enemy of every Tech Guy the the sales department uh I mean yeah I know we're not ready for nodded but this is urgent we'll pay you like 17 million dollars yeah I don't have really a good solution for that I mean I think the engineers actually need to kind of the kind of push back a little bit on that and like we're building like a lot of times these are like financial apps these are apps that are gonna hold a lot of money they're gonna interact with a lot of value so we actually need to be like really careful about it because okay we can push it this month it's going to be better maybe we're gonna catch up to the competition or whatever but if we get hacked two weeks later it's all for nothing so I think Engineers need to like push back from the business development side at like security comes first when we're dealing with with money and user funds I will actually take the Devil's Advocate perspective on that and uh say that when we talk about the blockchain space uh often when we hear the word incentives we think about tokenomics or the design of a D5 protocol but we have uh incentives in the security space too and that is unfortunately to process as many code bases as you can so even if we have some uh some audit firms who do follow proper process they have a checklist of requirements of what the code should meet before the audit begins you're still going to have other other firms other solo Auditors who don't have such processes just because of the incentives so uh unfortunately we need incentive better incentive design even here ideally well I mean the biggest incentive should be security I guess I think a lot of uh uh security people and security firms in the space unfortunately are not necessarily prioritizing the overall security of the ecosystem but more their their own benefit I think that's part of the problem I see with the incentives part one of the things I hate is the incentive design hmm I agree okay okay uh okay okay we gotten some of that frustration out now what what do you think what would you implement that would anger the customers the most but you think would be like the the most necessary thing in terms of improving security I I will give an analogy instead of a direct answer I'll let the rest of you speak for direct answers have any of you met a really really privacy-centric person who doesn't use a cell phone who uses some obscure operating system that doesn't support anything and when you want to even send them an email they're like no no I don't use email I use this protocol over here I feel like that is the sort of thing that we could do in the security space to really level up the game but it's just not realistic for the average person unfortunately so we have to somehow uh find a middle ground between the idealized end goal of security and what the average developer is willing to deal with so I think the security people want it to be here and then the developers want it to be here and we need to find where is a good spot for that maybe a good approach here like would be to look at it from the holistic perspective like we're now talking about like securing some smart contracts on the network but is are there things that we can do to secure all of them um currently like if we look at the Smart contract landscape and the byte code that represent them um they're like they can be basically anything like you can start running on the virtual machine basically anything and there's no really easy way to like ensure that they are behaving in the way that something fishy can can happen there uh but like the way that people in the ethereum ecosystem think about it is that we can like bring certain abstractions to the byte code that can help us um like better holistically check if there are any any like malicious or vulnerabilities in the smart contracts uh one of the pushes for that is the evm object format that is going to determine how the byte code is going to look like and it's going to allow like for a more holistic approach for this on the security of the whole chain and that's one of the things that I that's one of the things that should that could really help us in securing like from the more holistic perspective yeah I kind of agree uh what was said earlier like it's it's always kind of two sides of the like the the optimal security is probably just not building a smart contract right so that's like the starting point so it's always kind of uh a mashup like obviously the the auditor would want to have a as simple as possible to to have like uh totally tested totally safe simple code and it's really easy to audit and everything else while the developer on the other side is thinking about right all the features and all the requirements he wants to build so it's always kind of confining the good middle ground in general I would say that like to to improve uh General like a security perspective uh I would guess that like the developers actually need to be kind of more more informed on not just like to to write good code and write good tests but also kind of to think like Auditors so for instance every time we're writing some code we obviously it's hard to you know try and audit your own code it doesn't really work that way but we do try to kind of have if it's kind of a code that's kind of new and we we're kind of not sure of all the the things that can go wrong we do try and kind of do a small impromptu security audit before we actually contract the external security audit so so I actually think that developers need to be like semi-auditors themselves they need to follow like the security ecosystem if something is like gets hacked you need to kind of understand why it happened because it might happen to you also so I think kind of generally pushing that there's that there isn't really two roles like uh developers and auditors like it's even even the auditor I think in order to be a good auditor you kind of have to understand the space really well and in order to understand the space really well you have to be a user you have to be somebody that actually thinks about like how the protocol works at the low level not just like what are some security issues that are common okay so for for the developers in the audience what are some good steps that they can do right now you know they're coming home this evening and they're like oh my God I I really don't want to get hacked these guys scared me what can I do right now what can they set up right now to make their code better yeah that's that's a pretty hard question but I would definitely like first start from the basics so I have you have you followed like the best practices we kind of all know in in a sense like do we need to to write a little bit better because you can have like tests that are okay but you kind of know as a developer like this can be a much better tested so so starting from the basics and then kind of uh seeing it from uh like starting to actually learn as as a person that would audit or trying to hack something actually like going on on a platform like immunify I mean picking a protocol and trying to like white yet check it I think that's kind of a valuable experience that then you can learn and kind of I mean that has a bit of a high barrier to entry to to go to a bug Bounty like what's what's something really simple like you know one click setup or like 20 minutes one hour I mean maybe even just running some sort of like a static analyzers on your code if you haven't already most likely it won't find anything serious but it may be like point you in the right direction and obviously it's always good to check even the the basic edge cases so obviously like running some sort of a code analyzer there's a few that's open source so you can set it up run it and then kind of maybe go from there okay so like Slither yeah and uh Metro and any other tools yeah be a quick sub yeah there's even like uh vs code extension and probably something else for slitter but do make sure to update to the latest version because uh we just bumped up to zero nine which is way better uh okay uh engineer what do you think about mutation testing uh like you mentioned uh that you can you can sort of get an idea of how good your tests are as a developer but like yeah I guess if we're just trying to give developers General ways to improve I would I would keep it as simple as possible to make the lives of the Auditors easier because we can definitely try to turn all developers into Auditors but that's a pretty big ask if we can just keep the developers doing the developer things but doing it better and the auditor is doing the security things but doing it better that almost seems like an easier way to divide the responsibilities so if I'm going to beg developers to do some very basic things it would just be do things like have clear Nat spec you wouldn't believe how many contracts still don't have good Nat spec before going to an audit seems simple a lot of people can't meet that simple barrier even using clear variable names the I feel really bad doing a security audit and then suggesting better variable names because it really just doesn't seem like that's my job but when the variable names make no sense and sometimes are completely misleading it really just makes the audit process harder so I would even say those two simple things good variable names and clear Nat spec I would just beg developers to do mutation testing amazing but again this is like uh one of the more advanced things that you'd have to make sure you're setting it up properly making sure that you actually have an end goal for why you're using it if you're using some of these security tools and you have no idea why you're using it you probably shouldn't be using these security tools leave it for someone who actually has used them before and can more easily set them up and actually knows what the output should look like and how to deal with the results okay one of the like I wanted to add some of the basic things that we didn't cover like most of these things were like I would say a bit more advanced but like using the like libraries that have been audited before for example open sapling libraries that have been audited for the last like five or six years um like using those instead that of writing your own is much more safer and much cleaner for the Auditors when they're looking into it so that's that's one of the things and then looking for the previous hacks what were some of the pitfalls that the developer usually fell for like making sure that your code is not vulnerable to one of those malicious attacks very dangerous I think I saw it a few times is like using like an open sapling contract which is kind of the standard but it's changing one or two small things oh yeah that's something that's really dangerous and people should be really hesitant of doing like only if it's really necessary and obviously you you need to notify the Auditors and kind of be aware that you you change kind of the whole like the Auditors now need to audit even that like really solid and audited code even if it's just one line yeah I mean honestly a big issue is when you start chasing some clout with your project and you're like oh yeah we made this totally unique thing and you just took you to swap in your name at all to whatever Swap and now you enter the 70 new bugs because you messed up the search and replace there yeah so like good job you took good Library you could have just imported and you made a mess of it yeah uh okay uh regarding mutation testing I haven't aced up my sleeve because uh yeah my sleeve uh because actually there's a researcher from and this is going to be a con continued on the next topic but there's a researcher on on uh in the University of uh Colorado and she like last week updated a tool called Sumo which is solidity mutator and it's bloody easy to set up like 20 minutes to set up with learning how to do it and then another like between one and 10 hours to run the test but you know it's just faster running and uh it it really gives you good quality score of your tests uh so the reason why we're mentioning tests a lot is because you need to know what you're building right you need to have good requirements good technical requirements and then you get the tests and these tests are validating stuff and mutation testing tells you if your tests are actually validating stuff well uh because what happens okay we have an audit and we talked over your engineer like uh the audit checks the current state of things but what happens in the future yes unfortunately a lot of contracts have certain variables that can be set by the owner and sometimes when you're doing the audit you have an assumption that okay this this value will make sense when the owner sets it and then when you look on chain after the code is deployed you are shocked because in fact that assumption was completely wrong so doing uh mutation testing to check for these values and to see if there's preventative measures to avoid the owner uh basically shooting themself in the foot by choosing a wrong value that is definitely useful and I will say regarding this new tool released a week ago the ideas I mentioned earlier using good variable names and Nat spec that's like a three-year-old advice like anyone could have told you that a long time ago yeah but this space moves fast the tools are getting easier to use there are new tools all the time as we see so definitely if if you're already someone who always writes Nat spec first of all thank you uh but I would I would definitely recommend trying to keep Pace with other tools and the latest in the space because as we see it changes so fast even I can't keep up with it yeah uh okay I do have a question given that you guys are an Auditors you mentioned like the the owner can change a variable and that kind of messes up the whole thing uh when you audit a contract do you actually like after the depot let's say you're auditing something that's not deployed and when the audit finishes they deploy it do you actually check how they deployed it because I know there were quite a few really large bugs where the the code was well but they didn't record the initialize function or something like that do you guys as an Auditors actually do check the deployment after after the audit is done so usually when the audit is happening this is before the code is actually on the Chain because usually because you want to avoid putting code with bugs on the blockchain uh that's obviously ideal you don't want money in the contract and then you do the audit and you find it's all ready to be hacked at any minute that's terrible I will say sometimes what I've seen some protocols do is they will do an audit they will deploy it on chain and then they get another audit and I have to say the scenarios where I have done an audit where the code is already on chain and I can actually use like etherscan to query variable uh values that are set by the owner or in the Constructor I have found bugs that I probably would not have found if I was not able to see it on chain so normally we are not actually looking at the code after it's deployed on chain it's it's definitely valuable to do so if any of you are Auditors that's my little piece of alpha to uh consider that the difference when the code is on chain and to use that information when you have it when you're doing an audit from our side we have like a as a hard element of the audit deployment scripts so we require everything to be set in advance in the deployment scripts so the system can really fully run and fully be tested and nothing is going to change so we have a bit of a higher confidence info there but there's always cases where people change stuff afterwards yeah for example like even compound Finance or Ave they are regularly changing their Market parameters based on market conditions so the deployment script values might look great but one month later they change values in a compound fork and it's a different story unfortunately who should responsibility for this like for this like on-chain monitoring great question I think that's a great zone right now that's why I actually like asked because as somebody that's building in D5 we actually like see quite a lot like you mentioned component obviously they update their protocols and a few times the updates actually mess something up so there's obviously then questions like should it should the update be audited of course but it still happens that um like I kind of think like that there's probably like an auditing firm I guess the larger companies have high contracts with auditing firms which kind of can continual auditing continual kind of security advisement so I guess kind of that sort of setup where uh a a company would audit the code they would know it well and then kind of have some sort of arrangement where they they would kind of actually periodically like check I guess the if the deployment values are working correctly in in that scenario but I'm not sure even if that's kind of that I mean that's the ideal case but in case of a fact in case of a hack two entities take the forward the project and the auditor so who's going to do the monitoring yeah I'm not sure maybe 10 early maybe us yeah I do have a question for the auditing guys uh so we've seen the rise of the new ebm chains that say that they are evm compatible not evm equivalent what that means is basically that the execution might not be the same it's just that the interface to the client to the node is the same and we've seen like uh bugs occurring on these evm compatible chains that are actually like working on the for example ethereum mainnet but not on some L2 networks uh what's your approach in detecting these or are you going like really down not just on the level of the solidity code or down to the VM and actually checking if uh the code is compatible with the virtual machine so I mean for for every different system you have its own guidelines like recently I held the talk on Venom and uh it's evm ish but the because it's asynchronous it has a completely different set of bugs for example there's no re-entrancy hooray but there is a ton of other stuff that due to a synchronous blockchain stuff so you just sort uh I consider it a different audit you know a different technology and that's it you're approaching it as a completely new thing you see some similarities because it is code but you have these blockchain specific things in mind at all times and auditing and that's it and these will be provided by like company knowledge documentation because the guys writing it will usually know a bit about their own bugs and like special cases and maybe the L1 teams if your company has it because those guys are going to be more versed in that low level stuff mm-hmm I will say one thing I am now in the habit of doing every single audit is asking at the beginning which chains are you deploying this to because if you are deploying to ethereum and also some l2s like optimism and arbitrum they are not the same there are some differences as a very simple example uh block DOT number is useful on ethereum mainnet but on the l2s arbitrum and optimism it's recommended to use block dot timestamp instead you can look in the docs for the exact reasons optimism and arbitrim both both mention this but just understanding the differences between these different chains is difficult especially when you get to the less common chains because you have to specifically seek out what are the differences uh it's probably not going to be identical it's hard especially because sometimes the the chains themselves are updating like the l2s are sort of uh regularly being upgraded so we don't know exactly what all those changes will be in the future we have to keep up to date yeah I mean what even is a solidity auditor you know at that point yes maybe evm auditor so would you recommend for instance if our project is on ethereum mainnet arbitrum and optimism and we want to move to a I know okay coinbase maybe or to to ZK sync or something like that would you then recommend basically doing a another audit just for the like ZK sync deployment which might have a few kind of core differences there if in the original audit you did not specify ZK sync was in scope probably yes uh and as I would say that's especially important if you're using one of these chains that is so young that they are getting a lot of updates because uh I mean ZK sync I'm not even sure how many projects are deployed on there now but I can guarantee that in 6 to 12 months the chain will probably have some upgrades to it which that's even a scary thought if you did get an audit for a ZK sync project today in six months do you need another audit even though you didn't change your code maybe the chain itself changed it's a little bit scary I mean even the Ethereal magnet can change obviously like removing the op code like self-destruct that may be affecting your code so yeah it is kind of scary also from like a developer like app site it is kind of hard to cut like any sort of new chain deployment any sort of upgrades any sort of like you need to do an audit then you contact the Auditors and you're like okay see you in three months so it's kind of hard from a like a developer perspective we know like yeah you should get like any changes should be audited and so on but then in practice you're like okay now do we wait like three months to to release like a small feature like is it okay to deploy on a new optimistic L2 or do we need to wait for for three months to get in another like expensive audit so it is a bit hard to kind of like we mentioned earlier kind of balance between like security and kind of business requirements yeah that stuff uh I mean honestly from a businessy perspective I think for a reordered sort of where you're just not really changing anything and just checking any additional parameters for the chain it's not going to be very expensive like you're probably gonna get it for a few percentage points of the original audit cost I mean we also had like not issues but uh if a lot of time has passed even if you contact like the same Auditors they're like well yeah not that like the person that taught it to your code isn't working here anymore so we need to re-audit everything even though it's maybe just like an additional contract which doesn't really move a lot of functionality but we need to re-audit everything and then like audit the additional stuff I mean that depends on the company process like we internally have a structure with Auditors and a lead auditor and honestly I mean I've been in this company fairly consistently and the same audit after six months I'm the constant no so uh uh that's in general how it works in terms of project memory and honestly you do forget the project after like three months anyway yeah so it doesn't matter that much okay we have uh just a little bit more time what should we finish up on I think something more positive we've been talking tough topic with we've been talking dark stuff here what's a positive outlook for security like three days ago news 70 percent less uh money stolen command yeah or what excites you in this space for the security side of things um let's see I would say comparing to the like landscape five or six years ago uh the quality of the code uh overall on the mainnet on the for the deploy contracts uh raised quite a lot um we have like as you mentioned on the previous talk we have certain like steps to ensure that the contracts are more secure that there are no like uh like bugs that have been like in the past and that are like constantly uh repeating uh hopefully we're going to lose the re-entrancy a bug uh in the future completely um well yeah uh so I would say that the space is evolving and maturing like much faster than I expected yeah I definitely like just looking for an application signing like following different characters that happened I think we kind of now have like things that were new a few years ago now are kind of like standard issues you should kind of worry about and I kind of know that like yeah you're like price Oracle manipulation is possible re-enterence is possible and so on so as a default developer you do have like a kind of a checklist of stuff that yeah they already know that so even the like most kind of uh quality teams like even if issue arises it's usually something like way more complicated like the the simple issues I think like in the if I are kind of starting to go away we kind of learned in the past three four years like what are the the common uh exploits and issues and I think kind of going forward it would kind of it will be kind of even more like not standardized but uh I was just talking like to never show beforehand like in web 2 like in the beginning like simple like SQL injections were possible now that uh they are much less likely I think like in general like re-entrances were a big thing maybe three four years ago now much less because everybody's aware of yeah everybody knows the check effect interaction pattern yeah uh I I think uh a really positive thing is like the scientific contribution and there's been more and more tools and more and more highly qualified people in the space you know adding stuff like symbolic analyzers which require tough to design circuits to detect stuff and they're doing that and that's now something you can just use so it's really easy uh and it's a great addition a new engineer yeah I think just the awareness and prioritization of security in the space has improved a lot and uh yeah there there's still a lot of problems with security in the space but we are moving forward we are maturing slowly uh it's not going to happen overnight even though I really wish it would um so we're moving in the right direction it's slow but we're getting there yep well uh that is it from us uh so it has been a pleasure talking to you guys about security and I hope you all been informed about some useful things uh [Applause]