# Houston, We Have a Quality Problem: Why Security Auditing Needs an Upgrade - engn33r | yAudit

- Channel: [ETH Belgrade Community](https://streameth.org/eth-belgrade-community)
- Date: 2023-10-07
- Duration: 41:33
- Watch: https://streameth.org/watch/yt-grpy_GtIhRU
- YouTube: https://www.youtube.com/watch?v=grpy_GtIhRU

## Transcript

thank you today we are talking about Houston we have a quality problem why security audits need an upgrade let's see if this works it does a quick overview of what we're discussing today first I'll touch on why audit quality matters then we'll jump into why audit quality is actually variable and lastly we'll touch on Solutions and implementation steps perhaps the most important part of the talk if you want today's slides QR code and the link is there and we'll be also seen later at the end a quick background about me I'm engineer I do security stuff at y audit and Y Academy Psy Academy we offer free security training mostly solidity security but right now we're doing our first ZK security track as well I used to do web 2 security before that Hardware stuff why improving smart contract security matters I I think everyone in this room understands that security is important in the blockchain space especially for smart contracts when you're putting your money there you want it to be secure otherwise your money won't be there for long uh hacks definitely hurt a lot of people users in web3 protocols and generally the entire ecosystem if it's a big enough event um the last bullet here as the ecosystem matures the security should be improving and the security practices and processes we use should also improve along with it I would use the analogy of the early internet when the internet was first created we did not have https as a widespread technology and nowadays that's a very different story you could even say the same with early smartphones early smartphones were not exactly the most secure operating systems even personal computer operating systems but over time that changes so basically as technology matures security becomes more important and it evolves the problem that we are talking about today is the variability of audit quality and some of the key factors here that that alter the quality of an audit is the time spent with the code if you give an auditor one day with the code versus one month with the code the result of what they will find is very different obviously if you have more time to understand the code you will probably find more bugs same goes for the skill level of Auditors if you take someone who has never done an audit versus someone with a lot of experience the results will differ there are also some common gaps just generally in the auditing space one is whether the code that got audited is what is on chain and this actually is a very relatively frequent uh occurrence because when you do an audit often there are issues now these issues should be fixed before they go on chain but when the changes are made is there another audit review of those changes uh so there's actually a website that tried to track this um then for audited code whether it's checking on-chain values so often audits are just focused on solidity code or Viper code or any code but they don't consider the on-chain values once the contracts deployed so some of these values might be in the deployment script some of them might be set by the owner after deployment and you just don't see it during the audit it's only when it's on chain that you might see these finally risk analysis this is tangentially related to security sometimes with these D5 protocols there are economic values set and some of these might not be properly considered in the security analysis it's more of an economic analysis so there's a lot of ways that audit quality can vary and sometimes not for the best another very unfortunate uh part of security in 2023 in web3 is this term auditor uh I talked about this in my talk last week at block split but the term auditor doesn't mean much in web 3 security if you want to be an auditor you just put that in your Twitter but Twitter bio and you just start calling yourself an auditor so unfortunately in other fields auditor actually means you are certified you've gone through a certain process you follow certain rules not not really the case in our world this also means that some people who claim to do audits they will give reports but we really don't have a good way of checking like is this is this a good report is this doing proper due diligence on everything they should be doing uh so it's it's really the the wild west right now I will say specifically a very brief history lesson on crypto security Twitter didn't know uh that you didn't know that existed but it definitely he does maybe six months ago a lot of people on Twitter realize they can call themselves Auditors and do independent security audits so now there's a lot of people speaking very loudly on Twitter about how amazing how smart they are they're finding all these bugs and let me tell you if if that was the true story we wouldn't actually be seeing the hacks that we are in the space it's almost like this bubble where everyone thinks they're really smart and proclaiming they can find everything but then you look at the reality and we're still having all these hacks in the space it's just uh there's some cognitive dissonance there's a bit of a difference with what people claim and what's happening before we go to the next slide I have a question for the audience how many of you are developers or work on protocols and you've been involved in the process of selecting an audit or just seeing how this works can I see a show of hands okay so of you can put your hands down now for those of you that raised your hands how many of you chose to do an audit with someone that you either found on Twitter Twitter that you got a referral from or just someone you knew can I can I see how many of those who raise their hands use those methods uh no okay that's interesting because I suspect that in general people are using Twitter to find someone or a referral or they just know someone and that is leading us to what I would consider this issue where the audit shopping market right now is very confusing I'm sure some people are overpaying some are underpaying you don't really know what the product you are getting is at the end of the day you you just hope that you give uh some funds to the security person they sprinkle magic security dust on your code by looking at it and boom you are secure uh unfortunately that's uh not not an ideal situation to be in we should be using more data-driven metrics we should be knowing understanding who's actually doing uh audits focus on certain protocols who's a specialist in this area and also assessing based on your budget what's the best choice because obviously budget is very important so what I would say is that we have this situation where people the developers I really feel for the developers don't have great information on how to determine who can help them with an audit if I had a stable coin forever every time someone asks me what are some good audit firms to get quotes from I would no longer be a crypto shrimp that's for sure it's it's just a very confusing field that moves fast and there's not a lot of clear information on who's who's good right now um even if there's an audit firm that's well known uh the Auditors at these audit firms move around a lot it's not like one person's just staying there forever so things change that's the one constant in in this space now ideally what should be happening instead of shopping blind is doing some sort of data driven analysis and then deciding who to go with based on past results and that's sort of what we're going to talk about today related prior art so there are some people who have thought about this in the past but I think most of what I'm talking about today is perhaps the first time we're discussing it um so there is this dashboard created by zelik that tracked differences between on-chain deployed code and what was audited so if you see that the audit only covered 50 of the on-chain code that might make you worried because you would understand that the other 50 might hold a lot of bugs and the the audit that covered fifty percent of the code might not be very useful unfortunately the dashboard is offline now I do hope they bring it back finally we have defy safety they do independent quality and ratings for different protocols and the one thing here is that even though they are trying to provide a third-party assessment it's an assessment of the protocol not the Audits and is also not focusing on the actual code it's just focusing on are the protocols following a certain process are they doing due diligence in these ways so it's it's like a checklist more than an in-depth analysis so really not much has been done to try and assess the quality of audits at this time who is auditing the Auditors uh I have to say this is not just about the developers choosing who will do an audit of their code and it's not just the web3 users trying to not get hacked or rugged using new protocols but if you think of the greater ecosystem unfortunately a lot of D5 projects get in the news mainstream news for getting hacked and that just does not reflect well on the entire ecosystem and there's obviously questions being asked outside of uh just the the Degen space even normies are wondering if you did an audit and you got hacked like what what good was audit so even Regulators in the future might be asking like what what is this auditor term that you speak of it doesn't mean much so we should really be getting ahead of uh what might be coming down the road and think about this how to do this properly okay now for the important part of today's presentation some solutions these are just some ideas there are definitely a lot of others and I don't want to turn this into a three-hour talk keep it short but some ideas uh one of them is portable ratings for individual Auditors so even though an audit firm has many people specific people are doing your audit so if we could have ratings to indicate how skilled these people are it might be helpful second some metrics for audit reports if we can determine how good audit reports from a certain person a certain firm are and if it's automated then we can just have it processing online and you can actually see some numbers that might help you make good decisions third standardized best practices and checklists if we could have some checklist that's known to have a lot of good good ways to verify that you're not missing the most common bugs that would be great finally some reviews I don't see any reason not to do reviews for audits because we do it for restaurants why not why not the more important things in life okay so I'm going to jump into each one of these and give some more detail uh so first on the auditor rating idea uh we actually have some platforms in this space that do competitive audits so code Arena and Sherlock they allow Auditors to compete to find bugs on a single code base and then at the end of the competition timeline you see the results you see who found the most who found the least Etc so we don't really need to create any new project we already have projects in the space that are delivering data on this of course the data is limited to people who participate but still it's a starting point in terms of creating a portable rating we don't have to reinvent the wheel there's many rating systems out there I'm proposing uh the chess rating system just because it has some analogies that we might be able to borrow for example above a certain number relates can correlate to a Master Level auditor or above another rating maybe a grand master auditor so a really skilled person um and this would just be a way that you could have some sense of what this number means instead of a meaningless number of like 100. it's like my rating is 100 what does that mean if we can correlate it to something else that might help understanding now the the third bullet audit firms have variability uh so measure at the individual level this is what I mentioned before uh individuals move where they're working at some people want to Branch out and do their independent work so if we do this at the individual level and not at the audit firm level uh it's more portable and also more accurate because an audit firm can have a great reputation three years ago but their current team is completely different and that means their their output is completely different so uh individual level might be might be better for for the ratings okay I already mentioned this uh but briefly uh one of the difficulties with the rating system might be that the chess rating system it's for one-on-one matches now these audit contests are not one-on-one it's more like one against many you're competing against everyone for a week and then you just hope that your results are good uh so we need some way to modify the formula in order to allow it to work in this case we also want a rating system that makes sure that it it Compares your results relative to the other people competing so if you're in a really tough auditing competition and you come in third place that should give you a higher rating than coming third place in a competition with very low comp low level Auditors competing so yeah strength of competitors should be factored in there's also different rating systems I'm not an expert on this but we can consider other options this is one example of one of these auditing competitions or the the leaderboard of code Arena and you can see here that really it's uh are we going to have a laser pointer nope um so you can see that really the the scoring is just done by the amount of earnings that each person gets so it's it's not really a rating that's portable it's just like okay this person earned this amount in this amount of time it's not something you can transfer around as a useful metric so unfortunately uh nothing really implemented here that we can borrow but Sherlock another similar firm they do have this points column and yes that doesn't really correlate to an existing rating system uh if these were chess ratings all of these people would be beginners but of course this is the top of the leaderboard so that's not quite the case but if we could somehow uh correlate these points to a rating system that we can make analogies to like a Master Level auditor Grand Master Level auditor and if we can make the ratings portable to different settings that would be amazing so a little closer around Sherlock okay implementation steps how do we make this happen so we can actually use this quality metric as I mentioned we need to adapt the rating formula for non one-on-one competitions we need to create a database to maintain these ratings keep them up to date and ideally optional we have code Arena and Sherlock help out with this maybe it's easier to pull the results through an API or something okay this was just the first solution we have three more I'll make it quick uh metrics so one of the unique things in this space is audit reports are often open source you actually see the bugs after the Auditors find them and that's amazing you don't really see that much in web 2 security so let's use this data let's actually put it into uh some formulas and see like are these reports good or not so one way we could check is the percentage of critical or high risk findings over total findings so if you if you're finding a lot of high risk findings compared to your overall findings that's great but we also need to consider the second order effects of measuring something like this if we want to consider high risk findings as a good thing then some people might say well all these findings are high risk 100 I'm I'm amazing so we need to also consider misrated findings so we need to balance out the two if you're considering a useless gas optimization as a high risk finding that's bad so we need to disincentivize that so these two go together and these are just examples of some metrics we could look at uh some other metrics that might be useful um looking at the number of unique audit findings and what do I mean by this uh some audit firms that shall remain nameless may use scripting tools to find common bugs these bugs are built into the script and that means that most likely the audit firm is finding this bug again and again and again and putting it into a lot of reports so this would be a non-unique audit finding it's something that's scripted and you see it in a lot of their reports if you have a lot of unique findings that's probably indicating that humans are spending more time and effort on the code which is a good thing so if we incentivize finding unique findings that's a little bit like a proxy for measuring the amount of manual effort that goes into the review which is a good thing you want real smart people looking at your code not just a script that's going to miss a lot of things the last idea for a metric here is uh the speed at which the audit happens if someone is looking at your code very slowly and taking a lot of time that's a lot better than someone speed reading your code just saying like yeah it all looks good thumbs up no issues so measuring that is also something we could do based on reports here's some example numbers I just made up some numbers to demonstrate what's good what's bad based on these few metrics I came up with more unique findings is good uh more serious findings good uh fewer misrated findings is good and fewer lines of code per day is good the reason longer audits matter this is probably quite obvious but the more time an auditor has to understand the code in detail the better chance they have of reaching what I call this region of Max bugs and this is when the auditor is understanding at least of certain security aspects exceeds the developers understanding and that's where the auditor can really uh find some logical errors that the developer completely overlooked so that's sort of the goal of an an average audit let's say okay how do we implement this metrics sounds great we have the reports a lot of them are open source we need to parse the reports and there is actually a project out there that's trying to do this if you are working with any audit firm or are talking to audit firms you can suggest that they create a parser so that the results can be added to this project um then second Point here is to get more feedback on what are useful metrics I listed a few here I'm not saying they're the best but it's a starting point and we can discuss and create others there's a couple optional ideas organizing a group of audit firms to maintain and enhance the metrics over time we might see these metrics are getting gamed for example if we say high risk findings are good more people will say they are finding high risk findings even if they are not high risk last Point here uh audit firms to contribute their own parsers this is the ideal way that we would go about it we need to incentivize people to add their metrics to this dashboard and then create their own parser for their own reports because each report is very customized we need to parse them properly so anyway that's how we get metrics okay standardized best practices and uh checklist as well uh at this point unfortunately uh most audit firms have their own process checklist uh if you look for security checklists for smart contracts on GitHub you will find a ton of lists they're all different we don't really have a standardized go-to place so having an open source list that's like the main resource that's updated would be amazing and if we can standardize it across audit firms and we say that this audit did in fact borrow from this checklist that's great I think personally information sharing across the industry helps everyone level up including new people to web3 security if you don't know web3 security at all you could go to this checklist and you can start understanding some of the common bugs that are found for those who are worried that the the secret sauce of their own internal secret checklist should not be open sourced I would simply remind everyone of the Cathedral on the bizarre approach the famous closed Source versus open source debate uh personally I'm a bit of an open source fan uh this is just a example of what an information repository can look like everyone's familiar with the solidity docs and this is like a go-to place to look for solidity information so if we could make something like this for security common bugs then we have a go-to place that's updated and uh yeah I even thought of using the same format because I did start a repository of these common mistakes it's a very basic I started it two days ago but I will be flushing it out more in detail and basically once it's hopefully the best checklist that's open source in the space we can then recruit other experts to contribute to it and make it a go-to resource okay last last solution here um customer reviews uh I I don't think this one's very uh uh controversial having reviews of what other other people thought of certain audit firms would just be useful for everyone we do it for restaurants why not audits which are more valuable um some of the categories we could have for feedback you can think of like the one to five star rating system uh was the cost fair or was it too expensive uh was it too cheap were the Auditors communicating well sometimes Auditors are just looking at the code the developers have questions and the Auditors don't respond not not good did the audit Miss some findings did you get hacked after the audit that would be terrible um and then did the final report meet expectations or were there a lot of typos and mistakes so just some ideas here's a fun example the Belgrade Fortress we have this nice five star review uh looks great I just want to read out the one star review it deserves five stars but I was a bit nervous when Google asked me for my opinion so I put one to vent some frustration I think we can agree this sort of thing should be filtered out but actually seeing reviews with this information would help so uh again it's just one one step closer to better decision making so how do we implement this we need a reviews website that's the most basic thing to submit and view reviews we also need a way to prevent Bots uh spamming the reviews website maybe an auditor really wants to promote their audit firm so they put 100 five-star reviews we need to make sure that one review correlates to one audit also so the developers can't shill the audit firm on the audit firm's behalf optionally maybe we should allow the reviews to be edited at any point so if a protocol gets hacked maybe they want to downgrade the review of their past audit because they realize it was perhaps not so great yeah okay time to get close to wrapping this up why does any of this matter why should we worry about implementing these Solutions it sounds like a lot of work to be honest first of all what gets measured gets managed I think if we are actually trying to assess the quality of the output of audit firms people will improve the Auditors will improve if they see their quality is not up to average or up to par and audit firms will also try to improve their metrics it's just the nature of the game if you want to improve how you appear to the world when there's actual data driven metrics used you need to you know try and actually make some changes uh most importantly for developers doing audits the quality metrics will improve decision making on which audit firm you get with and make sure that you're not overpaying underpaying and that you're getting what you expect you don't want to end up in a situation where you Fork over a lot of your your funding and the result you get is definitely not what you're expecting that's just a terrible uh a very sad ending to the story let's say so helping developers make informed decisions about Audits and hopefully it will also enable fairer pricing in this space lastly if we improve the security in the space we reduce hacks and I think everyone wants that so uh that's that should be a motivating factor by itself okay quick summary I covered a lot uh first point was audit quality is variable not all audits are equal secondly quality metrics are possible at the auditor level so at the individual level and at the audit firm level and at the report level so there's there's a lot of ways this can go um and a lot of ways to slice and dice the data third better data does improve decision making for everyone especially those who are trying to make a decision about audit firms or how to proceed with their security journey in the space finally automated metrics that are automatically considering the latest reports we need something automated to actually keep Pace with this space because it's moving so fast that we don't want to have humans in the loop actually processing this data so automated would be ideal which is why we need the report parsers to collect the data okay last slide I want to leave you with a simple thought which is increasingly true in software and especially in this ecosystem and that is the question is not can this be built but should this be built and if you want to join and participate in this journey because it cannot just be a single person's Journey it has to take a movement then you can join this telegram coordination group I'm going to be discussing what I'm doing we can discuss other ideas on how to proceed and ideally once we get enough momentum we can get some funding to actually make this possible at a greater scale and at the end of the day if we can actually make these quality metrics happen it's going to improve security for the entire ecosystem so if you're interested feel free to join the slides that also have this QR code the link is right here so that's the last slide and I'll open it up for questions we have a question here yeah thank you for the talk it's yeah really hot topic so I'm an auditor so it's a subjective opinion but I completely agree with you with three your points from the last slide but I adapt that it's possible like to build some really fair like rating of uh findings of auditor because yeah the reason is pretty simple like if you charge a lot you're a good auditor you get a pretty good source code and it's it's difficult to difficult to find the really cool critical bus box there and if you're like a beginner you usually get uh some shitty quotes So with a lot of issues so you can be a leader just go into 3D quote actually but the question is uh maybe it's One Direction maybe it's better like to go along this like Direction which is uh used in other areas for instance I don't know if you build some serious uh system like I don't know in avionics for instance you need to follow guidelines how to actually develop code according to like hot uh guide guidelines you should like to I know use properly or like I don't know your repository right documentation document all steps like have all tests coverage then if you really goes like According to some security levels higher to like to the top level you need to do some formal verification of your key like components etc etc so maybe maybe like moving this direction would be better I mean so having this guidelines so if if you if your project like meets certain level of uh security then like yeah of course you need some auditing firm that can do or check all this like stuff for you but this is a better like representation maybe for I don't know that actually this that's not magic dust because of course you have you can have like the best Auditor in the world but I don't know he he was sleepy or something yeah uh these guidelines that you're talking about is this for the developers implementing the code or for security Auditors doing audit for developers mostly yeah okay uh I guess the perspective I take on this is that it's very hard to change the processes and opinions of a large number of people unless you give them some incentive so if you're suggesting that all developers should follow this process that's going to take extra time and it's very unclear what they are getting out of the process if they can instead save time pay someone to do a security audit and they think they're getting a similar result they're probably going to take the lazy route because let's be honest it's developers so uh I guess the the approach here is that if we actually have some metrics that's providing an incentive because it's public data people can actually see well this audit firm is doing a terrible job based on the metrics uh so maybe that's an incentive for them to improve and to avoid this firm I think the good developers already are familiar with what they should be doing whether they do it is another question but yes I think generally I agree it's a very difficult problem in general here so to continue on topic um say uh an auditing firm is forcing the the implementation of these best practices right of test coverage uh fuzzing whatever and due to that implementation bugs are found that normally you know a tired auditor like you mentioned or someone who's having a bad day may miss that day those should also be accredited as found bugs to that audit company because if that audit company didn't force the customer uh and the project to implement these measures they wouldn't find them and the audit company doesn't give a green light to the project unless uh they Implement all these measures right yeah I think that makes sense uh this can actually go back to the idea of a standardized best practices and checklists so maybe we have a standardized checklist of what developers need to uh do or criteria they need to meet before the audit even begins and if we standardize that across the industry I think not only will the Auditors be happy because the code they are looking at already meets some minimum threshold of quality but it would also catch these issues like you are saying yeah often uh often an auditor is going to spend a lot of time with the bad project just figuring out what is going on there and how does it even work because all the crucial steps are missing um onto the point of reviews uh Siri mentioned customer reviews but what about like bad projects reviewing Auditors negatively because they gave them you know trouble for their bad practices for their dark patterns for the stuff that they are doing that is coming that is actually an issue yeah I think we would have to use something similar to what you see here and we would just have to filter those out because uh every review website I know of has something like this happening so uh if you have a better solution let me know okay thanks okay any other one more um hi uh I have a question about the the metrics for the audit reports you have mentioned so um I I uh there's a total findings so that's uh how do you get the total findings I mean the that's a uh as a ground shoes I mean is two yes so the the total findings is per audit firm or audit individual so the idea here is let's say audit firm number one has done 10 audits so we would use the parser that's developed for their reports to parse all of the findings and we would have the total number of findings and then we would try to calculate how many of those findings are are high or critical risk for the serious findings percentage so we need the parser in order to go through all of the reports and collect this data but luckily there is this project that's already working on this uh masamune from a guy at zelik so uh we we just need to create parsers for all of the different reports but once we have that we can collect this data quite easily and automated yeah but the only problem is maybe one project just for example employee two or three Auditors or companies to do that we we don't we don't we cannot cover all the for example different odd different companies different audit or different projects right let me see if I understand the question so you're saying that some projects are doing three different Audits and that the later audits yeah that's the ground shoes but if other audio companies starting to join in you you cannot do it to measure their performance right well the way I see it is this is taking the the number of findings at scale we're not looking at one project or one report so even in a scenario where a project goes through five Audits and most likely the the fifth audit is going to find very few bugs uh that's that's not necessarily going to reflect so much because we're taking an average and if you're also trying to um mention or ask why total findings is even considered here we're not saying that more findings is good we're focused on normalizing uh let's get to the right slide here uh we're we're taking the critical findings divided by the total findings so it's not like more findings is good it's more like more serious findings divided by the total findings is good so even if you find 100 findings and they are all low risk findings that that's not necessarily a good thing for this metric this is just one one or two metrics though I'm sure we can think of many others yeah I see but the their scenario because I for example I just gave you two bar two total finance and they are all critical findings so there would be 100 all right yes so we would need to collect more data uh there is there are some ways we could we could add a a star when the not the amount of data points is below a certain amount just so you can see that it's uh potentially not the most accurate number um you're right if there are only two findings and they are both critical and the audit firm never publishes another public report that should be uh somehow that should be pointed out that they are trying to game this metric I see I think so I just can I just ask one question um I believe this crdm makes sense but the problem will be like if you go in a direction of the standardization the amount of findings should actually slowly go down and you should think more in a direction of creating a average of a total finding to see how the space is moving and then in relatively to this average normalize everything if you do it in a total absolute way you will have a bias which will skew everything we know that from the risk side so and the second part if you're really going to build this kind of stuff we are like really interested to work with you together on that side excellent [Music] yeah so my question is twofold I wanted to ask do you have any rough estimate on the number of audits actually performed uh compared to compared between the independent Auditors and audit firms and the second question is to me it seems that any standardization or anything along those lines should most likely come from the auditing companies actually working together is there any communication between them at all other than people just jumping between audit firms and whatnot uh so let me answer the second question first at this point I really don't see much collaboration between different audit firms it's more of a competitive environment right now which is very unfortunate because at the end of the day we should all be striving to make the ecosystem more secure and do a lot of information sharing we don't have that right now so that's part of the goal here with standardizing best practices and checklists the first question how many audits are being done by independent security Auditors versus firms I don't have a good estimate on that and it might actually not be the best to just count the number of audits because the audits done by the individual people might be much smaller code bases for example just a very small nft project that's mostly based on just ERC 721 default so those sort of projects maybe it does not make sense to go for a very large audit firm because the code the custom code is 100 lines but yes I have to say it's really mostly in the last six months that the independent security auditing movement has gained a lot of momentum okay I might be out of time yeah just to get the mic on uh thank you very much a great talk and you don't have to move anywhere because we'll be continuing with our security panel just give us like two or three minutes to get set up and a huge Applause for this [Applause]