Beyond the One-Off Audit: AI, Continuous Security, and the Evolving Threat Landscape | ETHSofia 2026

Awesome. [music] So, thanks to everybody for joining on stage for for the talk. Uh, even the injured ones. I think the setup here shows how passionate everybody feels about the topic. Uh, if we have Jonathan coming in, uh, a little bit, as you could see, slow motion.

[laughter]

Really sorry about that guys. This is a welcome to Sophia. You need to get injured first to enter the country and then show your strength. Yeah.

Thank you so much. [laughter]

Good. So, um, hacks have always been kind of a hot topic of the web 3 space and the crypto space. Um, and AI has recently been a hot topic of every other industry, right? So, we do the logical thing here and we're going to bring these two topics together. Um, and we're going to try to have a discussion on how AI impacts uh, web3 security and our day-to-day roles um, at each of these companies.

Uh, thanks for a lovely announcement to the host. Uh, so uh, without further ado, maybe we can start with uh, a little bit of intro into your day-to-day for each of you guys. um what exactly do you do when it comes to web3 security and how do you see AI changing that in the recent years and then we'll go deeper in from there but just to get everybody's day-to-day perspective first [clears throat] uh so maybe I have Joseph start

yeah okay so first of all thanks for having me here I'm really happy to share the panel uh with my panelists and um uh about myself so I started [snorts] like auditing I think in 21 but um we get to crypto thanks to Grant from Ethereum Foundation in somewhere 2016 17 and I was also a speaker at Defcon so almost 10 years in crypto and uh we are security auditors for the auditors I would say like the first um at least 8 to 9 years of uh my auditing career were quite the same but a big change happened let's say last year I would say it happened in uh end of November 25 when Opus 4 uh six was released and since that everything changed and that will be the topic of our talk today.

Nice.

Hi everybody. Uh really happy to be here. Uh apologize uh I had I had a an accident uh this morning so I apologize. Um I'm Jonathan. Uh I'm a blockchain oint blockchain intelligence analyst at Certic uh web two and web three uh cyber security company.

Uh I'm not an auditor. I'm more kind of an investigator. So I mainly investigate uh threat uh trace on chain activity, trace fun um analyze how the sorry [clears throat] how the the the attacker operates. Uh and yeah there is there is a lot with uh with AI actually uh AI as you all know AI is capable of proceed and aggregate a lot of uh information a lot of data and uh it's really useful in uh in my day-to-day job because I'm more efficient with it. Mhm.

It's uh you know it's uh easier to to to to analyze a situation to understand what happened, how it happened. So yeah, AI is uh it's scary but sometime it's great.

We we'll get into the scary part later. Chrome, what about you?

Yes. Okay. A very a very quick introduction on my end. Uh, I'm Pashov the Yes. voice is good.

I'm the a little bit up. [clears throat]

Okay. So, as a very quick introduction on my

There's a new mic coming in.

This is good. This is better. Okay. Third time. Uh I am Pashoff the founder of Pashoff Audit Group.

We're a web three security company. We work with uh the best security researchers in the space, security contest champions, publicly proven successful bug bounty hunters. We find the best security researchers and we put them on the right uh fit projects for them. For example, when a lending protocol comes, we choose people who have experience in lending protocols. we we put people who are um really proven to find the right vulnerabilities there and this how we work now um I was a security researcher before that I did 50 or over 50 so audits myself uh a few years back in 2022 and uh the things have changed so much since then right now uh audits are really vulnerability discovery specifically in audits is primarily done by AI really by automated tooling and this is what we do as well.

We have our own tooling that runs on every code base that we audit and we have our individual contributors all of them are using their own solutions and everybody is uh maximizing this new technology to really do vulnerability discovery. This is the most important part. This is this is what has changed the most. But still, it's still about finding vulnerabilities in the day-to-day, studying, playing with the new technology, learning code bases, studying them in depth. That's the same.

So, I I want to chat a little bit about kind of the conflicting forces of of AI right now, right? So, on the one side, we have the developer side where I can produce code much faster. I can ship much faster. Um the auditors can also build tools much faster and specifically for the project which may be might have been unfeasible before. Um and many other upsides like that.

Um at the same time we've seen uh an increased or even historically highest level of hacks was it in April this year. Um, so I'm curious to hear the perspectives whether AI helped make the industry more secure right now or less secure and why and how you're kind of seeing that from from your ends. Um, I'll go with Joseph again first.

Okay. So um the situation changed somewhere in the December 25 as I told and since that we are still like finding the new equilibrium between developers blackheads and white hats.

So far we can tell then in the first half of 26 the winners were the blackheads because there were so many hacks happening in the space. We did a deep dive on the data and we analyzed like every hack and every audit report behind that to trace what was the problem and out of the audited protocols 97% of the hacks didn't happen in the audited scope which on one side is like good for the auditing companies that they can tell no it is not our fault we've not been hacked our our code base is safe but on the other side we as security researchers kind of like failed because the black hacks were faster and what was the reason the only the 3% of the hacks that were audited by uh covered by audits happened in the solid code let's say in scope of the audit the rest was like infrastructure offchain components cross component communication all the bugs that if you would be on I don't know and report this kind of bug uh it will be marked as out of scope you know we don't care about it so we must start caring also about these uh crossconnected uh um issues which are the hardest to find but AI here is to help because if we would be doing this like manual audit review uh reading line by line like we were used to do the past years uh there's so much source code that human mind cannot uh make this review. So here the AI is helping us to uh be able to process much bigger code bases and find the relevances in the code bases. Uh but on the other side the blackheads started using it sooner than the white hats. So that's the reason why there were uh so many hacks in the past half a year.

So now uh we must kind of like catch up. It's also about the communication with the development teams what should they pay in the audit you know if it's just uh just uh looking at the uh 700 lines of solidity if or if the secretary researchers should take also responsibility for the whole code base and we are still finding the solution for that. I don't think so. It's it's settled right now.

Brings to an interesting uh debate as always. Yeah. AI enables hackers to do stuff continuously to monitor any kind of lowhanging fruit from older contracts that we've seen uh on chain that might have gotten less attention before. Um we can monitor for governance proposals. We can monitor for upgrades.

Right? So the time from an event to an attack got increasingly lower than it used to before. Um we also see a lot of auditors try to build tools right um pushov have a good example of the push of skills um and then open source them and kind of give them to the community to use and kind of help with this part of AI. But I wonder also how do you see this? It's kind of a individual contri community contribution from your end right?

But I wonder how do you see this in a broader ecosystem? Should we kind of as white hats unite more and do some continuous monitoring to report bugs before the blackheads apart from the obvious which is the contests?

Yeah. Um what I'm seeing is that there is a good business opportunity here and some companies have started betting on this. I saw Hexense have came out with such a solution which is for continuous monitoring really to they have a solution which certainly uses uh artificial intelligence really to uh they fed it all the previous exploits that have happened so far in the ecosystem um for the past five or six years and now they check for every single vulnerability that has been exploited before in every new change of a smart contract or if a new hack happens and if there is a new zero day vulnerability something that nobody has seen in the space they immediately uh add a checker for it a detector and it will check all the smart contracts that they work with. So this is a good business opportunity. Now, uh they have to figure it out with um building the product right, getting their distribution right, creating the right profits and everything make it sustainable.

But what's great about this business is that it's really continuous. It's uh for the long term. It's not like with audits, they're very oneoff and companies usually just want to get one audit and then they think it's safe. But with these types of solutions which are really they have to be software solutions because they have to be 247 now we are seeing that we are starting to get better monitoring better continuous security so it's good but I don't think that we will have too many contributions when it comes to open-source technology with this but companies have a good business opportunity and we will see more here I'm 100% certain from other companies

nice

if I can jump in follow up on this. Uh it's exactly the point that right now the audit companies are becoming more product companies because if you're auditor without some kind of like internal tooling basically uh you are kind of lost you know in this space and if we before we start building anything there are I think two important facts to mention first of them is of course benchmarking you know uh maybe we'll get to the later but the problem and also the very nice thing is that uh the existing harness I don't know codeex or cloud code are really really good and the models are also really good. So if you just type there uh please find all mistakes uh and uh enter you know and you you'll keep it working. It sounds very trivial but this is the baseline you know and you have to compare every tool that you build on this baseline because this is something that you have to beat with your tool. Second thing uh about past results we cannot just simply tell that with this tool we discovered every critical discover by humans in last year because AI is learning from the public audits.

So um we must be comparing it with the data that the AI doesn't know which are not the past competitions because these are exact data that the AI is trained on. So that's the reason for example when we are benchmarking we are benchmarking on unpublished code bases and unpublished audit results. So we know that AI doesn't know this codebase and in that case it had to find the bug. Um in web two security is a little bit different. Web two attack vectors are basically database of known exploits and the exploit path is like building one after another.

In web 3 it's not like that. most of the criticals are logical mistakes and you cannot reproduce logical mistake. For this you need to have really strong reasoning uh in the model. So that's the reason why the strongest models usually win. So if you benchmarked again sonet or early oppus you know it's now completely obsolete you know we have to benchmark always against the latest model because it is the highest intelligence and benchmark it just with like simple prompt you know find all bugs make no mistakes.

If you beat this, your true link is good.

Find all the bugs, make no mistakes. Yeah, that's a easy enough, right? Uh but still we see Yeah. And still we see a lot of auditors uh doing their own models um and well not their own models but kind of their own versions of AI auditors and creating their own harnesses um to make a little bit of a better result than find all bugs and make no mistakes. Um, I wanted to hear actually uh Chrome for you from you about this process and how do you think you build a good harness or a great harness for your AI auditor versus just an average one that matches and that Joseph would benchmark to as equal uh output as find all bugs and make no mistakes.

Very yeah really uh when it comes to building tools with AI I think one of the most important things is your benchmark. You really have to have the right benchmarks for what you want to achieve. Uh you have to test your benchmark that it works correctly and everything. But you need to check with each change with it with with each iteration of engineering of your solution. You need to check that you're making progress.

If you do not have really um testing around this, you you can never be certain that you're not actually regressing and you're not in a worse state uh than where you started from. And when you have uh such an evaluation framework, you can actually use it with the artificial intelligence itself, you can really turn it um into self-developing loop, a development where the the artificial intelligence tries to improve itself. It can check with the evaluation if it has improved. It can either revert the change or it can continue looping. This is a good approach that pretty much all software companies are doing right now like this this self-arning loop but yeah what matters the most is the evaluation framework and you really have to be very rigorous there go deep and when you compare two solutions you should see which one is the better when it's better keep going continue doing this

about what is better if I can add to that so we we described the benchmarking methodology I think we all agree on And now the question is like what to measure. So of course the first obvious thing is the recall. Recall for you all of you that's like the number of discovered issues of all known issues. So of course you want to make a recall uh 100% or even uh for example we are getting recall 13% which means the AI discovers more than was the baseline of known issues. So that's a recall but that's not the only one thing to measure.

Second thing is token spend. For example, when you increase recall for 2% but the token spent is 10x more, it probably is not that good tool. So token spent is another thing to watch. And if you combine these two things into one metrics, which I think is pretty pretty cool to watch. It's like uh price per issue because you can be using cheaper models, more expensive models and basically if you have if you achieve like high recall then you can optimize for the cost of the discovery.

So these are all the things that uh that is nice to know about your harness when you are building that and evaluate it and iterate and as so as special said I have one small thing to add as well of course everyone where who is building an AI security solution and wants to find vulnerabilities people also measure false positives and here is some very small alpha which uh the people who have actually been deep into building they have understood it already But for the more beginner people, they still do not know this. Uh when you try to really get your false positives to a very very low amount, you really will remove so many true positives that very often is the wrong solution. So you should always have some false positives and you should find the right balance. Um do not overindex on a specific metric. Make sure they're all balanced.

It should be a nice cocktail of balanced metrics. This is the right way.

Yeah, we do the same because you would rather trade off uh reviewing more false positives um and spending more time of that on that than dropping uh a false yeah negative

to be educative. That's called precision. Precision is the number of issues uh discovered that are true positives. So these things these two things kind of like go against each each other like increasing recall and keeping good precision combined it's called F1 score. So this is like the topic that you should be watching.

Exactly. Um and Jonathan here brings an interesting perspective of um blockchain analytics but also physical security. Uh so we talked a lot about uh how AI impacts smart contract vulnerabilities, right? But security is a bit more than just smart contracts, right? As we know um and so I'm wondering on your perspective and how do attackers use AI now to uh impact even and cause even more damage in that area or how do you use AI as well to defend against that and make sure that physical security is untouched by the bots?

Actually, that's um that's a very good question. Um yeah, as you probably know, uh physical security is really uh something here in Europe and especially in France where where where I came from. Um

he got injured in Sophia though, not in France, just on [laughter]

um so yeah. Yeah. Um I think honestly I don't know if uh I have no proof that articles really use AI but um but I think we could we can easily you know with with AI it's easy to to to to cross information um from various [snorts] database because actually this is the major issue issue it's it's the the data leak you U and with AI, AI is capable of aggregate and process a lot and a lot of information, a lot of data. So it's easy to to build um a good data set. So and and more than that uh you should try if you if you if you if you haven't but uh AI is capable of you know you you take a picture or a screenshot or of anything and AI actually is really capable uh to say okay this is this place at this location exactly so it's pretty pretty scary I think to Yeah.

to to to bypass uh to bypass that to to people have to be careful about the information uh they they post on social media but also the information they give to to AI uh like their address uh if they are married or not or those kind of those kind of stuff but more importantly don't flex on uh on social media on on

Don't brag.

Yeah. Exactly.

Be humble.

Yeah. Yeah. Yeah. Keep it uh in a way keep it uh low profile about uh your net worth, your your assets, your wallet and uh and uh and stuff and uh yeah.

Okay. So AI expands um and and it its impact expands through all of these environments, right? onchain, offchain, physical security. Um, it also impacts people themselves, right? So, um, I wonder how do we train new security researchers now with AI?

Uh, before you might expect a junior security researcher to find certain types of bugs. Maybe these are known vulnerabilities. Maybe it's a pattern recognition or stuff like that. AI can do these things much better, right? And much faster now.

So how do people enter the space now and what should their mindset be? What would you hire for um in a security researcher to make sure that they succeed um even with this AI impact on on learning and maybe I start with Joseph because you're also a professor at university. So maybe you see that in school actually let's go back to school.

Yeah thanks. So I listen to Pash of Stalk. So he's quite still very bullish on getting into the space which I think uh is nice. Uh tomorrow uh I have to head back to Prague because I have the first lecture of the semester and always I told the students like uh doing security is the best thing that you can do. It's better than development you know and uh everyone was quite bullish.

Now I must say I have a questions how to approach the semester. Uh because like the traditional approach was that uh students are doing a lot of smaller easier than harder tasks of course themselves which makes them the senior developers researchers and everything. But right now it's like very demot demotivational for example doing a semester work or assignment that is like one prompt solution from AI you know I myself as a student would probably not like just doing it by hand when it's like once implement this made no mistakes enter you know so I'm also thinking like how to approach to this educational um curve because when we on the other side let people use AI for everything since the beginning they they never go through the trenches you know of finding entrances and stuff like that uh because it will be too low level for them. So now it's really the questions in education how to make the new seniors you know without uh being obsolete let's say. Yeah, Krue, what do you think?

You also are kind of well, if I may say, recruiting the country of Bulgaria to become security researchers, right? So in the age where most of the junior findings would be expected to be found by AI, how can we develop more critical thinking or you know just recognizing these red flags really in web 3 uh that I think the question is actually not how a junior auditor should should study but maybe it's more what makes a good auditor or an average auditor versus an exceptional one right like how Because a good auditor now anybody can become one right let's say hypothetically if you have some uh desire to do so and some motivation and with help of AI you can be on a okay level right let's say uh but what makes it exceptional like what crosses the the threshold

yes so you mentioned desire I think this is one of the biggest factors and really when I was doing audits myself when I was doing security contests It was always about really it sounds cheesy but how bad do you want it? Because you go through the code base, you find a few vulnerabilities, you feel good about yourself, you report them, but then if you truly want it, if you truly want to impress, you go deeper, you do another level of of depth. You go through the code base once again. Now you would speak to the AI a little bit more. You try to upgrade the severity of the vulnerabilities.

You try to make two medium severity vulnerabilities. You you want to turn them to a high or a critical. It's just about putting more effort wanting it more. And from what we are seeing even with our audits, I have tweeted about this and it seems like still not too many people know this which is strange but it's a bit counterintuitive. It's very often that people who do not have that much experience can outperform people who are rock stars, people who are very famous, who are like on Twitter, they are like gods, they are legends and they're so so good.

But junior pe not not junior but let's say people with a little bit less experience. Um when they're crazy motivated, when they truly want it, they outperform. They find vulnerabilities that the other guys miss. We have been working with all kinds of security researchers, people with like 10 years of experience in cyber security, three four years of experience in web three security and they sometimes get outperformed by people who are just doing this for a year. Even now uh on my talk a little bit earlier I mentioned this auditor Vivvec who is a guy who I believe a year and a half ago didn't have anything to do with web3 security and right now he is the best like the top performing white hat.

He just this year he has over $1.4 million worth of bug bounties which is insane because bug bounties are much harder than audits. you find the vulnerabilities after someone has audited. So, it's so much harder. And this guy, how did he uh do this so fast?

Uh I'm sure he's very intelligent, but the biggest reason is incentives are set up correctly so that he's well motivated. He does not even have a um a ceiling to his upside, especially with bug bounties. So, this is the right way. create good incentives for security researchers and give chances to people who are hungry and have crazy desire. I think this is pretty much enough.

The smart people can figure it out themselves. The materials are everywhere. You have the blog post, you have the skills, the tooling, they should figure it out. If you have to fe spoon feed them every small little thing, every small step, it's never going to work. This is crazy competitive space to find vulnerabilities and be successful.

You have to be better than others which is really hard to do and it's for people who have very strong desire. So this is the right way. Do not do too much there. Just let them be. Let them do it.

Okay. Let them do it. Nice. Um good. Um so yeah we're a bit tight with time.

Uh, so I got handed audience questions as well. So I'm not the only one asking questions here. Um, I guess after the just do it. Uh, not sponsored by Nike. Everybody feels uh, encouraged to go and do um, security reviews.

Take action. Take action. Do not just listen. Do not just study. Practice a lot every day, many hours.

Outwork others. This is what works.

Yeah. True. True. Uh, good. Uh there are some spicy questions from the audience.

Uh so I want to start with okay very good one. Um so uh we didn't touch upon this topic but uh apart from manual audits there's also formal verification. Uh of course a question that comes up here is uh are you scared of formal verification taking away job? uh your job. I'm talking about uh firmware verification on the compiler level which helps shipping buff bug proof smart contracts.

Um is this something that we deem to be a fear or not really yet in the industry? Maybe uh Pash is a good start or Joseph you want to go. uh with formal verification we had a incredible battles with tur panels uh I am team fuzzing which is engineering approach formal verification is more mathematical approach if I summarize it and would tell what formal verification promises is basically that uh the codebase is behaving like it should uh behave in the documentation the problem with that is that they are not proving it on your codebase they are proving it on their model so whenever Whenever you check some uh critical that was uh missed by formal verification audit, they tell you that it was not missed by the audit because the model is right but just the transition between the codebase and the model was wrong. So that's the reason why I don't uh do formal verification and we do fuzzing. What about you [snorts]

Chrome?

Oh yes. Uh I would say that formal verification is a very very good approach to security. it there are great technologies around this um even pashop group is about to deep to dive deeper into this we have done a few engagements there actually still it's not something that I'm scared of and thinking oh no audits will go away still there is a right way and a wrong way to do it and uh it's still people that do it it's people even if it's tools it's people that set up the tools there can be vulnerabilities there So it will it will not remove all vulnerabilities even if you're uh at doing formal verification at the compiler level. This is not enough like people still write vulnerable code. This is certain.

So again, best do audits, do formal verification, do fuzzing, do all types of uh security approaches because each one finds different vulnerabilities and um you should try to [snorts] hack the smart contracts in all all approaches. Um but yeah, formal verification is not enough by itself.

Okay, so not the end of the uh of the audit business. Good. Um the bigger danger AI weaponization of zeroday vulnerabilities or AI weaponization to exploit the weakest link aka the human. So what do we think is AI going to exploit more zero day vulnerabilities or uh exploit more humans uh going forward? Maybe I take Jonathan uh to answer this one first and then we see if we have any disagreements on the panel.

Thank you Clara. But honestly, I don't know if I can answer this question. C. Can you can you repeat the question, please?

So, um, is AI going to exploit more and kind of uh do more damage to the ecosystem by exploiting zero day vulnerabilities or by attacking humans and exploiting them as the weakest link? It's a It's a tough one. Uh maybe I will say maybe AI will cause more damage. Um I can jump in really quickly just to say.

Yeah, thank you Pash because this one is pretty. [laughter]

I think AI helps a lot with um vulnerability discovery when it comes to code. This is where it's the best at. And there is so much code. There are some code bases that have 200,000 lines of code. With AI, you can really go through it in like a few hours or something.

So, this is where it excels with humans. While humans are not the smartest usually, and they still get fished, they still get socially engineered. Um, it's hard to do this just through AI. You still need the human there. even if they're using deep fake and they're on camera um it it's not good enough yet so that AI is enough to social engineer people but it's progressing very fast but again there will be more uh code vulnerabilities hacked through AI than doing social engineering

Joseph wants to jump in

the question was on zero days uh so

zero day vulnerabilities yeah

in crypto everything is a zero day you know we are not like web two curative and there are patterns that you combine together uh or not everything but most of the issues we discover are zero days. So that's just the reasoning that uh we need uh someone who is thinking logically which can be or human more than like pattern matching uh like is to security.

Okay. Um and do we have time for one more? One more. Okay. Good.

Uh spicy one. Uh what about auditor's responsibility? Uh is there any progress to hold an auditor accountable when contracts they audited suffered from critical failures? Of course the always on a security panel question any volunteers I can go and I can say that I don't think there is any progress. There have been a few companies that have tried doing something around insurance and they have not doing done it very well.

Still, there are a few that can say, "Oh, we'll refund your money or up to a few months, we'll um pay you like 2,000 $200,000 or something a little bit more than that." Uh, which is not really enough. We all remember that when Oiler got hacked like three or four years ago, it was a $200 million hack and they had insurance uh with another company which this company paid out I think $2 million where when you have a hack of $200 million two two million are just like nothing. They are 1% of the hacked amount. So the business model is not there.

And what else can you do apart from creating good incentives? You can just put the blame on an auditor. You can say, "Oh, they did a very bad job." But uh if they did a very bad job, the space knows it. They should get more less business.

So it's happening by itself. But yeah, we have not seen any changes here. I don't think we will see anytime soon to be honest.

Any other thoughts on the subject?

Yes. So just like in general nowadays creating code is uh easy task which means it's cheap. Making sure the code is doing what is intended to do is the hard task. It's expensive and that's why we are here to make this uh make this u happen and uh stay safe in the ecosystem.

At the end of the day, the auditor's reputation is really all they have, right? It's the it's the core of their business. Um and if this type of things happen kind of as as Pashov said, um yeah, it will get out and the editor will suffer. It will not make it easier on the project though because the project will also suffer as well. Uh but if you're doing your security setup right, you should also have multiple audits in place.

You should have a bug bounty out. There should be multiple incentives um to that. Um and then hopefully uh with all the right uh things on track, uh we should be good. Okay, awesome. Uh we are out of time.

I see zeros everywhere on the screen. So, thank you very much for joining and I hope everybody enjoyed the panel. [applause] Good.

[music]