Nosy Neighbor - Automated Fuzz Harness Generation for Golang Projects by David Theodore

Yeah, so I'm David. Uh, last time I saw you guys I was in better shape. I'm surprised I made it here. Um, more on that later. I'm going to talk about Nosy Neighbor.

Uh, Nosy Neighbor is a a tool I've been working on um, open-sourcing for everyone to use. Uh, hoping it makes some waves in the Go language community. Um, talk a little bit about the challenge. Um, finding bugs in open-source Go projects and a little bit of the motivation. So, Ethereum really loves Go.

Uh, some stats here. Geth accounts for 82% of execution clients. 42% of the consensus layer clients are running Prism. And this is as of like last week. These metrics actually used to be higher, like 90 plus um, on the execution side and like 60-70 plus on the the CL.

Uh, MEV-Boost is also written in Go. It's the only production open-source uh, MEV client for Flashbots right now. Uh, it as of like I think last week, 48% of blocks on mainnet were flash blocks. So, I won't go into MEV and all that fun stuff. But, what I'm trying to kind of hit home here is that uh, Go is like critical in the Ethereum stack.

So, somewhere I would say like maybe three quarters of nodes or more um, are using Go at some point. So, it's super important. Let's see. Lots of code. Got to run two chains, the execution chain, the consensus chain.

Uh, we've got the entire EVM, all kinds of stuff. So, there's like a a a huge amount of code here. Um, how big is it exactly? The pure Go Ethereum stack is 583,000 lines of code. So, as a security researcher, like a a deer in headlights, um I'm not going to manually review all of that, right?

Um we do do manual reviews here. Um all of these things have been reviewed at least once, most likely, if not twice or more times. But, they're moving targets. We've got uh every 6 months or so, we've got hard forks on both the EL and the CL. So, it's just kind of something that if we can automate anything, we should we should automate finding issues here.

My task, uh there's a few of us at the Ethereum Foundation working on the consensus layer security research team. My task is to kind of like look right now and focus on the Go stuff. Uh this is kind of a daunting thing. So, a few months ago, I guess about a year ago, started looking at this and trying to understand the problems that we have. So, I'll talk a little bit about it.

First of all, uh Go is memory safe, which is awesome. Um we don't have remote code execution issues, stuff like that, very often. Um the memory safety stuff helps also a lot in the I'll talk more about it in this talk, but making it possible for me to make like this. A lot of common mistakes in Go, though. Um there are some that are queryable, right?

There's a lot of these like little issues, like the colon equal sign, this like quick variable assignment. Um it makes writing Go really easy and and human-friendly, but it also means like you can have these shadow variables, where if you like declare a variable in a loop, um you override it and you might reference it later thinking you're talking about the first version of the variable. If you call a Go routine and and uh you kick off like which is basically like for non-Go people, it's like a it's like a pth create under the hood. Um the variables that are passed to that are not the same variable that you might think they are. So, if you have like race type things later, uh you don't really know and Go doesn't complain.

So, we can query for certain things like this. This is a good example of like one query GoSec. It just says like, "Hey, you need to manually review um all these uses of unsafe." Probably not the best example, but we have CodeQL, Semgrep, and Go-Securer, like just major tools you can look up for like automating querying static analysis, right? Um another big one is race condition.

So, Go's I think the the reason it was designed and the design decisions they made was to make parallelization very easy. Um so, they have like this concept of channels and they have this concept of go routine. So, it's very simple to be like, "Hey, just go on another thread, go do something, and then come back." And Google made it to be memory safe, but also efficient and human readable, but to process massively large data sets. And so, we can do a lot of parallelization very easily, but the problem here is that um it's so easy that it's really easy to introduce race conditions.

So, like one thing we have been doing is running these thread sanitizers. Um this little terminal prompt right here is an example of actually a mainnet um geth bug, a race condition that was causing some memory corruption, which if you see memory corruption in Go, um it's usually something race related or you're using like Cgo or some native library, because Go is relatively like almost 99, I would say, percent safe um if you're not in one of those conditions as far as memory goes. There's also some other sanitizers. There's ASan, MSan, um UBSan, things like this. We're running nodes on Robson, Sepolia, uh the Prater Görli testnet, and also on mainnet that have all these sanitizers running.

So, we're like we've kind of automated, you know, this querying. We've automated uh like sanitizers running stuff like using dynamic analysis. So, we've kind of checked both of those boxes, but uh you know, what else can we do? And that is where Noisy Neighbor comes in. Um so, how else can we cover these 583,000 lines of code?

Uh the solution Let's talk a little bit about the problem. I'll rewind just a bit. Um obviously, I already mentioned the huge attack surface. One thing I do want to point out here is that denial services are like critical for us. Um not just like for other Go repos necessarily, but a blockchain cannot have a denial of service.

So, usually a denial of service on the common vulnerability severity scale is like a three. It's not like a nine. It's not considered critical. Um there's no information disclosure. There's no remote code execution, and thus there's not usually privilege escalation.

So, people aren't like lifting keys. But, if you have, you know, more than 35% of the network running some kind of go um under the hood, and there's denial of services in those goes uh go routines or or or these repos, then you end up in a problem where the Ethereum network could like be brought to its knees. Um we obviously have a multi-client architecture, so these other clients would be kind of like carrying the network during that time. But, it's not something we want. We wouldn't have finality um for those of you that are like really familiar with proof of stake.

Uh it would be pretty big deal. So, we kind of have this like weird issue where we the worst kind of bugs we don't really see, but like we care a lot about these smaller bugs. Um the good, I mentioned RCE is rare. We have the source. This is really great.

I've been a security researcher for a long time, and I've not had the source very often in my career. So, this is like a whole new ballgame. Um Go is strongly typed. All of the panics and stack traces and failure reporting is like excellent. So, if you write a fuzzer, and it finds a crash, it doesn't bring down your fuzzer, and your fuzzer doesn't commit suicide.

So, that's really helpful. Um and I'm going to talk a little bit about the tooling. Uh this is a big deal. So, Go 1.18, I guess like maybe 6 months ago, they released uh native fuzzing support.

So, this is an example of like in the testing library, um I can fuzz this function. So, the function under test on the right side, and I lifted this straight from like the Go fuzzing like native fuzzing design doc. Um foo is the function under test here. So, we can fuzz foo. Uh we can say, "Hey, f.

fuzz, give it a function interface." I say, "I want an int, and I want a string," which are the two argument types to foo. We can add a test corpus of like right here it says five and hello. Anytime you want want like prevent like regression, um you can add like previous bugs into your test corpora this way, and it will automatically tell you if the bug is re-introduced. So, that's really cool.

Um it's automatically covers guided all the test cases when there is coverage. They get added to the test corpora, and they get mutated on. Um errors are super descriptive. Don't need health checkers because all this stuff is built in natively. It's awesome.

This is kind of the next piece of the puzzle. Uh discovering this kind of like opened my eyes to the possibilities of what we could do here. The AST is exposed um the parser library and the go types library expose everything about the source. So, when the compiler reads your code and then compiles it, everything that it sees you can see right here. So, like this is an example of the AST.

Um it kind of is pretty printed. You can see like there's a variable. You can see at the top it says like AST if statement. That's saying there's an if statement here. Uh the first directive is X.

The value is two. There's all this information here. There's more information than you ever want here. Um so, that's really cool. So, what can we do here?

We can parse all the go code in a repo. We can basically collect all the dependencies for the packages and the types. We can collect all the function declarations, the package declarations, um the type declarations, all the interfaces. We can see what every function looks like. Like does it does it take like, you know, array, slices, bytes, strings, complex structs?

Also, uh if it takes a complex struct, what is that struct made of? And recursively down the whole thing. So, you can basically go all the way down to all the built-in types um in go, and you can see all the information you want about something. So, using this we can generate valid fuzz harnesses for all of these functions that we have typed. And uh we can fuzz them.

We can round robin them. We can fuzz them uh while, you know, the target's running, and I can talk a little bit about that more later. Uh but then, yep, find the bugs, profit. I know this isn't like a security specific conference, so this is a reference to an old frag magazine smashing the stack for fun and profit. I'm not profiting off these.

I promise I want to clean up all of these bugs. That's what I'm paid to do. That's how I profit. Oh, and then you can repeat on every commit, which is kind of why Noisy got its name. It's the noisy neighbor.

You can integrate it into your CI. Hopefully people will do this. And then when bugs are introduced like the moment they're introduced, we can just like automatically fuzz a function before even the tests are written for the function and we can find these bugs. So it's like really annoying and really noisy to developers cuz there's like this old granny across the street that's like always looking in your business. That's where the name comes from.

I just call it Noisy. Noisy neighbor's way too much, so I'll I'll refer to it as Noisy from now on. So, Noisy in action. It basically has three main stages. There's an initialization, there's the harness generation, and the fuzzing.

You can put this all into like one seamless, you know, action if you're integrating into your CI, but for the purposes of like the tool as I made it, you might want to debug stuff in the harness generation. You might want to add test corpora. There's all kinds of like interesting things you can do in between these steps, so I have it broken up into these three steps just for like kind of sanity reasons. This is the input. Every time you run Noisy, no matter which of the three like actions you tell it to do, you're going to give it this YAML file.

And this YAML file contains a bunch of stuff. The most important thing though is the URL to the repo. So it'll pull down this repo. You can say I want this particular branch. You can specify different Go versions.

Like Prism for instance is like one of the big repos that I always look at and it won't build with Go 1.19 right now. So this is really easy. As long as it's like 1.18 and above because that's what supports native fuzzing, you can kind of use older versions and that sort of stuff.

There's also these like ignore declarations I put in there. Like maybe you have a bunch of test functions. Maybe you have um things that use networking, stuff that writes to the file system that you don't want to fuzz because you don't want to pulverize your file system. Uh you can declare at the package, the function, and the object level to ignore these kinds of things. Um it also has substitutions.

What this will do, if you're familiar with Go, if you put a substitution in here, uh it's you just put both packages in there and it will put a little replace directive in the go.mod file. And this is really nice if you like want to knock out all of your like signature checks. Um you get a lot more coverage this way. Like obviously your fuzzer's not going to be like signing ECDSA signatures correctly.

Um that would be a whole 'nother talk and and we'd be having bigger problems if that was the case. Uh the initialization it uses Docker. It basically makes this little fuzzing environment and building environment all in Docker. And there's kind of a lot of reasons for that. Uh one is that we don't want to pulverize our host file system.

Um we have a lot of ease with dependency. Uh if you're like looking through and you're trying to dynamically write code, writing all these fuzzing harnesses, then it's a lot easier to not use your own go root. So this makes a valid go root inside of a Docker container. It makes every it adds all the dependencies for Nozi for doing the the source parsing and the harness generation and all the fuzzing and all the stuff. Those it in this little Docker container and initializes it inside this repo.

Uh or initializes the repo inside this container. Um it uh has a shared like host file that you'll see like in a little bit. It's an asset directory that keeps everything that the host needs to get. So you do all your fuzzing inside this like protect protected environment. If you're a security researcher and you're familiar with like jailing uh a target, this is just like a chroot under the hood except that uh we get to stand on the shoulders of Docker and you know, uh we get to like potentially neuter all of the the networking.

We can like control things and like really jail stuff. It also makes it where you could like run this on your host computer and like only get a few cores and you could still like work and you know, maybe you're you're kind of like dual-purpose using a fuzzer that's also like your desktop for research. Um let's see, generate harness. Generate harness will copy all the assets in and then it will spit out this one-liner. And if you double-click this one-liner and run it, it'll start generating all of the harnesses.

Uh and so you can kind of see like the lower half of this terminal, it's spitting out all of these fuzz nosey_test.go files. Um all of those files are placed into the respective package under test directory. The reason I do that instead of having them all in one is that one bug doesn't prevent the whole thing from compiling and having go complain. The other thing is that we can fuzz internal uh functions.

You might not want to fuzz internal functions, so nosey has a flag for that. Um but if you do want to fuzz everything and get like this serious breadth-first coverage of your target, this is the best way to do it that I found. So, this gets to the fuzzing. Um you same thing, spits out a one-liner, it creates it adds all the assets to this asset directory, uh and then you start fuzzing. So, you can see on the right side here, um it started fuzzing, it found a a crash like right off the rip, it minimizes the test case that produces the crash, and then it spits out the like panic output.

So, kind of funny that one found one like right off the rip. This is an example repo that I have that comes with nosey so that you guys can all test this. Um I don't provide YAML files uh for all of the targets I'm testing because there's I don't want to give you guys three bugs and uh nosey is still kind of like a work in progress. So, I will release uh other stuff further down the line as I've like hammered out all the bugs that are there. Um I think what I'll probably do is kind of like have this like private repo that's maybe like three months ahead, and then as as we've kind of like shaken out all the bugs that this fuzzer can find, I'll open source the other parts of it.

But you can copy the example YAML file for this target repo that I've made, and you can point it at the go standard library. Um I haven't had time to do that. I'm sure there's tons of bugs out there. Like I have one thing Noisy does do is it causes like a decent amount of false positives, but it does find bugs. So, I have like a ton of crashes to look through before I'll release all this other stuff.

Oh, that was the This is an example of like the round robining. So, when there's not a test when there's not a crash found, this was what the output will look like. So, it'll fuzz for like 10 seconds on each thing. The animal file has a little variable there where you can say like how long how many seconds do you want to fuzz for. As you um generate like larger test corpora that are getting better coverage, you might want to bump this up to like, you know, 6 minutes per function or something like that.

This is just a little shot of like what the script looks like. Like what does the round robining? This is a shell script that's just like kicked out into the asset file that's run on the target. So, there's some reasons that this is not the best way to do it. So, I don't think I'll be doing this way forever, but if you can see right here, it basically calls the Go test fuzz on the function you want.

If there is test data fuzz inside of that package now, that means that we found a crash. Copy that in the asset directory so that it's available on the host. If the fuzzer either commit suicide or if you're done fuzzing, you don't lose this. A lot of people like know that you a stalker, your container might not be persistent if you don't have some like asset directory where you save stuff off to. So, this prevents you from kind of losing work that you've done.

All right, example findings. I made this little repo so that everybody can kind of like see Noisy in action and have an example so they can point it at their own repos. These are all the root cause of all of these bugs are copied from real bugs that Noisy did find. And I'll talk a little bit about like the type of bugs it finds cuz it doesn't find everything and it's really good at finding a few things and and I'll talk about that looking at these here. So, um this is just showing like the panic line.

Like this is identifying the type of issues that we have. Uh it looks like there's two index out of range. The second one actually has another bug. There's two bugs in that function. So, I included the wrong screenshot.

It should be like a divide by zero. So, these four functions on the right show uh the vulnerable functions. So, these are the kind of things that Nozi, like if you say like spend 3 seconds on a function, Nozi will find these things like immediately. And if you notice what these are, they're panics. They're not like remote code executions.

They're panics where things stop. So, if you have some uh blockchain software, it's highly social, it's listening to all these peers. Every time it receives a packet, if you have like a GRPC handler and it's meant to like panic gracefully and it just panics in that go routine, you're fine. But, if you have this in like core code, this panic can like make the panic go all the way up the stack and just completely bring down your node. So, these like true like packet of deaths, um they're a big deal and this is kind of the thing that like keeps me up at night uh because panics are not always handled handled gracefully, especially in uh like these huge systems.

So, like the EVM, for example, if you found a panic in it, uh you might actually crash that part of the process and then geth would just be like completely worthless, right? Um so, let's talk a little bit. I've only got 5 more minutes left. So, I'm going to talk really quickly about like what these functions look like. So, this is the most basic, right?

So, uh if you look at this, this is the same native supported testing. The input to the top is is a it's actually a test function. It takes in the testing object from Go. Uh you hand it a function interface, right? So, this function here, all I'm saying is, "Hey, uh I want to fuzz this l- log validator web auth."

Right? I just picked some random function. Uh it takes a string, a string, and a string, right? So, you know, this is the kind of thing I need to tell Go-fuzz, "Hey, I want strings." When you mutate, like I need a valid type string.

Super simple. This is something that go testing does not support, complex structures. So, if you see the second line there, what it accepts is actually a byte array. Um what this does is I use right now, Nosey ships with uh the open source version will ship with go fuzz utils from Trail of Bits. Um there's some reasons that, and I won't go through them, um unless I have time at the end, why uh that's what it ships with.

The biggest one though is that you see all of these fill errors, it will return if you don't have enough data. So, if I have like, a bunch of nested structures, and like you can imagine uh my function under test like needs basically like 2,000 bytes to fill all the data correctly, um it'll it'll say, "Hey, uh return. There's no issue or anything. Give me something larger." It doesn't really say give me something larger, but it it keeps letting the fuzzer mutate until it finds, you know, further paths.

And this is coverage guided, so the fuzzer will very quickly make it all the way to that last line. So, what happens here, um that last line is ACM import. It takes a context variable. Uh I need a valid context variable. Um I want to basically test import, but I need ACM.

This is a this is an actual like object. So, it doesn't just fuzz functions, it fuzzes meth like methods on receivers, which is Go's version of an object. So, methods on objects, we need a valid object created, we need the argument there, um that kind of stuff. Yet more complicated. Here, we actually have a constructor.

So, why make like an object and fill it with random data when there's custom constructors made? So, like, uh you know, these these large blobs for the EVM, um maybe like different peer structures, maybe like beacon blocks, things like this. We've already got constructors for them. So, like, why make them ourselves? We're going to get a bunch of like no pointer derefs and false positives.

So, what Nosey will do, and this is hugely inspired and borrowed from FCGen, Even like probably like 60% of the code for this type of interface is is borrowed from that. Uh but what we do here is we go look and say, "Hey, is there a function that returns this object and only this object that doesn't take that object as an input?" And, you know, it can either just be that object or that object and an error. And if that's the case, we say, "Oh, that is a constructor." Sometimes you see false positives for this, but in reality, um they're actually really good at still generating valid objects.

So, in this case, I need a new key manager. I didn't have to write new key manager. Some developer that made the new key manager the the key manager object wrote this constructor. That's what he uses. It takes, you know, this configuration deal and then whatever C1 is, a context variable.

So, what happens here is Noisy recognizes this, builds not only its own version, but a second function to fuzz this that relies on the constructor, and whichever one gets, you know, more coverage can find the bugs. Um so, right here we we basically find the constructor and we know how to see a function interface. So, we build everything we need for that constructor, and then we make the constructor hand us the object, and then we also provide everything for the function under test, which in this case is fetch validate something priv key. I can't really read from here. But this is just like a random example.

There's plenty to choose from. Uh one thing I forgot to mention, out of the 55,000 583,000 lines of code, there's over 15,000 functions that in the in those five repos, which is basically all of the dependencies for Mev Boost, Geth, and and um Prism that are supported. So, that's 15,000 functions that I can get coverage in. Uh that's coverage-guided fuzzing that I don't have to write the the harness for. That's kind of like the value out of this tool here.

Um And and yeah, notice here that like Noisy didn't only like create the valid arguments to fuzz here. It created the object and by doing that, um the way that it did that was it created the valid arguments for the constructor. So, everything that needs to happen here to try to get like as close to like a real test case as possible, we have. Mistakes learning, I'm going to go really fast. I got a minute and a half.

Version one, actually there's like a version .5. Shout out to Tyler Holmes, one of my teammates. He did a big CodeQL query for various things and we found like a bunch of stuff that just accept byte arrays. So, we wrote some Python that would like basically generate harnesses to fuzz those.

That was version .5. Then version one was in Python. This is just like to show you my pain. I'm grepping for regex here in Python.

All that gibberish, I felt like J.R.R. Tolkien like basically like writing Elvish or something. This was no good.

Trying to do this for for complex structures and stuff was like a total pain. So, I moved on to the AST objects that you get from the Go parser library. But you guys have seen this. This is kind of cool. Looks sort of pretty printed.

It still sucks. All that stuff is like you don't really know like this AST ident on the fourth fifth line that X's. You got to like do a type check on all of those. So, you end up with like this massive parsing thing that's got like, you know, a gazillion nested case statements. Still better than Python, but still really ugly.

Then I ran into FCGen uses the Go type libraries to write that like more complex function interface that I showed. Um I it blew my mind. I can't believe I wasted all my time for 6 months on these other things. So, I basically grabbed all of the code that works from there, threw it in here. It was a minimal rewrite for me.

Cool. Talked about why we use Docker. These fuzzers will find the Go binary and delete it. They will write all kinds of crazy stuff to your file system. If that happens, you can just restart Noisy and your host isn't screwed.

Let's see. Various fill libraries. I talked a little bit about trailerbits. I have a proprietary version uh that I'll talk a little bit more about here. Last slide that really matters, I know I'm out of time.

Uh things that we want to do, auto corporate bootstrap. So, you can imagine here, we already know how to dynamically cut right code, we can dynamically rewrite code. So, if I point it towards a repo and I say, "Hey, I want to run go-ethereum." Uh you know, let's say I support 7,500 functions in it, then what I can do here is I can say, "All of those 7,500 functions, instrument them. Run go normally.

Run it on mainnet if you want. Save off every valid call to all of those functions and and receivers, and then mutate on those. So, I can I can bootstrap a corporate that way. I can automatically fuzz in a separate go routine uh in a Docker image or something in real time. You can be like continuously fuzzing mutating on real valid uh test cases.

Um let's see. Auto object fuzzing, you can find race conditions this way. Uh you could say, "Hey, I know I support this constructor. It's got 10 methods on it. Uh write a fuzzing function that will kind of round robin those."

There's some work like this in FC gen that I'd like to copy as well. Um I think if you run that with the thread sanitization, you'll find a ton of race conditions that way. Um lockdown networking, you can do AST walk to say like, "Hey, I want to like look at all the reachability from this function. If it writes the file system, exclude it cuz I'm tired of something Something's pulverizing my file system and it's destroying my fuzzer." That kind of stuff.

Um final task, test case minimization at the end of a run and all the coverage analysis. This would be really great if I had this done today cuz then I could say, "Hey, look, you know, Prisms testing library has this much coverage and I added this much coverage automatically with Noisy Neighbor." That would be really cool. Maybe I'll have that in 6 months for you guys. All right.

Uh I will open source this in the next 24 hours. That's my promise to you. Uh my creative excuse as of all procrastinating engineers um has a creative excuse. I got bit by one of these snakes like 4 or 5 days ago. I wouldn't be here if my wife didn't like do so much to get me here.

Uh I spent a lot of time in the hospital. I've been elevating this foot, hence why I came up on crutches. I'm starting to be able to put weight on it, but yeah, I actually have like a real excuse this time. The dog didn't eat my homework. One of those guys like legit bit me.

It was a a whole thing. Follow Infosec on GitHub or Twitter. I'll drop the repo links probably later tonight or latest like this time tomorrow depending on how the rest of the day goes. Real quick, I do want to thank SCGen, Trail of Bits for that Phil repo, Zinshada and and Justin Traglia for various things that they added to to this repo and to this project. The Go-Fuzz folks and then everyone in the Go-Fuzz Slack that's been like super helpful.

Any questions? Yeah, with that David, I'm going to say you can go ahead and take questions over to the side, but thank you so much for A making it here through all of those different hurdles and for giving us your great presentation. So thank you so much. If you do have any questions for David, please feel free.