# Debugging data for Ethereum – ethdebug/format Overview and Project status

- Speakers: [g. nick // gnidan](https://streameth.org/speakers/g-nick-gnidan)
- Channel: [Devcon 7 SEA](https://streameth.org/devcon_7_sea)
- Date: 2024-11-13
- Duration: 24:22
- Watch: https://streameth.org/watch/67347e409dbb7a90e1a811ca

## Description

Building debuggers for EVM languages is challenging, time-consuming, and brittle because compilers do not provide enough information to enable robust tooling. The **ethdebug format** project, sponsored by Solidity, seeks to address this concern by designing a standards-track collection of schemas for expressing high-level language semantics in connection with low-level machine code.

Please attend this talk to learn about the status of this effort and a brief overview of its components.

## About the speakers

### g. nick // gnidan

Nick (gnidan) leads the ethdebug format project, a Solidity-sponsored initiative to standardize debugging data for Ethereum smart contracts. With extensive experience building advanced Ethereum developer tooling, including his past work leading the Truffle suite of tools, Nick is passionate about improving the smart contract development experience. He frequently speaks at Ethereum conferences, sharing his expertise and vision for more robust debugging capabilities in the blockchain ecosystem.


## Transcript

It was about a year ago that I was at the Solidity Summit in Istanbul and introducing the work that was just getting started to build a debugging data format. And so I figured it would be appropriate to come back a year later and report my progress and hopefully disseminate information about what I've been working on in hopes of getting people to adopt the standard. I'm going to try and get to the interesting technical stuff, but first I'll do some high-level overview. We'll probably run out of time, but I'll make sure to leave room for questions. Yeah, just to get into kind of the high-level, go over, you know, what is this project? What is it for? Why is it important? Making debuggers is hard and brittle, and it takes a lot of guesswork. Like, for... I don't know if you've ever looked at EVM bytecode before, but it is entirely opaque. And even if you were to peel back the layers of opaqueness, it would still be confusing. And so, I mean, of course, this isn't a problem just with Ethereum and smart contract computing environments, but also for traditional computing environments. But on traditional computing, what they do is they have the compilers provide more information, and that has worked very well since the 80s. We have debuggers. Debuggers, even in the last 10 years, are really quite impressive these days. But for smart contracts, the question is a little bit less, or it's a little more non-trivial. Like, what information do you actually need the compiler to output, and how should that information be structured for this, like, our unique computing paradigm? So let me introduce the project, maybe not get ahead of myself. So EthDebug, the EthDebug format is an effort to build a debugging data format. A debugging data format is the term that they use for the information that is output by a compiler that enables debuggers to exist, at least to exist without spending way too much money building them. So that's what the EthDebug format seeks to achieve. We are funded by the EF right now, and are proudly part of the Argo collective spin out. You should all go to stage two at 5 p.m. to hear more about that. That's the last thing I'm going to shill apart from my own work. I get this lot I put on there. Fdebug is not building a debugger. We're building standards. We might build a debugger in the future, but debuggers already exist. I just want to make it a little bit easier and cheaper to build a debugger. So who am I? Well, I'm Nick. Nice to meet you all. Thank you for coming. I'm the lead for the EthDebug project. I am a member of the Argo Collective. I have built a Solidity debugger, and then I oversaw it for five years and it is a nightmare. I have worked on projects that are now ancient history and I just kind of work in the dev tooling space. Stuff interests me quite a bit. So just to try and churn through these non-technical things. Project goals. We want to make a universal format. I don't want a slitted debugger only. I want a slitted debugger that is also a Viper debugger, that is also a Fee debugger, a Huff debugger. And I don't just want to target the use case of, I am a software engineer that needs to figure out why my code went wrong. I think it is also important that we target use cases such as, why did someone else's code go wrong and lose a billion dollars? So things like auditing tools, analysis tools are very important, and I would lump them into that same kind of category. On that topic, it's very important for Ethereum and smart contract platforms that we target not just debugging in development, but as I'm sure you all can understand, it is extremely important to be able to understand why a billion dollars disappears because of software. So that's a little bit more challenging than your GNU debugger and your local build of your C project or what have you. And because this stuff is hard, it's clear to me that we have to make it a goal to really optimize for adoption, right? I have the solidity team in the room here supporting me in this talk. And I feel very bad for the work that they will have to do to support a project like the fdebug format. And on the debugger side, it is only marginally easier. But again, it is orders of magnitude cheaper and simpler to take an approach like the one we want than debugging today. Ultimately, the idea is we want to lower the cost of understanding the blockchain. We want it to look at the EVM and understand what's going on or have a dozen tools to choose from that are all reliable and are well architected, right? Like this is a future that would be nice. It's a future that you know, it's like a present and like your x86 architectures with your traditional languages. And so I hope that we can move in that direction for smart contracts. So this is a so this is a series of steps that I have followed many times and I would imagine that a significant number of the people in this room have also done this where they need to understand how solidity works some strange feature like how do mappings get stored in solidity or strings or what have you and many times the documentation is thorough and many times it is like even the Solidity engineers don't exactly know how modifiers work all the time, stuff like that. It's quite bizarre. So what do you do if you're trying to do some blockchain like smart contract analysis? Well, you sit down with Sol C and a bunch of input examples and you compare it to the output examples and you scratch your head and you think, is this how Solidity does it? compare it to the output examples and you scratch your head and you think, is this how Solidity does it? I think this is how Solidity does it. And then you go and write your code and you implement it and you start like slurping blockchain transactions and figuring out what's going on and then you realize that, oh, actually in 0.8.6 there was a change. But turns out I was just wrong actually also. And then you go back and you spend another few weeks trying to make sense of how Solidity behaves and how your implementation should behave. And, you know, this is kind of annoying because people do these steps for the exact same problem. Like finding strings in storage. Like many people have done that same thing. And why? Why do you have to repeat the work? So, that's just solidity. People don't even try to the same degree with Viper. Yeah, there are some Viper tools, but people spend a lot of time, I would argue, wasting time making sense of solidity. And there just isn't that for Viper. But you know, Viper also is responsible for billions of dollars of assets. So I think this is a pretty significant concern that we cannot ignore. Here's an example. This one's at least documented. Like the Solidity docs are quite clear on how this works, but it's still very bizarre. Solidity has two different ways of storing strings. If the string fits in 31 bytes or less, it goes into one word. If it's longer than 31 bytes, it goes into at least one word, or at least two words. And those words are not consecutive, right? So, like, in the first example, it's what did I have, slot zero. And then the second example is slot one followed by a catch-hack hash. And then you start counting, you know, incrementing using the catch-hack hash. This is very weird. How does it work? Well, that last byte at the end tells us two things. It tells us the length of the string and whether or not that length is longer than 31 bytes. And, well, the second point is whether or not it's odd or even. I don't necessarily need to go into the examples. You can look this up in the docs or you can see the gist. But it's weird. And this is not the only weird thing that EVM languages do. So how do you accommodate that when the last 30, 40 years of debugger development has assumed that people are organizing data structures with normal ways of organizing data structures. We don't have that privilege. So here we are. Shall we get into it? Here's kind of the model that we've been working with. There's fdebugformat, and there are compilers and debuggers, and there's some interaction between the three. And I won't assume that the people in this room all know how compilers work, so you can imagine, well, there's source code that goes into the compiler in a high-level language, and there is machine code that comes out of the compiler in a low-level language. And in between, there is a long pipeline of many different steps, some very complex, some straightforward. The compiler does things like when you make a function call, and the compiler has to convert that function call into machine code. Well, it generates at least two jumps, you know, one jump to enter the function, one jump to leave. Or if you are storing a struct or an array, the compiler has to take your single assignment statement and convert it into a series of individual word assignments. And in order to, what you're left with if you're looking at the EVM, like the raw running EVM, is you don't necessarily see any way to translate back, right? Who knows? You just moved a word. I don't know if it's part of a struct assignment or what. But if the compiler were able to keep track of every single transformation as it performs it, right? Oh, I have to convert a function call into a series of jumps. Well, if you annotate those jumps to say this was the start of the function call, this is the return of the function, well, and if the compiler can manage to preserve that information all the way to the end of the compilation pipeline and produce a nice JSON object, well, then the debugger can read that information, observe the running state of the EVM, and basically see, oh, we're executing this instruction. What has the compiler told me about this instruction? Oh, it is a jump that is part of a function call. So with that, debuggers can actually make sense, like a coherent mental model of this fictional high-level world that is lost once the compiler is done. And the idea is hopefully this should be sufficient for debugging missing billions of dollars in optimized code. We do have challenges. The first significant challenge that has been on our mind is the fact that when you debug something, you are debugging runtime. And unfortunately, the compiler is not capable of predicting the future and knowing what runtime will be. So if you want to allocate an array in memory, you have to put it in memory. Where does it go? The compiler cannot guess what address in memory that array will have at compile time, because maybe you are allocating five arrays or N arrays. So we are limited in that the compiler can only know so much, and there are these gaps that we have to take these runtime observations. Like, okay, how do I take the running EVM, pair it with this compile time information, and produce a cogent model of the high level world? That one's not too bad. Maybe. Of course, optimizers make it even more complicated, right, when you have techniques like bytecode deduplication, right? I mean, optimizers are very important for smart contracts because you have to pay for every operation and it is significantly more expensive than most computers. So if you're writing a compiler, you might be tempted to say, well, oh, this series of source instructions, like this source code is very similar in its output to this other part of the source code, so maybe I can reuse the bytecode for those two same pieces, like the two different source ranges that, like, you know, maybe they're in two different files, maybe they're in two different functions, whatever, but they would produce the same exact bytecode, and the compiler would jump into that bytecode in either situation. So how do you annotate this low-level information? Suddenly you're going from, well, this instruction didn't necessarily just come from here. It came from either here or here. So we are effectively stuck with this disambiguation situation where a debugger will not be able to necessarily understand with full precision what the debug information is saying, but the goal is to ensure at least accuracy with that. Maybe we can't be fully precise, but we can at least target not being wrong. And so it's like, hopefully, if you have a block of bytecode that corresponds to two different originating sources, the debugger will have other information in its internal state to be able to say, oh, okay, well, I know that we were in this state, so we must be following this code path. And then another time when we are in this state, we must be following the other code path. What do we have so far? Well, we have so far? Well, we have a bunch of JSON schemas. I would estimate probably about 60% of our, like, working data model is implemented in schemas. And the 40% is a lot left. But at least we, you know, have been working pretty hard on it. Like, we have examples for every schema. Those examples are all tested. You know, I don't want to have to fix things that I didn't realize were broken in the future. And on the adoption concern, I do want people to use this format. I'm glad you're all in this room listening to me because maybe I will convince you to go and implement this. If you do, there are explainer documentation sets for all of the schemas. And we have right now at least one reference implementation for what is the most tricky schema so far. But the idea is that I think it's very important that we build reference implementations for all of the schemas, hopefully on both the debugger side and the compiler side. So maybe you're, maybe you're inventing a new language or something, and you can just, you go to the Fdebug website and see, okay, this is how an example compiler might implement this stuff. I will copy that, or I'm building a debugger. I go on the website, I see, oh, this is how a debugger might implement this technique. And, you know, I really just want to make this as easy as possible. These are the schemas that are furthest along. I'm running out of time. All right. Well, I'll have to run through it. So we have a program schema which represents a single piece of byte code and it's structured as a list of instructions with annotations for each instruction. We have a type schema for describing variable types. Hopefully that's self-explanatory. We have a type schema for describing variable types. Hopefully that's self-explanatory. And then the most complex slash complete schema so far is the pointer schema, which is to describe data allocation strategies from a high-level language. I'll go into a bit more detail on those. So the program schema here, if you want the direct link to the explainer documentation for that, this scheme is quite incomplete, but the foundation is there already. The key concepts of the program schema are it's one program is one bytecode. Programs have a list of instructions or instruction annotations. Those annotations describe a high-level context and allow the debugger to basically have a lookup table. When it reaches an instruction in the bytecode, it can go and consult the debug data and see what is happening with that, ultimately informing the high-level state. This is an example. It's like the offset's the program counter. I think with EOF, they're not really calling it program counter. It means something different. So I just called it offset. You can see the operation that that instruction is doing. You might recognize this as the start of every single solidity contract. In this example's case, I also added a source range so you can kind of see, oh, this is the corresponding line of code. And it looks like in this example, we already have a storage variable available to us. So that's, I got the first instruction. We can read X, which is of type string, and it's in storage slot zero. I'll get into the type and pointer schemas in two slides, probably. First, I have to give you this warning. Yeah, we haven't tested a lot of our assumptions here. Just full disclosure. But we think it's viable. But stay tuned. You might see me very happy or sad next EvCon. All right. The type scheme is pretty straightforward. You have to just describe types. So we got known and unknown types. There's like uint strings. All the normal ones are supported by the format, but we also allow compilers to define custom types if they have, you know, some advanced type systems. From there, types are divided into either elementary or complex. We do need algebraic types, but they don't exist yet in this schema. Elementary types are just like a single thing thing and complex types contain at least one other type. Types, you know, to deduplicate representations, like there's a lot of data that we're talking about, we allow types to be referenced by ID. If you don't want to just copy long descriptions of arrays or structs over and over, you can just give them an ID. And of course if it's a user defined type, it can point back to the originating source. Here's a couple examples. It's pretty straightforward. On the left is an elementary type for a fixed point number and then we have an array of uints as a complex type. There's more examples on the website if you want to find them. This is really what I want to talk about is the pointer schema. This is I think quite interesting and suggests that this approach might be viable. We need to represent where variables live. I should probably get this over with. So yeah, you guys can probably just go. I should probably get this over with. Yeah, so... Yeah, I mean, you guys can probably just go. Unless you like this kind of insanity, then please stay. But yeah, so we kind of put Lambda calculus in JSON schema. And that's because of the compile time constraint. Which is, you know, the compiler doesn't know where something's going to be before it exists. Right? So, basically, we get an expression syntax with this schema to describe, like, where is that string storage? How do I know, based on the state of the machine, whether or not the string is short or long? Let me show you an example. See? Oh, yeah, here's the example, actually. Yeah, don't look at this slide. Just grab the QR code. I did document this. There are comments there. But you can kind of see at the very top, this string storage, this pointer, starts with a storage slot, slot zero, and then we have a group of other data regions and variables. I'm really running out of time. Maybe I'll just ask for questions now. You can see, is there a laser pointer on this? Well, I can just point. I really can't. So you have groups of... Basically, the way the pointer schema works is you have regions and collections. Regions are a single continuous range of bytes, and then collections are this abstract thing. You can have groups of regions. You can name the regions. You can have conditionals. You can do things like check the remainder. Is it odd? Is it even? If it's even, we know it's a short string, so do this. Otherwise, we know it's a long string, so do that. Remember the example from before? Great. All right. I swear it's less bad than you think. I mean, the idea is that these are actually static representations, so the compiler will only need to output this once, fortunately. And then instead of zero, it would just be a template variable, and then the compiler can say, use this pointer with this variable in place of string storage contract variable slot. And this way, any supporting compiler can just output, like have a library of these pointer templates and just output them whenever they're necessary. And if they're not necessary, it can omit them. Yeah. Apprehensive. But we tested it end to end. I put a string into a Solidity contract in an automated test. And I step through the EVM and I observe after every string assignment that the value changes and matches the expectation. So, if you want to see how it works, you can look at the implementation guide. Hopefully it's coherent. All right. Anyway, there's a lot of work to be done. Feeling very good about it. You know, I think this is a solvable problem. If you want to learn more, please join our Matrix chat. We meet every other week. You're welcome to join the calls. You can also just watch the repo. Ooh, we got questions. Yeah, thank you. Exclude the optimizer. Yeah, so that's the approach we're taking. Will these get recorded? I should probably repeat them. The question is, can we exclude the optimizer and just focus the debugger at compiler level? Optimizer should not be altering code functionality. Unfortunately, I think it does alter the code functionality, but the approach that we've been taking so far is to start with just unoptimized code while thinking about what do we do about the optimizer and making sure that we're not painting ourselves into a corner when we do want to support optimized code. When is F2Bug format done? After DevCon. Others? This is so impersonal. Biggest blocker? I mean, the tricky thing with work like this is that it's periods of challenging, like, thinking, figuring out, like, how can I make this meet the requirements, followed by months of tediously documenting. And so it's like there could be blockers in either of those, and in both situations the blockers are quite different. Can the debugger identify the function arguments if the format is followed? Yes, that is the idea, right? So if I didn't have only a 25-minute talk, I would have bombarded you all with way more examples. They are on the website. You can kind of see. If you look at the program schema, I actually handcrafted a pretend high-level language source and the corresponding opcode output. So you can kind of take a look at what that looks like. How do I scroll down? It looks like there's more. Any plans on decompiling deployed by code with no original source? I mean, this is a cool thing. It's not really my area. I know there are decompilers that exist. I just haven't thought about that too much. Any more? Well, thank you for your time, everybody.