EVM Tracing in Geth by Sina Mahmoodi | Devcon Bogotá

foreign [Music] this is going to be a workshop so I will be working with the terminal I will show you some code and show you how to use gas like I guess tracing feature that's why we specifically needed my laptop so yeah sorry for for having to wait um also yeah because I will be showing code it would be great if you guys could come forward if you want to see what's happening because I imagine the phones will be small so yeah I'm going to give you a bit of time to come forward and uh while you're doing that I want to talk a bit something a bit personal uh so I come from Iran I don't know if you heard but there's some stuff happening there uh I just wanted to give a quick shout out to the people who are going to the streets they're they're purchasing for their civil liberties for freedom of speech um so yeah there's been three weeks of protests now internet has been disconnected um and it all started when the girl on the left who is 22 years old uh was in custody because of showing a bit of hair and she somehow happened to die in custody and this outraged people who went on the streets and during this protest uh close to 200 people have died among them Nica who was 16 years old and by like when running away from the police yeah she also died so yeah just want to give a quick shout out to them they're risking their lives asking for their rights and for for freedom but this is a technical topic and I don't want to waste your time anymore so let's get straight to it so what is uh tracing what would you need tracing um I just want to show you an example of this is etherscan uh you everybody knows etherscan right and the internal transaction section of etherscan is basically a trace of that transaction it shows you what happened within that transaction every call that happened there uh who called who and and so on and this was a this was actually probably not from a guest mode but I'm guessing from open ethereum or Aragon but we have a very uh we have a similar feature to this and this is not the only use case um but there's many of them just to name a few like I just just before coming here I was talking to somebody who's developing a zkevm who are also heavily dependent on the tracing feature of gas I've seen talks about from me researchers who are using tracing features and many more yeah this is also another tab in the etherscan uh the details where it shows you the the state like the state difference uh we also have as we will see later this is also something that you can get from get so before we get to in actually we get into tracing um you know that you can use guests to execute a call on top of a given state so you probably know East call you give it a set of parameters like as if it would be as if you want to send a transaction to the network but this is actually not sent to the network it's just executed locally and you see the result uh it's a simulation uh I actually you have to give me a bit of time because I had to restart my laptop and I lost all my setup I have to bring up the terminal because I wanted to show you how this actually goes so yeah I have a I have a girly note on my machine that is a kind of thing to uh to the network uh okay yeah is it behind me yeah maybe I move it here all right so um let's do a call wait I have them all stored in a file so I don't have to type it out because anyway I'm losing too much on time on this so I'm just gonna go I already lost a lot of time and you all know how eight call goes okay let's let's move on to the next one um I'm just gonna explain to you so basically uh this is the I'm sending um I'm sending like I'm assimilating a call to the wrapped eth contract on Gurley to get the balance of of an account of account zero right uh this will only give you the result like in this case the result is as zero like it shouldn't be I have to figure this out like I think I I have the wrong address here um like this address the the zero address should have some balance but anyway what I wanted to show is that when you when you uh do a scroll you only get the execution result what is returned from the contract back but this is oftentimes not enough information we want usually more information we want to see what happened inside the transaction right and that you can do by doing the same thing but using a debug method Trace call which will simulate the call for you and trace it at the same time I would give you more information so if I were to run this now I think this is the mainnet contract I was using um I was going to present with mainnet node but then I found out the Wi-Fi here is not so good so I can't SSH to my server here we can already see that there is a lot more information as it basically shows you all the steps that happened during the execution of this transaction yeah there is going to be a lot of steps I'm not going to go to show you all of them but basically you you will see okay like what is the first op code what is the programming Contour at that point which opcode was executed how much gas do we have at this point like what is the storage at this point and so on so it will give you almost the the full information that the evm has uh when it runs uh through the transaction you can ask questions sure this stack is before the instruction yeah so all of these information are from before we execute the opcode here I showed you in this example I showed you like tracing a call of my own craft like basically I said Okay I want to like do from to input and so on but of course uh guest supports tracing existing transactions the one that were already mined on the Chain um so historical transactions so to say uh those you can do via these endpoints you have a debug Trace transaction that traces a historical transaction you just give it a TX hash you can also do it at the Block level you can trace a Blog by using twice block by number or hash or you can do even a whole range of blocks by using a trace chain but Trace chain mind you is a bit different it's not a simple like it's not using the simple Json RPC request you have to it's a bit based on the subscription API so you have to use websocket for it so the usage of it is a bit different I'm not going to show you here but yeah it's cool for when you need to run over a range of blocks what what we saw um in the previous Trace was the default Tracer it's basically the opcode Tracer like it shows you every step but sometimes you need uh information at a different abstraction level like for example as we saw in the etheroscan screenshot you want to see what calls happen or like you want to get different information and get has a bunch of built-in tracers that you can just call them by name and and get the information you want so here like in this section I want to show you uh three of them the first one being the one that we saw I'm just gonna like here in this table you will see all the information that you will get from the opcode Tracer um which is yeah I'm gonna give you a second if you want to see what's in there but yeah basically you get the optical information all the gas related information like how much this up code cost to execute or will cost to execute how much gas there's left the the whole memory snapshot uh of that transaction the the stock return data of the last call like when a call finishes in The Next Step you will get the return data for it uh storages for the storage slots for the contract the depth of the execution refund and if there was any error but please note that if you want to use this Tracer there are some things to note as I said memory stuck as storage and return data these are Dynamic values and they can grow large especially uh memory you have to watch out for memory because uh mainnet transaction these days are very heavy uh so like if you have memory enabled it can kill the node basically like yeah somebody for one of our users reported that they try to trace a Blog with uh on a server with a 64 gigabytes of memory and it crashed basically but because of this um the the Tracer accepts options to disable all these features so all of the ones that you will see in this list they can be disabled some of them are enabled by default some are not but they're all like you can toggle all of them also if you if you really need to trace uh uh you you need a memory let's say and Json RPC is not sufficient for it and you control the node then there's an endpoint a standard Trace block to file which will trace the block and save it on a file alongside the node that you can uh yeah you can easily access so it's not like a Json RPC request the next one is the call Tracer this is arguably one of the more like most interesting tracers uh it will basically give you all the call information like all the internal calls that happened during the execution I'm going to show you in in the previous example like like basically when we call wrapped ether balance of uh what information we will get in return so you can see um that's like balance off is a simple call it's not going to call anything else and there's there's one call it shows you who is the sender uh who is the recipient who's the contract what is it call or is it create uh was there any value transfer and what was the call data what was the input but yeah this is a more like boring example we can um I hope these transaction hashes are okay I feel like the file wasn't saved yeah okay it works um so I'm just going to run it show you guys so here I'm I'm trying like I'm not doing Trace call but I'm using a historical transaction that was mined on Gurley so I'm using Trace transaction endpoint and this is the transaction hash and here like as an option you can specify you can specify what which Tracer you want to use so if you don't provide anything then it will default to the opcode Tracer but you can say I want to use the call Tracer as we did here and you will get a bunch of information uh yeah so this is this is going to be the main call and notice that we can already see like this call uh errored like there it reverted we can also see from since recently we can see the revert reason which is system time outdated this is the solidity revert error that was returned the input output and then here from here we have the nest that calls so basically the the the main contract did some uh called another one that we can see here which which also reverted and like up there you can see like there is um there's more nested calls but that it's not being displayed here but you can you can get that information also so yeah call Tracer you would use when you need information about the internal calls uh it also since recently it accepts a Nifty option so I'm gonna provide here like a you can you can basically configure the tracers you can say like I want only top call um this is this is just useful if you need only the top call information like only if you want to know okay why did this transaction revert then I don't need to see all the nested call information I just get the top call and I check the revert reason and I know okay there was a time outdated or something this one is the pre-state Tracer it has two modes of execution by default when you just you just say I want to like trace this with the preset Tracer then it will give you all the accounts that are needed to execute uh the call or the transaction you can think of it uh something similar to access list or Witnesses so if you somehow need to simulate a transaction and don't store the whole state you can get the state via the Free State tracer and then you can execute your transaction locally and in in the second mode you can see all the state state modifications that happen during during the transaction basically you can say you can see okay the balance of this account increased to such value the the storage slots changed such and such if if we were to see it in see it live it's this one so here I'm um I need to change the address I'm gonna execute the same wrapped ether balance of call and pass the to see the pre-state and yeah as you can see like the zero Atlas has like shows the balance of the zero address this is the the contract itself has a balance the code uh nouns and like the all the storage slots that were needed to execute these transactions these are not all the storage laws of this contract but only those that were required here and let's say you wanted to see the the div you just pass the parameter div mode and yeah basically we can see like the biggest change here is that the non so you have like a pre and post State and you you have to compare them so you can see here like we have the zero address the nonce is emitted because it's zero so nouns was zero and now it's one so this is the only uh the only change that happened in the state during this transaction but of course if we were to run it with a more complicated transaction you would see yeah you see like there's going to be much more stuff this is uh yeah and this is what parts of the state changed during this particular transaction like here we had also balance updates changed and balance of zero account change and yeah here we have like um the format works as follows basically whenever something is created let's say a contract is created it will not show up in the pre-object it will only show in the post object and whenever something is deleted it will only show up in the pre-object but not in the post object so by by comparing these two objects you can you can see um how the how the state was modified uh so now I want to talk a bit about how gas stores stage because it is very relevant when you're doing tracing like if you've been been tracing you've probably saw this error required historical State unavailable specifically when you want to trace a historical transaction one that was mined uh now now why is this so first of all how do we prepare the state for simulating a transaction or re-running a transaction it's basically like you have to we have to find okay the transaction in Block n so we need we will fetch the state for Block n minus one the pro state of block n minus one and start executing all of the transactions within the target block until we reach the transaction that we want at this point we have the transaction we have the state uh the pre-state basically for our transaction and we can we can execute it but what happens if the state for this block n minus 1 is not available in the database that basically that's hint that's what when you get this error so what uh so now comes the question the state would be which blocks are actually persisted um let's get the archive node out of the picture because it's the easiest so for archive node you have all the states next one is full sync by forcing I specifically mean that you start executing all of the blocks from Genesis up until they had here um it's worth to note that always guest stores the latest 128 blocks the state for them in memory so you always have like while your node is thinking you always have the state for the last 128 blocks now anything beyond that um is stored on like it's persisted to this only periodically and that is roughly every two hours so every two hours um the state for a block will be persisted to disk and because we think from Genesis that means we have the state of a block roughly every two hours from Genesis on until the more recent ones and the difference between full sync and snapsync is that in synopsis you won't actually start you won't actually execute the transaction you want to actually execute the blocks from the beginning but from some pivot point so let's say here is the point when you um start like you fetch the state from the from the network and you start executing transactions so that's why we only have the checkpoints from this point on and like so we basically we cannot execute any transaction that that happened before we we did the snap sync so it's important to know like based on your use case um which think mode you should go for and now we can complete the picture again so how do we prepare the state for for executing a transaction we fetch the state of the parent block from the database if it's not available what do we do death will go back in the blocks for for a number of logs and checks if it has the state available for any of them uh in on disk and if so then it will basically re-execute all of those blocks and prepare the state for your transaction so here you can see in the error who we have we have the error and we have some more information like re-exec equals 128. this means that guests tried the 100 went 128 blocks backwards to find some State on disk that it could use to to go forward but it didn't find any and this is a this is a um this is a parameter that you can give so basically when when you do Trace transaction you can provide this re-exact parameter and say okay I'm willing to go back two thousand or two million blocks to find a state that I can use to uh to execute uh this this transaction but of course the more you're willing to go the more you go back the longer it's going to take because all of those blocks have to be re-executed to to compute this state we have uh there's a there's a method that you can use to see States for which blocks are actually stored on disk and that's debug gets accessible state so let's say like I want to see I want to see like five five thousand blocks behind the head do I have any any state there Computing no State found so I don't have anything between the last five thousand and four thousand blocks okay I I can try again or I can enlarge in this the search parameters it's probably also going to say no oh actually no it found it so here it says like I have there's this block that I have the state for so I know that uh block number yeah so I know that I have the state for like 8 000 blocks ago basically so yeah this is this is the Nifty method for finding um which which states you have on your disk and then when when you when you know this then you can set the re-exec parameter accordingly and sometimes there is a range of blocks that you care about a lot like you you need a lot so you want to store the whole state for that what you can do is basically stop Geth add this parameter GC mode archive and and run for as long as you need and then stop it again and take it off because when this flag is there then guests will store every state like the state for every block on disk we went over the some of the tracers that are that come as like in in stock gas but that's not your only option you can also uh you can you can write a custom Tracer like you can basically uh get like there is some hooks that you can use to collect the relevant information for your use case and only that you don't have to rely on the the tracers that we provide and here I want to show you an example I um doing devconnect in Amsterdam I I met this guy who was an Emmy researcher and he was telling me that he found the tracing feature of Geth extremely useful and later on I saw his talk and he was talking about this so uh apparently there is there are some poisonous tokens out there or there used to be at some points that that wrecked basically uh Searchers like the automatic Bots and one of one of the those tokens kind of looks like this this I think this is a simple one it's not a real real one but basically it just checks uh to see if you're in a development environment then it does the the actual transfer of value but once you so let's say you stimulate that uh in on hard hat and you see like yeah I got some money but when you run it on mainnet then you actually don't get that money and you probably lose some so that's that's the gist of the token and like he he wrote a like a simple Tracer that can detect this uh I'm gonna show you that this is how a custom JavaScript Tracer looks like it's a it's a very simple one you have a few methods that you can Implement in a JavaScript object and you will pass this object to get when you when you invoke the API in this case we implement the step method which is run for every step of the execution and we Implement also the result function which is executed once at the end of the transaction so basically we want to the the idea of of the detection algorithm is that token transfer shouldn't use any of the following up codes coinbase number difficulty origin these are all block metadata right why when you're doing your erc20 transfer why would you need any of these op codes so this is like a very simple detection thing so what what I'm doing is basically like I have a global variable called fishy and as I'm going through the op code as I'm going through the steps I check that if any of these op codes appear then I'm gonna say okay there's small coming out of this I don't want to do this trade so I just set that variable to true and return it and now like here is my I you can't see that but basically the result is true so what so I uh I I run the Tracer over a contract uh and the result was true but let's let's dive this is this is probably the most part interesting part of the there was a few things here right so I'm I'm stimulating I'm simulating a call uh from and to to an address but note that this address is actually not um is empty there's no like there's no contract deployed on Gurley for this address uh so I'm going to introduce to you in a very Nifty feature and that's called state overrides right um basically when when you when you simulate a call using IF dot call or debug.trace.com you can give a bunch of State modifications to to get like temporary and you say okay apply these State overrides temporarily for stimulating this transaction so here what I did was I I have the byte code of of the contract the poisonous token this is the bytecode and here I say okay State override in this address please put this contract code so this country code will not be deployed anywhere but only when I'm simulating this transaction it will be as if this address has this code so this is very useful like if you're simulating stuff and and the Tracer object I'm passing also separately here so what happens is basically yeah there's a call to transfer to this contract and my custom Tracer detects like says that it's a it's official call because it uses the coinbase opcode now the JavaScript Tracer is um is what most of people know about it's not the only option for writing a custom tracer since a while ago we also have another option namely you can write your tracers in go this is specially relevant if performance is is a concern to you the the go JavaScript overhead is a lot so when when you want to do tracing over a long range of blocks and it just takes a long time but now you can actually write your tracers in go and it's pretty simple how it works I have I have one here that I'm gonna like walk through a bit but I just want to show you how you can first I want to show you how you can write one so basically you write your tracer you add this this is a very important part this is how it how guests will know that there's a tracer here to be found and you drop this file in a directory in yes so we have this directory is tracers native you simply have to drop your goal file here do the registration and you're good to go and of course your your struct has to adhere to an interface and this is the interface that you will see so the methods are capture start this is at the at the beginning of of the execution uh you have captured state which is run for every every step of execution you have enter we didn't see this before enter is basically a hook that is that is called every time we enter a new call frame so when there's a call or a create we enter a new code frame this uh this one is called and exit is when we leave the call frame and you have fault and and the result so the result is when basically what you return and what the Tracer will return when you call the method uh yeah it's a bit tight time wise but I'm just going to give a short overview so what this Tracer does is that it will tell me the method signature for all of the all of the functions that were called so as you as you probably know uh solidity functions like when you hash their signatures the four bytes that's how you actually invoke them at the evm level right so like when you when you um and I want to just collect uh the this this four bytes basically like the signature of the methods that were called so this is the gist of the logic uh uh I checked that we're entering a new call frame I check that is this uh it has to be a call or a delegate core statical some kind of call otherwise just return that's not important also we don't want to collect pre-compiled information so they're also written if in otherwise take the four first four bytes of the call data and this will be my signature and this these are stored in some object here uh yeah I don't have more time to go through this so that's how you write a tracer and I want to quickly mention uh what's coming next on this front um so you saw the trace call method I'm planning to work on the trace multi-call this is when you will have a list of call Objects they will be executed sequentially maintaining the intermediate State as they go to the next call and the other thing is the trace namespace or as other people refer to it as parity tracing so these are the things on the roadmap that have been worked on right now and on the last seconds there was a that was it basically thank you guys [Applause] hi most of these will probably be available in the docs but um does trace to file exist for transactions only rather than just blocks uh no it's I think what we have is only four blocks now got it uh is riazek re-exec available via Json RPC yeah so I didn't show it but basically when you do Trace transaction in the same object that you provide the Tracer name use you specify re-exec got it and does get accessible State return only the highest block number that has state or all it will return like in the range that you specify it will return the first state that it finds the first so the first one so if you want to get all of them then you have to iterate you have to narrow your range as you go and so if I did Genesis to now if you need Genesis to know then it will give you the first one like let's say it's block 10 and then you do another call from block 10 until now and so on hey um quick question on the um the state diff tracer so does it provide a kind of you know if there are multiple State changes in a transaction like at at the call level do you provide all of those State changes or is it just from the beginning to the end of the transaction what does the state diff I may have missed this oh it's yeah it's only from like basically throughout the whole transaction okay not you mean for code level yes uh you want the site to focus yeah no no uh it's all for all translate but this is something a custom Tracer but yeah you can you can definitely do that yeah yeah you can just check the code for the prescription it's in the same directory that I showed yeah uh I don't think it will be very hard to do yeah but thanks so much thank you