ETHDay - Querying and Indexing Smart Contract Data - The Graph

foreign [Music] yeah my name is Simon I've worked for a company called eternode it's a one of the core devs that are building the graph protocol or the graph by itself and today we talk about querying and indexing smart contract data with the graph the graph is a very technical protocol um although heavily used so I think I take the time to explain a little bit the problem and then kind of the approach that the graph protocol takes so that problem and if you want to learn more about how it works there will be longer workshops at East Amsterdam where we actually do live coding and uh where you can jump into this but yeah let's get started all right so like Eva bailin the director of the graph Foundation One tweet and said like yeah if you build a tap for the community but you start to using centralized components for your VCS or whatever reason then it's no longer a debt so you actually need to kind of put that d off I mean it's okay to do this but if we really talk about decentralization then like the whole stack needs to be decentralized otherwise the weakest link will kind of kill your product um so let's quickly dive into it with a very interesting use case so you'll probably all know about nfts and smart contracts on ethereum and how this works and but like let's look at the crypto punks example so we have uh the these are kind of numerated from zero to ten thousand or 9999 and then for every uh token you have one owner and every token kind of has this metadata URI and that's basically it that's the state that we know from the blockchain that we can query the blockchain about the crypto punks nft holders but when you look at this and think like yeah I mean like it's interesting to know who owns Which nft now but we are actually more interested in the so-called provenance so like who owned which token before and what was the price and this data although it is around it is kind of uh in the blockchain it's not easily retrievable so what you need to do is kind of you need to go into the blockchain and start to search for that data you can think about like when you when you send an nft around or if you sell the nft you actually submit a transaction to the blockchain which in the end kind of changes then this spreadsheet but the final state is still very uh just to tip the chain head and you also know that this transaction is stored Forever on the blockchain is kind of of a promise that we currently have so it should be around so what kind of example comes in mind is like the it's it's the time machine that we know from from Mac right so and also when you can also do this in in the Google spreadsheets you can see like the revision history and then you you start to work through the revision history and then you find out like who owned this nft before and in the example from the crypto punks also you know you will know exactly on chain what was the price that was paid so that's actually something very important but it's not something that the blockchain by itself was was designed to do so and this is actually the problem with data on the blockchain like the current stage is easy easily accessible but the historical state is kind of buried into this large thing if you think about ethereum if you want to run an ethereum archive node that actually has all the historical States stored then you need a machine with three terabyte of super super fast disks that you can actually store it and that's just a little bit too much so when we go back to the tweet from Eva from the beginning um so I think in the beginning when they came up with with this idea of tab or when they coined the name depth they thought about something like this so we have a user interface on top that can be distributed through ipfs or in the beginning it was a Moore's farm and now other kind of decentralized storage protocol coming to play because the front and in the end it's just like HTML CSS and JavaScript right and then the idea was that the front-end directly communicates to the blockchain in the best case people said like yeah you should run your own node on your laptop and you only communicate with your node I mean fast forward into kind of the easiness and laziness of all the users and also the success of the blockchain technologies that we kind of have now decentralized providers that provide us like in four hour Alchemy they provide us these and end points um that we can use but still like the premise or the idea the architecture should be it should be possible to run every dab like this um but we have actually this this architecture has a big problem because um the the blockchain is optimized and also incentivized for storing data it's not optimized for loading data so if you go there and you actually send an nft around you always pay a little bit of gas or actually you pay a lot of gas so that this act this happens so what you do if that gas actually you incentivize the miners to mine your transactions to put it into the blockchain and store that store it there forever so that you are sure like your nft is there and nobody can take it from you but now when you want to start to read from the blockchain there is no real I mean there are these Json RPC protocol where you can start to query for all stuff but like for the miners by themselves they do not have any incentive to actually mine you or give you that data because you don't pay right so they do it somehow or you have kind of this other solution providers that jumped in like in for our Alchemy kind of providing you this Json RPC interfaces but if you run a heavy app or yeah then it's not then it's not really a depth if you use them actually if you run a heavy thing then you have at some point you start to have very fast very high bills of those API calls and also because this this thing was not optimized for reading you end up with code like this I mean this is real world code actually it's a project that uh that came up and uh and and we worked with them and so what you see I mean if you know a bit about JavaScript then you know like each of these awaits they they do um they wait for something usually here you seek a contract or you have this these calls to this chasing RPC calls and let's say every change in RPC call takes 500 milliseconds to resolve now you see this for Loop and they they Loop through and they first get the contract and then they try to get the token ID and somehow sometimes they also try to get um like follow-up information about the token ID like the owner or whatever the metadata URI so for every 500 milliseconds that such a request takes so you have already three just for one token now imagine someone has 10 tokens in their wallet it is easily 15 seconds just spent by getting that data out of the Json RPC interface so that's actually a little bit uh I know not very satisfying so what should be the modern depth architecture is actually that you put in a indexing protocol or a data layer between the user interface and the blockchain so that you have like the data available in a way that is fast and snappy and like how users are are used to to travel the web if you open a Instagram or whatever kind of user interface a user experience that you're used to it's not like it's loading I don't know 10 seconds with hundreds of requests it's kind of you open it up bam it's there and you start scrolling sorry um so that that is why so so you have the user interface on top that stays and we have kind of this indexing protocol inserted here that is actually kind of solving that reading problem and the name below you have the blockchain but you're still directly interact with the blockchain also uh it's not only about ethereum it's also about the other other chains that that could be data sources like arbitrome optimism Salo um all uh what are all these names I don't know then also kind of these others Cosmos polka dot all these kind of blockchains are currently worked on to be integrated in inside of the graph as a data source like a lot of them are already done and others are kind of in the making um which all kind of in the end result in this very nice graphical query where you can have like this exact same data that we saw before plus even more extracted from the blockchain so that's very nice I mean me personally I love graphql I lost it before I learned about the graph so this is um this is how you should query for data in general so yes now we kind of saw the problem I already kind of set the graph somewhere in in here so let's dive a little bit deeper so on the website it's written like it's apis for a vibrant decentralized future like as I said it's about providing apis for those steps that we have kind of this user experience that will that we'll often want to see but also in a decentralized way currently the graph there is this hosted service which is a centralized kind of a proof of concept that was that is running for more than a year now and it's up to date serves more than one billion queries per day for multiple apps like like uni swap or enzyme or audios or M stable or whatever like there are tons of tons of projects building on onto the graph but this is just kind of this centralized proof of concept like a year ago rough a year ago they launched this decentralized indexing Network protocol so it's not only it's not just a centralized entity that runs it for you it's actually kind of now a network of of different indexers who all index the data for you and it's it's an incentive I think where you have like different roles and and the incentivization models uh kind of to talk for itself to really dive into how this is set up but for those that are interested into tokenomics I would say it's one of the most advanced kind of use cases of tokenomics and how you can set up something and incentivize the different actors the decentralized network a decentralized graph Network today has like more than 150 indexes worldwide so if you deploy your subgraph a concept that we go into deeper a little bit later to this network it will be indexed across the world from this individual indexes so your users will actually kind of directly create data from the index that's next to them and is able to provide the fastest response also because it's a decentralized network it's kind of an owner lesson permissionless Network everybody can become an indexer but also like even if the company that I work for etch node and all the other quartets kind of what that for whatever reason disappear or get censored or whatever like the graph Network it will run it will continue to run because it's basically a bunch of smart contracts on ethereum and then these these indexes which is work in this permissionless way also you have the redundancy like if you deploy a subgraph again to the network and one of the indexes goes down for whatever reason you're still the other 150 indexes around that are happy to serve decrease to your subgraph and the network also is kind of designed so that the query fees and everything is kind of is fast and affordable sure if you if you generate the billion queries a day you will still have a a high bill to pay um but but compared to to other services and also compared to what you actually receive um I think it's uh it is a fair price also there's it's still the market between the indexes so indexes compete with each other to provide the cheapest uh response so if one indexer just has kind of ridiculously high fees at some point it will not take anymore to serve any queries and yeah and the thing is it goes towards a global open API where everybody where all data is indexed and this is available for you at your fingertips and that in the end enables truly decentralized apps so I quickly talked about this concept before called the sub graph so what exactly is then a sub graph so in it in a nutshell the sub graph is code um that's not executed actually it's a on the top we see a database schema so you actually Define it's it's really a schema like how you do you want to store your data uh in in a database if you if you could because like the blockchain is not a database it's kind of just a collection of transactions basically so you define your schema then you define the mappings which is this gear below and then you say like for everything that happens on the blockchain for the events that you watch for I do this and that and then in the end store it into the database so it's kind of an instruction set that in the end the indexer will take it and execute and so they do index your sub graph so in the end kind of they follow the instruction set and if they index everything they kind of have this database ready and indexed for your users to use and as I said before like the blockchain by itself is not the database not a structured uh database it's more or less optimized for putting transactions into blocks and then on top of each other and if you look at the blockchain like all these protocols that are um interacting with the ethereum blockchain they are layered like every every line here is a transaction and they are just like on top of each other and even mixed so if you're just interested in for example what happened on maker maker dial like what which are the walls there which was the what was die price or type price is always the same but compared to if um then you go down and you just need to pull them out the ones that are interesting that's why it's so inefficient um and same for all the other protocols so what what would be nice actually is kind of exactly this that you say like for all these protocols I have one data storage and it's one defined data that can easily query with uh graphql API so that's that's how you you should think of it kind of starts to make starts to organize the data on the blockchain in a way that uh it makes sense for each of those protocols um and then this is more or less how it works so on the top you have the depths the interfaces that you already that we already know and then these on the right side you see it uh it sends transactions to the smart contracts and the smart contracts when they receive transaction when they execute those transactions they emit events and these events are then handled by the graph node according to what is stored into in the mappings that you see on the top left bottom left and then the mapping statements make a reason out of these events check also what's already in the database and then they start to update the database and put it into the store then on the other side where you see on the left with the queries so when you now you query you query directly graphql API which goes also into the graph node and gets the data out of the store and provides it to your um to your debt so this is kind of the the cycle of the graph node um we are already with questions but since I see uh I have much more time we can actually jump a little bit into how you can now start to interact with the graph um switch this so [Music] um foreign so for example there is this one subgraph the crypto punks that I showed you before um so you can go here it's kind of a Community member that built this subgraph and it indexes all the crypto punks with all the problems uh that it has I mean the crypto punks is actually interesting example because you have developed punks uh so so you never really know like where is it is it the punk rocked or not and where is the drop Punk and where is the other and when is the punk rocked and so these complexities on top of that uh the crypto punks contract has box so sometimes it doesn't emit the events that you would expect it emits and so on and so forth so this guy was actually able to um to to create one subgraph that are indexed is all in a cool way so if you go here this is still on the hosted service um you could switch here to the decentralized graph Explorer but uh for the sake of Simplicity for now we stay here you see the sub graph you also have a playground here at the bottom but I also show you actually kind of the recent the recent addition to the graph core devs uh the the group or the team called the guild they're very good with this graphql stuff and they created this or added this very nice graphic URL interface on top of it so what you can actually do here it's on the left you can open an Explorer and then you have like for that subgraph all the entities that you can query you can also read about here so you see queries we have like accounts or whatever punks and we can look into it which which Fields do they have but um let's think about something interesting and say like hmm let's look at the transfers so which are the last 10 punks that uh were transmitted or kind of that moved around so we would say I would order by timestamp or the direction descending and then you go to the the punk the ID token ID maybe also the image URL so we can look how it looks right let's let's do this oh the demo effect maybe that sub graph is not it's not finished I have to open all right let's do it on otherwise let's go with punks we can look at punks by themselves so let's go for the punks we say token ID the ID metadata we want to know the traits and also the image URI right okay so we have here oh maybe you want to know the owner so we get the owner also and maybe also only the first 10 so the query results a bit faster all right so we see the punk number zero um it has uh it has these traits I need to get for the trades actually the the ID so we know what it is so we have like the pump number one has a blonde pop an earring it's female and also green eyeshadow and also we see like there are 2450 85 ones with veneering and so that's not very rare but I think here the blonde bob already has some Rarity and here we can also um directly see like how does this Punk look like I didn't create this website see that's the problem with the centralized stuff but um yeah you get the point you get the point we can also see the owner but I can also say like okay let's go just into the first one and see for example like the previous owners let me check uh this is also another round the problem is if the sub graphs you really need to create them according to to your needs so um but but again with this subgraph because it's a community made one if you if you want to play around with it and there is information that you do not see that it just doesn't serve to you feel free to go here open up an issue and uh that they will hopefully react fast and kind of implement your need I actually just realize that I need to open some issues after that talk um but there is there is more to explore I mean like you can also see you can also see the sales for example the SE are usually interesting so they can also kind of order by maybe timestamp get the first 10. order action descending and then amount and uh this one let's see nah this makes more sense um although these are probably not correct but here we see we see this the sale prices and so we can start to inspect it uh you can see which phones have have uh which were sold for how much and when they're all sold and how did it move around and so on so forth so um even the the rops like we can create it from when was it Punk erupt when was it unwrapped and and how did it move even even erupt we they included this so um yeah and there are tons of other subcrafts that you can start to um uh explore I mean we have ens uni swap is heavily used by users also the usdc is probably interesting and actually kind of most big protocols or even smaller protocols have a subgraph here you can start to play around explore the data and then do whatever you want with it and if you want to learn about how to write subgraphs by yourself then come to one of our workshops there will be one that uh one at The Hacker House that the graphics co-sponsoring but also on Friday I think in the evening at Amsterdam I will give like a longer workshop and there will really do a subgraph development from scratch um but also feel free at any time to reach out to me directly um kind of if you scare if you scan the QR code you find my Twitter my DMs are open like if you have every you have any questions or or whatever then feel free to just write me um I'm always here and happy to help or link you to the right people or um yeah make sure that your problems are solved okay Simon thank you so much for the presentation I think we have five more minutes left so if there are questions or any open questions around this feel free to ask or if you have a question for the community yeah so um who in here already used subgraph's kind of consciously not by using adapt but kind of as I did just right okay and who wrote already a subgraph cool okay so then I think the the level of the presentation was appropriate um yeah okay any questions from the from the room anyone if not then a big Applause once more thank you Simon and we start with the next presentation in three minutes yeah just for for those who are willing to travel to San Francisco there is the graph day and hack uh beginning of June uh yeah I would love to see some hackers there thank you so much okay [Music] sugar