# Introducing The Graph Substreams for High-Performance Indexing by Alexandre Bourget

- Channel: [Devcon](https://streameth.org/devcon)
- Date: 2023-10-07
- Duration: 31:59
- Watch: https://streameth.org/watch/yt-K-nhC2FCB5k
- YouTube: https://www.youtube.com/watch?v=K-nhC2FCB5k

## Description

Visit the https://archive.devcon.org/ to gain access to the entire library of Devcon talks with the ease of filtering, playlists, personalized suggestions, decentralized access on Swarm, IPFS and more.
https://archive.devcon.org/archive/watch/6/introducing-the-graph-substreams-for-high-performance-indexing/

Substreams is a powerful, parallelizable, blockchain data indexing technology, developed within The Graph.
It enables truly composable indexing, with shared modules written in Rust, both in batch and in streaming. It reaches unbeforeseen performances by virtue of its parallel design that supports both mappers and stateful store modules.
In this session, we will see how Substreams solve indexing for all blockchain protocols.

Speaker(s): Alexandre Bourget
Skill level: Intermediate
Track: Developer Infrastructure
Keywords: thegraph,data,indexing,subgraphs,substreams

Follow us: https://twitter.com/efdevcon, https://twitter.com/ethereum
Learn more about devcon: https://www.devcon.org/
Learn more about ethereum: https://ethereum.org/ 

Devcon is the Ethereum conference for developers, researchers, thinkers, and makers. 
Devcon 6 was held in Bogotá, Colombia on Oct 11 - 14, 2022.
Devcon is organized and presented by the Ethereum Foundation, with the support of our sponsors. To find out more, please visit https://ethereum.foundation/

## Transcript

foreign [Music] ER I'm CTO at streaming fast and I'm also a Pianist a data scientist whatever that means I'm a father of eight beautiful children two of whom which are there I love designing and crafting software which I've done since I was 12 and I'm here today because one day in 2013 I read the Bitcoin white paper and that changed the trajectory of my life and fast forward to today streaming fast a company based in Montreal Canada is now one of the core developers of the graph and we joined the graph a bit more than a year ago and it kind of bizarre m a 2.0 fashion or lawyers they don't understand what happened but anyway we said thanks and goodbye to our VCS and shifted our Focus to make the graph the greatest data platform on Earth so today I'm here to introduce substreams which is a powerful new paralyzed engine to process blockchain data and before I can do that I just want to set a bit of context perhaps you can raise your hand if you know what sub graphs are raise your hand if you know you're good okay so slip graphs can be thought of an ETL process right extract transform and load and subgraphs add that little Q there that graphql layer to it and some grass today provide that sort of simple approachable end-to-end solution to blockchain indexing and the graph node is responsible for all of these components right the extraction is done through hitting Json RPC nodes and then transformation you provide some assembly script you guys know that composite wasm running in a distributed environment and then you have the load aspect which the graph node does puts that into postgres and offers you a rich you know and beautiful graphql interface on top and one of the reason one of the reasons we were brought in was that so we could push the graph to New Height in terms of performances so to do that we brought first thing the fire hose something at the extraction layer to our take to boosting performance by one two three orders of magnitude the first layer extraction it's the method of extract attracting data from blockchain nodes imagine prying an egg open where the data is exfiltrated as fast as possible and all the juicy data gets then thrown in a grpc stream as well as into flat files and you can think of that as sort of a the bin log replication stream for blockchains where you'd find in a Master Slave replication engine like in databases so we'll get back to fire hose in a minute then sub streams is sort of a reasoning of this rethinking of the second box the transformation layer here instead of the traditional sub graph handlers and assembly script you will write substreams modules in Rust and those can be executed in real time as well as in parallel with unprecedented performance so let me give you first a primer on fire hose because there's a lot of benefits of substreams that come directly from the fire hose so a streaming fast for many years we've been thinking hard about all these indexing problems from first principles and we needed a first a robust extraction layer we wanted something that is creamy but that was extremely low latency so I'm thinking that would push data out the moment the transaction was executed within a block within a blockchain node Json RPC was not going to cut it and we didn't want to have to deal with those large bulky nodes right hang on a thread occupied due to with uh managing high right throughput I kept everything in a key Value Store behind a you know a Json RPC requests and it was really heavy in Ram and CPU and you needed super optimize ssds this is really annoying and all these things are much costlier than what's needed when our goal was to get to the data inside so we also wanted proper decoupling between the processes producing the data so the blockchain knows and its intricacies and it's it's request response bottle and they're all different and the data itself we wanted the data to be the interface and we wanted something also extremely reliable in the sense that we could avoid hitting load balanced nodes that had all sorts of different views of the world and that we need to have like client code to like latency inducing code to to resolve what's happening there if there's another fourth you need to query nodes again and and you know for reorganization heuristics for example but also you know we wanted something better than even the websocket streams that pretend to be sort of you know linear uh that the nodes have implemented because when they would send you a signal that would say this block was removed it can leave you hang if if you happen to be disconnected for just half a second you'd reconnect you'd miss the signal so the reliability was not built in so we wanted something to address that and above all we wanted something thing that is able to process networks in 20 minutes well okay an hour or two but you know never three weeks or things where we're waiting linearly and that's still our goal today and when we say Network history I mean executing guests and extracting data executed into flat files that's the extraction layer but also any sort of indexing after the facts we wanted to be able to have massive parallelization like there was no other way to have reliable and durable in performance without parallelization so our solution was the fire hose and the fire host of all of these issues in a radical way we took a radical approach because we wanted to solve those problems definitively like meaning that there would be no further optimization possible except attempting to bend sort of space-time Continuum itself right so with streaming we with even multiple nodes pushing out data multiple nodes are actually racing to push the data the first through a consuming process gets the first to get out like you can't really add remove more latency there and um there can be nothing faster than immediately when the transaction has just executed from your node and then like regarding the staple processes and cost flat files flat fast for the wind we have a hashtag for that right flat files are the cheapest much cheaper than processes they're easier to work with there's nothing simpler nor cheaper in terms of computing resources these storage facilities have been optimized like crazy and it's also where data science is headed these days and there's one common thing to every blockchain protocol and that it processes data data is also the right abstraction for this technology not an API that's common to all chain data so fire hose clearly delineates responsibilities and and the contract between the extraction and transformation layers is again the data model fire hose creates and for every chain you can imagine the best data model the most complete and that's what he done for fire for for ethereum for example that both the the data model for ethereum within fire hose is the richest there is like you have in there the full call tree internal transactions you have the inputs and outputs as raw bytes you have the logs obviously you have the state changes like you see on etherscan down to the internal transaction level you have balance changes the same way with the prior value and the next value so when you're doing like navigation backwards or forward you get you have the data you need you have also gas costs at different places and there's that important notion of total ordering between things happening within the logs State changes and calls all of these things happening during execution are totally ordered so you get in there everything parity traces would give you and more and everything you would need to rebuild a full archive node from Flat files and everything there is goes to the transaction level not rounded at the Block Level which is crucial if you want to index with Precision right it doesn't rounding of blockchain information at the Block Level is sort of a was meant for helping in consensus right but it doesn't mean that what happens mid block is of less value than what happens at the boundaries so okay so that's very interesting and now regarding reliability whoops no not too fast regarding reliability the fire hose erpc stream provides reorg messages like new block or undo this block or this block is now final accompanied by a precious cursor I think that's really key here with each message so if you get disconnected and upon reconnection you give back that cursor you'll continue exactly where you left off and potentially receiving that undo say signal that you would not have seen were you disconnected right so you will get it so with the guarantees or linearity of the stream so no websocket implementation would do that because it doesn't make sense for a single node to track all the force possible even two days after the fact and undue messages come with full payloads so you get all the Delta so you can just turn around to your database and apply the reverse or you know pluck again in the the full payload of what happened in the block and just like what you do is so it doesn't pose on the reader to store what happened like at that previous block if the signal was just remove block 7000 right okay and when you commit that cursor to your transfer database well you get through that you know finally some consistency guarantees within your you know your your back end so some of our users told us they could cut ninety percent nine zero of their code reading the chain because they were relying on some that reliable stream and okay and it also lays lies down the foundation for massively paralyzed operation files plus the stream and so this is the future of the graph unbeatable performance and it's core to our multi-chain strategy because you know any blockchain can have that data model uh now let's dig into sub streams substreams is a powerful clustered engine to process blockchain data it's a streaming first engine and it's powered by the fire hose underneath and its data models of the chain so let's dig in here are a few quick facts it's invoked as a single Jace grpc call and within the request we provide all the transformation code like you'll have in there oh it's too low you'll have in there the code from some wasm modules relationships within the modules and uh you know all the transaction the Transformations within the request it's not a long running process except if you run it for long it's not a service you you spin up right and the backing nodes are stateless which provide nice scalability properties modules for Transformations are written in Rust they compile to wasm and they're running a secure sandbox on on the infrastructure they're similar to the sub graphs and the ultimate data source being the blockchain data being deterministic all the transformation outputs are also deterministic and the request if the request you send involves process prior history even if it's 15 million blocks well the substream's runtime will then turn around and orchestrate the execution of a multitude of smaller jobs in parallel fuse the is lost on the Fly for you and aggregate the results to simulate a linear execution so you would see a dime in the difference and all the results are streamed back to you as fast as possible with same guarantees provided by the fire hose with a block per block cursor and a transparent handoff from batching into a historical processing to the real-time low latency rewards aware stream of the head of the chain so let me show you if you're interested how we create one of these things raise your hand if you're curious okay you're good okay so let's start we start with a manifest like that do you see that down here can move the podium I can't so there's package information you know some metadata there you have pointers to the Proto buff that you'll use again contracts between modules are about data so there are protobuf models similar to the protobuf models of the root chain of the chains the layer ones and you have pointers to the binary that you're working on your drive and all that and you have Imports and imports are actually very interesting because you can import third-party substreams packages and these yamo can be packaged and so you can import from someone else's package you can write your own or combine both that means substreams enables composition at transformation time which I think is pretty unique in a pretty Game Changer and then follow up and there you have the module sections which defines the relation between the different modules and you see it defines it directed a cyclic graph you have modules that slowly refine the data and so there's two types of modules one the mapper the first up there map pools and this one takes inputs does transformation and outputs it's parallelizable down to its core block wise though massively paralyzable and then there's the store input I think it's awesome this one takes any inputs and outputs a key value store that or something accumulated in a stateful way and stores can then be queried by Downstream modules and uh okay so we'll see a bit more after then the name corresponds to the function in the wasm code and the inputs can be of a few things either the raw fire hose feed so for example The Source here that means the block with all transactions you know for for that particular block and it can be the output of another module like you see down here the input of map map pools so you'll get the data as bytes and it also be a store which would be a reference we'll see in the next slide there and on the store pools here you see there's an update policy which sets constraints on what you can do with the store and it defines a merged strategy for when you're running paralyzed operation okay I'll get to that also a little later and the value type field will help anyone decoding understand what bytes there is in that store so you're going to UI you can jsonify them and your code consuming can automatically you know decode them with protobuf all languages supported otherwise the key value is just Keys as strings bytes values very simple and one thing to note here is that because it has deterministic input it's possible to Hash a module like the kind and all of its inputs and the pointers to its parents and including the initial block so you have a fully determined and hashable let's say cache location for all of the history similar to git right all the history of data produced by by and you'd hash also the wasm code right so it makes it for an extremely cachable system and highly shareable and cross verifiable output of modules which opens really interesting possibilities you know for uh collaboration within the graph ecosystem and imagine one has large disks and everyone has large CPUs or you know sleeping CPUs they could pull resources together to build something bigger than themselves okay you see the relation there so this gets piped to that and if we add another module here you see how the graph comes together this one computes the price this is unit swap V3 thing it computes the price but it you want to get them for certain pools because maybe you want to use the decimal placements in the pool we'll see a little bit more there and when you're running let's say you're running that at block 15 million well you're guaranteed the runtime guarantees that the store you'll have to execute code at block 50 million will have been synced linearly or in parallel but you wouldn't know but it'll give you a full in-memory store eventually backed by some disk but whatever and you can query the key value store at each block it's guaranteed to be synced for you that's exciting no okay uh so you see the dags fully being built huh right so the dependency so now let's let's this leads us to composability see each color here means a different author and modules written by different people ideally the most competent for each right like we would hope they would crop you know analyze what's on chain and refine the data and Abstract it to new heights and the contract between the handoffs is always data it's the model of data so you take a modulus bytes in bytes out and so you see here we can get the prices from using V2 and price this version of B3 and Sushi and chain link and whatever and have someone reality a module that takes these input at transformation time and then averages them out and whatnot right and then that you'd have like sort of one beautiful Universal price module that you can then hook on top and and feed to some who knows maybe if someone feeds that back onto the chain for some reason and then soon enough you know all of that while someone wants to build on top of it ah something like that if someone wants to compute you know the USD denominated volumes aggregation of nft sales on openc you know you'll take some sales you merge it with the price and we see here that little Trader Inc he seems he maybe he wants to feed that into his trading bot because it's a streaming engine we're not storing that in the database yet right but this begs the question where is that all that beautiful data at land where does it get piped that's where it thinks head up like substreams being limited to the transformation stage of the ETL analogy remember it doesn't really care where you load it and that could be anywhere these are just a few examples you can load that in databases we already have a sync for postgres and you hook to sub streams and it just loads it into postgres with a data model that we've agreed upon right if you write it in a certain way it just thinks over there and or message cues or whatever you know data lakes or some Bots or some trading already you know so I don't know some whale detector you want to hook directly on the stream or also so something I think big for doing some ad hoc data science because you know you have a really fast engine allows you to process the whole history in like it can take a few minutes to process the whole of it here image to go and pluck some new insights so you can write your code send it to the network and then you know stream out the results similar to for those who know big query you know the cluster the big cluster the service by Google that's what they do you send the request they just shot at everything they send you back the request well all of a sudden substream's engine can allow you to do some things like that ad hoc and it you can write any program uh that that supports grpc in protobuf which are many and the last one here not the least subgras through graph node well so we're working to make substreams feed directly into into graph node to then provide the same loading experience and then querying experience that you've come to know and love and you'll be able to deploy a sub grasp this time not containing assembly script but a stud streams package with an entry point and would process the history in parallel and load that in your database in Crazy speeds so stay tuned for that that's not out yet but you know soon okay and so this is a simple example in Python it's not really longer than that you have two one or two dependencies like grpc so that you can use the query so we're leveraging a lot there and you can uh you know see that spkg there we can use that to code gen python classes and helpers and all of that because it turns out that the Manifest through the spkg there is for those who know protobuf is they file descriptor set it contains all of the things all the protobuf definitions so the spkg also contains all the wasm code the module graph information you know the dependencies the inputs and all that and even some documentation everything is needed is in there so you can pass it down to the you know modules you take it from from the disk and boom you send the request to the server and it's running so you can deploy packages also very easily and consume them very simply this way there's a few Imports we've omitted there but it's simple this is the show okay and let's look at a simplified data model for unionist swap V3 and I'll show some code making use of it okay so this here the pool is a list of pool this is actually what gets handed off from you know our mapper which finds the pool that were created down to the store pool which we're going to look also and so it has a list of tools and the tools you can imagine an address and the two tokens that are concerned here and we have a reference to the Token also which is going to be very useful to enrich the data down so we will have the decimals right at hand like we won't need to do much loading it's going to be very very close so we can enrich all these U ins 250 79 000 and you know put the the comma where it belongs um so let's see what happens in the mappers so this is a sample rust code raise your hand if you love rust where's your hand if you no rust okay it's very simple here I'm gonna go through you have the map pools function corresponding to the Manifest there has one input the block this is the fire hose block with all transactions all logs All State chains you can craft your own triggers in there as you wish but we have a simple Voyager and see that that line there blocks events you have a thing that goes through transactions and it's going to trig on pool created and that pool created object in Rust was actually code gen from the Json API so you can just give the instruction and we're going to filter it for only the V3 swap Factory and then that beautiful filter map will give us the log okay and then we'll we'll output we're going to collect some of these things into one list of pools and it's assigned to the pools object there and and notice that little thing here this is the RPC create unit Swap tokensing this actually hits you know an e-call on a node behind similar to what we have in sub graphs that's actually very important it means that once we've processed this layer once and we've done it for the history it can be cached very efficiently so anyone relying on that thing will never need to reprocess it again you can give the package to someone and they can access the stores that's been cached by other people immediately so you could go to block 50 million and you'll have the list of all pool created that you can query super fast you can depend on it also so I think that's pretty cool and here that's the store modules the store module is pretty simple here it receives C the pools from the output of the prior module and it does it Loops through the little pools there and calls output set and see the key there is pool colon the address is going to store the Proto buff encoded stuff of the pool with the token decimals for both right did I say that to be constrained so the store here is constrained in two ways in order to preserve parallelized ability the stores are right only you cannot read your rights otherwise that would make some potentially cyclic and they expose only the function defined by the update policy in this case it's set so let's see what happens if we run things in parallel so here we have two jobs covering two segments of the chain you know one million block each and to see those ugly arrows there they correspond to a pool created event and so in our code you've seen we would write a key for each of them and uh so in the first partial run we'd have a what we call a partial store with four keys and the next one we have two keys and so when we run the merge operation we would apply the set merge policy which says basically if you take one store take the other store cycle through keys and the last key wins if you do that you can paralyze endlessly so we'd have here a complete store with six keys and now at that place we have a snapshot we can have periodic snapshots and so if you want to go and explore the chain at any point in time you have a snapshot plus a little partial you can have the state synced at any block height so so this one you know has the last key win policy but you have a few others like min max ad and another one like first key wins so if you merge them you have set if not exist and then that that allows us to build different aggregations right you'd like to see that running live every few minutes now I need to bring that other window up so you see that okay that's good enough huh okay so let's imagine we want to see that pool pools created thing okay do you see that I want to see the output I'm going to run that I hope everything is good you know the demo of gods are connecting okay okay whoa not too fast so this is going through uh starting at the beginning and we have there a pool credit event and see that we have everything decoded because it's the protobuf thing we have the thing to decode it we could feed that it arrives on the wire as bytes and it properly serialized bytes and then we see that the token address is there we have a decimal we have the address and so what that means is pretty crazy already oh do I see that okay what that means is that you can inspect the chain with your code at any place in time for a mapper especially you can go there and I could run it again here and say and say I want to run the mapper let's say a block I don't know uh something more recent give me a recent block what's the block yesterday fifteen seven million I know something like that okay and see is there anything recent so there's some stuff right some things are recent can I see that someone still created a new pool and I can inspect my code to make sure it works where is that come on right this one was rap ether and infinity they were just created that address as a new pool so you can go and test your code everywhere what's that's done well you're set right you can go then to the next data dependency so this really changes Dynamics for debugging and this can also work for stores they can you can ask for a store and it's going to process it in parallel and then you can inspect all the keys that exist there or see the Deltas coming through okay and let me show you something running in parallel uh-huh graph out now this is very interesting because oh no let's start it at 15 million let's say I want graph out at 15 million I didn't run it again before and I want so let's write it so this starts a whole bunch of parallel processes and you see up there the number of blocks per second yesterday I had about 8 000 on Solana blocks I had 16 000 depends on the power you put behind there but this all like you said the pool counts as a dependency on the pools the pools is further down so we're able to schedule things and all that just massively parallel and once that's ready let's say everything was done I would start streaming and get all the content and so let me show you the graph out it's very interesting because because graph out graph out has refined the data up to entities now we're talking about database tables and Fields and you get and you get out of that do you not see it okay that wait a second here so let's imagine see we have token and an update and you have the field derived if and you have the old value and the new value I don't know if you've seen this thing in the data Science World this looks very much like change change data capture cdcs that can power a lot of large-scale systems and uh and you have the prior after so you can feed that to your let's say postgres apply the changes and when you have an undo signal it gives you back that payload you then just flip everything and you have guaranteed linearities to your store with a cursor and it is Flawless it's just extremely simple to keep your things in sync you also want to have a slack body can have it undo message remove the message if you have a thing coming through right so I think that's pretty cool what do you think okay okay that's cool can I shut that down here do I have another window uh I'm close to them prepare your question please ask ask them succinctly we have a half a minute I just wanted to have a final final note there so as a final note I want to share with you a little bit of my vision for the graph okay I don't know where the window is so whatever it just says fine it's fine uh I'm imagining the graph becoming that sort of huge worldwide cluster of processing and storage capabilities and something like Google's bigquery but where people join because it's better together right instead of running it alone we need to have all the resources alone and I see also a new era of composability which means more collaboration and in a tighter Community working together more intimately in those in those uh you know with those data contracts and I see also a new mix of collaboration between indexers like exchanging data or sharing resources in terms of compute and stores and whatnot therefore introducing new value flows and also I'm seeing new products new Services being offered directly on the network you know to satisfy some needs that perhaps couldn't be addressed before and I mean there's a place for you in there as a developer as an indexer as someone who realizes the radical benefits of such a platform and who who builds on it and promotes it so my ask to you is you go to you go try substreams put pressure on your favorite layer ones so that they do integrate the fire hose natively that's pristine's Aptos has done that recently some others start wear I think so that makes it everything we've seen today becomes immediately available to them sell them on the goodies so also join a Discord I love follow-up follow-up questions and come see me afterwards I love feedbacks on these sort of things all of this is open source so let's dig and build something of the like together the biggest blockchain platform on Earth thank you for your time today do we have time for two three questions we're the last one so if you have a question hey uh so one question uh modularity and composability of these substreams is super super powerful but still if I look at this compared to SQL and like DVT models right it's a lot more complex so how can we enable people to really kind of learn this and like build these kind of hyper modular data streams so SQL layer like this is powering going through history it's an adult transform with stateful storage but you would pipelive into SQL store to do other things right you would have refinement you have Knowledge from the community as to how to analyze this and that protocol ever ever you know increasing refinements but then you might store that in your store with off-chain data and maybe that's best fit for you maybe you feed it into a subgraph that's what you need you had a total decentralized solution and you don't need to host anything so this is enabler at a lower level it's not a it doesn't seek to replace SQL but it puts itself at a place where we can feed all the systems on Earth with enriched data which you would need to do in SQL and it's it's really not fun so you leave that to the community right gotcha thank you we have our old subgraph which is pretty slow and we would like to transform it to the new type of subcraft Yes Should I only read some code on Rust and that's it or something else so it is not the same Paradigm to enable parallelization Nation you need to distinguish the data dependencies and that infers the number of stage of parallelization that is needed it's not easy at all actually it's pretty crazy to try to parallelize the subgraphs we try that that's what yield us to design substreams by cutting you know uni swap stuff so you will want to go and write in Rust modules and it's a different Paradigm so it's not just an easy switch I admit but it brings us to the next stage an evolution you know of blockchain index thing yeah I see thanks uh thank you so much Alexander my pleasure