# Shasper on Substrate by Robert Habermeier (Devcon4)

- Speakers: [Robert Habermeier](https://streameth.org/speakers/robert-habermeier)
- Channel: [Devcon](https://streameth.org/devcon)
- Date: 2019-10-09
- Duration: 23:41
- Topics: blockchain, smart contracts, distributed ledger, Science & Technology
- Watch: https://streameth.org/watch/yt-YDJCezRzaXg
- YouTube: https://www.youtube.com/watch?v=YDJCezRzaXg

## Description

Visit the https://archive.devcon.org/ to gain access to the entire library of Devcon talks with the ease of filtering, playlists, personalized suggestions, decentralized access on IPFS and more.
https://archive.devcon.org/archive/watch/4/shasper-on-substrate

Robert Habermeier gives an overview of Parity Technology's work on Serenity.

Speaker(s): Robert Habermeier
Skill level: Intermediate
Track: Developer Infrastructure
Keywords: serenity, parity, substrate, casper, sharding

Follow us: https://twitter.com/efdevcon, https://twitter.com/ethereum
Learn more about devcon: https://www.devcon.org/
Learn more about ethereum: https://ethereum.org/ 

Devcon is the Ethereum conference for developers, researchers, thinkers, and makers. 
Devcon 4 was held in Prague, Czech Republic on Oct 30 - Nov 2, 2018.
Devcon is organized and presented by the Ethereum Foundation, with the support of our sponsors. To find out more, please visit https://ethereum.foundation/

## Transcript

yeah hello everyone i'm robert i worked primarily on a project called polka dot but i work a lot with parody technologies i'll be talking today about parity technologies work on on not jasper very sorry on serenity basically what we've been doing so far we're building it on this platform called substrate first i'm gonna go into what that is a little bit of background on on sharding on casper on serenity Vitalik and justin and the others can explain that way better than I can so I'm sure you've seen their talks as well and then basically dive into what we're what we've been doing what we are doing in the future and what we're doing now on building out all of that so a lot of you might have heard recently there's been some buzz about parody substrate so what parody fundamentally is is a blockchain development outfit like we're we're a group that's built a lot of different nodes for a lot of different chains so the best-known one is the parody etherium node it's the other than the go aetherium client it's the most widely used aetherium node it's used by infrastructure it's used as a light client it's used in a lot of different ways it's not just in a theorem node it supports all kinds of different consensus algorithms and stuff as well we've done a Bitcoin implementation recently announced is that this is soon going to be also modified somewhat into the first alternative Z cash implementation so we're we're a blockchain Development outfit and at some point we realize like we've been writing these nodes and we're duplicating a lot of the same work over and over again so stuff like synchronization peer-to-peer networking database Merkle trees light client algorithms that kind of thing it's actually a lot of infrastructure that you have to build around ultimately what you really care about is a blockchain developer which is the state transition system that you're putting together so that's basically what parody substrate is it's something like a blockchain the box that you can build a chain with it's something that's going to be useful for for side chains and we're also using it in a bunch of our own projects like to build polka dot for example and as well to build the 2.0 client so the way that substrate works basically is you have lit p2p networking although you could swap that out for something like dev p2p my understanding is that ether 2.0 is heavily leaning towards live p2p right now so that's what I'll talk about for the moment will a p2p networking that lets you run a consensus process that does agreement over state transitions that are encoded in webassembly that we call the runtime so the runtime is the actual work of the chain and this is something that we've designed so that you can upgrade it on the fly for a theorem to a put only that's not really a feature that you need so you know you don't need it to be web assembly it could actually be native code in this case but substrates general model is around web assembly state transitions where the state transition logic itself can be upgraded and that's gonna be composed maybe of many different reusable components or modules like governance or slashing staking for polka dot we wrote one for para chains and for smart contracts and that sort of thing for a theorem 2.0 we've basically written a runtime from scratch because the beacon chain implementation is very very unique so what substrate gives you is a abstract block format it's agnostic as to what kind of backing database you use so it could be like a key value store in base 16 a miracle patricia tree it could be a binary Merkle tree it could be something like just a key value store column that's not crypt 'var cryptographically committed to if you were writing something like aut EXO base chain at its core basically all you have to do to write a chain and block in in substrate is put together a implementation of function execute block so that takes in a blob of data and it tells you was that a good block with the with the backing database so that's basically what we've done and it's got some extensible networking and other utilities as well so one big part of what we're doing is pluggable consensus so that was a feature of parody etherium for a while to support proof of work as well as proof of authority and a contracting system around that but for substrate we've come up with a pretty nice set of abstract that lets us do a lot of different complex things in substrate like we've we've done rhododendron which is a instant finality protocol that's derived from pbft we've got grandpa and or and which are a hybrid consensus together it's something like Casper FFG in the hybrid model you could do Casper FFG as well with this you could also do stuff like Ava that Eman was just talking about fairly easily in this model so it's very very extensible in that sense and what you get is hot swappable consensus is one feature we've talked about where essentially you can move between consensus algorithms for example if you deploy your chain right now with some relatively simplistic longest chain rule then you could upgrade it to something else and actually fork the chain on to another system a hot swappable state transition function I'm not going to go through all of these but one of the ones that I do really want to stress is that it's got a light client from day one I'm one of the few people who's written a light client for a theorem I can tell you that adding that onto a blockchain node after the fact is awful it is a lot of work so having that supported from day one having networking protocols for it from day one is really really important to make sure that you can actually have light client infrastructure in the end of course there's things like the module library and if you want to plug into something like polka dot that is that's supported as well but to get more to the direct point and the question and on the forefront of a lot of your minds with this conference is that aetherium basically needs to scale so the EVM can't process the proof-of-work consensus the EVM can't process enough transactions for the uses that we're already seeing that's just the case side chains and bridges which Buuren talked about a bit earlier on the mainstage are a temporary solution to this problem but they only mitigate it at some point even if you have a ton of side chains and a ton of bridges you're sort of fragmenting the ecosystem which isn't ideal and also you're just delaying the scaling issues but is at some point you won't be able to get all those bridge notifications onto the chain as well so you really need a root level chain that's scaling so the solution proposed by the etherium research team is serenity aetherium 2.0 so this is this is a name that's been around for a while I mean serenity has been on the etherium roadmaps for years the exact feature set that it's encompassing is just being nailed down now so the spec is about sixty percent complete now it doesn't cover everything like block authorship but it does get the core of the state transition system validator shuffling changing validator dynasties and collecting at the stations and that sort of thing so the main piece of the sharding design is a beacon chain and this is what we're building on substrate so it's sort of a shard coordinator it manages proof of stake validators it add blocks adds blocks to the chain it finalizes them it gets cross links from shards which are attested to by committees of validators so all the valid leaders will be split up into small committees each committee being assigned to a specific shard for a period of time the assignment is done by a randomness beacon which is why it's called a beacon chain and basically these guys just attest to blocks that exist on sharts it operates as a proof of stake chain the validators might take turns proposing blocks which would be finalized by Casper FFG so Casper FFG is is one of the Casper's that's floating around it's basically just a finality gadget so the term finality gadget is a bit nebulous but it's essentially part of a hybrid consensus process where you have a probabilistic consensus algorithm adding blocks to the chain and then you have finality gadget adding accountable safety on to that so then if blocks that are finalized by the gadget are reverted you get this process called accountable safety which means you can identify who acted in reverting those blocks and punished them so Casper FFG does that by voting on checkpoints every n blocks and if you get to check points in a row that gets super majority support you've finalized a block it influences the fork choice rule in charting of the block production system as well the cool thing about cast for FG is that you get information on chain about which blocks have been finalized and the slashing conditions are really really simple so this is sort of a diagram from the sharding website of what the cast for FG fork choice rule is doing so you have some you have some check points which are getting closer and closer to being justified like you always follow the most recently justified check point but then you also look at which messages people are passing around on unjustified check points and use that to influence your your your target block on the main chain so shards are basically freestanding chains they're secured by their own consensus algum they're typically proof of stake and they're linked to the beacon chain via these cross-links these at attested to statements that make their way onto the beacon chain so to look a bit at this diagram that we got from Sal way and Ben Edgington we can see that the the architecture of sharding looks something like this so the the proof-of-work main chain is the anchor layer this is actually like the etherium 1.0 chain it's got this one directional lock up stuff where you can basically lock up your ether on the existing chain move it over to the the beacon chain and then get the beacon chain tokens for for participating as a validator there other than that it's not particularly relevant the beacon chain is doing this random speakin stuff coordinating between validator sets and the shard chains are actually doing computations with a web assembly virtual machine so now I'll get into the the part of basically our implementation progress like wherewhere are we with that so I'll first I'll zoom back in time originally there was a plan to have a hard fork in aetherium to to implement casper FFG 2 to hard fork the etherion main chain that we're using now and make Kaspar FG hybrid consensus model with the proof of work that's going on there so we the first step that we took towards sharding shortly after the Thai Taiwan summit was to do this implementation so that's been closed because this plan has been abandoned but we did this work we spun up a test net and we showed that it was at least in a proof of concept stage so now that the the plan has pivoted somewhat to incorporate this beacon chain architecture and we are doing parody tech /jasper is the repository so the developer who's been doing most of the work on this is a guy called wait I'm from China we've the rest of the substrate team has been contributing bits and pieces through architecture and pluggable consensus and that sort of thing basically this is a repository that contains more or less everything that's pecked out in the state-transition aspect of the beacon chain block authorship etc is still to do but as those parts are further Specht out those will be added so most of it exists as a runtime that piece of web assembly code that actually processes the logic of the chain so what this is is processing new at two stations crystallizing them handling new validator set changes and that sort of thing substrates native block formats are a bit different than the ones that the beacon chain uses so the beacon chain will have fields like the R and I'll reveal or a pin to the proof-of-work chain the anchor layer that it's attached to but substrates native lock formats don't really have that stuff so we use an adaptor so basically on disk we've got a like one one set of of block types which are the substrate block types where these extra values are encoded as extrinsic s-- and on the on the networking layer we've we've transformed them into Shaffer or serenity blocks so that when you communicate with other nodes like those perhaps built by prismatic labs or where lighthouse that they're all using the same block format but actually under the hood it's using a different block if you're into porno is very likely to use lib p2p for networking so substrate has been using lib p2p from day one I parody we even did well to whine back a little bit lib p2p is a set of networking libraries first designed built and described by protocol labs of ipfs and file coin so this is a intended to be a set of cross-platform and future-proof networking libraries at parity we did an implementation of those in rust and we've given that back to to open source and we're in the process of moving that over to to parity etherium as well so the parity etherium can use lib p2p as a backing network but we're also building substrate out with Libby TPS first-class networking so the next steps like what's what's next in the implementation side of things for if your IAM to point out for us we are going to see this through the parody is 100% committed to finishing the etherium 2.0 implementation thank you so the next step will be basically better pluggable consensus so we've now just in the past few days merged the commits into substrate that generalized the consensus framework I think I've got a picture of this so we've merged this this pull request you've got a few more coming what's next is basically to implement cast for FFG on substrate start to do that block authorship work so that not just processing the blocks is done on our on our Shaffer repo but actually authoring them as well one other thing that we'll need is multi tree support so the validator set might grow pretty large right now it's just stored in this ssz serialization simple simple serialization format that's described in the sharding spec but if the validator set is large enough and continues to to grow dynamically then you would need to probably encode it in some kind of tree structure so this is a feature where basically rather than having a single store root in the substrate header you have one that references a few others but it's also a valuable feature for for for example building and if during 1.0 client on substrate because this is roughly the same model that you would use for the accounts tree versus account storage trees in a theory in 1.0 so one possible application of that could be to probably fairly easily put together a light client in substrate for a theorem 1.0 or for a theorem 2.0 shard chains so I guess I'll conclude it there get substrate give it a try continue to follow what's going on with parodies if your name 2.0 work we're doing it a bit quietly in the background but slowly plugging away at it as the specs solidify we really intend to keep it up to date keep our code up to date with what the specs are saying so that we're always on the ball with with the sharding stuff thank you very much it's time for a few questions hi thank you very much my question is is there could you maybe elaborate a little bit about the crystallites states like in what form are they happening and if it's kind of the benchmarks that you haven't proof of stake and like yeah if this benchmarks are something centralized or is done in a decentralized way yeah so so benchmarking the the crystallized state is the question so the crystallized state is just something from the sharding specification it's basically the state of the chain is spit into two parts you have the active state and the crystallized state so the active state contains sort of Atta stations that have not been processed yet and the crystallized state stores at two stations that have been processed and checked and and as well like what the head of all the shards are that sort of thing as well as the validator set we haven't benchmarked it yet it's a bit early stage for that I mean there's not really much to benchmark until you have something like block authorship going but you know as soon as you have a chain up you would start to see maybe how many transactions you can process but right now is sort of the stage more for experimentation like there aren't even really none of the beacon chain implementations are really at the stage where you could attach shards to them yet so I don't think benchmarking sharding makes that much sense and so you actually have shards you said that the substrate works with the beacon chain but does it work with the shard chains as well yeah exactly so you could you could use substrate to write the beacon chain which is what we're doing now and you can also use it to write the shard chains like those are most likely going to be proof of stake chains on their own that are running you azzam as the state transition from the perspective of the beacon chain the chain the the shard chains are just blobs of data so we sort of step aside from that problem for now but you could definitely write in aetherium to point out client for the shard chains in substrate in that's what we will be doing as one of the next steps so do you have any plans to upgrade some of the other products you support like you know the Bitcoin is e cash when you just now if they're in 1.0 with substrate once it's up and running and ready to go yeah so so it's it's quite possible that you could do a Bitcoin implementation on substrate I think with the existing products like parity aetherium particularly with parity aetherium there's a lot of extra code around it like things like warp sink and the light client protocol and stuff like that that's a bit different than the way substrate looks at things so just with that amount of like three years of code porting that all over to substrate would be a bit too much work probably you could definitely write in the theorem client and substrate but it's unlikely that the parity etherium codebase would migrate over to substrate in the near future the Bitcoin client is a lot less complex simply because bitcoin is a lot less complex so I think it's very possible to do a at the very least a you TXO based chain on substrate where you execute Bitcoin scripts transactions the question at that point is really just optimization like it's it's for sure possible to optimize a substrate node to get to the level of like a specialized Bitcoin node but that would just at this point be duplicating efforts since we've already written one I think substrate sort of added snitch is more focused on less migrating over existing blockchain products because then people have already done all those pieces of work that substrate is supposed to save you from duplicating and rather focus more towards like new blockchain products to get people off the ground a bit faster like we found it to be immensely useful for the charting stuff they wrote up the state transition in like two weeks which you would think is you know if someone told you that beforehand you would think that that's impossible but managed to do it just because all of this stuff was completely taken away from developers are you using G RPC for the interface gr PC yeah is there because I think I'm actually not familiar with with gr we are using JSON RPC like there's some json-rpc methods but I'm not familiar with G RPC okay Google okay okay so like as like a JSON RPC server yeah no so we've we've written like parody has written our own JSON RPC library and rust actually the guy who wrote most of it is sitting right behind you talk to him but yeah so a lot of the stuff doesn't exist in rust yet when we started to use rust like these libraries like we had to write a lot of them ourselves so mostly our own code just another one I'll top my head you mentioned alternative networking protocols is i2p in any sort of plans or anything like that I to P what's I to P sorry say it again for the mic over efer Manero suppose to have additional privacy here at ease over p DB okay yeah I don't see why not basically what none of the code that uses networking assumes anything about the networking protocol that's being used so it's fairly generic like usually at least in the consensus algorithm stuff for the block propagation stuff it's like you have a stream of incoming messages in a stream of outgoing messages on some topic or some period of your connection so it doesn't really matter what you're using for that loopy TV is one option and that's the one that we've coded up but it shouldn't be too hard if you have like an IP to P library that you can link into rust or do an implementation of that in rust to slot that in as well we're not probably going to do an implementation of that ourselves but it's an open source project and that kind of contribution is the stuff that we actually really want to see so if there were to be a pull request open to implement ITP support and substrate we would definitely review it and merge it yeah will do all right that about wraps it up for this room I'm Corey petty from the Bitcoin podcast and status and there's still some more stuff on the mainstage so thanks guys [Applause] you