Reth: Rust Ethereum client - Dragan Rakita | Paradigm

uh hello nice hello everybody my name is Ryan rakitam I am one of the core developers that are building great it is modular contributor friendly and basically fast implementation of the ethereum protocol uh execution layer basically what I want to talk about is basically these four things a little bit of the history while the new client is important thing but are we missing currently in the present state are we focused a little bit on the modules basically but but are but red is composed of while they are important for us while that modularity is important for the new clients uh I'll give a little bit of developer insights to just show but basically Pat that we took what decisions that we made and I did talk about a little bit of the current state of the client a recent history ice let's start for the present diversity in ethereum ecosystem is very important in essence it allows you more security it decentralized Network it loves basically if there is some potential problem in the one client or the clients should in should take over and just move the network forward but the present state is the get is the majority client that everybody uses so if the get has some of the potential processor bugs something like that it will break the network network will stop and all the client will would just report the problem they couldn't they they don't have any weight to move ethereum form forward other things that other clients bring is innovation it's a little bit experimentation it provides users and developers new ways to use ethereum a little bit history 2022 uh is the year that I started working on ethereum in this time we had get as even more major ethereum client and the second biggest one was imperative ethereum paired the term that time basically moved away from Material space and just open source the client and rename it to open ethereum 2021 it's a brilliant hard Fork here we see fork of the get name turboget there is nethermine embasso uh parity renamed to open ethereum London hard Fork or the time open ethereum was deprecated it was basically the client that was not well maintained and it basically time got over it and at that time we decided to deprecate it to not Implement merge basically Paris hard Fork we see I reckoned that was renamed for the Turbo get Fork of the get and see Basu and undermine taking traction Paris was imported one 2022 people started noticing and basically focusing more on diversity and as it was one of the major points of the merge as the merge was one waiting because it when you switch consensus you cannot go back in that sense diversity was a little bit more focused and present Netherland besso are the major clients get is still the major client now we come to the last year September to 10 2022. uh Paradigm started building their own client it's building the rust it uses some ideas from the Aragon uh idea is to have like very modular very well made documented client that we hope to um will improve the ecosystem and develop an experience it's built for us last basically modularity is one of the major points of us because we want to be able if needed to just rewrite components if they don't perform well enough or to be able for Community users anybody just use components add their modification and use it with the rest of the system clean obstruction Standalone rust libraries or basically that hikability of the code and allowed to for clients to be used in different ways you want to be flexible and in configuration to sportbot to dark heavy node and the full node you're currently supporting on the archive node but there is we will add full node support and most importantly it's MIT Apache license so anybody can use it it's like open I'll focus a little bit on components but inside the client database footprint I wrote here database is most important aspect of any term client because uh have you write the database how you read database matters a lot in terms of how fast you are thinking how fast you're doing things and the database layout is the most impact impactful thing in the ethereum client for the plane State we are using diary on flat key value database our archive node is under two terabytes we are very pleased with that uh we'll probably shrink in the future a little bit there is some idea that you want to implement we are still work in progress and currently we are very satisfied it's under two terabytes uh optimizational data basically optimization memory footprint is very important as any byte matter for example to build intersections give you if you save hash of the section it's additional 50 gigabytes up 64 and still optimization depends on how you read and write data first module that can be important is provider uh because everything not and retic loading has simple key value database fetching data and fetching data in proper order is very hard you need to know layout of the database you need to know encoding that's used you need to know basically where the data is provider is the one Library that's going to wrap that it will give you high level obstruction on the low level database access basically to it will allow you to fetch one block where the block is split into multiple tables it will just one Library call other than that it will allow you to have access to the history State without worrying how the state is indexed one good thing is that we can have multiple readers and basically you can run your client and client will push data right data to the database then you can have another process that's going to just read and that allows new flexibility yeah initial sync with the pipeline with nvme SSD hard disk is under two days uh pipeline is basically used for initial Sync It's optimized to handle that multi-block execution a multi-block uh preparation of execution and localization after that it's one of Dia that is pioneered by Aragon uh it's um yeah we have in the our database uh we are just saving the canonical chain that allows you a little bit flexibility where we know that the data and blocks that are inside the the main in the database are the connect class block and we just if needed pipeline is optimized for the ranges site Penny block we have different module that handles that pipeline for example stages that are inside the pipeline or the header body that are special specialized for the fetching of the blocks headers basically data that's needed from outside or from the P2P Network there is some send recovery total difficulty that they need realized preparation for the execution execution that executes creates receipts everything around that and there is a few stages that do mercurialization and history indexing and after that there is transaction ID there is some place that we want to improve uh one of the things that we could introduce is ETL is to read the right data a little bit differently but either way we are we have a very fast even without it a long stage longest time for execution is of course execution interface is one of the core libraries it's basically list of traits that obstructive a components basically we implement the trade specify the trade and somebody is going to implement it that allows you very like abstract Avail modularity between the libraries so you can just Implement your own trade and just be sure that you comply with rest of the system P2P downloader module P2P is one of the things that does Discovery management of the nodes timeouts and sending receiving requests from the P2P Network it's one of the things that every client obstructed in some way because it's very easy to abstract it's it allows you to handle peer Management in one place and don't think about it downloader is wrapper around P2P it's glue between the rest of the system and P2P module that allows you to download headers blocks and full header blocks and bodies whatever you need in future maybe we switch downloader to not fetch P2P but idea is to fetch from some server or something like that it could be very interesting blocked into you uh it is memory structure the stores validate blocks that are received from the excess layer it is the it's needed as the separate entity that allows us to have multiple multi-side blocks that can potentially be received and it does reorganization on the r canonical change that's database Mercury is of course calculated to check everything uh why this is possible why it was not possible with proof of work is for proof stake as we have consensus we have finality finality allows us to have a limited amount of blocks that we can reorg in the practice that is just one block but in the theory can be 642 epochs engine is the thing that drives the red uh basic engine is the starting point of the red engine is the one that handles engine API that is consensus lab blocks that you receive um and focus updated we receive is switches between history sync that is in the pipeline and the live thing that is in the tree it's basic translation point of the node and it basically requests if needed it payload building basically creating new block other module transaction pool is expected it's like it's very complex thing but it most of the most of the devs just spec to work as like it needs to be DDOS protected it needs to be it needs to have like uh ordering of the transactions by base fit priority it needs to have buffer for the nose gaps everything around that sounds simple but it was not Ram is the evm basically that I built two years ago it uh it is tested it passes all the term tests it is used in The Foundry additional IPC basically standard APC that you expect everywhere and we have metrics and parameters this looks that's very rough uh component flow this looks something like this engine is at the top it drives three Pipeline and building pipelines uh basically takes blocks from the downloader and fetches from P2P but the blockchain tree and pipeline uh touches provided to send write data or your gift needed development insights even with evm model done it was very hard to to figure out how to do it we had few ideas that that was not when we started implementing it when we started basically doing it we figure out it are not good enough and we needed to switch up at um there there are a few dead ends that we approached and basically reverted back uh we initially started with the pipeline and just wanted to be like pipeline to be the main component that does everything uh very soon we figure out that we overburding one component of our system and we needed to build multiple multiple bands uh we wanted to basically use pipeline for the finalization and like the using and like integrating pending side blocks everything around it but figure out the three and separate components are basically better option we wanted to pipeline food to be the main entry of the red basically the pipeline starts everything and figures what needs to be done but we move that logic inside the engine so dungeon drives everything pipeline is there for the initial Sync It's optimized only for one thing even that was very hard to do that initial sync and everything about it we tried to use all the build mokuchi implementation but there was a problem uh having optimized way to McClain's full State and one point of time we had a lot of memory uh problems there so let me build our own and the biggest Insight is basically speed depends on the data how we Access Data hybrid data it matters a lot if you're writing if you're just a padding data to the table you know the order you know have block numbers that you're receiving just push it there is bigger there is even difference if you are writing randomly and if you are writing um a lot of data that is pre-sorted even that get can get you like uh big Improvement in the speed one of the things that we started is the sexual average granularity granularity in sense that I have not could access a history State on transaction level if we initially thought hey this is a good idea it will slightly increase the database but it will have better performance but after she researched and talked basically we already built that and so that the database was not that small it became a little bit bigger and we figured out that for the history access users mostly use providers everybody that uses that uses Block Level indexing and they even like optimize for the multi-block logs State accesses basically build it uh build Block Level granularity async our database a little bit uh impression data was fun one this on the main the section was we have our according that compresses a little bit of data but is encoding smart they have you basically remove zeros but for some Fields intersection receipts we introduce standardized compression you can have you can look at PR and it was like 300 200 300 gigabytes smaller for the section and around 200 200 for receipts we was we was very surprised it was this much current status basically starting with this sentence just leaving this sentence was like convicting uh for the seven eight months we build that client from scratch and we are thinking in it even that is enormous job enormous success uh project feels good uh in a sense that we figure out what module how they're connecting uh we figure out how is going to work there is some bugs that we need squash but in the way the structure of the project feels very good still it's not even Alpha to be honest it's pre-alpha even even before that we we are going to break database a few times we know it we are working currently in stabilizing some things cleaning up some code that we know that can be better um either way it sinks it is work with consensus layer everything around it but so so you know database is going to break uh in Broad Strokes we are there we want to consolidate that libraries that I talked about to add a little bit more documentation just to make it a little bit easier for the guys that just want to use some components just to use it and that's very powerful thing that we can do we open for the feedback contribution I think people don't notice but working on open open source project I worked from open ethereum in 2020 to now feedback is important that's how the communities build that's how you figure out your next bet you you figure out what's important for the people that use your software you can figure out what's the next step for the project so the it's very important other than that bug reporting and just reporting problems it's one of the major things have you stop stabilize the software we are around 100 contributors Community is very important the basic the guys that build The Foundry and Foundry was became something special in ethereum X system are now building red and you can expect same thing that you can find The Foundry to find the red so the same Community feedback same community outreach and everything about it the team already knows how to build it I think that's it thank you for listening questions um hi so for the initial State sync is it like Aragon that it downloads the Statewide BitTorrent not exactly we are still figuring out how to do that snap seeking snapseeking State downloading around that first it our first interaction was like let's our heisting from the beginning to the end after that we will figure out what's our next step we want to consolidate what we are currently having and then talk about is it going to be BitTorrent is it maybe is it possible to do it like Snap sync that does that get and nethermine does it maybe there is some different ways we can do it but either way it's now open question but this is something that we are probably going in some way to implement in the future okay thank you uh we should practice here yeah so I quite like that breath was doing uh transaction level State um you know for replays it makes life a lot easier is it irrevocable now that you you're moving to block level State yeah we removed that part basically it was harder to support both of them and we figure out we you can still execute all the section in the block to get the state of distraction but for the history access the main important thing is like the block Cloud you can you are basically iterating over logs of multiple blocks irritating over the blocks of multiple section particular they will be ability to have like give me traces of that suction it will be a little bit longer because you need to execute pass detection but either way we have fast evm we have the state you can expect performance as The Foundry has it basically funded does it in same way yeah question there all right or left depending on is it working yeah yeah um thanks for the talk you said you wrote the code from scratch but you know you use a lot of code from sporting VM in akola uh not exactly evm I built two years ago this not new component because uh there was drama related that's fun Monument drama that unfortunately it happened but at that point of time we had like 20 of the code it was I'm not sure basically but it seems Twitter just come to the train and just I'm not sure what happened Aragon basically deprecated a lot of things at one point that point of time they removed polygon they removed binance and they deprecated their client and I'm not sure but it seems that the main developer just wanted to introduce drama so there's no code sorry there is some code we use mdbx wrap around it and I think we use LP that's very small part of the code we are not forking akula that basically that's how the Aragon turbo gets started we built it from scratch and it was like what else to say thank you one quick question about the uh I mean I also talked with the Lesser Minds teams and the other developers and they told me that why gas is so popular because more like EF I mean is a foundation is more likely to support them so well this case also be a situation for your projects I mean well EF I mean they provide limited help for Netherland but I mean for this one are they interested or they will also provide limited help yeah uh basically this undertaking is mostly driven by Paradigm I'm not sure about future uh maybe that basically a talk with ethereum Foundation the guys from Material quotation are testing the client as the a lot of providers you're in contact with them basically I was the guy that was on open ethereum uh Paradigm is normally the space and they're basically Building open source client and it's never you can be you cannot be more person than MIT in Apache um I think that's it thank you very much [Applause]