# Year working with Geth on a production Ethereum application by Adam Link & Don Mosites (Devcon4)

- Channel: [Devcon](https://streameth.org/devcon)
- Date: 2019-10-09
- Duration: 21:01
- Topics: blockchain, smart contracts, distributed ledger, Science & Technology
- Watch: https://streameth.org/watch/yt-vl-ZCOEoYio
- YouTube: https://www.youtube.com/watch?v=vl-ZCOEoYio

## Description

Visit the https://archive.devcon.org/ to gain access to the entire library of Devcon talks with the ease of filtering, playlists, personalized suggestions, decentralized access on IPFS and more.
https://archive.devcon.org/archive/watch/4/year-working-with-geth-on-a-production-ethereum-application

AirSwap is a decentralized trading network. As Infura can attest, these applications end up making quite a few calls to Ethereum nodes. Most of the calls are simple lookup calls just checking balances. In addition, because the network supports trading there are many transferring token calls as well. This talk focuses on the evolution of AirSwap's geth infrastructure through the course of the year and how the setup has changed. We started from using Infura, to a single node, to making an in-house cluster. Along the way, we've experienced down times, slow syncs, and code issues. While, the infrastructure is bound to change as the ecosystem shifts, we'd love to share what we've learned and a bit of our setup.

Speaker(s): Adam Link, Don Mosites
Skill level: Intermediate
Track: Developer Infrastructure
Keywords: airswap, trading, geth, infura, infrastructure

Follow us: https://twitter.com/efdevcon, https://twitter.com/ethereum
Learn more about devcon: https://www.devcon.org/
Learn more about ethereum: https://ethereum.org/ 

Devcon is the Ethereum conference for developers, researchers, thinkers, and makers. 
Devcon 4 was held in Prague, Czech Republic on Oct 30 - Nov 2, 2018.
Devcon is organized and presented by the Ethereum Foundation, with the support of our sponsors. To find out more, please visit https://ethereum.foundation/

## Transcript

all right well hello everyone I'm Adam link I'm an engineer with air swap and today we are going to be talking about running a guest node cluster in in production Netscape so as was alluded to about 15 minutes ago there is no open source way to run a guest node at scale we will be open sourcing our production infrastructure in about 20 minutes yep so this is the work about three developers over about two months to develop the infrastructure with a lot of the pitfalls that we experienced along the way so it's at least something for you guys to reference as you're building a Gotha node cluster or perhaps use on your own so at the core this adapts need to keep state right when you run a DAP your data store is inherently distributed and distributed datastore means that you need to stay in sync with your peers to decide what the correct view of that data actually is there's a whole host of issues that can occur if your front end or your back and application does not properly reflect what the consensus of the database state actually is and there's nothing worse than users not trusting your application because they can't trust the underlying data but you also can't sacrifice the responsiveness of your application when you're waiting on an eventually consistent database users on the Internet are generally not accustomed to waiting five minutes to do a state change operation and while we in the industry may be okay with waiting for transactions to settle if we want to actually get mass adoption we need to we do understand the general public wants which is a lack of leg when they're doing a state change operation there's two types of data that you need for state management the first type is the balances on your accounts right this is the eath that you hold in your wallet and for many depths this is a critical piece of data in the user onboarding process because they may need to swap their eath for your proprietary DAP token in order to use the product that you're creating the second type of information is the the state tree this is information about events within your smart contract this is things like the users token balance smart contract function executions and token balance transfers however contract events which is the state of your smart contract are actually implicit and computed based on the data in the blockchain so this means that your DAP needs processing power behind it when you're running your ethereum notes if you're out of sync with your nodes obviously you're misrepresenting the state of the world and that's a dangerous place to be in as an application so with the importance of staying in sync now for your DAP kind of resting on your shoulders as a developer some of you guys and the industry as a whole we've wondered can we just outsource this and pay somebody else to do it and the answer is of course you can there's many companies out there that actually offer services where they will run nodes for you not all of them are geared towards OLTP processing that adapt needs some of them are more analytics based you know Google recently came out with a product like this but the question we need to ask ourselves as an industry is do we want to move towards the MasterCard Visa model right so right now with credit card processing there's only a few companies in the world that have the technical expertise software and hardware to actually keep up with the number of transactions that happen on credit cards and as we move state management towards centralized parties we move the entire industry towards that model again and the question is do we want to do that do we actually want to be there the solution in our mind to combat the centralization is having people run their own nodes but running an ethereum node at scale in production is really not well documented online right now there's also costs associated with running your own notes not well-documented either so not only do you have the overhead costs of the actual infrastructure for running your own node but you also have costs on your team to create the infrastructure and create the software that manages your nodes make sure that your nodes stay in sync there's a lot of resources online pertaining to optimizing nodes for mining but there's not many resources optimized for reading and read heavy workloads that adapt needs it gets even worse when you start going into data querying and trying to understand how you process this data there's very few resources online for data querying for taking block data and storing it internally we as well as everyone else who was up here before I have created tools for doing that we're not outsourcing those yet but you know that is something that we are all creating our own data querying layer on top of the blockchain and there are some companies right now like the graph that are actively working on making a way to query using graph QL but they also still require that you run your own nodes or you find somebody that can run nodes for you to get the underlying state data so right here is a simplistic view of our death architecture we run a production cluster and it's basically a group of load-balanced death nodes behind RS re in an auto scaling group behind an application load balancer on AWS the application load balancer allows us to upgrade non-secure requests to TLS we can also run very tight health checks to cycle out nodes that are not healthy out of the load balancer target group so if they fall behind for instance like the head of the chain and usually what we found in our internal test is we recommend studying the frequency of those health checks to a time that is less than a single block propagation time so right now that's doing a health check inside of a 15-second window this allows nodes that fall behind to be dropped at the pool quickly so they can sink back in so you're not querying nodes that you know are behind in state the load-balanced nature of our clusters also means that if a single node does fall behind our overarching service doesn't actually fall out of sync and become unavailable so we use our goth nodes on the front end you I on the air swap platform as well as in the backend for state management and storage on the front end this takes the form of balance checks in the UI wrapping and unwrapping width and also approvals an approval state for the atomic swaps so if you've used the air swap platform you know that I like on the new spaces on the right hand side you know we have a bunch of information that comes from our Gotthard state we also use a heavy polling method to handle trade confirmations so if you've ever popped open the network tab in chrome while using air swap you'll notice that we do heavy polling against a gap cluster basically when you do a trade to make sure the trade actually goes through and gets executed we also do allow limited very limited third party access to our GAF cluster we do have some of our maker partners who use our GAF cluster as well for balance checks both for their own wallets that they operate out of as well as counterparty balance checks to make sure that the counterparty taking a trade is a valid taker further our parties and also use the GAF cluster for market data and price information in addition with a couple of their sources but we we do provide chain state to them for pricing information when we were setting up our Gotha cluster for the first time we ran into many issues with the way gift was written and our particular use case of heavy reads this is not a knock and the guys that built Geth it's a great piece of software but we just operated it at a scale that really no one else had been publicly talking about so the first problem we had was GAF wasn't written with modern pub/sub methods this meant that you have to do heavy frequent polling in order to get state changes and you can't just simply subscribe to the events being emitted out of the blockchain that you want to get now that that's kind of true because you can with filters but in and I've heard a rumor here that this may no longer be true but at the time filters were dropped on a node resetting and you'll find out in a couple slides whoa that was a huge issue for us but basically the filters being dropped was a hindrance to us properly getting a pub/sub system setup and then further complicating that is filters aren't a single node only so when you cost your nose behind a load balancer the round-robin aspect the load balancer means you can no longer guarantee which node is being hit and if the filter even existed there one of the solutions we found at the time was a node.js library that basically allowed you to do a multi node filter setup and all it would do is iterate over an array of guest nodes and set up filters on each node it was a heavy client-side solution and we really didn't think that that was what we wanted to do we were hoping for more of a server-side solution rather than relying on a client implementation so the underlying issue that we found in some of the people have alluded to is the overall ecosystem of ethereum is meant to be fault tolerant when it comes to chain State however there's no real thought that's been given to the notion of creating a subset of nodes that are also fault tolerant in and of themselves and so with adapt this means either querying a single node and praying that it doesn't go down using a bunch of third party nodes and hoping you have the right state or doing what in fira did which is writing a really cool piece of software and hardware called ferryman which basically takes introspects the request coming in and then it sends the request off to the proper type of node also provides a modern pub sub layer on top of that and then allows for you to subscribe to client events the problem is the inferior solution is very proprietary and we make sense for their setup so they can't really open source that to the community our story on our gaff nodes we do about 800 requests per second on our gatherings requests per day that's over to clusters of three nodes each so that's a fairly large amount of traffic and under this load death actually broke we had a massive memory usage issue and a recurring memory leak that caused each node to slowly utilize all the RAM available on the box we tried scaling up our hardware - this is AWS terms but m-52 X larges so it's a 32 gig of ram 8 v cpu piece of hardware and with six of those in production you should be able to do substantially more than 70 million requests per day but as you can see from the chart here after about 15 hours we had topped out our memory right at 32 gigs and the drops you see where we actually went in physically and restarted the goth process the memory leak meant that we had to cycle our gas nodes before the GAF processes used all the available ram in fact if we didn't catch it in time the box itself was so the ram was still used up you couldn't even SSH into the box to kill the GAF process you had to restart the whole node itself and this meant that we were waking up every three to four hours to check memory consumption and manually we start the process to make sure we didn't get locked out of our boxes and once we figure that's the usual cycle of RAM usage we were able to write cron jobs to then go in and restart our Geth process and we staged that over about an hour across six nodes and it worked but for any of you guys that work in infrastructure trusting your entire system to six cron jobs is incredibly scary so we started tinkering with some of our guest settings and we actually dropped our max peers setting and the memory usage issue was largely resolved and it turned out - the memory leak with the max peer setting really only occurred at the scale at which we were operating and it wasn't publicly documented prior to that as you can see in the charts here those those drops like that's when we were restarting death the stability that we did finally get came at an incredible cost to our operations team so for at least two of us we were getting nothing done during the day other than goth node work and I can't that your sleep quality certainly suffers when you're waking up every few hours to make sure that your boxes aren't freezing or running out of memory also your morale really suffers when every morning before breakfast you wake up to 50-plus pager duty alerts and they just keep rolling in throughout the day so we wanted to fix this and we embarked any massive experimentation to find the correct hardware and software combination that would be the most stable and cost-effective we ended up iterating over ten different instance types spinning up everything from t2 micros to four Excel boxes and we tried everything from compute focus memory focus instant store focused trying to find and profile the way that the Geth process worked under the loads that we were seeing we made some concessions along the way that we have built into our open source solution unfortunately no pub/sub method meant that we were just gonna brute force it and that was one of our big decisions out of the bat out of the gate we couldn't guarantee that a single node was up so we needed to load balancer nodes we couldn't guarantee there are nodes were in sync and up so we needed to write health checks we also couldn't guarantee that a node was at the latest head so we wanted to do pure checking to make sure that we were actually at the the head state that we thought we were so that combination was kind of the secret sauce that no one was really talking about online which was aggressive health checking aggressive peer checking load balancing and improper hardware so let's talk about the infrastructure we actually run in depth is what we're gonna be open sourcing so when when we run or knows you run on top of Boone - so we use the service construct to start our proprietary tasks here we're actually starting the death process so this is in the open source repo you don't need to worry if you can't read the actual code but let's break down the options we use so we brought the max peers down to fix 50 this solved memory leak issue and also helped with the data transfer costs that we'll talk about a couple of slides we use a custom data Durr parameter to take advantage of the nvme drives that AWS recently made available right now our default is cores turn everywhere you may want to restrict this for your own implementation we use a four gig cache to speed up the initial sync process this is the health check that we use to keep our nodes in sync it's probably pretty small but what it essentially is is a node.js Express app that just does the waterfall queries over our peers to determine state so we compared to in fira and ether scan but you can certainly customize this to run on additional peers we consider ourselves to be in sync if we're within 10 blocks of the head and you can make this tighter but we found that if you tighten it up too close to the actual head state like 3 blocks or less you will occasionally drop entirely out of sync with all your nodes when you get a small chain split at the head which happens on occasion so we decided to rather stay in sync all right sorry available then it completely in sync all the time we returned back HTTP status codes that do correctly identify whether the service is up or down we also returned back a network timeout because we found out occasionally in fira just times out our requests we've got a couple of the nifty features health check is started as a service we've run into issues before where our health check was down but the node was actually off we have cron jobs that reset the max open file limit this was something that Goeff had an issue with earlier where it was not respecting the max open file limit so we just wrote a cron job to fix this and then we wrapped this all in a cloud init script so it automatically spins up the boxes installs all the required programs death mounts the nvme drives and starts the services so your boxes spin up and they start syncing you can actually check that in a health check that we publish as to whether or not your node is running one of the biggest takeaways we had was the hardware selection that we used so we're gonna talk about IO right now magnetic and spinning disk drives just don't work you just don't stay in sync you become IO bound and the the node will freeze similarly provision die ops is actually not cost effective you over allocate your I UPS and you're only using your max I it's about three percent of the time so this means you're just wasting money so we try moving back to births based EBS drives and you have to actually allocate three terabytes of storage space to get the provisioned I ops that you need but again because now you're below where you'd want to be in AI ops you're actually hitting caps when a large block comes in and so now you have a right to that will build up over time you know it falls out of sync EFS is something that AWS provides an infinitely scalable network attached storage and the network overhead actually makes it so your node will never sink as well so just don't try that and nvme drives were announced a few months ago that's actually what we use it is a very good io based instance store drive familywize death is io bounded incoming data and compute bound and serving requests so m4 and m5 general classes over allocate memory we don't use those any other memory heavy family is actually not going to be cost effective I three classes is actually what inferior uses or use the time I talk to them so they work well but nvm you drives are now back out and that's a better instance store than the i3 class we run C 5 D is in production this is the proper mix of compute power with IO 4 what we what we run we found anything smaller than a large instance is actually not going to give you enough power to run your nodes in production at scale and any other family than these it it's just cost too much so we run 6 c 5 d 4 excels in production and we have not had a single instance the downtime since August last factors cost so what do you pay for you pay for the ec2 instances pay for the load balancers and then when you also pay for outbound data transfer something we didn't know when we had our max pure set as high as they were was your outbound data transfer actually runs into the thousands of dollars per month so we've dropped this down with 50 peers we actually end up with about 2 to 3 terabytes out every month real quick slide why not parody when we did parody about 5 months ago we were getting a bunch of false nodes so we were not actually staying in sync appropriately with enough peers so we just dropped them but version 2.0 we certainly want to go ahead and looking at other clients to bring out and I think that's something that we you know kind of call for help on the open-source side if you guys know how to properly configure parity like this month that would be great and lastly here's the CloudFormation stack like I said we do run an AWS but this URL will take you basically to our github repository where you can go ahead and click with a single launch stack spin up the exact stack that we run in production that's all I have thanks guys [Applause] you