# Trains, Planes and Network Upgrades by Danno Ferrin & Tim Beiko  (Devcon5)

- Speakers: [Danno Ferrin](https://streameth.org/speakers/danno-ferrin), [Tim Beiko](https://streameth.org/speakers/tim-beiko)
- Channel: [Devcon](https://streameth.org/devcon)
- Date: 2020-10-09
- Duration: 16:40
- Topics: blockchain, smart contracts, distributed ledger, ethereum
- Watch: https://streameth.org/watch/yt-vsA57h052hU
- YouTube: https://www.youtube.com/watch?v=vsA57h052hU

## Description

Visit the https://archive.devcon.org/ to gain access to the entire library of Devcon talks with the ease of filtering, playlists, personalized suggestions, decentralized access on IPFS and more.
https://archive.devcon.org/archive/watch/5/trains-planes-and-network-upgrades-a-regular-release-cadence

How does an Ethereum Improvement Proposal (EIP) get deployed onto the Ethereum mainnet?  At times it feels like an ‘80s buddy comedy, but as part of the Ethereum 1.x initiative, the process should become as predictable as a clock. In this talk, we will propose a regularly scheduled process modeled after Linux distributions and the Java platform that Ethereum could use for its future network upgrades. Frequent and regular network upgrades will remove the time pressure to submit and review EIPs, provide predictability to applications building on Ethereum and reduce the burden on core developers to begin implementing improvements that are not ready. Once a network upgrade leaves the station, another one will be coming right around the corner!

Speaker(s): Danno Ferrin, Tim Beiko
Skill level: Intermediate
Track: Execution layer
Keywords: EIP, upgrades, implementation, technical

Follow us: https://twitter.com/efdevcon, https://twitter.com/ethereum
Learn more about devcon: https://www.devcon.org/
Learn more about ethereum: https://ethereum.org/ 

Devcon is the Ethereum conference for developers, researchers, thinkers, and makers. 
Devcon 5 was held in Osaka, Japan on Oct 8 - 11, 2019.
Devcon is organized and presented by the Ethereum Foundation, with the support of our sponsors. To find out more, please visit https://ethereum.foundation/

## Transcript

Aloha I'm Danno Ferran and I'm Tim product manager at Pegasus I am a network protocol engineer and our talk today is to propose and talk about a proposed cadence for network upgrades which you might also call hard Forks and changing it from the old model which is kind of like buying a plane ticket everyone gets on the same plane you pick seats you do a months and years in advance and things don't change it flies out or it doesn't and instead of replacing with a train model where you show up at the train platform and whoever's there is leaves and whoever's there doesn't but first let's start with a little bit of history what did network upgrades look like before this it was a lot like a family vacation frontier homestead Byzantium constant Opel mom and dad would tell us where we're going we'd pack the car you sneak in your favorite toy we drive and drive and drive we'd get there eventually and it would all be happiness in sunshine in the end except when it wasn't there were a few network upgrades that were a little more emergency all-hands-on-deck type situations the first two were the Shanghai attack and the second one was then there was a security flaw discovered as part of Constantinople in those situations we all came together we all got the network upgrade out in a timely and fast fashion laser focused on what the problems were so where does that put us today so the beginning of the year we proposed a plan for instant Bowl where we would take everything from the beginning and we'd have landmarks along the way that we would do things like have the IQs ready and all that other stuff go from step to step down the waterfall and it was gonna be rainbows in the sunshine it was gonna be awesomeness right yeah well like everyone in this room probably knows Waterfall doesn't work for software development but we tried it anyway for a spam bowl and so at the beginning of the years we set ourselves a bunch of deadlines we would have a kickoff in January give ourselves a good amount of time to review all the eeap's so that around mid-may we could have a final list this would give us two months to work on client implementations which brought us to mid-july this way we could deploy a test nuts around mid-august and finally have her upgrade go live on main net by mid October which was supposed to be two weeks before DEFCON we're forking next week no we're not and so how many of you have these deadlines we actually it we hit the first one the kickoff and obviously a lot of things went wrong here and one of the more interesting ones is that the theorem community had grown a lot since the previous upgrade so when it came time to review the eats instead of having only a handful to review there were so many that I wasn't able to fit them on this live using the font they gave us and so this means that around mid-may when we should have had a finalists of Eve's for the upgrade we're still trying to wrap her head around all of these they were in completely different stages some of them were early drafts while others had working test nuts and there were dependencies across all of them so this was not ideal and we obviously missed that deadline so we actually got her and her final list of eeap's for Istanbul around mid-july which is when we should have been having the clients implementation done this pushed back the clients implementation to the end of this summer around August which is when we should have gone live on the test nets and it pushed back the test net upgrades to October which is when we should have had a Romanian upgrade at this point the minute upgrade will probably be at the end of the year or early 2020 and so as this was happening a lot of people realized it was not going super well and there were suggestions made around how we can make Berlin the next upgrade go much smoother than Istanbul so will not walk you through some of these ideas that the community you've brought forward and try to bundle them together into what we've called the train station model the first person to notice how poorly things were going was Alexia coup nob the brains behind state rent and stateless clients and he posted a blog post or outline basically where things were at where things could be and where they should go on the first picture outlines what was going on with the first set of network upgrades there was a everyone would come together they would write their their tests but there was one special client that had special privileges a left the C++ client was the one client that could actually produce the reference tests that were needed by all of the other clients to verify that all of their consensus critical code was correct it was going to function properly so before anything could progress out left had to have all of their stuff written in the tests written and then they had to go work with the other clients and work back and forth they created a gating function this is why a lot of the network upgrades moved in lockstep before because there was one tasks that had to be accomplished so what Alexei proposed is instead of focusing on as a group and having one client and doing everything at once that we split these these upgrades the details into various working group so to have various areas of concern and interest that they would recommend proposals in in these network groups they would produce reference implementations for what they thought needed to be changed now what he proposes a very important thing for the reference implementations it was going to use a new tool written by Dmitry from the etherium testing team called retest F what Dmitry did is he when he got all of the testing code that I left needed to produce these reference tests and he took out the need to have to have a left run these so you can create a reference test from any client that implements the api's that retest death uses death has implemented them basically was implemented them and obviously Alice's implements of them so from these you can create your iips and you can target any of those three platforms right now to generate your reference tests to create an EIP to test so we are not tied to just one implementation and one implementer to produce the reference tests anymore the second idea that came up as a need for EIP champions on this is a picture of tweets from the berlin all-court devs meeting and from there alyx different different alexei but alex proposed that each AIP needs a champion and boris echoed it said yes the IPS need a human being that we can go to then we can talk to and say hey what is the status of api p1 human that you call into the alcor devs call and say hey state rent is going more into stateless clients right now so you won't be seeing this for a while it needs one human who coordinates so when we say hey we need more tests for a IP 2200 that we can talk to that person that can go back to the developers and they can create those new tests it is not just one person do all those things but if the IP is small enough where the person is dedicated enough it could be the person implementing it but it is one person who coordinates and is accountable and champions the EIP through the process taking it through the steps as available and another big idea that was introduced this year is the concept of a IP centric 4 key this was brought forward by Martin and the theorem foundation and the general idea is that instead of having all our apes move to the various stages of the upgrade in a at the same time you have each a progress independently and you only schedule them for an upgrade when they get to a point where they're ready to do so so using this model if you wanted to get an e plive on main net the way you would go about it is this you start you first start by writing your EEP and then you'll go on all core devs to get what is called an initial acceptance this means that the core developers are generally positive towards your idea and that assuming everything goes well on the next steps it except a PR for a well-written implementation of your EEP in their client and this can be a really useful signaling mechanism for organizations that fund some of the teams working on these eats organiz I see organizations like the EF like Malik Dow or like consensus can use this initially acceptance status to note that their funding in efforts that has a fairly high likelihood of making it on main net so once you have this initial acceptance stamp you go and work on your reference implementation this just means implementing your EEP against one of the major clients and then generate your reference tests using region instead this would kick off a security evaluation period of your EEP where you want to reach out to people in aetherium who are familiar with the bits of the system you're changing so that they can poke at your eek as you're going through all of this you want to feed everything you learned back into your each specification and specifically under the security consideration section once you've done all that so you've got in your initial acceptance your reference implementation and your testing you want to go back on our core devs to get your eat move to accepted at that point all the other maintenance lions will implement it and then they'll be scheduled for a test net upgrade once that upgrade goes live assuming everything is fine then it'll be scheduled for main net and will be deployed on the maximum net upgrade and James Hancock put together a really good diagram to explain this process so you see it step 0 you start off with your draft of an deep and they're the only Gaming function is you need to get it approved by EIP editors this just means your EEP meets the basic templates requirements from there you get a green light from all core devs or an initial acceptance which means that they're generally positive towards your idea and that did accept a good PR from it in their codebase you start working on your implementation and then you own the testing for it so not only are you generating the reference tests and trying to test both the happy path as well as the quirks and edge cases very specific eats but you also want to get in touch with the EF testing team and other people who are more knowledgeable about what you're changing to make sure that all of the important security considerations get fed back into the spec once you've done all that you go back on all core devs get your eat move to accept it and it'll be scheduled for the next upgrade at that point it'll move into last call and we'll keep running the reference test against it all the way up to the test nut block if it goes live on the test nets and everything is fine then it'll be scheduled for main net and finally your equal relies on main net finally as Tim mentioned the etherium community is growing there's more people that we need to take considerations into rather than just the developers and the researchers who are creating these items we have people running exchanges that people running their own personal Nords for corporate for personal information we have corporations running nodes and we have service providers such as in Ferrara that run these aetherium nodes and when you know take into consideration that they need some some standard amount of time to react to these node upgrades so a few months ago I proposed a IP 1872 and set of upgrade windows to tell us when we should do our network upgrades rather than just picking a date random date when we're ready to fork so there were several recommendations in there some of these I feel stronger about than others the first recommendation that I feel the most strongest about is that we should fork on the third Wednesday of a month kind of like how Microsoft has their Patch Tuesday where their security patches will come out on a second Tuesday of the month unless it's emergency patch that way so submit administrators know that they should not take vacations during the second week of a month because that's when a lot of the work is going to come around to testing and upgrade in the clients similarly note operators would know that the third week of a month is probably not the best time to take vacation because that's when a network upgrade could occur now these could occur in any of the twelve months but the second thing that I recommend this is more of a recommendation is that we put four specific months that we prefer to launch main nets main an upgrades those will be January April July in October those are three months apart and those who have the advantage of missing a lot of US and European centric conflicts around vacations and holidays so that the operators in those areas won't feel any conflicts with those however sometimes it comes it's necessary to delay a network upgrade has happened with Constantinople in those cases the delays would go out in month increments if we have to delay it we pick a new third month the third Wednesday of the month probably the next month to shift it out a month and the aim here for this is to do these network upgrades twice a year we would do one in January and one and not July or one in April and one or October so once you schedule one you would do one six months later I mean of course in this recommendation you could always fight fires whenever and wherever they happen if we have another situation like Constantinople or the Shanghai attacks we can upgrade it on very short notice or we couldn't cancel a fork and move it out on very short notice if there's an emergency those recommendations do not constrain us on acting on those emergencies but if it's not an emergency the discipline that comes with the time frame like this will be appreciated by the people who have to operate the etherium nodes that all of our networks depend on so it's synthesis from all of these four recommendations we get to the Train model so rather than a model where we're all flying on an airplane and we're land we're going for a specific flight at a specific time instead it's like we're going to a train station and as the trains come and go you come and go yeah and there's four main ideas here so the first is that the each should progress independently like we said before you don't want to have all the eeap's moving in lockstep towards towards the various stages of the process instead every epin working group moved at their own pace and whenever it's something is ready to ship only then do you schedule it for an actual upgrade the second point is that EAPs have a champion a human being who the all core devs can contact to say hey is this EIP ready for this network upgrade should we move it to the next one it's a single point of contact who can answer these questions and help move the EIP through the process maybe motivate the team to make deadlines and maybe just let all core devs know that we found an issue and we're not going to make this network up and the third idea is that whatever is done is whatever shift is whatever shifts sorry so this way an upgrade consists of basically the eats that are in a spot where they're ready to go live anything that's still in progress gets moved to the next upgrade like we said if there's any last-minute issue then you can obviously kick that eat out kick that EEP out to the next train as well and the final recommendation is that we move to semi-annual network upgrades rather than the annual or less upgrades that we've doing in the past one of the big advantages that we get out of this is that when we tell an EIP that you're not ready for this network upgrade the next upgrade is only six months down the road rather than a year or more down the road so they know that they're there the work that they put into will still show up in a reasonable timeframe rather than some unknown timeframe in the future you'll reduce a lot of the anxiety when it comes to scheduling and moving stuff in and out of a network upgrade and from this we can make sure that what goes into the network upgrade is the good stuff and like we said earlier we didn't come up with all of these ideas the community mostly did so we wanted to make sure we linked back to them so anyone who's interested can go and dig it a bit deeper to the original sources yeah if you google alexia to knob that's the article that you'll find and so yeah that's what we had thank you all for listening and we have a couple minutes for questions if anyone has any hey thanks for the thanks for the talk I was just curious what's required to get miners on board at this like plan of doing regular hard Forks for upgrades and are there any like difficulty adjustments that are happening with each of those hard Forks yeah on board miners and miners I don't think you would change anything for miners so the only obviously like the the difficulty bomb is one thing that comes up and that's usually solved through an e p-- so if we want if we want to delay the difficulty bombed again we simply choose whatever upgrade we want to do that in and and and and add that to that upgrade so the idea then that the difficulty bombs would be changed to line up with this schedule that you're proposing to be determined in historically for the past two upgrades it has picked it one fork but my understanding was that they were trying to kick it out more than one pork it's just we've had difficulties getting these network upgrades out thank you and one of the other reasons for proposing the regular time frame is so the miners know hey third Wednesday of a month is coming up we should check aetherium are reporting are we not so the miners are another important remember this community that we need to take into consideration any other questions okay back to everybody you