# Validating Designs and Finding Points of Failure: Testing ETH 1.x and ETH 2.0 Against AI Agents

- Channel: [Devcon](https://streameth.org/devcon)
- Date: 2020-10-09
- Duration: 17:50
- Topics: blockchain, smart contracts, distributed ledger, ethereum, Science & Technology
- Watch: https://streameth.org/watch/yt-usKTfMamSNk
- YouTube: https://www.youtube.com/watch?v=usKTfMamSNk

## Description

Visit the https://archive.devcon.org/ to gain access to the entire library of Devcon talks with the ease of filtering, playlists, personalized suggestions, decentralized access on IPFS and more.
https://archive.devcon.org/archive/watch/5/validating-designs-and-finding-points-of-failure-testing-eth-1x-and-eth-20-against-ai-agents

We’ve introduced the notion of machine learning algorithms in our network of simulator: Wittgenstein. We explore the different strategies that can be taken by participants in the network to attack the system or manipulate protocol’s design to increase rewards. We focus specifically in reinforcement learning, and set up different agents that engage in different byzantine behaviours. We present results and guidelines to improve the design of protocols such as PoW, Casper and others.

Speaker(s): Olivier Bégassat, Vanessa Bridge
Skill level: Intermediate
Track: Execution Layer
Keywords: technical

Follow us: https://twitter.com/efdevcon, https://twitter.com/ethereum
Learn more about devcon: https://www.devcon.org/
Learn more about ethereum: https://ethereum.org/ 

Devcon is the Ethereum conference for developers, researchers, thinkers, and makers. 
Devcon 5 was held in Osaka, Japan on Oct 8 - 11, 2019.
Devcon is organized and presented by the Ethereum Foundation, with the support of our sponsors. To find out more, please visit https://ethereum.foundation/

## Transcript

[Applause] so today I'm here to talk about validating verticals and then a neat one and when they need to level using AI agents so we focused on the ant no matter what I'm learning and you may be wondering why are we looking at this how does it fit I am Shila me and I'm gonna explain all of us and talk you through that so in the past we've looked at something called selfish money which is a strategy attack on IBM through one boy next vertical where essentially you're trying to attack the network that employees the proposal calculation and you do this by essentially keeping like a chain that it's a head so essentially you don't send your blocks as soon as you receive them but you will hold them and sometimes repeatedly as you keep it heads from other - and so the end you're gonna make the oyster time by looking at a loss that you just sent and you're always keep ahead so they do ensure that you're winning and so this is a problem that's they've studied and different papers and there's different algorithms that you can follow as I described before you can hold a number of Bluffs and whenever you how elite if someone sends you a block then increasing the general heights you publish a walk that and essentially always keeping a heads up in to ensure that you can maximize aware that you getting because when you something you either get a maintained reward or an awful reward which are almost essentially the same and so I learned with the Pegasus team at consensus and we focus on research so make develop a framework to build simulations for different types of particles so we look them through from work for the state and so we decided it would be pretty interesting to move implements a selfish attack and look at the data that wouldn't return so this is a bit like oh there looks like we actually took the combination of the difficulty we're looking at how it's done if you're really just a kind of a sense of what their reward time and so the only what you're doing as I mentioned before you can get a reward from mining a block which is right now to eat when the transaction is included and if you include any animals that you receive so your ordinary - 175 125 or - and since the transaction fees that you receive this quite small sometimes we decide to not include this in our own simulation and so we decided to build a test case where you only have a network of nine miners were always fun and liner that is selfishly mining and so what happens if that there's some variation and we decided not to internal warrants of transactions and some of their results on murder this so essentially what we found is that once you have to like this time of the hashing power in the network you can start attacking it and doing selfish mining and this is more beneficial for you because this manipulates the difficulty calculation and it increases your own work so they're more hashing power you have the better for you and so if you wanted to test how would this prove if you are giving it to a reinforced not an agent and since we have actually active back it up we can compare the results and see what we would get with it and so that's why you might be wondering why are we doing reinforcement learning when we were focused on watching but if this will give you like a sense of what reinforcement learning is in case you haven't heard it is the concept where you have a piece of software it's an a gendered construct for an environment one of the most famous ones is deep mind where they implement in a deep Q network and essentially interacting with different attendings and it learns how to solve it so we thought okay we can take this concept and implemented into different protocols and just see what happens and then just like make a sense of why we chose that too is because it's an automated system that essentially has proven to beat humans and many other elements and strategies so it just made sense to go that route so how does that work you know this room or you have an agent that is collecting ice box the environment so it's partial information on the current state and then you have actions that you can take which will return a reward based on those actions and so the process into using some loop where you just keep it's raining two different actions in the ten states and so what are the different observations that you can get in terms of our product over looking at the number of blocks mine the number of luck receive or something all those different things that you're actually doing in the network and feeding that information into the agent so they can learn the impact of their actions and so it's really still blessing but they don't want to be too broad in terms of wider minor can do so we decided that you can only combine up to ten blocks hidden in your own private chain and you should be able to publish up to three blocks in a row so you could publish one two or three based on whatever it's rented al seems the best and then you would get some reward based on that and so I think I'm also making a part of setting this up and strength understand what is the best promoter can you go to an agent because you wanna give her more such that it understands what's the optimal strategy to think it also cause it was golly selfish mining without necessarily implementing the entire Center Institute and so what we decided to do in our case is to incentivize staying ahead just minding as many boss as possible faster than the other miners but there's other stretches that you can take we try to test in a few to see what is the most optimal result but eventually justice intervention using the metric of staying ahead that's probably the best and so going back see what this looks like we have on this agent that has been implemented in Python and his colleague code that is implemented in Java this mountain simulator and it's essentially going through this loop over and over again so I think actions and then collected the information that he gets from the simulation and so there's a number of different alternates that you can use as I mentioned before like one of the most famous is the network's many fiddles 103 Q learning algorithms policy 3 based algorithms they all play on different sets of statistics and modeling and use of light chains as the back of how they calculate different states and wards and so I'm excited that probably the simplest one to go for was Q learning and so illustrate and explain a little bit what Q learning is essentially so we use this Python framework that's developed by an opening eye and it allows you to create your own environment and interface that you can use and so we developed that calling the Java code and then we tested for men so we wanted to see what was the best strategy when you have 10% of the hash around 40% and 60% and then we set some hydro parameters that are related to these Q tables and how you calculate the reward so you look at the learning rate which the session is just telling you how much you love you and new information that you receive versus information that you've recorded in the past and we also have a discount factor which also pertains to the reward you want to look at the current reward that you have for an action plus there were work for making it a next option because you're trying to discover what is the most optimal path and so it's something that you have to - and in terms of how do you value the future of the Lord because you can think okay I kept mine a lot right now just have it as soon as possible and again or if - but if I decide to wait a couple of what's ahead publish it there's traffic was buying some money might cost less because of inflation or you can just think well I lined the four blocks in one minute that's a negligible it's worth the same so based on what you value most you will tune on this parameter and then finally you have epsilon which is just related to randomness how many choices do you do that our random person's time a choice if you do that you're maximizing your strategy by thinking that they and so there's two states as I mentioned before the first one is driven by Epsilon where you're just trying to make random choices so the different actions we can take and reporting the results that you get in this table and then this will allow to generate different valleys so that you can reach multiple in deciding what is the best action at each state that you reach and so then there's a theory during this that you saw some people development equations and it just said that basically by maximizing at each state that you find yourself what is the optimal action you will maximize for the entirety of your decision process and so you're just looking at maybe two before their reward and then you're also adding the value above the future award of the next state in action but with an aesthetic factor so that's what I was talking about before that zero point about you and so does the reason is filling with more concrete because it can be very abstract we can think about this Q matrix being something like this where you have four different actions so you can hold your block and not publish it just keep it in the branching you can polish one block two or three as I mentioned before this is the difference some actions you can take and then you can hold one two three four and more but for simplicity in this example and that's it three and so when you start to initialize it and all the values aren't zero and as you keep without going around them the connections you'll change the values so let's just walk through one of these examples so let's say that you start and you start in that chain you decide to hold and so you're gonna calculate what happens when you hold and then maybe the next step you want to publish it so that will generate a reward let's say to me and so you continue doing those few random paths until you generated enough values this is like random values not the actual ones but just to give you any cable what happens and then after a certain point what you start doing is that okay I find myself that I speak about holy - so I have two blocks wise my next next action sure polish them or should I call them and so you think the maximum value for each action if that's how you can find a possible path and so I don't mention briefly there's a the marker change crosses bet that's all this information and ensures of this equation holds true and it's just an ATS so that you can also visualize this in wait you can have a chain where you're mining and then you can serve you won't write a chain and keep switching between one and the other and that essentially because of the properties of our chain use you reach the steady state and that's why this come table is useful because you can assume that to exert a number of steps now you've reached the same and you were off to infinity and so that's how you can maximize the values and so there's a lot of attention challenges we can get this approach because you need to understand what's the best data that we eat you need to test for all these hyperparameters and keep changing them until you can find something optimal and it's also a consideration of how do you build this what are the behaviors that you incentivize how do you to find something that's visiting and so on there's a lot of things that you can buy challenges with and so it just to go back to where you can look through some work we took a survey shion's we were distracted what is the number of blocks that you've published in the public chain private chain I'm keeping track of all calls and forts and then letting it know so we can make the best actions and so we ran this similar to what we were in in the self-contained simulation lying on its miners and their one is my nerdiness top date through the reinforcement learning agent so we measured it at 10 40 X 60 percent hashing power and we collected the results and so as I mentioned before we use the simulator and John we use the spray wearing company I - so that's a big publishing memory libraries and then I ran all of these tests in a double us so that we could test it collect all the data and so this is just an overview of what we were doing and the different hashing rates we collected the reward only a total of one hour remaining we also collected over the total of episodes or simulation so essentially all raw element my agent run for an hour or until it has mine a thousand watts and record that behavior and I run it for ten thousand episodes while you're changing the actions that you select and we also measure for three different strategies we just wanted to make sure that we weren't performing better than if you were just mine honestly or if that your mind just randomly and we wanted to prepare the results so whether there was also play we found it when you ran for even just like a thousand episodes the agent that was incentivized to hold his chain a ten percent for plumbing work because they're all ideas that I've done percent you're not getting are gonna be ahead of all the other miners who don't have enough power so you're just in the losing one however for any person they already start gaining an increase in your reward after maybe like five hundred sites you can already see a little bit of an improvement in terms of what you're doing and then at sixty percent it's just like taking over the network so their results I never had that here anywhere else for looking at this nitrate which is the ratio of lost of mine so this is how many blots that you mind versus the number total watts mine so what you see here is that for ten percent you're doing worse than if you were just being honest but forty percent though you you started moving we set two different reward systems just testing and for one of those starting to be better at twenty percent while acting selfishly so he's getting like almost 50% of the watts which is gonna slay increase and at sixty percent is just mining all of the boss because you get such a huge advantage from selfish money that essentially you're publishing most of the bots because nobody can catch up to you and what happens is that you're also decreasing the difficulty of lot so you can my investor with the same I should rate and so what are some of the key takeaways of all of this is that at sixty percent you're doing a lot of damage but we all know that even like I thought this interview should be able to move a lot of damage a forty percent you start achieving some interesting results but one of our hosts was about this automated wait we would find perhaps other lost something different than selfish fighting a different strategy and then a type person is just doing very poor thing and so I never mentioned before that 60% your producing over 90% of the boss but what's interesting is that you're not actually making any more money if anything you'll be making around the same amount of etherium and so I mean you're just attacking the difficulty but it's not necessarily beneficial and yeah we couldn't find a better strategy that selfish mining but it's interesting because then what you can do is use these types of systems to test different particles and understand if there's something going on so perhaps you can see CNN signal detection system that there's something wrong with the protocol because if you don't follow the normal strategy you're doing better and so this can help you understand evaluate in different ways what's happening and you shouldn't just think that this is not a wholesome solution but rather a good indicator of what can happen and so this is an open source project it's all I can get hope so if anyone feels like they want to test their own protocols you're willing to do it and yeah that's it [Applause] you