Reinforcement Learning for Query Pricing in The Graph by Tomasz Kornuta | Devcon Bogotá

foreign [Music] good morning everyone how are you so let's start I'm Tom I work for simiotic labs and today I'll have the pleasure to present you our work on reinforcement learning in color pricing in the graph so first let me start with the outline I will tell a few words about semiatric what we are doing then I'll introduce what what's really we are doing in this area automated price Discovery I will formulate the problem I will add a few words up when it comes to modeling how we model that problem then I'll briefly went through reinforcement learning Basics let's say it's just a refresher for those ones of you who know or maybe do not know enforcement learning then agent-based modeling is the the core of the talk and I will show how we're using that for testing system properties then and for testing well different behaviors when it comes to using single and multi-agent setups and then I will show that actually we deploy that that solution in production what are the results and finish with a small summary all right so let's start the first semiotic symmetric Labs was founded in 2020 by Ai and cryptography researchers uh we are focusing on applied research and we are one of the core depths of the of the graph protocol also we are developers of others the optimal text aggregator uh actually the the lead of that product is giving a talk at the very moment on a different stage so bad luck okay so our expertise we we combine the cryptography with AI That's our core expertise and of course there are software engineering involved and we focus mostly on building infrastructure we focus on crypto economics today's talks actually focusing on this on crypto economics all right so what was the scenario so the scenario in here is that I don't know which one of you know the graph actually who knows the graph protocol all right okay 50 50. good so basically in this scenario we just focus on on on one part of the graph system right of the graph protocol but imagine that you've got some customers that are sending queries right and you've got indexers that are indexing blockchain and our mechanical serving serving those queries so between those guys there is an entity called the gateway which is doing we're just gonna go acting as a query market right so depending on the price the price depending on the quality of service a given query sent by the customers when we would be distributed on this or or that indexer and indexers earn money by serving course right so they can control the prices of the service queries and the core idea that that I'm going to talk about in today's basically wanted to have Dynamic pricing based on query volume received by an indexer and we call that auto Agra and then I will explain the second why we are calling that Auto augura so this is how we model the the scenario right it's it's simplified from one point of view because as you can see ah as you can see there are no no let's say agents in here we get rid of the customers for the purpose of that flooding of those simulations instead we've got a traffic generator right something that just says okay at given timestamp this is the query volume that you've that needs to be served right we also got query distributor so something that basically look at at the bits of the agents so the agents are putting their bids and then depending on those bits the queries will be distributed across those different agents and those bits are expressed in a domain specific language called Agora that's why the I the the the product we call it altag all right because it automates that that building otherwise the users will have to the index cells have to basically create those those price models manually which is kind of tricky right uh all right so some assumptions selected assumptions what we're discovering in here so this is only about how we generate the dot query volume so we've got in here so this is basically the shape that that we we model so the x-axis there is this price query price slash this is a budget of of a customer that customers are saying okay this is how much I can pay for that query this budget is unknown right that's important then it's not sharing that with the agents it's not sharing that with the indexers it's something that that that is setting on its own so this is something that we want to discover right and the budget can move right from left to right which means that if the if the if the indexer was set its price its price bit higher than this well he won't be picked right also there is some kind of like a noise we we added a noise to that volume to that query volume and important is to know that there's there's a game happening right in here and the game is suraj's goal is to maximize the revenue but gateway's goal and basically protocol goal is actually quality of service right we want all the queries to be served you know uh five nines and so on right so there's a game happening all right so if you know how model let's not now switch gears to reinforcement learning just one-on-one most classical thing so down there in reinforcement learning we've got two main entities agents and environments agents interacts with environment by executing an action then agents actions change the state of the environment I just got reward in the observations after after oh okay and I just can up their its policy based on the received reward in short that's reinforcement learning so we're using different types of agents in our simulations uh in here I put two most important criterions one are trainable reinforcement learning online learning and the other ones are like rule based so there are some predefined behaviors and also we've got this second Criterion Solutions can be stochastic or deterministic in here I put some some simplified classification of of the reinforcement learning algorithms and I highlighted the two that basically we will be using in our simulation so we'll be using vanilla policy gradients and mostly most of our agents will be using PPL important there is one important type of ideas that actually we we are mostly focusing which are we call them gaussian Bandits so the idea that this is sustainable uh stochastic agent with policy that is represented with a gaussian and then when an agent is is supposed to perform an action in something from that gaussian right so imagine that there is this distribution and when that when the digest is supposed to make an action it's just sampling right and in this case Okay the agent sample is 0.8 and from that blue distribution second agent sampled python 24 right and why Bandit Bandit is because the agent is not building any an internal model an internal representation of the environment all right so going to the most important part so testing based testing properties with edges-based modeling so in here so already kind of explain this but in the first set of experiments we focus on single agent simulation just testing the properties of an agent and this is a single gaussian Bandit with a modified PPO policy update Rule and the query distributor is like a super naive you know let's say version of that what could happen if here is that we've got access okay depending on the price business is just inverse inversely proportional and in the first step we've got a fixed customer budget with some noise of course right and what we are checking in here is the following we've got market conditions we've got two criterias for every experimental custom criteria and a property that we want to test whether the Bandit is fulfilling that that Criterium is is operating the way we want or not in this case the Criterion is customer budget discovery as I've highlighted before and the Agents do not know the budgets right but so we want to discover that so first so that's there will be lots of things happening right because we are tracking many things there are in simulations so let me try to let's say unfold that and go step by step explaining what you will be seeing in a moment in in those animations in those videos that we captured during the experiments so first there are five plots right in a single video first one is the most important so what we've got in here is the is that budget right the query Volume Plus Plus customer budget the solid red line is the agent's current policy the dashed line is the initial policy that that agent is starting from second in the second plot from from the top this is basically query volume server by the agent at that given time step so what happened in here is Agent actually didn't serve a query right didn't get any queries because it's he sampled somewhere in here right like few steps ago he sampled some like the the the price was too high right so the agent didn't get any queries so which which also is shown in here so this is the aggregated query volume served by a given agent in red and the blue ones shows basically how many queries were dropped weren't served at all right so you can see in here there's a tiny tiny step which is which actually is associated with this then in the next plot we've got agents revenue and aggregated agents Revenue over time all right so let's run our first experiment right so once again what we are testing in here what we are trying to discover is word up that where that that budget is right so as agencies is learning right so first the gaussian is was super wide right like it was like much much wider and once the agent as you can see it's something from time to time somewhere in here right above the budget so it's getting more and more sure that okay this is the max that I can get right that's the highest price that's that's the highest bid that I can make so as you can see the agent is dropping some of the queries right uh what is revenues is nicely nicely growing all right so first proper property test right that was the static environment so how about Dynamic environment right by by Dynamic I mean that the customer budget can change can vary over time right so in this case we are also testing the same property whether the customer budget can be discovered by the agent but in more difficult market conditions let's say right so once again we are starting agent is kind of a converging capturing the the right the price right which is kind of optimal and then the price changes so what happens is the agent is is not getting getting any queries right so it's saying okay I don't know where am I so it's press the gaussian and once the queries are once again captured it it moves to the left right so it's like okay I was above the budget let me let me go back right and that happens that is happening over time right so there are plenty of queries queries dropped but still we are we are making some nice Revenue right and we are reacting to the to the dynamic dynamic Market cool second property check so in further experiment we said okay so there are some subgraphs that actually you have no queries at all right there is a moment where nobody's is asking for queries so what will happen then so that's a different scenario in this case we call that the marking conditions basically there is no demand for your services right so we wonder how we should add so it will happen in a second right so there will be like three iterations we're recording few of those those videos but as you can see already there's a hint in the right hand corner that something is not right all right gaussian spread this is great and now time set 400 because every 200 steps were like you know to write it okay so what just happened there are no queries so according to the design the agent just said okay let me spread right so I can sample from wider distribution but actually spread all across the the the the price Biz domain so we said okay actually this is not a good behavior right we if this is happening then the agent can can sample at the same probability like super some super small prices price bits and at the same super high moreover recovery from that will take a lot of time so we said okay can we improve that so we implemented something that we called graceful initial pull which basically uh the idea is that okay there's this initial distribution that the agency is starting from this is something that the index service will parameterize the agent when when when the agent is deployed and where when there are no queries it just slowly is pulled towards that initial distribution right and when the queries will appear once again it will get from there and once again start there's something I'm losing from time to time the voice I think um Okay cool so we pass the property so now let's move into multi-aging simulation so this was great at that point you know we aren't we aren't competing with anyone up till that moment right now we are starting with with some we're competing with some deterministic agents the idea is that right now all the all the indexers have that the Agra models that is fixed right they are not changing the models so it can be seen as a rule base right so there's like okay that's my bid and I'm sticking with it no matter what right so this is what we are trying to model here competition with deterministic agents and the property that we are testing is called discover of prized bits of competitive agents right so we want to make sure that we we also will uh we can let's say compete with them and discover what are what were their bits without actually knowing them without having That explicit information so those three lines in here actually those are the three three competitive agents with deterministic policies that they're just set and the blue agent in this case is the one is the Bandit right and as you can see it after a while it's set up like a price just below the the the the cheapest agents which means that it's trying to once again snatch the whole Market if you will see a number of total queries right you can see clearly see a difference right that agent is dominating right so I'll say okay this is nice check cool so how about stochastic similar experiments we also want to discover the the price bits of other regions but their policies are stochastic so we model them in the singular way as gaussians those gaussian distributions right but they do not change right so we observe similar Behavior this is great right so the agent our agent right our band it moves somewhere in here and of course it's sampling you know that the red the red the red agent uh stochastic heuristic agent is also something from that distribution so it's changing but as we we can see direct moved slightly below and shrank once again to capture the market right all right right I think it's we check that property awesome so how about it will right now compete with like make a competition where we've got plenty of those bands deployed and we hope that actually this is the most inter that's the most interesting case right because we want Auto agara to be run by all the indexers in the graph ecosystem so same property slightly different market conditions all right so what is happening in here sadly it's not good it's not good so that I just are acting actually as designed but as you can see they're like fighting fighting fighting going moving moving to the left hand side which means that they're lowering their prices all the time right and as you can see in here oh you can see the labels but this is this is the revenue that they're getting right at a given time step basically it's going it's converging to zero which means that after a while the digest will be basically serving for free right and so that's not a good situation for the indexers right because in here we are not even modeling the we are modeling the revenue not the not the we are not modeling the costs right so at that point agents will be well indexes will be actually uh well paying for that instead of making money all right so we call this this this uh phenomenon or this outcome raised to the bottom and we run many simulations we discussed that a lot and actually we realized that well this is the expected Behavior right in this setup so we've got some images that are basically trying to snatch the market right and they're purely driven by by the by the query volume right so if this is the case and the environment just natively distributes you know the queries based on the the price bit rest of the bottom is the expected is the expected outcome right so it can be addressed in many ways but we look at the graph protocols what are the features right and down there and what are the assumptions so one assumption is that all the elixirs should have freedom with their pricing right there should be any limitations so that if they can if they want to serve queries superb that are super expensive or super cheap it's okay and all the indexers should be able to make any profits so the conclusion was that actually is the Gateway right that the thing that controls the market should implement this kind of anti-dominant policy and so happened it does it already does so in our next step what we did we wrapped the existing isi right the Isis index selection algorithm is one of the components of the of the of the Gateway right so we just wrapped it in here and using the simulation we are once again testing the same conditions right competing you know many versions many government is competing with each other and the property okay can we discover each other prices and maximize our profits so the outcome is totally different right we ready by changing changing the way the the queries are distributed actually you reach the like a consensus right so that all the agents are discovering the markets right what was the market at the same Gateway is looking the ISA is looking at their their quality of service uh at their prices and trying to like feed everyone right not just okay you're the winner you're you're searching all the queries all right so this is really nice so please note that the gate will also leave some kind of budget right that is the this design such a way that actually it's it's not enabling the agents to to consume all the budgets right so the budget is there is a portion that is left to the to the let's say to the customer okay so we're on some more simulations of course it's like okay free can we run that with 10 20 agents right will that still work it seems yes it does so we are testing the same properties right which are just a little bit changing the market conditions right this is looking really good if you look at look at the the query serve and the revenue it's like all the agents are you know are are making money which is the desired the desired uh Behavior right and they are rediscovering the market cool so what we also did uh we we started we wanted to see whether okay so how about different initial conditions right so we're assuming that different different in Xmas that okay that's my mean price that's my variance I want to maybe move a bit more expensive for like less expensive other being wanted to see whether the the the the the system will converge to that equilibrium and it seems that this is really happening right so disregard the initial conditions this is good we also run some additional experiments with with frequent a stronger policy updates so this one is our modified PPO agent as you can see it's much faster than okay the red agent is the vanilla policy gradients like parallel mental policy Elementary update rule and the other one is just pure PPO right so our agent is kind of like a dominating and it's kind of interesting to observe that it's serving less queries right the the blue or CN line is clearly below and this is the number of queries that are served but but it's making bigger Revenue right so in short is making less work for more money so policy really matters right all right so having those results we are quite quite happy with with those we said okay so can we deploy that in production and actually that's our solution you know on a Battlefield so once again there's there are many there are many things to unfold in here so let me try to focus on one thing at a time right so first this is the mean so this is gonna go where the gaussian is on that on that x-axis right so what happens is that the initial value was too high so the mean was going down first but then insteadly it was going up up good so it was moving to the right at the same time variance was steady going down which was which means that the gyrogen was more and more narrower which means that I just were like okay I'm I think I'm there and I'm good right I'm more confident that I should sample from that our smaller distribution so the outcome is that the reward goes up maybe you cannot see it in here like super nicely but that's the revenue right that's the total revenue and you can clearly see that it's it's going up there is the tendency right so we deploy that the results were great so I think we are well this is the amount where I should start telling okay if you're an indexer please go ahead there's a repository to download play with it and we there are some additional materials okay but you know actually in a second okay so let me first summarize so agent based modeling for crypto economics that was the core of that talk and in here is focus we focus on the diamond Dynamic pricing for the applied to the graph protocol to to automated price Discovery and we use reinforcement learning and and a multiple multi-agent simulation multi-agent modeling for for Revenue maximization so we can show how to use that for testing the properties of the protocol right we came with that framework that okay those are the market conditions those are the properties and systematically start the test right uh our our agents are solutions against against those and finally we have deployed out our in in in production and have shown that actually it makes sense so Future Works of course better policy better updates better policies agents of multiple rewards of taking quality of service into account we are not doing that right now just we are just looking at the at the query volume uh modeling and putting consumer agents into play right so that right hand side that we simply fight actually putting that back in the simulation it will give us some more insight in what is happening right so that's kind of that will give a simulator with higher Fidelity and uncomfortable redesigning the game right so in this case there was no information the agents were like acting totally independently the questions it will redesign the game redesign a little bit the graph protocol would that kind of perfect information will help right there are some here are some additional resources so my friend Alex is he there is he wrote this blog post which describes more the technical side of deploying our Target in production he also gave a talk around this about this during the last track episode in June I think so there's a recording on YouTube and of course this is open source so I encourage you go ahead play with it write all those visualizations are there basic you can reproduce that and try to deploy your your your altarger in production right uh finally I wanted to highlight some of the of the other words that samiotic is doing so their stock actually that's happened right now that that our friend Matt is giving his the product and research data for us he's giving it right now so that's uh that's that said and Savvy is giving a talk today about our work and or let's say our our our our our our our trip in the to the snarks world and we are focusing on on this is more like cryptography uh we are focusing on verifiable payments for the graph protocol as well so if you're interested in those those topics please come to us we've got plenty of positions open we if you're interested please go ahead thank you for your information excuse me there's a gentleman who wants to be back [Music] hello and thank you for the talk I was wondering if the customer has a very very low budget will you try to match it and make a loss or will you shut down that's a great question actually I wasn't showing this in here right because it's shaky like I said the goal of graph is to have five nines so right now we are working with with the guys that are you know implementing the guide when updating that a little bit because for now if the customer will say okay I'm I want that for basically free the Gateway will still try to distribute that right it's not the best outcome I would say for the indexers right but we are changing that we are we've got a tool right now right so you can think about I was showing you that no we're working in this work we are focusing on the agents right but we developed a tool that enables us to test the properties and the outcome of the whole thing right when there are different players in the game including Gateway so great question it's not tricky right the tricky thing what do we want to achieve we want the indexer to make money or we want five nights right the quality of service to to be super high I have another question sorry uh maybe more technical when you show the experiments whenever the budget of the client was going down the spread of the ocean was was widening but the mean was things the same which means that you are still sampling to the right side where you know it's not the right price would it be more effective to move the gaussian to the left as you spread it yeah uh so what what's the question actually because you just repeated that the design Behavior right if we overshoot if we overshoot with the the the the price then we don't know whether the reason was there were no queries or we were too expensive right so we spread because we want to sample from wider distribution right all right I've if there are any other you know additional questions I'm I'm here I'm I'll be delighted to answer your questions but it seems that my time is up so thank you for your attention