Automating MakerDAO's Economic Risk Assessment - Jan Osolnik | Block Analitica

okay so everybody hey I'm Jan and today I'm going to be presenting project livon uh a way to automate makers risk assessment and also how to expand it across the whole default ecosystem and I work as a data scientist block analytica we are an r d company that specializes in the economic risk intelligence and the aim of this project is to assess the individual border risk from the economic standpoint to go through the topics I'm going to begin with introducing credit scoring and also contextualizing it in terms of both tradfi and also how to map it to defy and some differences there then I'm gonna go a bit more specific when it comes to economic risk engine and why that matters and I will continue with project live on which is the the product that we launched and in the end I'm gonna also give a short uh introduction to the product and and how you can use it yourself and finally I will also present some ways that the product can be improved and also uh integrated into various protocols and how it can be implemented so to begin with uh traditionally in stratify credit scoring has been about assessing uh the uh the the capacity of borrowers to repay their debt and that meant different different way to evaluate this like the the fecal score uh in the U.S and there are different systems in the U EU origin and on the other hand we have defy which is uh which through the transparent data layer we can do a similar thing but the difference is that in traffic we have a legally enforceable rules and in D5 we don't have that we we comply by the code is uh code is law principle most of the time and that's why we need uh different systems and in this case we've seen a proliferation of of over collateralized of Highly over collateralized Landing um protocols such as maker compound Ave and uh they the way they operate is that they have liquidation systems in place and in case of uh of uh over under collateralized positions or like when when when volts or different kinds of entities go go beyond a certain threshold the liquidation ratio then they are to be liquidated and the collateral been sold and that means that we need a different system in place that we have in threat fly and in this case uh this means the importance of economic risk and management in defy which means that that we are trying to assess given a certain let's say price drop how can uh how can the system respond to that given also the current state and also the historical behavior of different users interacting with that system that means the importance of both assessing the user risk and also the portfolio risk and we can we can assess the historical Behavior the behavioral patterns on the user level and then we can bring a bottom-up approach uh to the to the portfolio risk and we've seen two different major metrics uh on the um both on the collateral level and also on the on the portfolio level first one is valid risk which comes from stratfy and something that we've developed in our team is capital at risk which is uh in this case a maker specific metric we can see it on the slide it's uh it has this this nice view of of riskiness over time and and we if we see a spike in that metric that can inform also our uh ways of of uh proposing the risk parameters of the system so it's a very nice way to to use Advanced analytics to bring up signals of how the systems operate in general again because of the economic because of ethereum's transparency of the transactions we have this really good insight into behavioral patterns and there's many ways to uh to use computational techniques to to derive internet from that one way to split this into simulation modeling which is uh basic methodology that we use in in a lot of our use cases and machine learning which which we are using in in also in the project Limon and we applied it to the to the problem of decentralized credit scoring the centralized component here means that it's applied to the to the centralized protocols and there are two main reasons why it matters the first one is the ecosystem robustness and that means that by by better assessing uh the risk we can also then understand different risk vectors and potentially also or necessarily also mitigator against those risks and then on the other hand we also have the the improved Capital efficiency which means that we better understand the the riskiness of the protocol and and through that we can if if the if the protocol is assessed as let's say safe then we can decrease the necessary uh over qualitarization and through that improve the capital efficiency and ideally let's say also move in the direction of under collateralized Landing but that we haven't really seen successful use cases in in that area and in the end this also brings a very important topic when it comes to decentralized identifiers and when it comes to just tracking different behavior on chain and the same time it can bring this kind of like a really nice Financial reputation primitive into the equation to get even more specific uh when uh about the the project given he's been launched into public beta and I'm gonna also share the website at the end but in general it's an ml based framework then that uses liquidation says a proxy for default so there is no technically default in in uh in D5 that's why we have this uh liquidations as a proxy and that makes it a binary classification problem that means ideal for for machine learning modeling and additionally what we're doing under the hood is that we estimate the probability of default given the worst historical price jobs so so we get the training data of of all the history of maker and then we take the the worst price drops and we we can also expand that you know make it a bit worse or make it make it a bit uh better and then we can see which volts would be liquidated uh given the data the the fitting of the model and at the same time we can go both from the user level and we can bring it up to the portfolio risk monitoring which which we can see on the slide uh the the chart shows the the proportion of that that is graded with a certain grade in this case we go from credit grade one which is very high risk to create a grade 10 which is very low risk and then we can see the proportion changing over time and that again tells us about the risk profile of the portfolio without going too much into detail this is a simplified representation of the system diagram that we're using for the 11 risk engine which is done which is the logic the the system behind the computed credit grades we we move from the maker specific data which is on-chain data and then we we build signals or features on top of that we build a vault liquidation ml model and through transformation and aggregation we got the wallet level credit score um because of course like one wallet can have multiple walls and then on top of that ammo model we also apply certain heuristics based on our just domain Knowledge from years of experience in D5 such as credit mix which is the riskness of the underlying collaterable and externally held Capital because of course let's say given a certain quantization ratio or like the protection against certain price drop if the if the wallet has a lot of capital elsewhere they can use that to repay their adapt or just increase their collateral and through that improve their collectorization ratio and when we bring all this together we get this this credit grade which is the final output of the methodology and also something that you can see in the product then we move on to the product itself this is the landing page of the of the maker credit rate we also see uh basically a more more uh D5 wide uh landing page but that's basically also that we we're developing for it for also compound and Ave but this one just presents maker specific data when it comes to uh how to use the product and one way is to interact just by by adding the the wallet into the uh into the entry or given that there's many wallets and you need to look on chain or like on some on some dashboards to get the right wallets you can also have sample wallets per per each of the credit rate and you can click through that and get the and get the reasoning behind the the credit rate as well and that's one of the key components that I'm gonna go deeper into this is an example of one sample wallet and we can see that it has a credit grade of nine which is very low risk and the important thing as well here is that we see a portfolio distribution comparing it to other wallets so here we have a wallet count on the on the wire uh on the y-axis and the credit grades on the x-axis then we also have some certain basic stats around uh about around the wallet in the in the maker system such as the number of volts collateral and depth and we can of course also expand that to to see to get even a more holistic view of of the wallet inside of a system and then we have then we come to the to the most crucial component of all this methodology because one thing is okay you get the methodology and you get a grade and then it also has to do with explaining why that grade is actually there and and the reasoning behind it right so we have again the ml model which is the Baseline score in this case we can see and then we have other heuristics on top of that such as the qualitization buffer which is the basically it's computed by dividing criticization ratio over liquidation ratio then we have the two heuristics that I mentioned credit mix and external capital and then we also have certain other interesting components such as Protection Service which means that whether the wallet is using uh third-party services for for for the automation of the the managed position uh with deviceaver or wages or this kind of uh services and this is very important because like what we want to do is okay we have the on-chain data we have the modeling but then like we also need this this uh core component of the explanation uh to to increase the transparency of the systems I'm not gonna go too much uh into this but we also have this uh analysis section which goes into different components of the methodology um and if you have some questions I'm definitely happy to to answer them later but here for example we have the correct analysis that's again the the the target value the final value that we compute and we see the the split across different uh categorical estimations of the high risk low risk and so on we see on the left the top left then we see the the wallet count distribution similar as we did in the in the uh in in the product itself and then we see as well a temporal a temporal Dynamic of both relative depth and also relative wallet count over time and that again we can see a nice portfolio risk profile to that then we have the factor analysis which is uh just again the signals around it this is not the deflection analysis in the in the typical statistical analysis this is something that just means different signals in the system or features in machine learning lingo and we can see the based on the number of wallets the qualitization ratio buffer that I mentioned then we have also the tenure because intuitively the longer somebody uses the maker system because we're taking the the time of the first open vault by that wallet uh the the longer the more experience the the wallet owner has the less likely it is to be liquidated like and then we can derive some kind of insights that give us uh signals to the machine learning model another one is how recent was the last activity which can be also relevant and also interestingly uh how we can also estimate that that the user uses some kind of Bot and through that we are checking uh how how we equally distributed let's say our our uh the the events across all the time zones because we can see that if the if the wallet interacts with with their with the wallet with the Vault on on a continuous basis throughout all the time zones that's likely to be predictable of being bought not necessarily but still it can be it can be informative when it comes to the the capacity of the borrower to protect their position and something to mention here as well is that if the wallet has a very high qualitization ratio then the behavioral component is not that important because it's it's protected against the price shocks but if it's a high if it's a low qualification Ratio or a low consideration buffer then the then the behavioral component matters a lot and then it really matters to what extent how responsive is the is the wallet owner another component is the is the is the heuristic that I mentioned before the external Capital uh external Capital that we bring into the model and we split that into two into two parts one is the something that we call passive Capital that means capital or assets that just lie in in the wallet uh themselves uh and they are not using any other any other protocol other than maker and then on the other hand we have active Capital which is capital that uh that is used in other protocols and then on top of that we have certain adjustments of this Capital with uh when it comes to slippage or just general domain knowledge when it comes to the haircuts because we know that certain uh certain uh Protocols are just more riskier than other and so on but because obviously a coin like a huge amount of coin cannot be used that usefully in uh in the uh to repay their debt because of the low slippage usually and here we here we also see the the the the chart on on the bottom left which which compares the distribution of different wallets and and what it looks at is the strong capital of the wallet and also the depth and that's what really matters when it comes to debt repayment or increasing the collateral because what we're interested in is the the relative relationship there and through that relative relationship we can see how much uh how much Capital power basically that that that external Capital has for the given position right because for example if somebody has millions uh in uh in the wallet like basically external Capital but it has billions in the in the in the position that's not as as relatively important for the for improving the the the collateral issue we also Dove deeper into segmentation analysis that's under the hood it's just uh some clustering analysis uh going to the behavioral patterns of the users and splitting them into different categories there there are categories such as whale uh High qualization ratio whatever the the whether the wallet is automated whether it was liquidated before there's just a lot of this kind of categories that they're interested to to both estimate on a on the current state of the system and also to look at Through Time finally we have the idea of how to move forward right because the 11th grade is something that is applied at the moment just to make her but this can be expanded to other protocols as well and the the most interesting use case is of course extension into cross cross collateral protocols such as Ave compound again like or spark and through this we can have a more defy wide score when it comes to an implementation there are many ways to go about this one of ways is white listing and white listing waste game is that if the wallets have a good credit worthiness then we can give them better uh uh or basically the access to to certain services and then on the other hand we have we also have preferential treatment which means again good credit for Fitness that means that they have low liquidation ratios or lower stability fees or let's say slower auction times depending on on the specific system or uh basically the the protocol that they're using and finally we we also have a more long-term Vision when it comes to integrating that into a more decentralized uh autonomous service through different Frameworks uh that are just popping up more and more and getting more and more mature one of them is autonomous which provides exactly that but that means that the the credit rate can be integrated into protocols through various ways right there's also ocean protocol which we which enables just privacy preserving ml models then we have also um let's say bakalia with the computer over data so there's a lot of this kind of infrastructure that's been developed that we can integrate this into right like the methodology itself is one thing productionizing is another and then the integration and and really building a use case is is uh something separate but extremely important to work on to make it useful finally this is the QR code for for the for the app uh you're very welcome to uh to scan this and just play around just inputting different wallets or just playing uh with with the app and the important thing is also the explanation and we are very happy to hear any kind of feedback that you have uh around just how we can improve this how it can be amazed maybe integrated into some of your services uh and yeah just very happy to to hear your feedback thank you [Applause] questions hi um how fast does my credit score update so if I see okay I want to get this privilege recruitment my credit got so low I may be fixed on things and then do I have to wait for a week to you guys to scan the epoctrine again to re-evaluate that so at the moment we updated on a daily basis but obviously this can be you know improved and just depends you know if you offer it as a as an oracle service that can be pretty much instantaneous semi-stantaneous right so there's many ways to go about this like one thing is building it you know but then it's like really optimizing for a specific use case because you also want to um kind of think about all the ways that somebody can game the system right and that's something that you want to really integrate into into the solution itself to to mitigate against those risks thanks for the presentation my question will be how to assess the quality of your risk engine so what we do is that given our methodology then we also have certain metrics right for example something that I didn't mention is that one of the biggest problems that let's say are this kind of modeling has is that or uh this kind of problem is that you have a huge class imbalance which means that you have way more walls that are not liquidated than liquidated right and then you have this class imbalance so it means it really matters which metric you're using accuracy doesn't work then you need you know F1 score Precision recall or Rock AUC and like that final score is is what is how you basically evaluate the model right but at the same time it's you know different companies can can model this in different ways so it's very difficult to to uh to compare these Apples to Apples so you know there would need to be some kind of framework that all the you know Solutions use the same methodology and then you you then you can compare it to Apples to Apples but I can give you the score you know but that always is context dependent right the way you model the problem the term is also like how how good uh how how easy it is or like like how difficult it is for you to to get a certain grade like certain um performance score right I say that like the the huge variety of models like is a problem but my question was more rather around um you get a certain number for a wallet and how do you understand if this number is reliable to have some like back test of your system or something like that yeah I mean technically what we're doing is uh in the in the ml methodology what we're doing is a trans train test split and then we do k-fold cross-validation and through that we get the final grade right like the the final probability and through that we go forward right but that's basically like that probability this is something that that we that we're getting the model performance from right hopefully I answered your question thanks but but I mean the standard advice for security is that you know if you open a vault at maker in one address then you use another address to interact with curve and another address to interact with compound because each smart contract is well we're still experimenting so they're all black boxes so rather than blow up you know have everything in one place you tend to well many people have you know 50 60 100 addresses and so are you analyzing the linkages between all of them because otherwise you're just seeing one place and oh they're doing maker here but you don't know what they're doing over here so it doesn't really give you a picture of a user's it doesn't really analyze anything it just takes one snapshot of a part of the the elephant if you see what I mean yeah I I definitely uh it's a very good point Sorry good point and the way to do this is once you build the solution you want to incentivize people that they bring more and more wallets into the picture right that the more wallets they have or let's say you know they can build a certain portfolio wallets you know and that's basically creates the final the final grade right like at the moment this is just on the wallet level but just like we are combining different walls together right into one wallet you can also combine different wallets together into one bundle of wallets which is the actual user right so it just depends about creating the writing sensor structure to for for people to to uh to share their their wallets right and that you know ideally is I mean like uh to basically prevent kind of doxing like you also want to make this privacy preserving right so and that's kind of like a final solution that would be more aligned with also like what what you said right but just to follow up at the moment you can't present a grade for the bundle if I put it that way or can you because I noticed for example metamask portfolio manager is is sort of bundling and linking many addresses now and and are you doing that or you still haven't got a solution to bundle things together and say oh this guy this person is using this and this and this and this here and this here you can't do that I know you can't yeah so at the moment we don't have that solution in place so what what we can do is so what we have in the moment our solution is that for all the users that have an open voltage maker we create the credit rate right and then if somebody has multiple wallets within the maker system then we can bring that together into one that's something that that is already been brought in place right but then a separate solution is that you go into all the ethereum wallets right but then the problem there is that it's much more difficult to to detect really single there's just a lot of noise and not not much signal you know and then also it's it's not that instantaneous you cannot you know you you cannot update the the credit grade for for everyone for all the ethereum Wallets on a daily basis right that's something that is definitely not scalable so it really matters that that you're able to to narrow down on a specific problem like on a specific uh set of wallets and then go from there right um yeah if another question um you're using machine learning right um do you anything um in order to make crafting adversary samples harder or maybe impossible so because all your train data is open right I could train my own model do you think like a similar architecture and then use that to come up with an specially crafted input that would for example give me a very high scoring when in reality I'm not really not trusted um I mean the problem with transparency is also that it's easier to gain it right so there's obviously this kind of trade-off that you have okay how how transparent do you want to be that's why also like there's a lot of features that that you know that are important you know but some of them are showing some of them are shown and some of them are not like exactly how they're crafted in the background right it's just a high level idea right so so you know and often like you want somebody to like if you design the system the right way uh it it uh it makes sense for them to game it because gaming gaming it means okay like they increase their external Capital right it's like okay like it's good right in the end so it's like so there's a lot of ways look at how you can show the right data to the to the users and you increase the resilience of the robustness of the system and then like their gaming this is like their pursuit of their own goals is is the is the pursuit of the goals of of the of the design of the system as well right and then there is you know just incentives that are aligned um but like to get like really really specific to your questions like we haven't done really like proper like uh you know like really attacks you know that can be done on the system like I think that that's really like you know solution down the road oh okay so uh yeah just just uh um there's something you guys should think about right if the credit scoring um I don't think the incentives are really aligned um with the people that using and um at least it could become very fast that like a centers that is aligned and you have still have Oracle access to the system so it would be probably easy to press something like that yeah so so in that way like I think it's it's a it's a tinkering process right so you begin with creating a solution then you in integrate it in into into you know certain uh it's a protocol and that can give you then the data on how how you protect against that right one thing is okay we can predict in advance or like I think about how they can game the system and another one is okay like like we have it in production people are using it and then you can detect you know like fraud or you know those kind of ways so I think it's it's a it's an evolving system right thank you no more questions a big Applause [Applause]