Building ecosystem health metrics with open data sources | Devcon SEA

[Music] all right thanks everyone for coming uh we have a session today on uh building ecosystem Health metrics with open data sources um should be accessible for you know anyone whether you're super super deep in the data uh kind of more of a data fan or just getting started in like pretty pictures of charts on screens um this is all kind of based on our work building within the optimism super chain ecosystem but should be applicable for any to any wide ecosystem maybe even like any business so uh yeah so with that we will jump in uh so I'm Michael uh data analyst at op Labs been there for about three years uh seeing the wide range of things from one chain apps you can count on your hands to uh now where we're at with many chains and trying to keep track of everything um and uh I'm tran I've been working at op labs for two years and has always been navigating onchain data oh so um for today's today's agenda I'm going to talk about um what is ecosystem Health what are we going to build there's going to be a metric tear down so you know all the metrics that we're going to talk about and then we'll spend some time building in the workshop and at the end of the at the end of the um session we're going to talk about the collective data vision and we'll have some Q&A time for everyone as well oh so before we start I would like to start with a multi-chain universe l trivia time so um going to get some hands um how many chains are there anyone has any guess just feel free to raise your hand shout anything how many chains 200 200 it's close there's more um any other answers it's more than 200 give me a wild number say 500 500 oh it's lower but okay we're still uh so okay so this data is as of like end of last month which is October um there's already 322 chains um according to Def Lama so guess how many rollups are there give number there's already a reference number now anyone shout 200 200 M interesting um actually is much slower so according to defal Lama it's about 30 8 and also l2b is about 48 so different providers last question just guessing how many chains are part of the super chain ecosystem so it's another subset but give it a number again please anyone 30 30 this what you're saying Lizzy okay so it's actually 21 we and we have this data available in a Google sheet with uh all the super chain ecosystem members so now we're going to talk about what is ecosystem health and I'll hand it to Michael yeah so how we think about uh measuring the health of art ecosystem uh the whole you know super chain L2 space uh can kind of a very very simple question to kind of try to answer is are we doing well um this being about chains seems like a very very easy question to answer um but it actually becomes super super difficult and it'll go into a little bit why one we just went through how many chains there are super hard to keep up you know who can name 300 chains you can barely name 20 chains um a lot of things going on even mentioned youed to able to count chains and apps on your hand now absolutely no way to um so a lot of information to try to keep up with uh second there's often a gap between what you think you're measuring what you're actually measuring I think one tangible example I can give when I first started at optimism um trying to find what things us users might be interested in uh decided to take all the users on L2 take those addresses look what they did on L1 try to get some understanding of what they were using uh at the time we saw some real asset apps we saw some like fixed straight lending things and I made this assumption like oh it's all these fancy defi things that's what these users want that's we should kind of go after and build uh turned out those were all just apps that hadn't launched a token yet so I thought I was looking at what US just wanted to do it was actually looking at people you know doing their farming activities uh and this is actually kind of a you know dangerous assumption to make especially for making decisions or figuring out where to invest engineering resources product resources if it's based on thinking you know something that is actually not what you think it is uh yeah put you down a bad path so ideally we kind of know what we're measuring and don't get fooled by other things uh third I think throughout the week there's been a few kind of you know data sessions data conferences uh heard a lot of people say like this is the metric that we should be looking at everything else is bad this is the one uh and depending on who you talk to what their perspectives are like what areas they focus in there' be many different ideas of what doing well is and the first question are we doing well many many different ways someone could look at that and make that uh judgment and so with that I'll go into a few of these ways of what doing well could mean uh I think our approach is generally you know start with some intent we want to measure uh like some something we think is healthy for our ecosystem and then what metrics are kind of proxies for that so A few of them you know very early on we like user engagement and adoption you'll hear that everywhere some sample metrics transactions people doing stuff gas used compute stuff uh and then users so a good outcome of going of focusing on these things is you know you scale your chains you reduce fees let people do way more stuff on chain you grow all good bad outcome is if you're just focusing on making the transaction number go up all kinds of programs things you can do to artificially inflate that potentially lead to some short come outcomes or short-term outcomes um and kind of waste of time there the asterisk is on users a soap box for another talk but no one knows what a user is uh we could talk about that later I guess hopefully getting there soon uh second one so so you know talk about the demand side about the supply side uh developers building things on chain mind share people coming to you to build their cool applications so some metrics here onchain contract deployments maybe look at GitHub for Stuff a little further Upstream good outcome build good developer tools kind of simple there bad outcome is you're just like give me as many contracts as I can get you know kind of lose sight of you know quality or long-term sustainability there um and yeah that's a bad outcome there economic activity is another one you know total value locked so it could be you know tokens deposited into Finance applications um or any kind of volumes like transfers trades things like that Devcon rough on The Voice um so good outcome address real the real would be good real world fanal opportunities uh bad is you know again short-term liquidity stuff comes in stuff comes out low retention right back where you started financial performance the last one I'll go through you know why go through all this complication why don't you just look at are you making money are you making fees you know we already have some some metrics for that kind of well known across any industry um what Revenue profit are good outcome General measure product Market fit people willing to pay you can measure that uh bad outcome is so focused on that you're like keep fees High money's coming in like we're good um could be at the expense of user experience and people churn you lose them uh so a theme Here each indicator has some good intent to start off with measure something useful you can you know tell your stuff it's all good um but over alizing for one has some bad outcomes um so yeah so we kind of like realize there's not like one key metric for everything uh so we had to figure out how we move forward with that so what do we do Tu should help us next slide okay so um this is kind of like ecosystem Health metrics framework that we've been using to approach the dashboard that we're going to show you later um but generally you want to start as kind of like a fun analysis and you want to think about what are the um what are the drivers or um kind of the levers that you can pull for the ecosystem and I think this framework actually not just only apply to chains but also apps or basically any other product you want to build um and then we want to come to the network effect so for example like brand product decentralization how is it going to impa impact your network and here as a chain operator what you want to look at is as you grow the chain you want to attract developers more develop developers building apps and they will attract uses and more chains come in so this is a flyware effect and um the proxy that we want to use to measure this file flywheel effect is onchain usage and this is not a static metric because I think everything is evolving and when there are better metrics coming we'll just measure them and right now we're just using transactions and gas usage as a proxy for onchain usage and down and at the very bottom of the funnel is just Revenue which is measuring the sustainability but is the impact of that is usually seeing at um I think at the very last so before we share anything more does anyone have any questions are you guys still following don't shy okay I see some people nodding so we can keep going um so yeah now I'm going to introduce what we build and I'll hand it over to Michael again yeah so some of you may have seen this uh so kind of going with the theme of there's no one metric no one big number we really want to focus on um we decided like you know we have many many things that uh we think are you know potentially good proxies for ecosystem Health um so rather than over optimizing for one they all matter in context of each other um you see one one metric going up one going down that tells you something uh everything going the same direction may give you more confidence that uh you could be seeing like real growth or or real health of your ecosystem um so this is like one view into there not not don't don't worry about what each metric is now we'll go through some of that now um but there is something too you'll see like a lot of people will you know make things or or make statements about like what are your incentives that drives what you do um and so I think there's some role when you're building either your ecosystem dashboard or whatever your source of Truth is or what people are looking at as a canonical thing they rally around whatever metrics you put on here in a way will drive what people are doing what they're optimizing for so it is a very powerful kind of role you play here um which is why it's important to yeah or at least our views it's important to show many many many things so you're not creating some some unknown outcomes everyone wants to be the top of a leader board so it's very important uh you know how that is sorted so now we're going to uh quickly go through each of the metrics in detail we won't spend too much time on there because that's not the most important part um but yeah the first metrics that they want to show here is transactions um and then yeah so is very self explan explanatory and the reason we choose transaction is because it touches like every participant on the blockchain with touch transaction for one reason or the other and um but the transaction by itself doesn't really mean anything if you don't put it into the context because it can go up for various reasons there may be boss there might be that the macro trend is just going up in general so you want to compare that to the the market or the market share that you want to measure against and anyone can Define what your metrics is for example like you can just say I want to measure against L2 or I want to measure against the market share as the whole crypto ecosystem or it could be vertical wise as well so that's something that the that's something you can kind of like Define what the market is looking like and you want to know like oh am I growing is my growth outpacing the industry versus just like I'm growing because I'm writing the trend um and then usually when we see this like kind of transaction numbers going up the natural question that follows is um where is the transaction growth coming from and we have a very nice dashboard that kind of like tells us uh where the transaction is coming from by the project or the contract and the very important thing here is about the contract mapping because without the without proper like labeling is just really hard to understand what's behind the meaning of each of those like hex code um so yeah so this is just one example of how we kind of like disset uh what's happening in in each of the chains next one I'm going to talk about is uh the L2 gas use per second and this is just a metric that measures like how much compute a chain is doing and usually this number will go hand inand with like transactions but there are also examples where uh you will see like the gas per second going up and some plateaued but the transaction number is not going up and that's because uh usually for example on any of the l2s when it when gas per second hits the limit the network will be congested and people don't want to pay because it's expensive and transaction number will actually go down so uh this is just one example that you can't just look at one metric and say something uh because you can have lots of stories behind uh next one I'm going to quickly talk about uh is the app tvl and also the assets on chain uh the difference between this two number is just that um usually I think I we kind of like Define the the app tbll as the economic value that is all the liquidity that is that is locked inside ecosystem and then on the other hand the asset on chain is just like the sum of any kind of tokens that is locked in the economical Bridge um and also any of the Native tokens that is on L2 and the other one that we show is just Collective revenue and the revenue definition at least like we're using here for uh for the super chain ecosystem is any any chains that contribute maybe like 2.5 the higher of the 2.5% of the sequencer revenue or the 15% of the net onchain value and the last one is governance which is something that uh we also focus a lot and here we're just measuring like the total vable op Supply the reason that we care about it is because it's about capture resistance and we also want to see like actors acting rationally to make economical decisions um yeah there can be lots of analysis that comes behind this looking at concentration of voting power behaviors of voting Etc uh for the optimism Collective and back to Michael yeah so now we've gone through all this you can kind of see some of the intents we wanted to you know view as health for the ecosystem we mentioned those all before and the metrics that we chose to represent those uh I think another good kind of case study here is if you were you know we talked about over optimizing for one thing or not uh you can kind of see how all these charts May Trend in similar directions different directions one obvious kind of Divergence that you may have seen as we went through is you know wire transactions going up gas use going up Collective revenue is just you know kind of straight down uh and that's kind of like been an interesting case study I think we saw after EIP 4844 the whole blob thing that launched uh in March of this year uh transaction costs on l2s went like straight down but you know great for user experience way cheaper to to do things on chain uh but then we saw a lot of these L2 chains hit congestion of what they could actually process on on their chain and that led to fees going up price of skyrocketing uh opportunities for you just to go churn uh so where the decisions were made try to keep transaction fees down let there be you know way more transactions that could happen on chain uh you did see Revenue go down and that sense so if you're only optimizing for that you'd be like this is crazy what are we doing um but if you're kind of taking a longer term lens looking at ecosystem growth um this may be a you know the things you see to make it seem like a rational decision there um and so some questions we often get our made a nice dashboard like why does it matter I think some influence that we see is there's um you know it's source of Truth rallying point for the whole Collective hard enough within one company to do this um then we have a whole bunch of stakeholders in the ecosystem those like the labs company we both are Foundation teams running chains delegates voting on governance grants councils all these kind of things huge huge long list of people who you know make decisions within the collective um helpful to have one chronical source of Truth Versus I heard this here I heard this there um it helps kind of bring some order to our decentralized chaos that exists uh growth opportunities so you know whether it's the raw number or something of looking at how you're comparing versus total industry some of these market share metrics uh could open up some growth opportunities like where are you lagging where are you leading um could open up yeah areas of where to go deeper uh and like we kind of briefly touched on with going a layer be under transactions into what contracts are being used what's driving that uh opens up opportunities for for future research um you know see something moving or see something changing you know why is that happening leads to one question to five questions to 10 questions and uh then you're going going deep there but hopefully helps you prioritize towards the most impactful things versus what may seem kind of interesting at the time uh in general goal setting um you know whether whatever your targets are it's helpful to have this to rally um behind and I think you'll often see sometimes numbers change or metrics change kind of represents where the direction we all we're all going is is trending towards uh very briefly so this is a public dashboard here I I'll go through back through everything here um so I'll have links in in GitHub that will show soon how we built this very briefly using a bunch of you know public API that exist to is down here some chain indexers we use you know not a complicated uh system database table business logic um you know building metrics joining stuff um then the dashboard there okay so now before the fun stuff after listen to us talk uh any other kind of yeah questions comments uh on what we've gone through so far yeah go ahead any Anis I can I can repeat the question it was like uh analysis on secondary markets for SWAT value um yeah I think that's like maybe goes into the economic oh sorry is it something different lock oh lock value so mean secondary what do you what do you mean by secondary markets of lock value um values the to Val cont oh yeah yeah yeah yeah so it's like yeah it's like some some Nuance on top of like value being locked I think there's something that's interesting we've tried to do is um so like use like defy Lama as our like total value locked and apps Source um and there's like I guess like a maybe it's similar but an issue there with kind of like double counting like you can deposit here do this do this do this do this and you make like one token represented 700 times um so at least like def does a good job of trying to exclude that double counting which helps maybe not totally related but um maybe it's in like the yeah business logic segment of like is that what you described like you know good or bad you want to filter it out or include it or you know call it out as being important um I think definitely different perspectives whether you're like a yeah generalized chain ecosystem like a specific version like specific app um hopefully it answered a little bit but we can talk maybe after about it too anyone else yeah go ahead I think my question is how accurate like how do you like know that your data that you um show is accurate yeah good uh prompt for uh some stuff coming soon so yeah so there's I guess like a few different data sources but one is yeah I mean like General trust in the ones that we do use like ones that are you know the examples of like the defama gry l2b groups um that do the Cur data so he's pulled that that raw result um yeah there's an element of like trust in there too um but think as well like there's you know if their data was wrong it's like that's no one would use it I guess uh but yeah it's definitely like a a point of even like General data quality something pops up it looks weird figure out like go deeper into it I think for a chain index perspective like you know we're building things that are checks to make sure like are all the blocks there does the amount of gas match up and things like that too but uh yeah I guess trust trusted sources and verify stuff I when it comes up as well but yeah a good point it'd be bad if it was wrong um yeah anything else cool fun stuff time uh so kind of a workshop here if you have a a laptop with you you can you can start uh building on some things together but if not no worries like you know find a friend next to you and you know we're going to go build some cool charts together um so find out the best way to navigate people to a GitHub repo um there's a QR code but you know phones to computers Apple's done some stuff it's kind of cool we'll see if that works too um so yeah so we have like our op analytics repo we build everything in it's getting cleaner um but there is a dedicated folder for you know demos Devcon 2024 here and try to find that uh there's some yeah optional work for if you're like beginner intermediate with you know SQL or python you can uh jump in and see the code but we're going to go through what exists there too um you and for everyone we'll get to a spot where you can make pretty charts with curated data so maybe give a few minutes for that and walk around if people need help doing things anything cool um and once you get a QR code um there are two ways you can do it if you are very familiar with GitHub you can just do git clone and copy the entire repo uh we'll have we'll make constant updates there but if you just want the demo folder you can just go to like download directory github.io put our put the QR code link there and then you can just get the folder in a zip file um do we also have a link for the oh we don't have the link we'll show the link later okay yeah back to the QR code and then fun stuff maybe get like a minute and we can kind of walk through what is in this repo while people pull it up too in a minute for okay maybe assume it's enough time if you can't uh don't want a Forker clone like on the GitHub UI you can probably see enough to kind of scroll through anyway U so we can talk a little bit about what you'll see in there before we run wild and build stuff okay so I'll walk through a few data sources that we use you know curated some some scripts kind of to go run through now shall We join them together um build new ecosystem charts and potentially do uh yeah call out some of the interesting stuff people go build so what you'll see in this repo General read me with you know links you'll see two at the top that are uh we had like a Google sheet up now with the end result data that you can just use if you want to skip to that part um as well as the the links what we just showed here before um we've kind of curated here is for each of these like apis we're using so defile the LTB right now some python files you know functions there was told it was boring to go through those so we'll skip over those um and go through some notebooks that we're using to pull the data join it together kind of show walk through of how this all kind of comes together uh then yeah get to the final data csvs that we can then build cool things with so walk through some of these data sources um similar stuff that you'll see on our health dashboard but just for what comes through here raw So Def is what we use for the total value locked metric here's their chart for across the whole crypto ecosystem um if you break it down by Chain this is the actual data that we pull from their API uh what is very cool about this is it gives you a like total crypto ecosystem wide view not just l2s not just EVMS you know has everything on here see like top three ethereum sonotron Bitcoin farther down bunch of other ecosystems as well um SOA is good to kind of yeah compare not just within the ethereum world but everything else outside as well uh other one we'll show is l2b I think what's cool you'll see here is there's like a similar kind of approach of showing multiple metrics together um what's useful here is like a huge wide list of layer 2 Chains um you see like total assets on chain total transactions what's useful here there's a lot of metadata for each chain too so some element of like security level with the stages there um what type of chain it is what stack it is you know all these kind of things that are useful you'll see for showing some sample charts later but hopefully it Sparks some ideas uh third one's grow the pie another layer two specific one uh smaller list of chains but definitely has a much you know kind of wider set of metrics so again I was like pulling these today I was like oh cool you know more metrics all kind of together up top uh and then a whole wide range of stuff but yeah from their API we pull all these metrics as well um so you could use them pull them all together so I what you'll see steps here download the data don't expect you to go through the code here call those API functions uh then we try to join it together um I think one thing interesting to call out here is like eventually it' be nice to figure out some like interoperability between all these kind of data sources um whether it could be some use chain names differently some use chain IDs differently um but you know do the best we can to some of this stuff now uh then the end result you finally see is like processing it uh making these metrics so Group by month now average sum all the fancy stuff you want to do um but you'll see like most of the metrics that we show on that Health dashboard you know transactions you know onchain profit all the things um come in through here and then I think is cool this like big CSV of data um We join all together we have all these Mets we talked about we see all this cool kind of metadata that we can kind of slice things by um it kind of interesting too is like we you know our experience building stuff we kind of filter this to optimism but there's definitely a lot more to think about from an ethereum wide perspective whether it's like all the L1 and all l2s maybe like a certain you know stage of decentralization for um these l2s but yeah a lot of different kind of filters and things you can do so kind of inspiration some things I was playing around with a little earlier um you know take the whole ecosystem show transactions per day cut it by you know L1 L2 L3 um using these data sources here um it was kind of cool to be like you know we know that after March transactions increased you we see like L1 is always kind of flat as expected l2s start to grow what was surprising to me is seeing from like March April as well how much the L3 started to cut into share of all activity um so maybe have some approach looking at Health where you want to look at any one of these kind of subsets um but I thought was kind of cool it's kind of fun thing uh second one so like from LTB we can get like the current latest stage of each chain um and kind of a cool joining experience is grow the pie has gas per second so plotted those two together um and this is just the the latest stage of each chain like you probably go back historically if you wanted to um but it was interesting to see like you know majority of gas use now is like currently stage zero chains I know some have plans to go to stage one but I'm sure you can make a case to say like let's look at stage one plus I think LTB made that case you know earlier at this stepcon um so there could be a case where if you're looking at health of the L2 ecosystem maybe this is a split you want to do but now um if you go through if you're in the GitHub rep you'll see the link to this Google sheet with all those uh there's like a tab for the the raw data there so your turn if you want to go and go build and and poke around at some charts um you know we use Google Sheets cuz it's kind of easy to share stuff but you know definitely any analysis charting tool of your choice whatever you enjoy I think we you like streamlit plot leaf flourish bunch of other cool stuff um yeah and I think do we have stuff for this oh yes so we have uh we have some special op swags to give away which is super Limited lied uh we have three toad backs there and we are really looking forward to seeing some really interesting nice charts insides you don't have to build a full dashboard use the data show something nice and um yeah we'll have three uh tote bags yeah I think maybe let's give it another 30 minutes for people to play around we we can walk around answer questions yeah yeah so well come back back in 20 or so minutes but yeah we'll walk around let people go and go build stuff yeah and I'll walk through a little bit too just pulling up the data here so yeah hopefully you can all find this link uh this is like the raw data tab that I showed earlier uh so you can build pivots on this download it whatever you want to do little sample tables we went through earlier um but also if you're in the GitHub you can download the raw csvs and you know put it in wherever um yeah we'll be walking around [Music] back back [Music] back back back [Music] back back back [Music] back [Music] back [Music] [Music] back back back he back [Music] back [Music] back back [Music] back back back [Music] back [Music] back [Music] back back he [Music] back [Music] back back he [Music] back [Music] o [Music] back [Music] yes [Music] back [Music] back back back [Music] back [Music] all right we'll come back here is he right all right I think we'll go through uh things everyone kind of put together uh after towards the end um we's go through into some other kind of closeup thoughts and things but yeah thanks everyone for joining participating uh was cool walking around and seeing uh everything cool uh so I'm going to wrap it up with uh the collective data Vision um so as you can tell cross-chain analysis is still as hot as as cross chain messaging and data source inability is also very important I'm just going to highlight here and also completeness as well so it's really hard to connect the data from so many different like open source data providers and also chains have inconsistent names um like is optimism op mainnet or whatever and also not everyone uses chain ID um the other one we're going to talk about some of the specializations that we wish to exist as open data s open data sets such as whether something is an active human is that a real user um yeah like you can use any trust algorithm or maybe like um world ID or anything else um also active developers there's so many different ways to measure a developer is it just like an onchain address or is it going to be GitHub uses or something else um there's also some interesting discussion about cross-chain activity and also asset transfer especially where uh when everyone is trying to develop intable inability across the chains and also continue building on the Chain ecosystem level data okay so I'm going to wrap up with this workshop for three takeaways um one is why should you build um so metrics matter in context this is one of the takeaways I would like everyone to have so don't just take everything for granted when you see a metrics be skeptical is is that real or not uh always question that and also you want to always have a holistic view of the market um and think about things in the ecosystem wide like why certain things are going up and how is it doing in the ecosystem and also metrics are not static they're always evolving there's no metric that is perfect so um keep building on that the next takeway is about uh like some of the open data sources that we wish we could have including your own ecosystem real uses stable coins decentralized governance cross chain assets and also any other emerging trend and also everything we build is open to anyone uh you probably have seen the op analytics repo and we've been constantly updating that so stay tuned for more to come oh yeah and one more thing if you staying until the end uh so this is another QR code that we have and we have this optimism super chain raw onchain data that anyone can get access to uh in Google bit query so click that and subscribe um right now we have all the super chain data sets here including blogs transactions logs and traces so the dream is just to select star from the super chain uh right now it's still work in progress and we only have like there's no promise of latency or anything it's just like still testing but I would like to welcome everyone to just try it out give us feedback uh the data set right now is only in October and September but this is the most complete super chain data set that you can get and yeah go wild with that do we have anything else okay uh that's all but uh we are here for questions and also we're going to oh for anyone who wants to who wants to get the toe back raise your hand uh show us the dashboard and yes uh [Applause] yeah okay so I can do I'll read the question the yeah I read the question out uh what guys do day to-day op Labs work with protocols if so what tools use to analyze protocol specific data yeah I could answer answer wide for this um yeah think it's like our I we kind of like a bunch of stakeholders or some internally whether it's like our you know product teams like Business Development teams growth teams uh so yeah so a lot of kind of like projects with them I think the main kind of questions there is going deeper into what's driving onch activity or you know what kind of things we see as as you know potential kind of growth areas uh that's we do kind of work mostly with like you know beyond that uh you know other kind of chain teams like some of them do have data teams that we like work closely with um some don't and you know use kind of stuff that we produce um so we started doing like a weekly is kind of uh like super chain Health Report that we share within op Labs with all the other kind of chain uh teams as well um that kind of goes through our just like rundown kind of what happened on the week like what's in what's going up what's going down um our takes there anything to look out for so just trying to like use our data as a way to kind of like show the reality to the ecosystem uh you know it's easy to see you know how many transactions there were um but it's hard to kind of be like go the next step and be like what does that mean and um yeah how do you kind of go on beyond that anything okay anything else this this is a long long yeah long session was cool cool I we can leave it at that um yeah like we said I think we'll go around and look at um yeah some charts and things and uh yeah beyond that oh there's a hand oh there's not a hand you do have a hand yeah okay cool um I don't know uh cool yeah thanks everyone for coming uh I think we have some time left in this room so we can walk around and and keep talking about stuff yeah thanks everyone