# What on-chain, social & development data tells us about Ethereum's growth (Devcon5)

- Channel: [Devcon](https://streameth.org/devcon)
- Date: 2020-10-09
- Duration: 1:22:06
- Topics: blockchain, smart contracts, distributed ledger, ethereum, Science & Technology
- Watch: https://streameth.org/watch/yt-uauHr49NG-U
- YouTube: https://www.youtube.com/watch?v=uauHr49NG-U

## Description

By Serena Tan

In the spirit of sharing knowledge to move the ecosystem forward, this is a collaborative space providing demos and lightning talks by tech leads, researchers and data analysts from Amberdata, Alethio, Coingecko and Santiment.   

Scheduled talks include: 
How to export ETH data into a database and query it; 
Exploring Ethereum Dataset with Google BigQuery;
Accessing the Python library for data scientists and researchers.

## Transcript

[Applause] going today by our data electorate I'd like to say the theory foundation and all the volunteers who before we get started with our sneakers just like you know that this is going to be an interactive session so if you come here today really challenges or something that you can look you tribute to the community please feel free to write on the post-it notes that are and then we'll make sure we get through today just a quick context hi everyone what's in the matter of when I see on sentiment before for a teacher and crouppen 2016 I spent like seven eight years man yo community excited you're in this edition few things in the big difference here what the size of the one the second thing we do see a lot of challenges all of it for some time statistics and ones of this elasticity or two but the way we destroyed you know just try to make stuff out step by steps for this one and that we have here is the different projects and we are doing one thing all together we're trying to make better and easier for and talking out people with the data doesn't matter is still valid one thing and we won't move to make it better show us it was how can they jump in mode very fast how can they depart what we're trying to change this won't be all us be inclusive at the same time stinky everyone will conversation please share what you've got give it Oh today is PMA the founder of Veneto which is an alligator platform for tracking cryptocurrencies and blockchain assets since about incoming Castro in 2014 yemeni has closely followed the development of crypto economics and the real-world applications of blockchain blockages technology apart from that team isn't contributing what there are two books on the topic of digital currencies published by Elsevier he holds a Bachelor of Science in computer science and a minor in psychology recently he was also listed under 30 for 30 under 30 agents like 19 please join me in welcoming Thank You Serena but it's opportunity so what's up you guys follow quite yet could be CPI data for writer for this case we had some business movement needs to be done on UNICEF so this is more like sharing the experience that these terms of finding all these data points until from Chamberlain to share with you just admitting to detection so we can contract call changing the scale of the into F of G which is gonna get down to exploring the data that is already want to change and typically sections smart contract how much balance a contract event these are addresses and some some events that they were long from the front check when something happen so when we've decided the most obvious thing to do is to run a who knows right your options are like Jeff apparently and then you have to note money have instinct out which takes like a couple weeks in advance of RPC calls and show them how to CSV or sort of the data that you can use against you so this is way you can pull control yeah or access data by the spiky days for us we just want to get down to dinner and get money immediately the second best option is to use an order to service so alchemy is divided on yourself and then Prague in the same way through how to the food on using a PC calls some code which are not sees me this is great and again it might eat wheat it won't be expected so the third option is to extract everything out and consider using third-party API service so what they do is that the third party interface will probably quite upset then key things that people are interested in and abstract it out so it's only internet the mess with API again I guess I know cSPD for us to work and analyze those data so a few example you could choose some other either scan the data you probably could pay some sort of money subscription for some of this great thing is this is no set up is no need to work with four nodes and this data index so it could be just grab bodies data immediately based on what the third party movement so the last one here which is something that I think most people are not aware of shall have to share the Republican this is what we ended up using outside is the bigquery data set here so I think this anonymous name and of last year where Google has initiative where they will index the entire property on to bigquery which makes it very easy to extract certain set of data so the great thing about Google bigquery is that the data extract which could be as much as you want can then be integrated with all the other Google services that you need to do so you could also export it out into a CSV or JSON just that how you would deal with an API so this is quite nifty the difference here is that the data updates every six blocks but this is kind of unofficial because we can't really find account documentation of any of this every 24 hours but based on my experimentation and updates almost every six boxes and is good for not about a sweet Venus tens of people what not technical they could just get down to this bit of any set of API to get this data up so this an example I want to find the addresses that have significant outbound transactions transfer from USD T I could easily talking a bit query sequence I structured make the call keep in mind that the charges is five five dollars per kilobyte so if you think it is a how much data you extract you could be a little smart in how you choose the the cue that you want to commit in order to state costs but pretty much this is your constraint in terms of cost and the options to an export to CSV in JSON you could do so and the hotkeys from that data point that I extracted are basically the large circle that you see which is already here but those massacres those addresses that has large amount about transactions so this may point out to be exchange addresses if you will these are the kind of things you can do it could easily extract some data pump it into a coughing solution and guess of data out of it and then before I give for that part it's somewhat sharing something quite interesting when you make it something a bit queries but it's social where you can share the queries and restart around and it's a second book book like a Jupiter no I'm not sure give you some of this Python to be the book we put layout text together with an explanation of data and that service is called genetics I highly encouraged to check it out as well because all these data are available over there for free you can just be make movies extract it up and there's one example on this about how the fairy content in fact the the price of gas said these are some interesting insights that people share so I highly suggest you guys check it out so the key takeaway is that is plenty of truth to transform all the way from having as much contrast 1 over 2 as easy to get started so it's always to be trade-off but it depends on what your objective is for us it's going to get in extract some data except analysis we went to that side but if you want to get more into our data we could leave towards the other side of the spectrum so really there's no barrier entry for the analysis solution of our developers to exploit your data set just go out there who crazy and the bodies do that and I saw you want that for us the data API not want it yeah not much a year ago but marketeer upon me you can use it if you like and that's all i have put up some quick highlight on the options for exporting LSL i hope this is useful to you we're going to continue to learn about these cases thank you I'm open to any questions and I'll try my best to answer [Music] what was the costs of using acquit how much that big by consumer yeah so for example for this one when you enter the query on the bigquery they were actually one estimation of how much data you would process at the bottom right to this before you actually executed Google will tell you that how much bigger by the table as a data you are using and then you have to do the math yourself five dollars I think anyone is a data warehousing solution so it's not supposed to be popped into a fund and if I imagine most at me if you're going to use this one application it would be like a background batch processing where you could control let's go clear the air about it being queried by outside world so if if Amazon site I think you have intership of storage but then you don't have to pay for critical because case you put the line in there it's free but every query do you make they will charge you $5 in other words did you operability or any eyes before since we generated activity maybe toko transfers just a very standard I think that is support but is it possible to get by yeah so so that's why terms of control maybe we could we may not give you marry in a one so this is just some extras back so if you go into the cooler you see the list of tables that they have indexed already on your site so clocks is pretty obvious this drops data contracts our contract with the byte code so I think you need India on top of it once you extract data blocks contract events token transfers which is one that we use these are tokens metadata is not much terror in their courage changes I haven't used them so I do believe what is last one is transaction data which is every human transaction of their so you are from stream within this set of tea but if this gives you what you want you could use it in another you might have to go up one level and the box here at included fiber so the API called yeah yep the decode user it's fine function probably happen to meet the stuff so like one example of the kind of data you would get would be back yeah yeah you probably get it in this form like so you cannot develop a year to all that work she punctured at home and decode this into something that human anesthetist chop it up any other questions I guess I guess I goof it up okay thank you thank you and just while we're getting set up with the next but now like to introduce Sean Douglas from at Mozilla a platform for monitoring and searching and analyzing public and private cluster prior to a unified building and operating the company's rapidly expanding expanding across authority and management analytics and reporting yes he'll roles as board member operating executive is a graduate of the carpet please join me in welcoming Shawn Douglas all right so I shut down this room every day no we combined watching day and market data you get a single data platform and serve up a RESTful API as well as WebSockets as well as our PCC connect access data understand what's going on a champ we support seven blockchain today as well about 2018 is a fact about complaining about chain by year so I wanted to kind of quickly strain it is going on happening right now with crypto area so everybody even believes this is a massively disruptive opportunity there's it's about 300 billion dollars in market cap right now across all of the total assets of New York creating an open financial system we're creating a potentially in the Internet and this this is going after it disrupt opportunity that actually discussed cloud computing remittance payments store value fiat currencies so why is that work and why is it so disruptive it's because because of everybody here in this room is building we're able to delegate processes that are radically transparent that provides transparency is not available in traditional financial markets that drives network effects aligns and sentence drives behavior and has a very low barrier to option that we can all develop against and I used to be a venture capitalist leisure to say here is you're always looking for things that drive network effects have social networks incentivize behaviors crypto massively enables that and you know it's crypt electric it's 50 left of Krypton economics is the fundamental reason why this is so massively disruptive is because because that we can create these mechanisms it incentivize behavior and because data is an input data as an alpha we can understand what's going on so with that said today do you think about aetherium the theory was about 18 billion dollars in market cap that's like AMT it's it's become an industry buddy here in this room is creating a district if you look at the top 50 tokens that are trading on etherium that's about eleven billion dollars in market capitalization and those about seventeen billion dollars per day trading of evidence - almost two times the total market cap of those tokens is trading on the top 50 so it's it's become anything it's not what we have been doing last year now I'm going to dig into the data to actually show how to crypto economics in action and the evolution of what's being built on champ so this chart is pretty interesting because if you look think about the most simple form of crypto economics looks like I'm going to be a miner on the vine blocks I'm a get rewarded and that's going to keep the system in balance to keep the system safe when we can see here is that over time when the ice age 7 is last year and you were starting to actually see the block time slowed down there was a lot of congestion in the network and they had to do that constant will hard work and then once they did that the system went back in balance and so it's a pretty good depiction of crypto economics back Cooper works so you can also see we're starting to see now to go back in time during the big ICO bubble or what have you there was a lot of transaction activity on the chain but really that was mostly single token transfers today that we're starting to see we're almost at the same point before we were previously the number of transactions on chain except the composition of those is very different the composition of those witness is showing here is it the actual transactions on chain today here this is wallet - wallet transfers these are smart contract transfers and including smart contracts and tokens and theory starts to be a global computer and mobile computers run code run software and what we're seeing here is that the actual majority of activity that they're in is actually interactions with smart contracts it's not just transferring value which is great because that's what we're all here for now the whole network runs on gas what we're seeing here is that in fact people the consumption of gas is driving these smart contracts is the kind of the lifeblood of was happening on area and you're seeing it is becoming true this is uh this is you know what we set out to everybody's room is set up and participating in building we're actually starting to see it come to fruition and you can measure the adoption utilization of our contract by gas consumption but let's gets a lot more interesting everybody here has probably build orchestrated systems and micro services and what have you well last year what you would see is you would slitting a smart contract would execute a transaction that would transfer Tony and that was that was done right but what we're seeing now is things like DX dy DX and 0x you're now seeing orchestration across the pentacle contracts so this hasn't cost at about six calls deep which means there's a lot of interactions with different us smart contracts so we're seeing this is you know going back in time there is literally two spoken transfers over here over now we're seeing orchestration or causing multiple smart contracts at five six levels deep so you're seeing much more complex applications with interactions and dependencies being built it's it's really an evolution of maturity and realization of much more powerful gaps being built you know back in the day was crypto Katie's was this little ecosystem that maker Dow was this little evil they really were Islands amongst themselves didn't interact however with the in bat advent of 0x where you could actually swap one token for another we can now start to see people connecting independent apps together in orchestrating transactions across these two enabling token swaps with compound coming into the ecosystem where we start to see lending we started to see borrowing we're seeing leverage being brought into the whole device and enabling this we're now starting to see you know people doing 250,000 transactions in a single week where they're taking compound interacting with 0x levering up their positions or or doing what have you so we're seeing much more complex interactions additionally with protocol bridges coming in you're seeing you know transactions of 300,000 transactions in a week where people are taking and building a protocol bridge between compound and a CVP protocol ICMP pull together it's also a lot going on eunice walk where you have a single token being swapped for another we've seen 510 thousand transactions with a single week where unit swap is interacting so you start to look at the complexity the complexity of these deaths in the orchestration across the system across grants action across contracts it becomes really really interesting and multi collateralized I just kind of think about where we're gonna be a year from now where you can actually take pieces and parts of smart contracts and start to orchestrate your new idea across those it becomes incredibly powerful where we start to have scalability and more efficiency and people start to share and we're building building blocks that we can build on top of each other the data is there for everybody to see the thing that makes blockchain and crypto so powerful is that crypto economics allow us to create these systems incentivize behavior measure behavior and then have data is an input and data has an output so you can measure your system instrument things and how they'd work like I said earlier we're equipped to economics data platform we had market data blockchain data metrics insights web services WebSocket web stuff it's restful api you can kind of do is just like and you can't go full Novaya RPC so feel free to go check that out [Music] alessio alessio as a powerful blockchain data analytics and visualization of a lot more Adrienne is a software senior software engineer building systems architecture with a focus on reliability and scalability is a Software Architect with the love of distributed systems and he says he played with Lego for money so please don't involve me in a bucket okay so yeah we certainly try covering transparency to the blockchain so I can take that platform as well having this guy they have stools but we never come about now this we're gonna talk about how you can get no third party required so it's just kind of and the graph that had a shock to the first slide where somewhere in the middle of assessment tool that essentially it's open source on github you take it you get any RBC an aquarium clients and our tool and what you get is the Postgres database and api with indexed chain data like something you can set up very easily on your own and control it what you get to get rocks get uncle's you get transactions index by account which is something taps key and get events log connection we don't get prepared messages also known as internal transactions or traces and you don't get reward polity so thank you for your help one thing that gets replaced you can't see what that we work what protection is for you and you don't take a little Cody and I just wanted to like show it actually might the network's does so after following the repo which I'm not the right now this might be problematic you get this configuration file where you put them like a note in this case I'm gonna start the journey and you have here like a feature for the tag which if you don't want to handle George you can set up for lack of dead and we'll just wait for like why not be there so you can get that button to pause right now so once I started up to just going to start getting data what's that committee set up just do this starts up postfix turns on the witness [Music] he wanted this good though great [Music] [Music] [Music] device [Music] [Music] it might actually work [Music] [Music] sit down parsing books so nothing guys this year [Music] okay [Music] okay so so this gives you like an index of potentials for their account if there were like it bends looks [Music] [Music] yeah so just like we pulled with grease use it to just index your own chain and there also if you develop you get that we don't want to pay for any papers and you don't believe a lot of thickness just like one command again that's all look for more complex things like internal transactions and the more dynamically you're handling so for example you get an information of you can send a query with the dainties that they have that you know about and our API can tell you exactly what category the new type of what transactions got throws back but all of this visit ilithyia and check out the bigger TM for small means you can just take it lose it Thanks it seems I can be challenging to use their internal transactions because especially if you go back to a previous felony showing how many transactions or contract all these days is that you're using yeah but whatever you wear you have about that so you think it's a difficulties without yeah they cure them many many connection because they are diamonds I guess it depends what the risk is is right like of course at this state if you need to understand internal transaction this wouldn't be as won't work given that it's open source could be extended to by supporting something like a fight it would lock you into parity that's what they are is something that gives you traces in a good way but this is like this is built to be like super agnostic just use any party see yeah that's kind of like if you if you can avoid using Internet transactions that this is fight use if that is that we try to give this to the community and wanted to see how this could evolve even bigger mechanism of you just be also more complex you said yeah but like non-assertive maintenance just any any immediate network or best name that basically just connected to specify the note no and that's it for our case in the demo we use the [Music] is my comment valid seokmin all-inclusive intelligence software he loves building rates on the Oregon's optimization map software like AI and scalable systems when he's not cracking in the mountains or kite surfing on a beach please join me in welcoming so I hope everything goes according to plan and make sure that going to start with the light they all because it's going to take some time to compile because I want to show it how it looks from an instant so dude this tool is very similar what the guys showed for getting clear all the analytics of pepperoni so poets like names or links to miss boiler but it is vehicle to export events out of the term mulching into cold and have them in a database where you can prepare them with SQL so we can build your own API you don't rely on any third party and it should be very easy to sell so in order to start so first there's the project generator so we need to install this thing which is project generator within Vienna and then this is the generation is moment of it yeah so you need to install the youma which is like a project generator for india and then this is the generator for finished project then you create the folder in transporter so I'm going to start doing that [Music] so I'm going to create all the threads and then I'm going to you know my tractor and it's nonstop doing okay now wow it's duty so why wait to do this so wait [Music] so watching right and in the bottom we hear some kind of events or transactions that are coming in and the poaching has some so it's going to hear some API and we need to have some kind of connection so we we need like when we do that we need to have some a gradual do because we want to create the UI like for example if you compound you don't to have a website for compound to be able to see when somebody deposit something to show that oh you two posted here is your balance or you want to see what is the current state of the market how much money has been invested what is the for example the amount of money being deposited every day so it's not only about fetching what's the latest transaction but also you need some kind of aggregation on top of this and you need such kind of aggregation not only for UI cases but also for example from to do fraud detection let's say or you know whatever your needs are so usually the foo nodes don't provide you with an API for that what who knows gives you is give me all the data for given book or give me the data for a given transaction or every what is the latest so they don't have any aggregation capable they're not able to say like select from all the deposits group them by day and sum them up and stuff like that so the current API is that you can rely on are either centralized like our or they're not in real time like bigquery that's it actually I was quite surprised that right now it's six folks behind before that it used to be like 24 hours but also as going to get people mentioned it's you need to have some batching on top of it so it's not like the real time thing that you can query all the time so then it's not going to work so if you are in the case where okay I built that I won't clicks or the data somewhere and I won't just run something scare on top of it to give all my needs then there are not so many options out there also but it's great now they are it's amazing so I look at this pipeline is going to look like something like so we have a phone then you have the data extractor it's going to extract the data from the photo and then it's going to put in some kind of query system in order to be able to wear it so why it's hard to be centralized it's hard because it relies on a lot of state this is like the biggest problem like it's very hard to this this state needs to be like the typhoon all they have the statement but it's organized the specific method a lot depends on you like how the poaching works and it's very hard to do a maintain this estate so the only way we can kind of figure out how to decentralize this is troll constructs let's make tools that are open source and people can use them to extract the data and compare their needs so if you are able to just have an easy way to run a tool it's going to extract the data put it into a database you can do it guys then everything Google do now why would you think as a data provider well we provide a more simple type of analytics and then on top of it we're gonna provide some more complicated things that we're gonna do in cancer so this is quite good but there is this data should be available to everyone so it's good to have operator so this is the URL to the project that I'm just showing so let's go back to the demo oh it's done Austin muckerman so directly working so far so we need to provide the URL to some opponent for this we're gonna use stuck in Europe because this is a in this way it is find your project when copy the URL and pour it into my consideration here here like HTTP okay and here are gonna specify from which walk I want to start sinking this is how many confirmations I won't wait for so that we don't need him the reorganization of same thing before and here's some that you can stuff so let me specify like this one because I think the more actions get me there and now I'm gonna run it picked over pause so now the pipeline start [Music] so this project here is generated uses the API of compound I found the compound API the compound documentation to be very good so it's very nice to work with so I started to sink the events now and it's going to decode all the events put them into database and this is all happening in real time so let's see what we have here so first we have a very simple let's look at the code or maybe open the database so we use a flip house as a database it's very very fast analytics database it's colmar and we are managing to handle like billions of records with that so most probably is going to so so I have a this is the main table events and I'm going to select advanced I'm going to take the five records and what we have is for every event we have times time the address from which the event is coming from the cache of the book and here we kept the Dakota Tibet so in this case we can secure interest so this as I said compound and it shows you like what is the by borrowing the ex interest accumulated for this event another so if we scroll up you can see it mostly [Music] [Music] so we have four small rules basically we put the API the ABI in the start all the events across all in this case I think it's limited to certain addresses but you can expect across the whole ballgame and it is going to as I said okay let's look at the code so this is how the exporter looks like this is basically so we mentioned condensity to is like an icicle so you get the angel okay so you need to have the API in this case compound API you instantiate the class you say extract events with a pencil and here's a simple REST API that is built with micro it's a JavaScript string and right now it gives you what's the total amount of dependence and most of the events over time so I can now go home and say curl I think it's important mm yeah I events and here's the aggregation that they get so it's good at the same thing so you're going to take some time to it seems good but yeah I mean this car works and you can take this thing and deploy it on your costume like you only need docker you need appointed digitalocean or something like that it gets for any smart contract you just need the API actually last night I was playing with the dy/dx api image to extract it so here is like an overview the floaties in theory the data extractor in this repo and scalable system is the silica so that's it the events only focused on events yeah this particular project is focused on events because we wanted to make it as simple as possible the truth is this open source who is using another open source which is more or level and for this one we actually use it for all kinds of things for example I can show you on our github it's again like an open source we use this war level library that extracts all the trades from centralized exchanges and it's basically game like JavaScript it's a bit longer it's like 69 spoken but it extracts all the traits from these exchanges in real time [Music] this is also open well I think very great your data where you say I want to take the latest transaction the latest or whatever Oh so I want to get some aggregation because we disobey the basement secure a big house like the reason we use this post bracelet is that we found it's very fast in doing aggregations it's developed by Yandex so it has a very strong community behind and we've been able to run a lot of the things pretty much online without any need of doing batching like cron jobs or anything like this so I would say now you need to be careful though there are some failures like the software engineers there is always failing so one of the trade-off is that we dis date the basement if you want to some very complicated joints might be a bit tricky especially if you have to joins them with tables which billion records then yeah that's not going to work but if you have like a billion records and maybe a million records that define so still like if you want to join all the transactions with all the works it's going to work but maybe it will be slower if you want to go for the whole strip so it might engage down and the other thing is the way it works in third Alan it's possible that there is a specific syntax that you need to use which is different from Paul's prints and so you need to be careful because database might return to some duplicate records so in order to avoid that you need to use for example select from final and then you don't get so there are some kind of catches like can you start the pipeline multiple duis or is it like one eighth favorite since perhaps maybe an API ABI yeah yeah laughs it never will or just one person yeah currently it's one for instance but you can share the same Eclipse so I mean that if you look at the docker compose so this is how the docker compose so we have like zookeeper and Kafka to storage click houses the database and exporter is basically the script a string so you can run much exporters here and connect them to the same database and this server is the API that I shall be the girl [Music] what you can do a chili you can merge the api's together into a single API because it's just enough super okay yeah super a bad because the API is just the list of description area and you can just merge you to list and then the super idea [Music] okay so the name spider valve section up here is gonna be interactive we're gonna be and three parts of the game dataset so if you have a posted at the end of your like seat area you might want to note down three things so something that you need something that you're looking for in terms of data supply any challenges that you're having you can kind of knock them out here and yeah anything you can contribute so if you're looking for opportunities this type of stuff scary you'd like to contribute we share those types of things and just before we do that try to like to gauge the level of the audience here is anyone looking for something more advanced than what presenters have presented so I'm looking for a little bit like entry-level were advanced good well watching mempool transactions so pending [Music] anyone else looking for something specific yeah I can contribute something maybe I'm events and people in China two injections actually and we also have body a the ice so we know that function calls so we have over 90% of what function parts covet and be put in the database and elasticsearch trusted so you can use either whatever if it's you need and grab all the data for free you could actually do the math and time series on database that create X data with us so anybody wants to use us feel free it's like if events and you can start and also use analysis analysis data and you need only once that the whole server one week for synchronized north and you can use all your oh sorry data on your server this that the place is under with lessons and you can use it [Music] all right so we have a challenge here I'll to prove how to prove that the data of metrics are not faint that means we want to have those you guys back up to your chairs or anyone in the audience so for this question like that that's actually one of the reasons we've been thinking about how you decentralize Canadians and like if you're using an API home provider you can't really be sure if they're not giving like well it could be like I'm speaking to the data right I don't be like deliberate and the only way I guess you know how if the center I so you don't need to trust a third party well draw contours like you need to use your own open source pipeline so if you want to do that and using some kind of tool and iran locally and to connect phone only to trust that would be the way to approach like i don't think there is any other okay yeah so I think the blockchain is the source of truth so anybody the aggregated data like you were like us or like in Europe how can you verify that that data is correct so what we do is we allow every single API call a return and we do the proof is you can always change it's correct but the offtank data because a lot of people that hire in detail because of the locker informations they collect yeah so if you don't continuously scrub that and be able to verify it and you could be serving that data and so by serving your work will proof with your data you can verify that's what we do so because otherwise people are going to protection but if you serve an aggregation like let's say you want to show number of transactions per day and there is another interest equation then we can actually measure that in the different ways I mean if you talk about number of transactions for a given table Internet transfers or last some other operations like does to hold also like rules yes illiterate they make it become like if you search but given talk then you can verify that for me won't get attached to certain transaction yeah when it comes down to aggregation it becomes I want to agree with you if you count things system some things several ways if for example this one will be Genesis there's many many more with anymore so example you count attachment is really common potatoes very different than them and also when you start aggregating the data to be Rwanda finds really weird so one of the things that we found is that you know during the Ice Cube's cost all these pre-sales sometimes they don't advertise events or the pre-sales on the book chain so you start aggregating for example one find what are the four coders of dope and the results doesn't match and when you go down the run code you actually find that in the smart contract there is a list addresses but they're going to receive a bunch of tokens when the contract is created let's say and this is not advertised on the blockchain at all and even more interesting thing is some these top coders are not even shown it again and what happens is that the first time you open an address forgiving token emitters in there when to create the funnel figure out that they had some kind of balance with all them inequalities of holders so we've found no folders with millions of dollars of tokens that hasn't even checked in yeah I mean it's great well disability also it'd be interesting how they and the dog park like and yet more like they are called change the balances material find without meeting any transactions and my director just what that refers to something is called count Amos and we actually have implemented accounting yet so if we ask note for that but the question is who or what Atlas owns what balance of tokens origins especially so and that's explicitly hard and we think to most robust way would be to carry the node for each dog penguins pretty costly notice your only truth that actually has to say otherwise if you do the math yourself so if you compute the state I think you can say something of that as well we said we can't recompute the state of the node correctly because there's all sorts of quick yes and this adds up to this so you actually have to ask and the dark-eyed note which is like four terabytes or petabytes of storage you can actually ask the defendants of that dress yeah but you know yeah but in front of the bonds of death this address was created without making a transfer event 2,000 times for the case of daca how do you know that you have to ask for interpret what that you don't then you don't have the most basic information well but you might be asking I want to get for a given address all the balance in this address has the most angles also the government uses when you and then it becomes complicated you want to say hey with my time series historical account balances for life you know all of my trimming lies just my life 360 days change of token balances get into like call 2 million bucks aggregating paying all the year logs being extracted your token value market prices we have that available today - how did you get it you asked them both why we built that and the reality just like you would probably a few other people here know he feel like we played every transaction since the Genesis block you know extract us we don't do to our Nationals across the funeral business but we can save this law so for the so we basically reviewed most of the contracts such kind of great ones it's actually it's very easy to find because when you start like if you look at not the top holders but the small holders if there is if somebody to give yeah if somebody gives organs under the table that's the same and this office gave the token somebody else weakening Isabel so when you see a negative balance you see that somebody's doing something funny on the contract so you will never have all that you can't do that yes [Music] [Music] I see no events that like her they said that some money transfers like the amount of partying dress the transaction failed but the event was given in the and you can see a cycle that's why those two appointments lens for County is bad choice events is totally optional for development and you essentially can write in there whatever you want you can actually do something different to a call and emit something different you could rewrite what and that would be another way that someone did but that was well I think this particular case my design there was a book in the smart contract then they were emitting events even though the action fails like they check oh boy if enough money to transfer oh I don't but I admit that event before that and and this is like I think in the end of the day this contract is unusable basically this is a cat we needs to be mark this this is garbage and throw away because if you get a wallet then that needs to like like work with these things the wallet there is a high chance of them to live offense and it's going to show like crazy so events should do it like they're built for having reliable view of the state of the things how they change not all contracts are by editing the other interesting Fitness there is not really good standard like years 20 is not a good standard at all now there are no events there for meeting and forwarding table so how do you calculate the total supply outlet events you come there is a better standard which is like a doctor but no pie exactly what we learn - and I don't think that events ever they meant to be a very common combination I think it's a totally optional the hands-on developer how to use event rather than the function it's not a precise function point it's something additional but then what's your connection with the outside world like the smart contract needs to have some kind of a connection to the outside world and like the events are the causes comes back up to the developer to use an event and it doesn't have to be the same as the function call in a tight relationship can be loose relationship and that means room for interpretation it's not a definition well on the bright side I see like for example for safety I see some very nice events like the spark contract that are built like the latest generation seems to be much more well behaved from events perspective so if you want to build some - part about define that tracks like interest rates or all and all that events movie make it as like we had stuff well maker I mean I mean very weird so it's all like yes like very weird few things yes that's how it stops they actually have to write a bigger positive for events which this sucks this is actually you probably know you might know as well but if you go down retro and input in epic events and Houston has needs of communication the more complex a smart contracts get the worse it gets nothing tends to be some sort of civilization in that process as well the key value few names is just stop and there will be way more and make sure if you go for sane or move over for cheaper change and actually use that means interface like heavily rely on events that would be a total move actually change how to interact with prop chain and I think this could have like one auntie James for example the transaction is cheaper well something that we noticed is that once you remove the proof work like use then a lot of spam starts here so this can analytics it starts to take a huge amount of space for example for us if you want to crunch the whole book chain and put it into like analytics database and you put all internal transactions into the mix so basically throw out everything like it's like almost four terabytes compressed Jason like the whole thing and it's been running for one year I wonder what these guys are doing you have like insane rates rates no worries I worry about you two then four LEDs yeah so it seems a shitty us we mistake you get try outside circuits I practice but every PDA or stack and who they get lost in the spam and that value do you think well then will they they use case continue on despite despite this map or this movement everything here is one it's not decentralized with ER staff as well used to be so the spot is more like people posting right understood right there is a culture that they're not even doing opens like this man is not even transfer some tokens just people publishing some binary data of the poaching or yeah I mean it's really weird but you need to keep it because you know you're in regular Linux so we need to keep everything and it's insane like I think for one year of yours history we hit built like took us maybe six months in order to build a special fooled like we need to work into the full vault itself in order to be able to extract the data fast in to one month everything because it's about four terabytes I suspect everyone big change the beat where she started up on the chain as I mentioned Solana and new challenges how do you like how it is approaching if it's hearing this amount of danger like you won't be able to run this anywhere like it Syria right now it's possible for events like it's not like archive code you can run out or walk of it it's probably going to take like 200 gigabytes but still like their clothes become a device of this so it's two possible but if it's something like these other teams I don't see it's not this answer my story can I stay yesterday there's many others big chains they compass in a private environment machine to machine communication so there will be many big chain things and I suppose there's many others they all have just to produce case they're talking of topic Chinese name but they all have the archive and the monitor as well you have mounds of data tend to chalk it open all the way at some point so the only thing that has the tool is no in the state with power on if you lose power my means we don't mind I will see this guy that was building an extra piece can so winters came before Riku and he was boasting yeah I run my own node now and like some pizza pockets datacenter like it took me three months to see it like yeah come on like you need like ten terabytes of storage or something like that and they actually write this in the documentation if you want to sink a food rip oh no you need then I think at least ten terabytes of storage and it adds 10 gigabytes per day this is only for the boat and it's not just like for trends then go to three times [Music] [Music] that was the only one that we give the net meter button as it speeds is going to do it so we use their own by the way Stalin from Google betray I can ask it questions because other the business was a very first presentation was using bigquery how many how many pick reducers are there in here do we ever plan to use a tea eyes and well state government is not different it's down so there's an API to transfer expiry data to do you want if you want to decompile the DVR is to be no to execute the take the data input and translate into something else yes I think you can do this by using the are there some ethereal JavaScript libraries that can be used to do this afterwards myself what she means miss me when did he say his bill when it's partly bigquery if I understand Facebook so I remember a Nick Johnson did something like what you're describing with the fjs library by calling it from a so all of the big for UDF's our weekly javascript and he found a way to shimmy in fjs into a UDF slice that he could run an EVM emulator inside the query so there's a lead for you might know how hard he thought on that but he was doing something like that I'm familiar with the terminology but that sounds like this kind of thing yeah you have this temper user / yeah yeah so you call the JavaScript you included JavaScript libraries that you can then call these functions yes apparently is is using his part of the registry there they got some contracts right cuz I wanted to reproduce the results of the contract falls inside an incredible family down here and probably not because they I guess the right thing to do would be to take the if you know what contract was called and you had the API you could automatically process it and produce the output but the problem is is that the structure of the inputs and the outputs are very flexible so we couldn't really define a schema that would easily support all of that that would be easy to clarity it would just be kind of nest of or repeated field of blobs which doesn't really buy you anything other than having done the execution with API so you know that again yeah yeah it's not a service structure know it's tough key that but every project yeah yeah if you index like thousand in the contract they have different keys and the joy oh yeah it will be flat so with this sub I find that I showed so currently the schema there is unified for all the events so here this few codes decoded which contains a JSON is critical table and then you can function in the database called like Jason extracts straight instead you specify the key is going to extract the key so you can then get this way but there is also a possibility did something that I didn't show like with this five lines of code you can work him go back and take the decode that they take the myth like the flags Jason out of it so we can rearrange the data whatever you want and then there is a SQL script use for utilization to create a table and you just need to create the tables with the same schema so you can adjust it in a way that okay i'm parsing let's say compound events like like these events and this is my scheme a lot of support so i'm going to translate it in the script and insert a normalize so it's possibility to that but yeah I mean it would be nice to get some that can see this actually solves any problems for people we use despite point for our own needs for many of the things and it's been working pretty well so we extracted it from our pipelines but yeah using a little bit different like me tend to normalize David views we don't want to extract Jason's as its lower so but for the devil and for the project sous-chef like chemically encountered any kind of such kind of problems some Ford apps that you write like you're right attacking molecule tiger you eyford or maybe some analytic spirit like giving conferences or is anyone in the room who built in Tubbs visualization aggregation in stuff like way that finds the cheapest highest level of long-range at the moment of death relatively somebody was curious about men pool talk about detective imposes statuses thank Jane so what anybody else interested in discussing that when problems your topic anybody that would currently look at the gamehole I don't think we have exposed any mention of itself but I believe to them always useful for frontrunner predictions because you can see basically what's a changing it yeah it's about what it's like to go by per day and what one location is something like this but um I I need power because it might be like because I was recently working on the trades from centralized exchanges to extract that is that was about like for this five or six exchanges about one gigabyte per day so maybe I'm so I need to double check here but we you know use case from that pool for wallet ad they want to display for their customers that transactions are submitted and it's not good enough to have first block which we have but what about pimples anyone do such a walkable or a little bit you want to say something we were just I just nice yes the body acid and post but then left I guess yeah I mean for us we do basically is to show the Explorer like the stink of transaction that we've seen we take the determine the time spent things like that but nothing fancy aggregation we're thinking about it but we didn't really find it with use is that apart from what you've mentioned like showing the transactions as soon as see that I anything but it's like basic use case and there's the real-time aspect so you don't keep any data or about where tribes divided oh we do we do keep some workers of it but which has just have a dozen people other things on that yet yeah we also used that to figure out that's actually get expired from the pool and I kind of it's just like that and as your difference and that's actually another if you want prediction you need to have at least I would say a bunch of confidence so you need to have to graphically stir this is for lines and almost they're different and they say actually last question but that's how much do they defend and usually they shouldn't believe that much but they might in terms of when the transaction is first seen where is the mentor as far as I know is not equal over there there are different methods of different rocks it is readable but also kind depends on where the transactions are mostly coming from slight if you do not I'm asking for if that's the source then that's going to be your main from there yeah it's hard to know when you see a transaction you're known kind of the what hop of chemical distribution that transaction is like five hundred milliseconds will come to the first note that's possibly it is just season abroad as soon after you get the phone [Music] TV right [Applause]