Bring any Web2 data onchain - Filip Koprivec | Flare

Cool. So hello everyone and welcome to this workshop uh on the flare data connector. Um this workshop it's titled bring any web to data onchain. So first of all um what this workshop is about it's about the flare data connector. This is an enshrined oracle on Flare that allows you to do well many things, but the one that we'll be focusing on today is bringing any web to data onto the Flare chain.

Um and well the main advantage of that is that the data brought on in this manner can be trusted because it it's done so by all of the flare validators also called uh uh uh data providers. So I think I'll just start with the example. What we'll be looking at is uh Star Wars API. Um we'll be looking at the second link. Um and I think I can also zoom this a bit.

So this is a simple API that allows you to query Star Wars data. And uh one we'll be interested in specifically is the one for people. So what you can do is you can go to this URL uh or post a get request to this URL with a ID uh appended to it. So for example, if we query Star Wars API.info API people-1, we would get Luke Skywalker.

And so what the flare data connector will will allow us to do is get some of these data onto the flare chain. Um so what we can also do because the data in this shape might be useless. There is a lot of data. We don't need all of it. Um it perhaps isn't even in the correct shape.

So what we can also do is modify it or uh yeah modify it by passing it through a jq filter. So I'll open up a website called jq jq playground. Um this is basically a website that allows you to preview how uh your jq filter would behave. So if I just copy the data here into the JQ filter field and I think I also need to remove these two characters. Uh then it will allow me to for example do name equals dot name and yeah you can see that um we can get uh field v val values out of the jq very simply.

Uh but jq is also a more powerful tool than just this. Uh we could also get something more complicated and for that I'll just take a um take a pre-prepared example uh and I'll paste that in. Okay, cool. You can see that I've just been copying and pasting this from the Axel show code. So it doesn't work out of the gate.

But yeah, you can see that if we take the URL, split it over the forward slash um and take the second to last value from that array and convert it to a number. Uh we can get the UID, the number one. This is the thing that appear in the URL. Um or we could also um we could also get the length of the films array which would then give us the number of fields films that the character is appearing in and the number in this case is four. So we can get a JSON response from an API.

we can apply a jQ filter to it. Um the last part is storing it uh in a form of a solidity strct which can be written on chain. Um and um I'll speak to that in a second. So before we continue, before I run the code, let me talk you quickly through how the um Flare data connector works from the users perspective. So if I go to the Flare developer hub um and specifically to the FDC and FDC, sorry FDC.

Um I can find a diagram that uh demonstrates how the workflow for a user looks like. So first of all we have some external data that the user observes. Uh in this case that would be Star Wars API. Uh and the user then desires to bring that data onto the flare chain. Um now what the user first has to do is formulate a request in such a way that FDC can process it and fulfill it.

Um and that can be done manually by hashing a few values or hex encoding a few values. But a more convenient way is just posting the data to a verifier server that is provided uh by the flare network and the verifier server will pop out the correct AI encoded request which we can then send to the FDC. So uh the FDC is a protocol that encompasses many things. it is enshrined into the flare chain which means that it is shipped which with each version with each node so it is replicated by each validator um and I'll explain what that why that matters in a second but basically once we get the AI encoded request from the valid uh verifier server um we can post it to the FDC hub contract on the flare chain and that will start the whole machinery in the background. So what will happen in the background is that our request will be get um spread throughout the uh validators the data providers on the flare chain.

each of them will fulfill the request uh go to the URL that we have provided get the data um and then crunch it together to get a proof and only the proof will be written onto the chain. So then in the next step once the um voting round has finalized we can go to any of the data providers and request the data from them uh and then they will provide us with the data and then we can do the same thing that the validators did. crunch the numbers, get the proof and we can compare the proof that we got with the one that was written onchain and if they match it means that they gave us uh the correct data. Um so in that case we can minimize the um storage space taken by the data by only storing uh the the the proofs for it and then we can of course get the data offchain and feed it into our contract with a as a proof and the proof also contains uh well the proof that the data is valid. So um in that way we have trustworthy data on chain and we can feed it to our contract.

So let's look at the code now. Um the code is already available in the flare hardhead starter on GitHub. Um this can serve as a um base building block for your project if you decide to build on Flare. uh and I'll speak to how you can modify it in a second. Um but basically what we have here is a simple uh Star Wars character list contract.

Um and all this uh contract does it it stores um some character data on the flare chain. Oh, I should probably zoom in a bit. Um so yeah um this contract has a um as a a mapping that maps an u a UID to a Star Wars uh character strct and this is simply the name of the character the number of movies they appear in the ID uh on the URL that we provided and a body mass index which is um yeah it's a bit funny but I mean the the Star Wars URL uh sorry the Star Wars API provides us with height and mass so it's a fun thing that we can calculate for the character um and so the only function I mean the only important function in this contract is the add character function um which takes the uh web to JSON proof strct which contains both our request the data we requested and the proofs that are written on chain. Um and from that we can extract the uh character information. Uh we can also verify that the proof is valid.

um and then add the character to the list while we also calculate their uh body mass index. Um and there is also a utility function called get all characters. Um now before I run the script, there's one last thing that might be worth u paying our attention to and that's the AI signature hack function. So this is the easiest way so far that we found um of getting AI signature of uh strcts that we want to save our data as. So right at the top of the uh file I have a data transport object strct with name, height, mass, number of movies and API UID fields.

Um, and this is all the data that we want to extract from the Star Wars API website. Um, and so I define a solidity strct and then pass it as an argument to the API signature hack function. And the reason why I do it do this is so that when I run uh yarn hardhead compile I will get uh the contract artifact and within the artifact for this contract. So um let me just open that up. um within the artifact for that contract, I will also get the um the AI signature hack function as one of the functions on this contract and within its parameters or within its inputs, I can get the um AI signature of the data transport object.

And so I can copy that and provide it to the uh verifier server as one of the components of the um at the station request that we'll be making. So now let's run the script um and I'll explain what's happening as it happens. So let's just do that. So first of all we are making a request to the verifier server with um a few values that don't really need to be changed if you just take this code as base um but they are basically hex encoded strings of um for the attestation type I think it's web to JSON u padded to 32 bits and source ID is public uh uh public API I think let's look at that um public web to JSON um but that doesn't really matter what's more important is first of all the URL is the Star Wars API uh the character will be number three and you can guess which character that is um then the HTTP method will be the get method we won't be b uh giving any headers or query parameters or a body. Um but the pro postprocess jQ um will take the character's name, their height, their mass, number of fields they start in and the UID um and the AI signature is the one that I showed you within the artifacts.

And so we post that to the um verifier server. uh we get a valid response and this here is the AI encoded request and now we can post this AI encoded request to the uh FDC hub contract on the flare chain. Um we also need to calculate the voting around uh ID in which we posted the request just so we can retrieve the data later. Um and so the machinery has run in the background. It never takes I mean it cannot take more than 180 seconds though usually it's much much quicker than that but 180 seconds is the top theoretical top limit um and so we prepare a request and uh yeah the proof is generated for us.

So what we get is a response hex and this response hex can be decoded into a correct solidity strct which um also includes the um the data that we wanted encoded as the data transport object that we defined. So the AI signature that we provided is how our data will be decoded. And here are the proofs uh for that um uh for this request. And if we then decode the data, we get an error apparently. Um okay, cool.

Very very cool. I know what the problem is. Um but I won't be debugging this now. Um unfortunately what we uh what we were supposed to get is uh that the character number three is R2D2 um with a body mass index of something above 30 if I my memory serves me well which qualifies him as obese I think. Um so yeah this is very quickly how the flare data connector works.

Now let's talk real quick on how you can modify the two provided scripts so that um they can serve you as a base building block for your projects if you decide to build with them. So first of all, you would of course need to modify uh the contract that you will be deploying that you will be using. Um and how this one is supposed to work. Um I cannot tell you because that depends on your project. But the only thing that it needs to contain is an IE web to JSON proof um and a function that can accept that as a parameter.

Um and then of course um it's uh also um useful if you uh define your own data transport object so that you can get the AI signature from that. Um the real modifications need to happen to the web toJSON uh TypeScript script which well first of all you need to provide your own uh API URL with your own post-process jq filter and AI signature and you can also tweak or play around with additional parameters and there is an example on the Flare developer hub um as well as within this hard head starter on how you can um use the uh web to JSON attistation type to for example get weather data and use that to uh automatically uh resolve the uh insurance policies that depend on the weather conditions. So this is the first thing that you need to modify and then uh the second thing will be deploy and verify contract. You might decide that you actually don't need this function. Uh in that case you would just replace it with a um uh with a standard um yeah you would just return your uh your um where's the thing?

so I can point to it. Um, yeah, you would just simply Oh, uh, what you would do is just, uh, create your, uh, take your instance type at a certain address. Um, and that would be the correct thing. Okay, I see why. Now, now I see why it might have been failing.

And that's because I've been writing a um a guide on how you can modify this and I have uh pasted in the modified version instead of the correct one. Anyways, um yeah, you would um you would just do return your contract uh type at and you would provide the address. So that's the first part. If you wanted to um to only to only to deploy a new contract every time, you would of course need to replace the Star Wars character list V2 instance with uh your contract instance. And same for the other thing here.

Um and then as for interacting with the contract, you could also define a more complicated way of interacting with your contract. Um but you would basically access it in this manner. Let me now just quickly discard the changes and try running the file again. Um yeah. So um that's really quickly on how the web tojson type works.

Now, let me just point you to a few useful um to a few useful uh sites. So, the first one is of course the Flare developer hub. Um here you can find guides, written guides for uh other attestation types. Um so if you go to FDC, FDC guides and hard hat you can find guides for um EVN transaction. This is a an attistation type that allows you to get the data about um an EVN transaction on uh XRP uh and uh two additional chains that are both both flares.

Um but there are also examples of how you can incorporate the web to JSON type in a real world scenario. So one is proof of reserves which um basically collects the um state data of stable coins across different chains and compares that to the reserve amounts they claim they have. And the one that I find even more interesting is the weather insurance example, which is an insurance agency where you can create policies that um can be resolved automatically depending on weather conditions uh at certain coordinates. Um and yeah, there are additional resources here at the Flare developer hub. Um, and there's also the Flare hardhead uh starter and that is a GitHub repository where you can find all the relevant uh uh example code uh for example uh for the thing that we've been doing here today.

So that's very briefly what we do at Flare and how you can use it. Um, I'll let the code code run in the background, but I think I'll uh turn to you for questions. So, does anyone have any questions here? Well, if not, I'll just let the code run until it send and hopefully this time it will work. Um, I mean, the one on the flare hardcut starter is supposed to be working.

Um, I think I've broken it when I was uh changing things around. Um yeah, anyways um if there aren't any questions, I thank you for coming to this workshop and I wish you good luck uh hacking and hopefully you'll decide to hack uh to work uh on Flare for your project. Yeah, thank you.