POT and Databases in Swarm

I unfortunately I'm in a bit brittle state throughout this conference so I didn't really have chance to prepare the slides and so the context of this this session is that we as the as the swarm team as the project scope grow we kind of under - undertaken the task of providing some base layer services for web 3 what does this mean we want to provide some services which which cater for web developers to migrate their projects to from from the web to stack - to web 3 this includes node to node messaging as you as you might have seen Louis or Brooks talked on PSS and also includes some payment solutions like infrastructure to to run service networks using the source window paradigm and thirdly and very importantly some sort of database support now so now basically the the the swoon so the swarm team kind of launches some different working groups around different topics one one today today's session kind of nicely Maps to do to these working groups we have one working group around PSS messaging one around streaming integration with with live peer one around the swaps where and swindle games paper and fourthly that is the database group currently the most active participant in this is the company would walk where we are very privileged to have them in the audience right now so they dedicate a lot of manpower to this to this project and she's just launching so I kind of want to want to briefly touch upon you know some aspects that we that we trying to target with this project it's as you might know swarm underlying lis is just a chunk store so the underlying network of storage just takes care of of distributing chunks of size to particular nodes in the network and retrieves those and all all the higher level API functionalities like it's built on that so for example when you when you retrieve an asset in your browser when you retrieve a file there's a there's a mechanism which takes the takes the root hash of the swamp trees and retrieves intermediate chunks from this it recovers the the next level of of intermediate chunks and finally it resolves the files into into the actual data chunks which when retrieved they sequenced sequenced one after the other end and give you the give you the asset and that you can render in the browser now an even higher level yes there's a thing called manifest which is basically the the tool that enables URL based addressing of content over small it it's based on having having a so the root hash of website is you retrieve a manifest and the manifest basically contains an index from URL path to two two hashes which corresponds to a to a routing table in the in the you know original you know what you normally see on servers when you when you retrieve a domain then it's it's basically routes certain paths that you see in the you era and you resolve into ashes but in the same way this index can be thought of as a as an index of a database because it's its bit simply just so see it's a particular key that you can it's the same as in the URL string to an asset word to a database record so that's this in this sense there's already one very simple way to represent indexes or databases in in swarm using manifest what what's this what the scheme is missing really is a is a smooth way to to prove that for example an index generated from a from original database is is is correct so it does not contain spurious entries and it it does not contain omissions so so one one aspect that we would like to work on is to have this provable provable databases and prove queries so so so basically this this is the this is a crucial feature that that will allow database services to be fully decentralized because if you if you have if you have the database processes for example indexing is provable then you don't need any kind of reputation system or you know any other mechanism to to select your service providers and it allows for for a trustless so there are there are a few tricks which we can catch or dis problem so one one solution might be simply to to take SQLite and SQLite has has a file structure which which you can simply store on swarm it has it has a configurable page size so you can configure it to correspond to thee to the small chunk size which is 4k and the trick would be to have the whole SQLite implementation simply compiled to wasm and through solidity and then you can use true--but to to simply prove that that that query resolution is correct so that'sthat's that's one possible Avenue the other one is to use the the latest interpretation of the spam hash which use this binary mark trees for hash with the by the fact that that we use binary Mercure trees over over the chunks for for the chunk hash is gives us the opportunity to have very compact inclusion proofs in to a very high resolution 32 bytes so so any kind of inclusion in in a chunk can can be can be proven and the short proof can be verified and on the blockchain so the so we kind of we're trying to come up with with a clever serialization scheme and and together with the canonical series a serialization and the inclusion proofs you can directly map for example statements like you know you have you have an original data source which let's let's say get an example so for example you you are you operate a flight aggregator site and and you your customers are all your data providers like travel agents which publish their flight offers and they want to make sure that if they publish write offer then their their offer will be included in the in the current search results for from on the flight aggregator site so what you do is you register for example your your your primary data source where you publish the offers and and the the indexing service that that you know aggregates these data sources and and produces a common aggregate index can can be tested against statements like this particular offer from like Cancun to Mexico City this price by this airline is is actually included in the in the in the aggregate index and the in if we find the clever canonical serialization scheme for for the database layout and and use inclusion proofs then basically this statement can be translated into into into into into an inclusion proof that you can submit to to the blockchain and a smart contract can can evaluate that that that particular statement is true and since since that is a mechanistic process that you know you can you can simply answers yes or no it can be used in a in a in a witness contract in a swaps parents window game so when the then when the database index registers their service to index particular primary data source then they they commit to a service contract which then you can you can challenge that service context if you find that that the data item was not included in the in the index so I don't know if this is more or less clear so that's that's that's one aspect that we what we're trying to integrate and this is this is kind of very important because all the all the other solutions that we see in the in the space port fault for this data gateways including big chain and and and also IP ID the ipfs the solution there they are very very nice in I'll give you some features that that are very important in those paradigms but they they they seem to have this like missing piece where you can actually have you know the of the database operations and improve ability of query query responses so so IP ID for example is it's a very evolved you know database structure which we definitely rely on so we have as is basically an extension of that what what what what's included in that it's basically a JSON like attribute value structure scheme which uses linked linked so so the embodied structures in the in the in the Jason's can be referred to with with the content address of the of the of the embedded object which means that it's it's it's genuinely provides integrity protection over your data and and the important thing is with this with this content address links is that if you if you if you map these these data structures in turn in memory structure then then these these links are typically pointers and using content content addresses these pointers can be directly mapped to two references in a distributed storage so so given given given this database layout it immediately defines how to how to store such huge graph databases in this central storage system so and and this this gives this this leads me to the next important point which is also going to be a relatively novel feature in our in our framework which is that since since the since the pointers are you know references to them in to the content analysis in in the distributed storage all the other database operation or the data structure operations like like merge or delete or add they can this similar similar way to to have the defined on on pointers as like recursive recursive functions [Music] structures they simply map to to to network protocols so so when when you when you when you define a merge function in memory database and you use pointers to pointers like children to to define the next level in their in the recursion the same way you can you can define protocols over the over the swarm network to carry out those operations and this is very very interesting approach because it allows for basically outsourcing to the network and an update to the database so you don't have to pre-compute yourself addition huge data graph but you just just simply send a message to the to the to the address that corresponds to the route - of the database with with with with with the request that please add this extra item to the database and what it does it simply looks up in his children where where the data has to be added and simply looks up content address that points to that that intermediate node and and forward the message to them so it's it's like basically instead of instead of recursing on in memory structure its recursing is a sequence of mass relayed messages so oh so like to basically as you can see this this topic kind of links all the other components we were talking about today together because this this this database operations can be database operations can be described as spss protocols basically and in the same way as as PSS messaging is incentivized these beta based services are also you know when I send a message to update it's basically a service request that we talked about in substance window game context the desire kind of promissory notes basically that that become mature and and become reward double when when when the update happens and you can you can formulate proper you know verification criteria with with with the witness contract when when to verify that particular update happened and in the same way as the services if you if you fail to comply with with with with SB the service promise that you that you index data source then then you can be challenged on the blockchain and you you you can you you basically stand to lose your deposit as a result so so these are these are the things that I wanted to mention and sorry I'm a bit running out of steam now so I hope you don't mind that I will just you