# Building an Ethereum assisted Decentralized Data Marketplace Breakout

- Channel: [Devcon](https://streameth.org/devcon)
- Date: 2018-10-07
- Duration: 19:47
- Watch: https://streameth.org/watch/yt-T47CneAXJVg
- YouTube: https://www.youtube.com/watch?v=T47CneAXJVg

## Description

Visit the https://archive.devcon.org/ to gain access to the entire library of Devcon talks with the ease of filtering, playlists, personalized suggestions, decentralized access on IPFS and more.
https://archive.devcon.org/archive/watch/3/building-an-ethereum-assisted-decentralized-data-marketplace-breakout

Data has become a valuable digital currency in its own right over the past decade, but despite its utility, it remains consolidated at the hands of a few large organizations. We propose a new protocol layer for creating an open-access data marketplace. The protocol is decentralized, and allows anyone to publish data in the system in return for incentives. We will discuss the hybrid architecture of the protocol, combining Ethereum smart-contracts for managing data access and subscription, alongside an off-chain network for storing and transmitting the data. Finally, we will introduce Catalyst – the first application to make use of a decentralized marketplace of crypto-data.

Speaker(s): Guy Zyskind
Skill level: Advanced
Track: Developer Infrastructure


Follow us: https://twitter.com/efdevcon, https://twitter.com/ethereum
Learn more about devcon: https://www.devcon.org/
Learn more about ethereum: https://ethereum.org/ 

Devcon is the Ethereum conference for developers, researchers, thinkers, and makers. 
Devcon 3 was held in Cancún, Mexico on Nov 1 - 4, 2017
Devcon is organized and presented by the Ethereum Foundation, with the support of our sponsors. To find out more, please visit https://ethereum.foundation/

## Transcript

hi everyone good morning thanks for coming at this early hour hopefully no one's too hungover so I'm gonna talk about enigma and specifically our decentralized data marketplace protocol all right hey everyone again good morning I'm gonna talk to you about enigma enigma' is building a decentralized data marketplace that is part on chain on ethereum in part off chain so the background to enigma starts at MIT in 2015 we launched a enigma project from there where we basically conceptualize at the centralized computation platform that can guarantee both privacy and correctness and with better scale than just blocks and themself allow how many of you here have like seen that white paper like our familiar with the enigma project from that sense all right not not bad not bad so cool yeah it's been sighted quite a lot I think it's one of the more sighted white papers in the space download it quite frequently so I guess there's a lot of people who share our vision and yeah anyone can help me with the slide I should go closer nice coupe so some apart of motivation so our motivations you know data is everywhere it's the most valuable asset right now in the 21st century but right now the situation is that only a handful of companies can really capitalize on that you know they're hard you need the monetize you need but that's not something that the open public can take part in and that is what we are said to protocol I'm gonna give you a quick outline of my talk I'm gonna first of all speak about you know what is the data marketplace at a high level how does the arm chain works with the of chain for the stakeholders and how we're thinking about designing this I will at that point I will not focus on you know how the network stores data how computation is done that would be the second part of the talk when we like go deeply into that again given time constraints this is going to be pretty high level and finally I'm going to give you a taste of catalyst calories is an application that we're developing that is going to be the first one to live on the data marketplace all right so at the high level you know the enigma data marketplace it has three main stakeholders which are pretty pretty obvious and trivial you have data providers those who want to sell data data consumers those who want to consume data and on the off chain you have worker nodes which are basically you know just nodes who don't have any stake in the data buying or selling but there are the ones who are actually storing the data doing computation answering queries and so forth so my focus right now is going to be mostly on the on the kind of the contracts between the data providers and data consumers first so starting with the data provider a data provider comes has the data said that they want to sell what they do is first of all they provide enough context to the oft a network they basically need to register the data set now that could be in one of two ways either they come they upload the data to the network and I will show later on how that's done securely or they can actually store it in some some of their own nodes and just provide enough context to the network onward that daily lives the network then computes some kind of address like a permanent address kind of like a DNS that the network can then route consumers to that data set the data provider then takes that address that unique address some meats transaction to the blockchain also puts like the price of the data in some other metadata and also stores a deposit which I'll touch a bit later on now that the day that that data set is registered on chain there's a reference to it off chain a data consumer can actually come in and send a message to the blockchain saying hey I want to subscribe to this feed of data and obviously they need to provide payment and they also need to provide a deposit a deposit is there in case the consumer defaults in their payments in the future so one note about the incentive structure here really what enigma is trying to do is is it's trying to kind of tokenize or capture in a token the value of data we're trying to make something that's really implicit today and kind of I guess stuck behind walls we're trying to make that more open and very very very explicit so there are two ways in which the incentives mechanics kind of work the first one is the value of data directly that's an explicit metric so a data provider basically sets a price to what the data it's worse and then the market either buys into that or no that's like the explicit value of data and they have to share their earnings with the nodes the worker nodes in the network that are doing computation that are doing the storage and that is really where the implicit value of data is also reflected another note about discovery in ranking so we expect to have you know eventually many thousands of data sets in our system of all kinds you know we can employ some statistical methods we can tag data cells but in the end of the day we need to find some mechanism to rank datasets that are kind like in the same bucket and the best way we see to do that is by directly correlating the economic incentive that is locked in a data set with its rank what I mean by that is that simply it's the amount of money that people have bought into that data set plus some factorization of the deposit that the data provider has put in so basically the idea with the deposit here is that data providers need to kind of put their money where their mouth is that sets the rank of a data set and kind of use to break ties now this is a good way to transition to like the option computation and storage again would be pretty high level but I want to go through some of the main ideas this is really where the heavy lifting happens off chain and we'll talk about that next so the main idea is and that's the main idea that we also set in the 2015 white paper is as follows global consensus is expensive I think all of us here are aware of that it's not scalable it's very very pricey we need to figure out a way to basically segment the network and you know and four different computations for different storage different data set data blocks of restoring we need to make sure that only some of the network is utilized and we need to do it in a way that's secure enough again it does it's not as you know won't be as perfect security is like global consensus but good enough for you know decentralized needs and scalable enough there are a couple of good ways to do that you know first of all let me let me let me touch on the on the stake on the stake holders here so you probably know this is shardene that's kind of what has been popularized in the space I'm gonna use quorums from these more of an academic term saying that you you have a large network you select like a committee and that committee is really in charge of like some portion of the data some portion of the state some portion of the computation so what we're doing basically you know for every data block every data segmentation every data said that's coming into the network the option network needs to go into a randomized protocol where they kind of like flip a coin and use that to randomly select a subset of the network likely small it's big enough that it's secure but small enough that it's scalable and to select that forum there are several techniques most of you them use a threshold cryptography or secret cheering which is very similar you can also use random beacons on the blockchain which are very fast but they have some trade-offs in terms of the entropy that you can get but we kind of like this black box you can actually for each data you can select only a handful of nodes that would be in charge of that and that's where you kind of get like your scalability properties sorry but that raises the questions once we have a quorum for one one data set right and that core needs to make we need to make sure that that quorum always runs the computation correctly over that data set and we also need to make sure that that chrome itself cannot actually see the underlying data if that data is sensitive these are the three key questions that we have and I'm going to talk about that now so this is not an exhaustive list but this is meant to give you kind of like a quick overview of different techniques in the literature for handling outsource correct computation with privacy so let me go very quickly over that first we have you know blockchain blockchain is you know fully decentralized gives us a very strong integrity and consists in consistency but it's really bad at keeping secrets like there's zero privacy in it and it's very and it's not scalable at all then we have something I like to call partial encryption not a scientific term but that's like an umbrella for like order preserving encryption deterministic encryption kind of like encryption methods that are not perfect but give you some confirm the confidentiality for the data but zero and and then we really get to the interesting stuff of like the really heavy heavy machine guns of cryptography we have fully homomorphic encryption which is would be the best if we ever figure out a way to do it fast enough we right now we we're not sure how to do it there's snarks which are great if you want to prove a statement and then verify that many many times but for other use cases it is unfortunately limited and you have multi-party computation which we feel is a good trade-off multi-party computation gives you you can basically compute anything with privacy with correctness under some assumptions it is more scalable than the other cryptographic solutions because it really only uses symmetric cryptography but I do want to emphasize that it is still much slower they like hardware based solutions and of course computing over unencrypted data and this is why actually we're to make mobile focusing it's a bit of a refocusing on both of these technologies MPC and secure hardware so secure hardware is stuff like Intel SGX and trusted execution environments we feel that MPC is is great for like applications like identity where you want to have zero trust but if you okay with having some minimal trust in the vendor which is what secure Hardware requires you then you can really get full functionality great scalability and privacy and with that I'm gonna yeah I think I have enough time I want to walk if you kind of like the basics of MPC right MPC is an amazing technology it's something we've been working on for a while and that also would help to illustrate kind of like the strengths and weaknesses of it so MPC says something like this imagine there's an ideal world where we have like this godlike computer that we can trust and we can outsource every computation to it now that trusted machine would never leave the data it no-one can breach it and we can always trust it to run the computation correctly but in the real world that's not possible so what MPC comes in let's simulate that trusted machine let's simulate that with the network kind of like the argument that blockchain is making by the way let's simulate that with the network and the statement says and this is a simplified statement that you know as long as there's at this one honest node this one no that is not a bad actor you can actually make sure that the computation is correct every computation and that no node in the network no one in the network can actually see the data the raw data and that basically means that data remains encrypted and to end let me walk you through a simple example let's imagine we have a network of three nodes a very small network and let's say we have you know the data provider on one on one hand and then a data consumer I'm ignoring the blocking here I'm just talking about the process of the off chain storage in computation so let's say a data provider wants to store some number X what it does locally it basically goes through a protocol called secret sharing that protocol essentially splits that data into shares encrypted shares and sends one share to each node okay so node 1 has x 1 no 2 X 2 and no 3 X 3 these shares are completely encrypted no one and the others cannot see anything about the row X then we do the same thing with Y and then actually the data provider can go offline data lives in the network now the consumer comes in wants to run some computation the state of the network is that they collectively hold X and y but it's encrypted as shares in each node now the consumer comes in and wants to compute just X plus y so because of the properties of secret sharing actually that's that comes for free we can just each node can locally compute their local shares which give them the encrypted summation and then they can just send all all of the shares back to the consumer and the one thing about secret charity is that if you have all the share in one place you can reconstruct you can decrypt the data but the important part to realize is that there was a computation going on in this network and we can extend that network as much as we want but none of the nodes in the network has have actually seen the raw data they have just operated on fully encrypted data which is pretty amazing so let's run another example with multiplication I'm gonna go first over that it's like more complicated I just want to give you the main ideas so company I'm sorry multiplication requires communication between the nodes they can just compute it locally but we know that if you get all the shares in one place then you can decrypt the data so we need we need some trick we need to avoid that so what happens in MPC and that happens here but that really happens in every like if you're writing a VM and you implement in any kind of protocol you're actually going to use this the same trick over and over and the trick is simple whenever you need to share some information with other nodes you're going to re-encrypt it using some like a one-time pad and then you're going to send the information so that's what the nodes are doing here then they're communicating that information with each other and then there is some more local computation that if you kind of plug the algebra in you would actually see that the Z's here end up being exactly the encrypted shares of the product of X Y and then you can send that back to the consumer and the consumer again having all the shares can pull them back in and decrypt the data and the network so nothing which is and that's the point now why did I show you addition and multiplication other than them being like relatively simple well there's a nice theorem saying in computation theory saying that you know if you have addition and multiplication you can build any theory you can compute any circuit essentially you can compute anything that you want so these are the building blocks that we need there are I mean a lot more like more complicated protocols how do you compare two numbers you know how do you do like floating points and fixed point operations many of these obviously I'm not gonna get to hear that was a lot of my teases actually doing in working about improving these protocols we got like between 10 to 100 X speed-up on quite a few foundational protocols but unfortunately I won't have time to discuss that here so that's it on the hard stuff I want to finish with like something simpler and kind of nice we feel that in order to bootstrap a data marketplace we need to also introduce some use cases right we need to set the demand side we need to boots up it with some interesting data and given the space we're in given what we're interested in given the state of everything that's related to like blockchain and cryptocurrencies data we felt that building I'm sorry data-driven investment platform for cryptocurrencies where all the data that that comes in and all the data that is like stored and is going to be kept on the enigma data marketplace and that kind of sets in boots up the network and also allows us to stress this the protocol as we are building this out now catalyst is still centralized because the protocol is not yet live but catalyst is live and operational if you're interested in algo trading or just like using crypto data for doing some research I really welcome you to try it out it's on a website this is one cool example so someone from our community has actually used catalyst to build a model of Markovic portfolio optimizations on crypto assets which i think is fascinating where a few other examples like that in our blog and I welcome all of you to really look into that and with that I'd like to finish if you're interested in you know off chain computation the future of data data marketplace is anything that I discuss right now please come talk to us I'm here John is John show your hand John is also here John is my co-founder please come talk to us and if you're excited about this and like you really want to be at the forefront of solving like really some of the most interesting and hardest problems we're like we're hiring so come talk to us about that as well we'd love to hear from you thank you [Applause] [Music]