# Streamr stack for P2P data transport and off-chain analytics

- Channel: [Devcon](https://streameth.org/devcon)
- Date: 2018-10-07
- Duration: 26:07
- Watch: https://streameth.org/watch/yt-40TWO0xL0BA
- YouTube: https://www.youtube.com/watch?v=40TWO0xL0BA

## Description

This talk is a technical overview of the Streamr decentralized stack which makes it possible for machines and connected devices to share and monetize real-time data. The stack is built on top of Ethereum smart contracts which are used for access control, permissioning, and secure licensing payments. Data transport is implemented by a P2P network where nodes are incentivized by payments in DATAcoin, the native token of the project. The network implements scalable, robust and low-latency pub/sub messaging infrastructure for data delivery. The layer above consists of data market where data streams can be showcased and shared between producers and consumers who transact using DATAcoin. Streamr Engine is an event processing and analytics layer capable of reacting to incoming real-time data and taking action. Finally, there is a graphical a visual programming environment where users can easily create data processing systems by connecting modules in a drag-and-drop fashion. There’s a number of interesting technical questions on how to achieve sufficient throughput and scalability and how to ensure node coordination in the P2P network.

Speaker: Henri Pihkala

## Transcript

[Music] okay hi guys my name is Henry I'm founder and CEO streamer and it's magnificent to be here I've like watched all the talks from the previous dev cons and I decided that that's what I want to do that's what I want to be and I'm here so thanks for having me I'm gonna introduce streamer to you briefly tell you what the vision is how it works I'm gonna do a live demo and spend most of my time on that if the Wi-Fi gods permit it will go well and in the end I'll talk about how streamer fits into a decentralized service stack consisting of streamer and other projects that offer services for the centralized application development so let's jump into it so streamer aims to tokenize the value in real time data for example let's consider a self-driving electric vehicle now to deliver the best possible passenger experience the car needs it constantly needs data from other machines such as traffic congestion information from other cars electricity prices on nearby charging stations weather forecasts and so on so streamer provides a single interface for the car to go out and subscribe to the data it needs and pay for it using a cryptographic token called the streamer data coin which lives on the ethereum blockchain now the car is also a data producer it can sell the data it produces such as traffic information to other cars road condition measurements to smart cities your blood sugar level to nearby advertisers or whatever so a data stream economy emerges streamers implemented as a decentralized peer-to-peer network for data delivery so data source can connect to any node in the network publish a message and it instantly gets delivered to valid subscribers so we aim for the low latency high throughput use case in a pub sub in a massively scalable pub sub pattern the streamer network is designed to horizontally scale up to potentially billions of events per second using a clever sharding scheme so sharding works by dividing the whole data flow going through the network into independent partitions and assigning asymmetric responsibilities for nodes in the network using a reputation mechanism so the streamer network is an if it's an off chain companion network which uses an underlying blockchain for security critical operations such as value transfers and access control so decentralization is achieved by allowing anyone to run a node in the network and contribute bandwidth to the network and storage in exchange for four tokens as a reward so the network is run by its users instead of giant corporations which means that you own control and monetize the data that you produce of course decentralization also makes the network more robust against attacks and node failures helping guarantee the delivery of data in all circumstances now in addition to the scalable messaging infrastructure we are also building two applications marketplace which allows the data producers and data consumers to find each other it's kind of like the App Store for data streams raw data is usually not very useful as is right some kind of computation is needed filtering aggregation combining different data streams visualization stuff like that and for this purpose we are creating the streamer engine and the streamer engine has a user interface called the editor which is based on visual programming and it looks like this so it allows developers to get their hands dirty with the data very quickly promoting quick prototyping and allowing them to seamlessly combine powerful scalable of chain analytics with smart contracts for example and we'll do a live demo about this right now so the demo I'm gonna do touches all the layers in the stack but of course the most visible thing will be the editor because the rest are just infrastructure and they look like api's you know so so you will see the editor mostly but it's good to bear in mind that the engine and the editor are just one application on top of the network so any application can connect to it and produce and consume messages so let's see what happens so let's see if we are okay so this is the editor and we're gonna start with an empty canvas obviously I'll go to the search and search for a data stream I'm gonna begin with so this stream is called public transport demo it actually contains telemetrics information from public transport vehicles running in Helsinki Finland which is where I'm from you can see on the box I'm gonna zoom up a little so there are all the fields contained in the objects in this stream so to see what the data actually looks like let me pull up a map and I'll just connect the vehicle ID over here to ID input on the map latitude and longitude fields on the map as well and I'm gonna hit start and see what happens so we instantly get a real-time view into where the vehicles are at the moment in Helsinki right okay so what does this have to do with a theory and blockchains now let's take a use case so let's pretend I am Helsinki now I'm the mayor of Helsinki sounds better okay and the king of Helsinki and and you guys you run a transport company right and I want to hire you to run the public transport in my city but I don't want to make a fixed deal I mean pay a fixed sum per year because that gives me zero guarantees on the quality or quantity of of your service so instead let's make a data-driven smart contract I'll pay you 100,000 way for each meter traversed by your vehicles sounds fair okay let's do it so what we need is a smart contract first now a cool thing here is we have something called the leti module it has an edit code button so I'll first select which account I want to use I hit edit code let me pull up a little template from my text editor it's a smart contract called pay by use it's also available on the on the platform ok I hit apply and it should get compiled I have my constructor parameters over here so let's initialize the contact with your address so I'm paying you right so I'm gonna just get some address over here ok paste that we agreed that I'll pay 100,000 way and I'll initialize the contract with some test ether and hit deploy so I can create and deploy smart contracts directly from the interface and later on I'll show how to connect data to the smart contract that I deployed so let's wait awhile for that to get mind and in the meanwhile let's add the computation so now we have raw data how do we get from that raw data to actual transactions that we can send to the smart contract something is needed in between right there's like hundred events per second coming from that stream and we definitely don't want to push a hundred transactions per second onto aetherium so we need aggregation we also need to calculate the distance that vehicles have traveled from the raw raw data so I'll show you another cool trick here I'm gonna add a sub canvas over here so we have abstractions yo dawg i heard you like boxes inside boxes inside boxes inside this box we have more boxes we have a lot of boxes okay I'm going to connect the vehicle ID and speed we're going to calculate distance from speed multiplied by time instead of difference in GPS coordinates because that's that's shady and noisy okay let's add a chart let's see what's happening we have two outputs I'm gonna zoom in again one of them says batch and one of them one of them says current I'm gonna connect current to the chart so what this box does in addition to calculating the travel distance it also aggregated it accumulates it up to a threshold after which an event comes out from the batch output so they should keep rising until my set threshold which is 1000 it's a bit too much maybe I'll have to change it to 500 and see if it works just a runtime change so now we're 300 come on come on give me some more so this when it reaches 500 it should go back to 2-0 and start from beginning and that's the moment when we can commit to the blockchain so we're aggregating and every once in a while we'll commit to the blockchain and I'll show you in just one second how how that works okay so this is what we're after right one more thing to do and then we're gonna be ready so this batch output will send out the amount before resetting so that's what we want to report to the smart contract how do we call this smart contract I'm gonna add a module called aetherium call let's make the calls from the tram demo account which contract do we call this one okay which function do we call function called update so what the function update does is basically multiplication so we report how many units we have traveled and it multiplies that by the preset price 100,000 way per meter only for you my friend so we connect batch to added units and we are basically done so let's add a couple of more things to see what's happening I'm gonna add a table I'm gonna add the batch output to the table and I'm gonna add another table so we also have the events that the smart contract might trigger there's one called paid amount I'm gonna connect that to the table so we make the transaction and we also get a feedback back into the user interface now this is ready let's start see what happens so sorry got the Mexican flu or or whatever so when we reach 500 we should have a line here indicating that something came out of the batch there it is and now there should actually be a pending transaction on the test net there it is and when it gets mind we should get one back over here wait for it these are the longest seconds of my life standing here in front of you just waiting for mining to happen okay here we go a few of those actually got got in mind so so this is it [Music] so just to reiterate what just happened a sensor or a bunch of sensors hundreds of sensors each in in a different vehicle produced data to the streamer Network the streamer network delivered the event in real-time to an application in this case the streamer engine the streamer engine called the blockchain the blockchain answered with events went back to the streamer engine the streamer engine produced user interface events which are which are also messages in a stream to back to the network and those events got delivered directly to my browser via the network in real time so application developers today they use various cloud services typically they use ec2 for virtual machines AWS they use hosted file storage hosted databases hosted real-time data pipelines hosted micro services now the hypothesis is that each one of these layers will be replaced by decentralized counterparts creating the decentralized cloud so for example Golan will steal business from from ec2 swarm and ipfs will slowly start replacing how s3 and other block storages is used so where streamer fits in the streamer network will provide decentralized data pipelines a bit like Amazon Kinesis the streamer engine will offer similar functionality to to Amazon lambda for example and blockchain of course that's new right there's nothing on the cloud side of things and what the blockchain obviously does is it offers untreatable state transitions making the decentralization of the other layers other services possible in the first place so what would this look like from the application developers point of view so let's say we an app we have an application it runs on a container which is hosted in a decentralized computing service for example Gollum now this application could be the streamer engine it could be anything else web server micro service you name it so what we want is that we have inside the container we have access point to these decentralized services which which allow the application to connect to the different decentralized our networks for example the streamer clients and it cerium node and more ipfs whatever and we're we're collaborating with with golem and researching and prototyping how to build this kind of decentralized able application development stack that allows you to write pretty much any application in any language and make that decentralized by running it in a decentralized computation container and use decentralized services for it okay so I'd like to invite you to meet and greet us over beer from five to six today at the Grand Fiesta Americana lobby bar which is just across the street would be great to meet you all if you like what you're seeing follow us at streamer ink on Twitter and join our community chat at chat streamer dot-com because we make your streams come true thank you so there's plenty of time for for questions yep first of all really cool UI and you know because there are incentives for providing data how do you prevent someone providing a lot of fraudulent data yeah okay so of course we need to prevent flooding the network with for example random right so there will be some kind of usage cost for the network for both the producer and the consumer so if you're a data producer it's possible that there's also a staking mechanism so you may you need to make some kind of initial investment or either in terms of usage fees or staking expressing your belief that your data is valuable to someone if that hypothesis is wrong you basically lose money so it's a bit like entrepreneurship in general you have an idea you think there's value in starting a cafe over there and then you make an initial investment and see what happens [Music] yes so basically if there's a fixed cost for providing this amount of data on to the network per day you need a certain amount of customers to break even right so so you should achieve aim to achieve that and that this incentivizes posting or publishing data that doesn't have an audience at all or if you use data for for private purpose purposes say you're an organization and you use use the network for internal data it's not even intended to be sold on the marketplace you use it for internal analytics or whatever then you use you pay the usage fee for for the network and those usage fees go to the nodes so it's kind of like mining but we're not solving this artificial CPU or GPU problem instead we are providing bandwidth to the network so are there some mechanisms for cleaning the measurement streams of malicious measurements with some assumptions that at least part of the measurements are to be trusted yeah so I mentioned a reputation system for for the nose which helps distribute the responsibility there but there needs to be another reputation system as well which is at play on the data marketplace kind of like you have reviews for for apps in the App Store you're gonna have to have a reputation system for streams on the marketplace now if someone is producing malicious content or infringing copyright for example so basically this is an unsolved problem so if you could buy a stream from there and then publish it for cheaper or for free and this kind of activity needs to be this incentivized but of course it's hard to exercise censorship in a censors censorship free Network right so so if you post a movie on YouTube they can take it down but if you do that on the network it's it's impossible so we can have like community community voting for unwanted content going on the producers might need to stake stake value in there as well or we can simply retain a certain amount of centralization on the marketplace so that it can be curated by by a group of of curators and actually then we can have a little bit of censorship going on in there okay so I may have missed it but where exactly is the data stored and you know in the network so what we have currently is a centralized solution built on on cloud technology and the decentralization of the system will take place from now on and we haven't yet solved all the details there so we're early on on that the basic idea is that the data is persisted on the nodes we might also use like decentralized time series database or no SQL database if one that suits our needs pops up but as this is very evolving space at the moment so it might be that we need to actually build them as well so it will be stored at the nodes and we need some kind of indexing going on there so compared to for example static files if you have a static data set or or a video file for example you can a little bit easier handle that but what we need to do is we need to have a search functionality into the data in order to retrieve specific messages or ranges of messages from there so we'll have that implemented as part of the network cool great thanks guys for your attention [Applause] [Music]