# Swarm Development Update - Channel: [Devcon](https://streameth.org/devcon) - Date: 2018-10-09 - Duration: 34:35 - Topics: Science & Technology - Watch: https://streameth.org/watch/yt-kT7BgOH49Sk - YouTube: https://www.youtube.com/watch?v=kT7BgOH49Sk ## Description Visit the https://archive.devcon.org/ to gain access to the entire library of Devcon talks with the ease of filtering, playlists, personalized suggestions, decentralized access on IPFS and more. https://archive.devcon.org/archive/watch/3/swarm-development-update An Update on the Swarm project. Current status, upcoming changes, roadmap; introducing project PSS, the FUSE driver, the Network Testing Framework, updates on encryption and encoding. Previews for the p2p breakout session on Saturday. Speaker(s): Viktor Trón, Daniel Nagy, Aron Fischer, Louis Holbrook Skill level: Intermediate Track: Developer Infrastructure Follow us: https://twitter.com/efdevcon, https://twitter.com/ethereum Learn more about devcon: https://www.devcon.org/ Learn more about ethereum: https://ethereum.org/ Devcon is the Ethereum conference for developers, researchers, thinkers, and makers. Devcon 3 was held in Cancún, Mexico on Nov 1 - 4, 2017 Devcon is organized and presented by the Ethereum Foundation, with the support of our sponsors. To find out more, please visit https://ethereum.foundation/ ## Transcript [Music] yeah hello everyone I'm very happy to start us off on the update from the swarm team so today we're gonna talk a little bit about well what swarm is just make sure we're all in the same page what the status of the swarm project is and also where it's going we've got some new sub projects it's going in different directions and the general scope of the swarm has grown and so we're going to talk about where we're going but I'm just gonna start you off with what is swarm to make sure we're all you know understand each other you probably know swarm does a network of computers share data in a decentralized way to give us a peer-to-peer decentralized storage with all the benefits that come from from being a peer-to-peer network such as resilience against network outages DDoS attacks and the rest but what you may not know it's really the same connections as etherium so same with Nazareth because right now swarm client is still separate but the vision is that it's part of the same package of protocols that would run over there p2p so your swarm know your eath node could talk the eath protocol or the swarm protocol all of the same connections so it's really tightly integrated with aetherium so let me give you the basic first use case the sort of baby example of what swarm is for the decentralized storage of app data so the blockchain the iam blockchain is great to store critical information about your DAP maybe user name registrations or something like that but your dad still needs images HTML Javascript and all the rest and right now you probably need to host them on a server and swarm is there to allow us to have this in a completely decentralized way so here's how that would work you'd go to your swarm enabled browser and type bzz which is a swarm protocol the name of your DAP eath then the theorem named service would turn that name dot eath into a hash so the ENS record has an address field where you can store your wallet address and that's what address would get paid if somebody sends payment to that name but the standard ENS resolver also has a field called content and in that you can store a hash that's referencing content on swarm for example so you use this hash of content to query your swarm peers for the data and in return they will supply the data to you and anyone who supplies you with data that you want you compensate them using a theorem based micropayments and then the last step is to just use that data to render your DAP so the hash gives you integrity protection you don't care where the data comes from you know it's the right data and gives us a server less completely decentralized way to host Bond apps so that's great so let me also tell you what's happening under the hood on swarm when you have a file or any content swarm is agnostic to what the content is it's gonna take anything you throw at it and break it into little chunks but for kilobyte each and the hashes of those chunks are collected into a Merkle tree everything is murk alized and the root hash of that Merkle tree is the ID of this file or this content in swarm and the chunks they're all being sent to different swarm nodes for storage what's really happening is that the hash is the ID of the chunk and every chunk gets stored with whatever node has an address which is closest to that chunk hash so really spread out the data and your DAP data is uniformly stored throughout the swarm network the nodes share services with each other it could be data chunk retrieval and serving data or other services but so the accounting how much you have to pay how much you get paid is done by this so called swap so here's how that might work suppose node 2 is asking for a DAP and node 1 happens to have the chunks in question and gets to serve them so you know like that it's my own animation I'm proud of that and if there's more than one chunk both nodes keep count so there might be two four six after a while if node one is not consuming any data node two is doing all the consumption no two will have to pay node one so it would use our checkbook payment contract to send a check to node one at that point the channel we call it balanced you know whether it's chunk four chunk or check for chunk doesn't matter and when it's balanced we can go back and continue consuming services so what's happening is in within this form a swap channel in the middle where zero balance neither node owes any node anything and as soon as services get consumed the balance moves off towards one side at some point you're gonna hit the payment threshold and at that point the node that's in debt would have to issue another check to the node to the debtor so that to bring the balance back at zero so you know what's always working to keep the swap within that middle range and if you slide too far off to the side well then you get disconnected that's the punishment the ultimate punishment in a peer-to-peer network nobody wants to talk to you anymore this in payment infrastructure that swap is a part of is very modular it can go in many different directions it was originally designed for data delivery content delivery and storage long-term storage as described in our orange paper but it's it's become much more modular and we're gonna extend it and we're talking about this on Saturday in the talk called swaps we're in swindle games where we describe just how how this can learn well how I can accommodate all kinds of different services beyond just storage the status well some of you may have tried swarm already if you have tried swarm you've been playing around with our release POC 0.2 and that really is a proof-of-concept release it's not a final release so the the proof-of-concept showed us that the basic idea works and we've had some fun in a hosting content and trying it out and playing with it but the performance has been really slow we've had data availability problems throughout the network data that hasn't reached where it's supposed to go but so as a proof-of-concept it was successful and the testament has taught us a lot and as a result several of the core components are being rewritten or have been rewritten completely yes and additionally the scope of the project has grown as I've mentioned before people are using swarm not just to host content but for example the live peer group are using it for broadcast streaming video and audio really cool they're also talking on Saturday another project that's developed is about communication direct node to node communication and to say tell us more about that I'd like to introduce Luis to talk about PSS [Applause] hello everyone I'm Lois this is my first Devcon actually I'm a classical pianist turned cultural magazine editor turned funding bureaucrat turned band manager turned taxi driver and I was privileged enough to get a ride with some blockchain people in Berlin where I worked in my taxi and then told me all about that centralized consensus war on privacy and the looming dangers of opaque algorithmically driven societal conformity and this brought me back to the world of computers that's I've been in all my life in varying degrees now I work for Jack part of the Jack team Jack is a company first of all I very clumsily forgot to include the logo of Jack here but that's the logo they're a sponsor we are a sponsor so Jack built son swarm and we want to contribute to swarm as much as possible of course and therefore I was very excited when Viktor Tron came to me and and challenged me to do the PSS work and that's what I've been working on the last seven months now notice that among the things I said I was I didn't mention public speaker so that's why I have this not to panic I will be looking at more at this than it's you don't take it personally please now swarm is a network of nodes and not only that but swarm is network of nodes that speak the same language this language is communicated on top of Internet protocols and is regulated by a protocol framework and go a theorem this is called dev p2p swarm nodes uses this language to find their most advantageous place in the network and to figure out who to share content with but content doesn't necessarily just have to mean files a message for example is just as much worthy of the term thus comes the idea if we can use swarm to send bits pieces of file surround why can't we use it to send messages as well now wait a minute you might say there is already a messaging platform in aetherium and that's called whisper and this is true but the mission of whisper is somewhat different from PSS its focus is to serve the need of total anonymity and to secure freedom of expression at cost of performance now since the swarm network is routed it knows that the shortest way to the final destination is and therefore PSS can deliver messages fast and with a minimum of network load simply it serves a different purpose so this is what PSS is it's a postal service over swarm now this sense sounds easy right if you want to send a message to someone you just slap an address on it and away it goes well it's not quite that easy since messages potentially pass through unknown nodes to get where they're supposed to be they should be encrypted so that people can't snoop on the contents that means that the actual definition of a recipient in PSS is not only the actual address of the node but who can actually decrypt the message actually the node address in the end doesn't unambiguously define a recipient yes to route the message service swarm using PSS you pass it to them you pass it the message you want to send and an address to send to but let's say you don't want to tell everybody on the network who you're actually messaging with now this is easy just disclose a part of the address swarm routing will still route the message the best it can from the knowledge it has so it will land in some neighborhood of nodes closer to the recipient than you and ask before whoever in that neighborhood can decrypt the message history recipient or recipients doesn't necessarily have to be only one now whatever we don't give address at all this is analogous to what whisperer does and the consequence is that all messages gets passed to all nodes by all nodes this has nothing to do with routing at all obviously but it might be in handy in some stages of the communication like when you send a broadcast at probe to enable a pair to send you its address but encrypted or to get some temporary encrypted keys that can be discarded later now to make life even harder for screw toe files PSS also employs a redundant routing and this means that when a message enters into the neighbourhood of the destination the node that receives the message for forwarding will pass on the messages to more than one peer this happens even if the node can successfully decrypt the message this one anyone trying to trace a message path through the network will have difficulty finding out where the message actually stopped that is to say the node not passing the message anymore are not guaranteed to be the one the message was actually for in fact most likely it won't be so all good and well you got the message you can encrypt it what do you do with it well for this PSS has the notion of topics all messages that are sent belong to a specific part topic which could be any four byte value but intended to be the first four bytes of the hash of an arbitrary value PSS provides a registration method method to attach code to these topics therefore when a node gets a message with a certain topic on it it knows exactly what to do with it the topic structure is the same topic structure as whisper uses Forsyth's envelopes envelopes here meaning the entity that encapsulates contents in the sshh protocol in fact PSS actually cheekily steals this whisper envelope to wrap messages in reality all PSS does is just to slap an address on top of it or not the whisper envelope also provides an expiry parameter this is used by PSS to stop messages in from circulating in the network perpetually and finally PSS uses whisper also has a back-end for encryption and handling of keys as whisper psst cryptography as well as a symmetric arbitrary symmetric encryption so any 32-byte value is actually valid as a symmetric key and PSS and can be set at the developers discretion so to sum up to send a message in a to a PSS note all you need is a key the topic and an address or not so what exactly right now this PSS implement the encryption has said luminosity control which is a fancy word for deciding how many address bytes who disclose we have a generic handshake module which gives you a simple diffie-hellman exchange of keys in case you're too lazy to make your own and the keys generated by the handshake module are valid only for a certain number of messages that is to say they're ephemeral handshakes can be optionally also left out of the compilation of go etherion there is an API of course available over IPC and WebSockets for key handling message passing handshakes and common supporting functions that are needed a simple flood protection all nodes that get the message store the message on swarm and use the hash of the message or the chunk to determine if they've seen this message just before and can optionally have not forward it and finally you can actually use PSS with the dev p2p protocol structure that exists already the logic so this means you can reuse Stephie TP code that's already there this can be either integrated into the code of the node itself or the results to a module that enables you to use to do this in a separate process using WebSockets as the transport layer now I probably overstayed my time I'm not sure but anyway there is a more thorough walkthrough of PSS on Saturday in the breakout session quarter past two in this go through a bit more through a bit more of the internals what makes PSS tick the API will present that and there will also be a demonstration of a chat app that's actually using PSS right now I mean it's not a production ready thing as a demonstration prototype so and this is not only demonstration from the screen but actually the audience will be you guys will be able to install the app and chat with us live air photos well that is to say yes so if you can't wait till then there are some links on top is the branch where the active PSS code is now PSS will be a part of POC 3 for swarm so this means you won't find it in the ethereal master repo now it'll be later on there is a description of the API and also there are some code examples for both the generator generate p2p programming in Goa theorem along with a bunch of PSS examples on the bottom link there and I think that was it yes Donnie what a good scene hello everyone I'm going to talk about the first swarm based application that is aimed at a wide audience basically the general public so this is a cloud service that allows people to share files between one another and also to sync files between different devices so this is a fairly this is expected to be a fairly popular use case and what we hope to get out of it is that we are going to see how swarm actually performs at scale so we are really hoping to get extensive analytics and user feedback out of it and also eventually it will allow us the swarm team but also the etherium project in general to sort of use whatever infrastructure we have built in order to host our documentation our source code our web pages and so on in a decentralized and censorship resistant fashion and in particular I was always baffled by the fact that little store waltz has given us a wonderful distributed source control system get and then everybody started using github which is a centralized service it's a great service very happy with it but but it's centralized and there's already work going on for a module for get on top of swarm Alex barracks RC is working on it so hopefully in the not too distant future we're going to be able to host all the sources of aetherium on swarm and have the development repository basically be stored in a distributed fashion the first thing that we did you know for this application was a GUI based file manager it was released last September it's over a year ago it is itself a swarm hosted web-based distributed application and it allows people to manage folders and files in a very familiar and intuitive fashion it also has pluggable pre viewers for various types of content so you can view images movies and later perhaps PDF and whatever people can think about so that's the first thing that we have built then Rahul Mohammed has in this summer released a fuse module which means that you can actually mount a swarm volume as a directory into your file system if you're using a POSIX compatible file system so it works on linux and mac know in the support yet it already has read and write support you can already mount Els registered volumes and there's one thing missing which is sort of a safe - ENS kind of functionality where you can update the Els resolver with the new content after you have modified the content of the swarm volume that you have mounted so this is also available for download and you can you're also very welcome to play with it this is a big one so currently all swarm content is unencrypted and in the open and this of course makes it completely unsuited for any serious use but and this is what I am currently working on privacy and security encryption so here you need to understand that this fundamentally differs from a traditional cloud hosted server client application because obviously you cannot force people like not to share certain information so access control needs to be implemented in a different way so essentially right control so who is allowed to modify something is equivalent to the permission to update the root hash in ENS so it is something that is controlled by a smart contract a smart contract is what takes care of write permissions read permissions are even trickier because you know where swarm content ends up it cannot be controlled it just goes all over in the network you cannot hide swarm content you can only encrypt it so what you can do is you can encrypt swarm content and read permission means the ability to decrypt it and if you want to have access control lists so you want a large piece of data to be readable for some parties but not others then what you can do is you can take the symmetric key with which this data is encrypted and encrypt that symmetric key with a number of public keys of the parties that have reaper mission and use that as a access control list and this obviously affects how content can be referenced so in unencrypted Swarm the standard reference to content is the swarm hash of the manifest in encrypted swarm in addition to the root hash you also need the decryption key of the route how this encryption works so as Aaron has already told before me swarm chunks of content into small pieces and organizes them into a miracle tree and in swarm every chunk is going to be encrypted with a unique key and each edge in the encrypted Merkle tree contains not only the hash but also the decryption key for that particular chunk which means that if you reveal a key at some point in this tree so one of the Merkle trees nodes key gets compromised it means that only the subtree under it is compromised which is kind of a stopgap measure for catastrophic failures against losing keys and also it allows different directories to be shared among different parties as a sub directory of different directory structures and so I would like to invite Aaron back to talk about the network testing framework that we have built yeah hello again I'm filling in for Victor our team lead who is here but not able to speak right now so I want to talk a little bit about the network testing well the full title is the network simulation and testing framework and I have to admit at this point I'm not the right person to talk you through this diagram in detail but that means I'm the right person to give you an easy-to-understand high-level overview of what's going on so I'm gonna start from the right and talk my way over to the left in the top right what you see is our network visualizer it is it's a web-based or a browser-based view of a swarm network so all those dots in there would be swarm nodes and connections between swarm nodes this interface allows us to query any swarm node what's its current status stored chunks every connection the swap connection was to swap balance what are the payments and so on so it gives us a very great overview of a swarm network of a test network to see what's going on and it also allows us to input changes we can click on a node and tell it to go offline tell it to initiate a data retrieval whatever we want that would take us into events so these events we can fill in to network so simulating activity on the network telling nodes to drop offline come online do various things and these events get packaged into whole scenarios so we can write a network scenario of nodes coming online sharing content downloading content paying each other refusing to page other whatever we want and we can observe how the network behaves like what did what the network does what the emergent behavior is in the backend which is further towards left we can either simulate the swarm nodes in process very quickly and or we could launch a bunch of Dockers with various so we could host a cluster of swarm nodes and and actually run them in full or we can simulate them so these scenarios are portable so separating out what we want to happen on the network and the level of simulation whether we just want to simulate the process or actually you know there's also also one to rather like a back-end to our cluster where we can have our whole cluster of machines running a whole bunch of swarm nodes doing running through these scenarios and the reason why we're so excited about this or why we're developing this is to test emergent behavior of the network so normally when we talk about testing a client you're running code tests to see whether a swarm node behaves the way you expected to but a lot of the intricacies the subtleties of swarm arise from the interaction of the different nodes in the network the emergent behavior like how data is distributed one node requests data the other ones forward the request the data has found handed back all the nodes have to do the accounting and if you want to see how this network behaves we really have to simulate a full network and test the network so this testing framework allows us to do protocol testing you write a scenario what you want to happen one node uploads content and then what you expect to happen you expect the data to be handed from node to node stored here you know and all that kind of stuff so it's more than just testing an individual node is testing that all the parts work together and this framework is not is about you know it allows you to test PSS storage retrieval the entire you know the whole package and that's all I'm gonna say about this today will be give more about all of this on Saturday so upcoming stuff what's on the roadmap so one of the things that's changing is the hashing algorithm earlier I said when you have a chunk of data you hash it and that is its ID we're changing that to actually make the ID of a chunk be the merkel root of a Merkel tree that we make out of the chunks out of the 32 byte segments and this allows far more fine-grained resolution proof of inclusion and and proofs of custody this is where that comes in then the syncing protocol has been rewritten the syncing protocol is the protocol that governs how swarm nodes share data how the chunks make it to their destination and that's the one that was causing us the most headaches in terms of performance on the test network and why if you were playing around with that you got so many four or four errors where files weren't found because the data wasn't being passed all in the way it was supposed to so that's been rewritten and it will be merged soon indeed the entire network layer and the connection management has also been rewritten and is ready to be merged and these changes are breaking so that hashes need to change the ENS entries would need to change and in fact we're gonna have a complete reset of the swarm test net soon once these things are merged and then hopefully we'll have a formal performance test network to play with from then on yeah so ongoing work encryption erasure coating plausible deniability that is Donny's he talked about that that's what he's working on PSS under active development and yeah the contracts for storage insurance we didn't talk much about today but they were in our original orange papers and they are being developed and tested no they haven't been deployed yet all we have so far is contract for for retrieval and like the bandwidth incentives but the storage incentive part from the orange papers is being developed to and then the testing framework is undergoing active development and then also on the roadmap I want to highlight a few things a light swarm node we've been contacted by people who doing mobile development saying we cannot assume that every swarm node will want to store a bunch of data and serve it and earn revenue some people just want to join the network and only consume very limited data that they want you know if you have limited bandwidth so we talked about various light modes of operation nodes without storage or nodes without a lot of bandwidth so we've been developing the theoretical framework how those four nodes should fit into the network and we intend to develop those to some various light modes of operation to make it easier for mobile science and other light lines the payment infrastructure as I alluded to earlier is being developed to be far more generic and to allow for four different services than data storage and retrieval one of them being you know communications broadcasts and other things service networks but they will talk about on Saturday and Victor will also be talking about distributed database services and how they fit into both the swarm network itself and the incentive layer so that's that the update from us our team is Victor Donny Fabia who and Anton from the foundation and from Jack we've had a not just one but two Lewis's giving us great a lot of time and code so big thanks to them and this forum team is still growing we're very active development and adding people yeah so there's even more coming and so I want to make special advertisement to our breakout session on Saturday the session on p2p technologies blockchains in data and communication there'll be updates from the swarm team will talk about swaps where and swindle will talk about distributed databases there'll be a PSS update there'll be a interactive chat demo and updates from other teams live pier will be talking about streaming services of a swarms a lot of good stuff so we hope to see there and with that I'm going to end a little bit early but thank you for your attention [Music]