# Private and Reliable Data Sync for Messaging Over Whisper by Dean Eigenmann & Oskar Thoren (Devcon5)

- Channel: [Devcon](https://streameth.org/devcon)
- Date: 2020-10-09
- Duration: 19:00
- Topics: blockchain, smart contracts, distributed ledger, ethereum
- Watch: https://streameth.org/watch/yt-zxN-PqYkZ1M
- YouTube: https://www.youtube.com/watch?v=zxN-PqYkZ1M

## Description

Visit the https://archive.devcon.org/ to gain access to the entire library of Devcon talks with the ease of filtering, playlists, personalized suggestions, decentralized access on IPFS and more.
https://archive.devcon.org/archive/watch/5/private-and-reliable-data-sync-for-messaging-over-whisper

How do we achieve user friendly data sync in a p2p network for resource restricted devices? In this talk we go over the problem space, do a brief survey of existing work, and propose a solution combining several of the most promising technologies. This talk will include a demo of a data sync protocol being developed. We’ll also cover things such as:
- dealing with mostly-offline devices,
- dealing with large sync contexts, such as group chats
- making the protocol transport-agnostic.
Finally, we’ll show how data sync fits into the larger picture, from transport layer to end user applications, and highlight some of the work that still needs to be done, and who is working on it.

Speaker(s): Dean Eigenmann, Oskar Thoren
Skill level: Intermediate
Track: Privacy
Keywords: whisper, messaging, technical

Follow us: https://twitter.com/efdevcon, https://twitter.com/ethereum
Learn more about devcon: https://www.devcon.org/
Learn more about ethereum: https://ethereum.org/ 

Devcon is the Ethereum conference for developers, researchers, thinkers, and makers. 
Devcon 5 was held in Osaka, Japan on Oct 8 - 11, 2019.
Devcon is organized and presented by the Ethereum Foundation, with the support of our sponsors. To find out more, please visit https://ethereum.foundation/

## Transcript

hi everyone I'm Oscar welcome political research at status I'm Dina also work on protocol research at status today we're gonna talk to you about vac PDP so vac is family of protocols for developing secure private and sensitive resistant p2p solutions what that means in practice for us the status right now is essentially secure messaging when people talk about secure messaging usually focus on sort of enter an encryption forward secrecy avoiding Manimal attacks and so on but as other security properties such as privacy and protecting sort of metadata as well as sensitive persistence and all together these sort of properties help you protect against various types of threat models that we see around the world today as well as for certain high-risk individuals so one way of looking at the problem of sort of secure messaging is in terms of a messaging stack we're sort of each layer is roughly orthogonal to the other layers so starting at the very top you have sort of end-user semantics let's you're it's some private sort of encrypted group chat or maybe some public forum with some moderation mechanism then below that you have sort of data consistency data Sinclair which is sourced at sort of all the nodes have sort of right idea for the state of some context or conversation is and below that you have sort of your secure transport which gives you probably is a confidentiality forward secrecy and so on and right now we use double register for that status but it's others like mrs lacy security below that you sort of have the metadata protection layer where right now we use whispered sellers but so solder so just like pcs and tor mix nets that kind of different but but they they share some common design goals and they have various properties against very types of fret walls and below that you have sort of the overlay network overlay routing for natural and so abstracting awaited actual transports like TCP UDP and so on and this layer sort of helps a lot when it comes to sense of resistance because this is where use of God against various types of traffic filtering and so on and it's a kind of somewhat the focal piece that you also have trust establishment which is sort of how you establish end-to-end trust with its various models like trust first use and web of trust like PGP and key based and so on talking briefly about some design goals we have it vac so we were trying to sort of create these these interfaces at each layer where that allows for existing in new protocols at each layer of the stack we don't want to reinvent the wheel here because there's a lot of protocols and components that are being developed for people at this conference as well as sort of in adjacent ecosystems and communities that also solve all the problems it's more about sort of define these interfaces and then allowing these very several protocols to work together we very much sort of take the design philosophy from products like lead PTP and substrate in instead of enabling that and having kind of a protocol selection that depends on your specific design goals so I'd say you're extremely sort of privacy conscious then you might want to use the mix net such as one developed by NIMH but if you have different of constraints maybe use of care a lot about bandwidth or or or latency or sense of residence then you might want to have a different choice at that specific layer another design goal is in terms of better specifications and for status we started off with this or ad hoc protocol defining the app and now we're sort of moving to the more other extreme where we serve look at a protocol as its as an animal in its own right and sort of trying to make it maximum a useful for force of other Pods as well and it's also include sort of making more rigorous analysis of the exact Kent he's in behaviors of these protocols another sign goes in terms of sort of pro in the tools to do sort of simulation testing in beaufort scale and for adverse environments and this is for things like white lock and Jepson simulation framework and so on to make sure that the design actually makes sense and will scale to the to sort of have the properties that we want at scale so it doesn't sort of full under which is what we were seeing with with some of the stuff we had in production with whisper for example another big one is in terms of mostly offline so when it comes to key messaging and so on often users are sort of mobile phones and usually this is sort of an afterthought and this leads to people accessing the network through gateways and so on and we want to sort of think about this from the get-go because what makes mobile phones bit special is that for example I guess you can't have back from execution is very limited on Android it's raining a battery so you got to think about these things and how he actually sort of natively connect to a network and and make sure things work properly with recent stick to devices and finally sort of a lot of people use things like what's happened and telegram and so on and we do it internally it steps as well and and like people aren't gonna use use not only on this that you actually enable Nixon to use experience so that's something you have to so if you have to make sure it is support to user stories for the end users therefore people actually use this kind of stuff with that theme we'll talk a bit about our date Sinclair yeah so what we started off with that status is a very naive whisper transport and whisper comes with what various issues in the way that messages are sent they expire at a certain point so we need to go to save these messages on a mail server so that with mostly offline devices a message actually reaches its destination user so to solve for that what we did is we built a Data Sync protocol on top of whisper to already add more reliability guarantees to the lower whisper transport layer is this a general problem in peer-to-peer systems when you have unreliable transports where you need to add some form of synchronization protocol on top of it to ensure that we have these availability guarantees which are usually only achievable through centralized servers so how does this look it's a very tcp like inspired protocol that we now split up the way we send a message into multiple steps what we initially do is we offer a message to a specific user so Alice would offer a payload to Bob and then once Bob has received that message of the offer he will request it from Alice Alice then sends that message and Bob then returns an ACK saying that I received this message with the each of these steps what happens is things are retransmitted so if Bob is offline when alice is sending an offer Alice will exponentially try and offer that message again and again up until it receives a request from bulb and that continues for every stage so we we know that this entire process has happened and that a message is transported in the end successfully because we've retransmitted it up until the next step that's how the interactive mode works that's like a very bandwidth high version of the protocol what we also have is the batch data sync mode so here we don't have any offers we don't have any requests what we do here is we simply repeatedly send the message payload up until we have received X this is sorry this is the bandwidth high one the other one just has a higher latency up until you've received the actual message because there's multiple steps in between this still has certain problems because it requires us to keep forwarding these messages repeatedly so one way we can solve for that is we can add a remote log on top of the protocol which Oscar will talk about yep so so the idea with the remote log is very straightforward you centrally have a local log where you sort of have you messages on a local machine and then you replicate it on some decent-size file storage and that acts as kind of highly available caching layer so what you have is you have exactly two two new components you have a cache and then name system the cast is contestable storage and essentially you you you upload some content and then you address it by sort of its content hash mutable store and the name system that's something like DNS or ENS or swamp feeds or IP NS in the IP FS world and it essentially gives you some location addressing or mutable references so when Alice so they upload something to the cast and they get back to address for it and then the update stuff they they the name system and then when when Bob comes online when it notes comes online and they know about sort of this this kind of conversation or do this sync scope between Alice and Bob they have a fixed point they know that it can look at the name system even with Alice being offline so they check the name system and they see what message they've been missing and then it can go to the to the cast and and get those without Alice being online in terms of data format it's it's very straightforward it's essentially a mapping and it's just mapping from sort of the native methods identifier which is sort of what's determined by the upper layers to this sort of address at there at the Cass and then there's all sort of an address to the next page so you essentially have these pages with logs because if you have a lot of sort of misses you you want to split it apart and that's an alternative you can also embed this sort of actual wire payload encrypted content in the name system as well and this is just kind of a trade off in terms of what the name system specific name system supports so it could be a very small one in which case you can't actually embedded content and then the trade office if you want to sort of traverse the linked list and to go to the cast multiple times and so on this is sort of a very general design so it works it works swarm it works with I profess and also works sort of if you want to have it on a USB Drive to the backups or any others of similar thing that implements these interfaces in terms of privacy properties there's this very much depends on the underlying casts so if you have for example a mixed in it and you do posts or gets through the mix net then you actually get send and receive anonymity that way you can also do other things like you can ratchet the name system so you change the location of it kind of like how you do ratcheting in so forward secrecy schemes and so on Dean will talk a bit about how we achieve certain cuz this is e Kennedy's with the metadata format yeah so a problem of all this messaging stuff is that messages are usually related and so we need some way to provide some form of consistency to these messages so we need to be able to provide ordering amongst various messages not only a linear ordering through my messages but we want some kind of dag if I'm in a conversation where messages are linked to each other to show what is the what message depends on what message so here we have an example of this dag where what we want is we want linear lists of messages saying I sent this message before this is the next message and I keep linking the previous message that I sent but what we then also start doing is we start linking messages that other people have sent me so I start linking the last seen message that I've received into the next message that I'm gonna send and additionally what we can start linking in is we can start linking in things like remote log information so that when we go to this log we know exactly what position a message is placed that in this remote log so that if I've missed let's say 10 messages before my last message I know exactly which page to go to and from which message number to which message number to sync and the way we do this is relatively easy right now we just add a protocol buffer essentially to the message as a header where we have a repeated byte array of parents which is just the parent IDs of messages so that would be like messages that have been in this group context before my message this can be messages I've received or messages I've sent then the sequence number that is the one which relates to the actual number we're seeing on the remote log and then we have something like previous message which is my last sent message and additionally we also add a boolean to indicate whether a message is required to be act or not and this is important for things like user typing notification or that a user came online those are things that we don't need to really be consistent about because if that message gets dropped I don't really care I'm not going to retransmit that those are like one-time notifications that I'm just gonna send out and if if it's received that's good if it's not received I don't really care because it's not important to the actual in a chat context conversation that is occurring cool so in terms of problems and rough priorities as next steps so our initial focus being on data sync there's still some things we have to improve it specifically in terms of scalability in semantics for sort of large in context so many imagine if you're a group with a faster participants you want to reduce sort of the chatter and sort of have efficient joining of logs so you don't overwhelmed network another big three priorities in terms of a better transport so currently we use whisper and it sort of has a lot of issues when it comes to scalability and it's using proof-of-work for spam resistance what is not great for heterogeneous devices or nodes because your phone can easily be overwhelmed by someone speeding up an EWS node and it's also not incentivized there's no reason to to actually run their whisper node and it sort of doesn't map cleanly to the counting of the resources that you're actually sort of consuming and so on so that's sort of in in looking at various alternatives and it's very early in recent stage right now and related to that is also that this sort of concept of adaptive nodes right now you have generally speaking so if this this thing where you have full nodes that are participating fully in network and then you have light notes or even sort of gateways that are kind of like leeches and and you know don't actually contribute to the network and we think this is a bit backwards because if you look at sort of more successful like peer to peer a bit torrent as an example as one of the most successful sort of peer to peer deployments you actually always everyone's contributing to the network and health of a network that's a very nice property and not everyone will all nodes will be able to contribute equally but you sort of have the option to do that and it's something that swarm is also working on this sort of idea and like you can imagine that if you have a mobile phone with limited data plan you probably aren't gonna contribute a network so you might pay for through some other means but let's say that you you have a you come home and then you sort of plug in you use Wi-Fi and you plug it in for charging then you can sort of start to use that for relaying a messages or if you have an old laptop laying around then you can use that sort of store messages and help the network that way this idea for creating continuum between sort of light notes and full notes where you sort of have some capabilities and you try to contribute that to the network sort of make the network stronger we think this does some sort of something we'll look into him we would love it if more people also so thinking in these terms because it sort of creates for a better network in general and it sort of also more in line with how we generate sort of see successful societies and systems where all nodes contribute to the extent that they can so yeah that's it for us are there any questions yeah and you can check out these websites where we have some research logs you can read more detail about data sink remote log and the general stack we also have specifications and yeah feel free to reach has a Twitter yes a drawer there is a for invidious that they're solely spec and it's also in there its status version one app remote remote log disrespect improve concept it's still not deployed in ends of the app and then the other other serve products are still serving in various research data's oh and the metadata format is in spec and proof of concept States but not yet in in app there is some not going to promise anything because that's also it's a free system interesting ideas when it comes to using senior knowledge proofs and so on where you can sort of limit it that way but it's it's too early to tell if those things are actually will actually work out so we still need to yeah spend more time on that essentially I would also point that that that's a specific thing for multi cost networks as well where you serve have this implication factor so for other types of networks that that might not be as big of a problem but yeah any other questions all right cool thanks [Applause] you