Reducing Beacon Chain Bandwidth for Institutional and Home Stakers | Devcon Bogotá

foreign [Music] we work on the lighthouse client so we we're going to be talking about reducing Beacon chain bandwidth for institutional and home stakers and I'm going to leave it up to Diva who's a Bogota local and our peer-to-peer Network expert to kick us off okay so first thing that we need to check in order to analyze how to reduce bandwidth for institutional and homesteakers is how does the what does the bandwidth look like right now we are going to be looking at data from Lighthouse like age said we work on Lighthouse is one of the production Beacon notes there are many others techu prism Nimbus low star we are the second most popular one and while the data we're going to be looking at is lighthouse we are certain that this applies to any other client across the network so have that in mind first thing to notice is that bandwidth Associated to a beacon node is proportional to the number of validators is how it has attached um you can see that it increases when the number of validators goes up but then at 64 validators it stays more or less the same 64 is going to be a really important number in this talk H is going to explain further why and why it's important so yeah we need to remember simply that we are going to stop at 64 and this is where bandwidth is going to be more or less stable now if we look at how many validators does every Beacon node on the network has we're going to have like main three groups we're going to have those that don't have any validator attached only simple Beacon notes uh notice taking big notes now we're going to have those that we are going to come homesteakers which have less than 10 validators in this graph you're going to see a very big bar that's the one with see your evaluators I hope you can see it those are a very big chunk of the network almost 60 percent of the network then we have those that we call homesteakers most of them have just one validator and then we have those that we call institutional stakers those are the ones that have more than 64 validators which is the big board at the end now why do we care about bandwidth to begin with so there are a couple of downsides of high bandwidth the first one is of course an increased cost if this cost comes from running for example in the cloud where your chart charged for uh data data transfer and it's also a matter of diversity because this means that you're only able to run a full node in devices that support processing all the data that you're receiving from the network and for people that are able to pay for the services in which these nodes have been running um I see a few devs here more my colleges for example um we know that there is going to be an upgrade to the network that is going to bring a lot of big big increase in bandwidth like a lot so we need to start tackling this problem right now before this happens that's another big reason why this is important um also another really really important reason is when we say for example A diversity problem the truth is if you are for example a homesteaker and you're running your beacon note with your single body data attached and then your brother or your sister starts playing video games then you're going to have a bandwidth Spike and you might lose at the stations you might miss other stations blocks so basically you're losing money simply because very very short the spikes in bandwidth so yeah we don't want that now let's check where is the bandwidth coming from for the sake of this talk we are going to separate protocols between discovery and non-discovery so why is that because Discovery has its own transport its own encryption it's over UDP whereas all the leave P2P we're going to call them lip P2P protocols uh they're over TCP we have gossips of and the request response the one we are going to be focusing mainly in the stock is Gossip so now that we have this separation between let's say Discovery and not Discovery let's check where is the bandwidth coming from this is what the bandwidth total bandwidth looks like for one of our nodes for the last couple of days is it Discovery is it live PDP where is it so now we have two graphs the purple one is going to be Discovery as all the discovery bandwidth and the other one in the bottom is dollar v2p bandwidth now forget about the scales this is just like a non node very sciency analysis of the shape of the shape of the bandwidth so I guess we can more or less agree that bandwidth has the shape of the leaf P2P when whereas Discovery is well it's very magical as we will see so the total bandwidth is around 250 kilobytes per second is it better like this yeah okay about 250 kilobytes per sec per second then we have lip PDP which is around 200 kilobytes and then discovery which is between 40 and 60 kilobytes per second so for the sake of doing analysis over pan with Discovery is really not important so we're going to focus only with P2P so I was saying we have gossips up and request response we're going to focus on Gossip stuff so in Gossip so gossip soap is a published subscribe system protocol we publish messages to the network those get disseminated to other peers and we receive messages to subnets where are two topics to which we are subscribed we have large amounts of data and those will split between subnets again here we have 64 attestation subnets which again age is going to be talking further in a moment and then we need to think how many subnets are we subscribed to since this is since this is the reason why we get so many so much bandwidth and the reason is that right now is because now they're subscribing to one subnet for each validator that it has attached and yeah well subscribing to a subnet means a lot more of bandwidth so a very simple example of how gossip stuff works gossip stuff has a lot of parameters but one that we care a lot is the match degree so this example is the three three match degree so we are going to start with our guy in the middle that one I hope you are seeing the red circle just right in the middle so this guy has a validator attached it's going to publish a block so it's going to beg to pick three Piers it's going to disseminate the message and then each period is going to do the same with some other three peers and so on and so on so if you think about this for a moment you're going to realize that is very likely that I know this is going to receive one single one unique message a lot of times from very different uh notes across very different paths in the network from the source to themselves Okay so why do we like gossip stuff or why do we like the way we do this uh this message propagation so it makes for a very robust Network in the sense that we are sure that if we publish a message and we have this kind of topology the message is going to reach all other nodes in the network in a timely manner we also know that we have low message latency does mean that means the time that hap the is spent between the creation of the message in the source node to when you receive it is not very long and this is as I was saying because of uh lower paths smaller paths lower Hoops but then again we have a lot of duplicates and these redundancy results in higher boundaries this is a graph that we have for nodes so this is like real data for the gossips of duplicates now this is going to be weird but I'm going to ask you to focus on this number the beacon block so we know we need to publish one block each slot so we know for this topic we are supposed to receive a single unique message however we are getting six or seven messages across the network that means an application an amplification factor of about seven so imagine what happens if we were able to reduce that kind of duplicates that was like a huge huge gain there okay so a summary of the current state so bandwidth is proportional to the number of validators I know it has H is going to speak about how we intend to tackle this and the other one is the high amplification in Gossip so how are you going to do to reduce duplicates without harming the network so I'm going to leave Ash to continue this talk foreign yeah thanks Diva so I'm gonna try and have a crack at explaining how we can potentially reduce some of these so I'm Australian and I can see a few Australians in the audience I think they'll agree with me that we have terrible internet like ridiculous internet I run a validator at home and someone that watches Netflix in another room causes me to lose attestations so I as a disclaimer I have a personal Vendetta to try and actually reduce this bandwidth so that I stop losing out of stations yeah so there's probably two solutions we have to these kinds of problems which I just want to kind of briefly touch on and give you a feel for what we're trying to work on one of them is called minimizing topic subscriptions let me try and explain that if you haven't already covered some of this kind of stuff before so a validator in an Epoch needs to do some kind needs to publish the messages on these things called subnets in order to do that you need to have peers to be able to publish those messages on those subnets now the problem with this is that peer Discovery is kind of is a slow process it takes a while to actually find some peers and it's even harder to find a peer on a specific subnet that you need to publish an attestation so the problem we've got essentially is that we need a stable set of subnet a stable set of pairs that exist on each subnet that we can find them easily when we need to publish a message so it's not really an uh an easy problem to solve the the way that we currently have this solution I guess is that every time a validator is attached or sends a message to your beacon node that Baker note is required to subscribe to one subnet for a very long period of time it's about 27 hours or 256 epochs so what that means is the more validated you have attached to your Beaker node the more subnets you need to subscribe to and subscribing to a subnet means that you have to get all of the messages on that subnet you need to verify those messages and then you need to send them on to other peers so you're doing this there's a lot of bandwidth and a lot of processing so essentially you're supporting the network at the cost of bandwidth on your node so you're kind of being like a good actor so there's a number of downsides to this approach one is that it's not enforceable so what that means is you as a beacon node you can essentially lie to us and you say I don't have any validators even though you've got like 2 000 attached to you and you just don't subscribe to any long-lived subnet so you don't consume any bandwidth so you kind of incentivized to do this and there's no way that any other node on the network can tell whether you're lying to us or not the next thing is potentially our subnets are oversubscribed we actually have quite an uh a large number of Beaker nodes on mainnet today and when we originally designed this kind of process we we didn't really realize how many nodes would be participating in the network so potentially we have more nodes than we need on each of these subnets so that would lead us to think that we have excess bandwidth on the network that potentially we can remove so the idea that's being proposed is why don't just every single Beaker node on the network subscribe to one or many subnets the there's some benefits to this but the journal discussion is in is in this um specs repo as an issue so what would the bandwidth graph look like if we did this if one Beaker node were subscribed to one subnet rather than having it for being proportional to the number of validators attached to you you would then get this green line on this graph which hopefully everyone can see and that green line sits at around 500 kilobytes a second at the moment from what we've what we've measured so essentially everybody on the network that has a validator wins in this in this scenario except for those that have one so sorry for the people that only have one validator in this room you guys then have to your bandwidth will increase by about 500 to 100 kilobytes the other benefit that we get from this so from a quick scan that we did of the of the DHT or or the current nodes on the network we found that 57 percent of nodes apparently don't have any validators attached to them so they're either lying to us or they actually don't have validators attached to them and they're not they're not participating in any of the subnets so if we do Transition over into this state they actually have to start participating in helping out and it lowers the bandwidth for the institutional stakers or the ones that have high number of validators yeah so essentially all the beaker nodes will now contribute to to doing um to to helping with this subnet stability so it kind of sounds pretty good that we get a massive reduction in bandwidth but there is a cost and the cost is how many what is the density of of nodes that you can find that exist on these subnets so if no one was subscribe to there we wouldn't be able to find those pairs in order to publish publish our messages on these subnets so we still need a decent density so that if you just randomly look through the PSA you can actually find nodes on those subnets and kind of hold on to them and so this graph here is a distribution of the current density and what the density would look like if we switch to having one and one big node per one topic so at the moment it's it looks roughly like eight percent so you randomly pick a node there's an eight percent chance that it's going to exist on any given subnet if we switch to one Beaker node per one subnet it drops to about one and a half percent which is one on sixty four as you would expect um the benefit of that is that it's configurable we don't have to say one bigger node one subnet we can say one Beacon no two subnets one bigger no three so now so we can adjust that that kind of differential and whether that's feasible is dependent on the number of nodes on the network so that's something we need to measure but the benefits that we get from this is essentially for the institutional stakers which represent roughly like 10 of all the validating nodes on the network their bandwidth will drop by about over 90 which is kind of Handy and as I mentioned before everyone that has more than one validator will have a reduced bandwidth so fundamentally we're talking about do we need all of these nodes on the network to support these subnets at the cost of the huge amount of bandwidth they're currently using and so potentially not and it's customizable the second thing is it's also enforceable because we would tag a beaker node's node ID to a subnet and so when we connect to a weaker node and it's not subscribed to that subnet that we know that it's being naughty or relying to it so we can kick it off so we actually have the enforceability property yeah so that's one solution and that's kind of a low hanging fruit it's kind of easy to do we get substantial gains the other thing that Diva was talking about is message amplification in gossips app so if there's anything we can do about that the idea that we want to try and push forward is a concept called appisub so gossip sub is a protocol that exists in lib P2P as Diva was mentioning and the lip p2b guys which is kind of run by protocol Labs have had a Evolution stage of Gossip sub they've talked about Epi sub for quite a period of time they've done a bunch of research in particular viso or Dimitri who works for protocol Labs has has had this kind of vision for episod for a while but has never had the push or the drive to do it but now that we're seeing quite substantial amounts of bandwidth it's it's probably time we try and realize this thing so I just want to briefly go over the concept and what it's what it's going to do and how it could help us with bandwidth okay so app is up you what the biggest problem that we have is that if you have a high mesh degree as Diva was saying that that means you have a high connectivity of nodes in your network so every node could be connected to another 10 or eight other nodes and then you get huge message amplification so every time you send one message most of the network will probably have to download it eight times so if you increase that message by one megabyte you're actually increasing it by eight times eight megabytes because that's the amount of boundaries that has to be downloaded and then propagated so natively you would first think okay why don't we just lower the mesh degree I did try to do that at one point but as everyone points out that's not very it's not a safe thing to do we don't really know how the network is going to behave we can try it on test Nets but the topology or the structure of a test net doesn't look like mainnet if we just lower the mesh degree to two for for example a mainnet you might just not restart receiving blocks so we have to do that with either great care or with a lot of testing so it's it's an interesting idea but we could probably do better so another idea is to dynamically adjust adjust the connectivity we also run into a kind of a similar problem and typically a lot of the people that use our client they they suggest lowering the P account which is um it's a fault it doesn't quite help you so that's also not the best solution so the idea with appisub is to try and minimize duplication and latency by making the mesh it keeps the same connectivity but making it a little bit more efficient and by efficient I mean we're trying to remove the number of duplicates and at the same time either maintain or lower the latency inside the mesh and the mesh is just a subset of your peers that you're connected to that's where you receive your messages from so that kind of sounds like we're trying to win on two fronts we get less bandwidth and higher latency which kind of doesn't sound like we should be able to do it but let me explain the the general principle and and maybe it makes sense so the general principle is you're a node on the network you're receiving all of these messages a lot of them are duplicates you want to just start collecting Statistics over which nodes uh are you sending you these duplicates and which ones are sending you late duplicates and you'll end up having some distribution of um the the number of duplicates you get and the latency so it could be that Paul over here in the front constantly sends me some late duplicate three seconds late whereas Sean's sending it to me straight away instantly so I come up with essentially a choking strategy is what we call it where we look at all of the distribution of people sending us duplicates and the ones that are late or the ones that have lower latency and we send a message called a choke so in this example we say I'm going to choke Paul because he is sending me late messages all the time what the choke message does is it indicates to Paul to no longer send me these messages over the mesh instead he does a process called gossip where he just sends me like the message ID which is a much smaller thing so over time if eventually based on your choking strategy you should have a more efficient Mass where you're only receiving the messages with a lower number of duplicates maybe from just one or two peers where the rest of them are choked and you still receive gossip from them so if the two peers that you have in your mesh are slow and the other ones that you have choked are started sending you message IDs before you send them in the mesh we have an unchoked message so we can uncheck them and put them back into the mesh so ultimately the idea would be is that you're you're dynamically changing uh your mesh and how many what peers what peers are sending you messages and how fast they're doing it in order to try and make it essentially more efficient hopefully that makes sense I take questions afterwards does this work I guess is the question there's been some preliminary simulations on this uh as I mentioned viso from lip P2P and protocol Labs has done a work done some work on this he's built a generic simulation uh from for the Go version of this with 250 500 and a thousand nodes and so this is this is with a mesh degree of six so there's roughly about a six times amplification if you look at the messages in these simulations but pretty much in all of the simulations what this graph is showing you is that you get roughly a 50 reduction in the duplicates from every sub now as I said there's a choking and a non-choking strategy which are left somewhat generic and I think we can tune this to be significant especially if we target it for the ethereum network I think we can get better versions of this but at face value it looks like you can reduce the GPS by 50 just by adding these kinds of messages um the next thing we probably should talk about is latency uh I kind of suggested that we could get a reduction in duplicates and reduction in latency uh the initial results that uh visor has uh completed in a simulation seems that the latency increases so on the left is uh Gossip sub latency distribution where the buckets are like milliseconds since you received the message and on the right is uh episode so you we receive more messages with higher latency in these simulations but as I was saying uh these are somewhat generic the the topology of these does not look like the ethereum network that we have and these can be highly tuned to what we need there's a lot of like research work going in there there's a few people in the audience where I think that I've promised uh results for a mainnet version of this so uh asset Sigma Prime we've built uh essentially a production version of episode inside gossip sub the advantages are that it's backwards compatible so we can release it in in our clients it'll work existingly with every other peer and every other gossip sub um node on the network but if there happens to be another episode node on the network we can start getting this bandwidth minimization so the fact that it's backwards compatible is super handy we can kind of just release it whenever we want we're also working on a mainnet simulation so that we can actually simulate the bandwidth that exists on mainnet and then apply Fe sub and then get essentially more specific data that's more robust and how we can actually apply this to what it would look like on an ethereum two main net these will publish results from this very soon and if you're interested just let us know so the title of the talk was reducing if I can remember the title of the talk was reducing bandwidth for institutional and homesteakers so if you're an Institutional or a homesteaker and you came here being like how do I get all these bandwidth gains what do you need to do in order to get a 90 reduction the answer is nothing you just have to wait we'll release it maybe run a lighthouse node and hopefully yeah we'll we'll be able to reduce the bandwidth but that's the end of our talk thank you [Applause] some minutes for questions um great talk guys is this something that's bad was compatible meaning that clients can gradually roll this out as time goes or is it something that we all have to upgrade at once here we're handling basically protocol versions so we can run against nodes that are episodes compatible and if they're not then we're just going to run episode version one that one so yeah that's what we mean with that yeah we add a protocol ID into the gossips app and so when you connect to appear you can identify which protocols they support if they don't support episode you don't choke or uncheck them if they do support it you can choke and uncheck so it works perfectly with 1.1 notes yeah I was trying to ask why do we want to reduce bandwidth for institutional stakers like isn't it kind of a nice property that there is no economies of scale at least if the um if it's enforceable that they have to subscribe to things um it just seems like I don't know it's weird to me as a concept to shift uh the load from institutional stakers to like normal Beacon nodes or like um I don't know homesteakers even if it's like a little bit but just it generally seems like we want those the lack of economies of scale good part of this is the part that edge mentioned about being enforceable so having 60 almost 60 percent of the network being knows that don't have any validators we're not so sure that's true those might as well be institutional stickers so the truth is that this is more like about being fair across the network so it's more about that than targeting something uh that is going to be better for institutional stakers it's just that they happen to get more gains because of this so I know with four eight four four we're going to be exploding our bandwidth costs I was curious if you guys knew if that scales with the number of stakers you have or number of validators you have so I just the question is that we're going to increase the the block size in 4i44 and how that applies with here if that scales the way that under current gossip our bandwidth um increases with the number of validators you have I was curious if that was the case for the increase in 4844 as well yeah yeah it's not so the the bandwidth increase is due to the subnet subscription um when you increase the block size everybody feels that because everyone subscribed to the Vlog topic yeah so that's that's felt uniformly here the part that matters for what Mark asked is mainly the amplification factor and not that much um how it behaves regarding the number of validators so since we have an amplification factor of about six so now with this uh this Improvement we're going to have huge block sizes which if we continue doing things the way we are are going to be sent across the network six times each time each block so that's that's insane but there is yeah so it's related more about duplicates that part than the number of validators for each node so you showed that by modal distribution where there's a lot of one validator nodes and then there's that sixty percent that were just regular full nodes like not validators and then you know uh the institutional notes with like more than 64. so that uh I understand how the Epi sub like helps everyone and it like helps the homesteaker it helps the Netflix problem for the institutional stakers their all running like data center nodes anyway so I I don't understand as much how the like subscribing to fewer gossip sub topics uh helps so I think I think this is a very similar question uh over here so I think fundamentally we were looking at as as a whole the the network may be consuming more bandwidth than we need to be you depending on how how you build your client You Can Be Clever about which peers that you connect to um when you have so at the moment you have these institutional notes all right they've got that they're not just institutional uh some people there's a usually parameter I think in most clients which is called subscribe or subnets so you can even if you have one validity subscribe to all the subnets and the reason you do that is because you get some benefit from seeing all the attestations you get a slight increase in performance so it's not necessarily just institutional people um but the the institutional I forgot what for my train of thought um the no I forgot what I was saying and I just have a point it's part of what age was saying is that well subscribing to a subnet we also advertise that we are inside the subnet inside Discovery so that means that we need to find peers using discovery that are useful for us so one strategy that people who have the bandwidth for that used to be sure that they have publishing all in time and everything is timely is subscribing to all subnets regardless of how many validators they have this is what age was saying about maybe it's not exactly true that all nodes that have more than 64 subscribed softness actually have more than 64 validators attached to them yeah sorry I remember my point my point was that we have these institutional Stakes or people that are subscribed to all the subnets and they become more valuable than every other node on the network if from a client perspective if you have peers that are connecting to you and one is subscribe to just one subnet and you have another pair that subscribe to all 64. you're more inclined to be like I want to stick keep a peer connection to that guy because in case I need to I need to send a message on one of the subnets he's he's uh he or she is connected to that subnet so you're left with like uh 10 of these nodes are super valuable nodes across the network whereas everyone else you kind of just throw them away they're not they're not all that valuable whereas if we transition over to this thing one it's enforceable which is something that we want two we're not entirely sure whether the amount of banter for using across the entire network is necessary so we reduce that as a I guess maybe a side effect and the the third example is that all be all the nodes are equally kind of valuable to you and because we tie it to the node ID it makes the node ID specifies which subnet they're supposed to be subscribed to so when we're doing Discovery queries we can actually search more efficiently for nodes on a specific subnet because it's tied to that node ID so there's a number of benefits it's not just uh we're helping institutional stakers that's just kind of like a byproduct hey uh cool could talk to you guys thanks um so you mentioned uh reducing bandwidth um and reducing latency to like avoid missing expectations something that we've looked at as well is like affecting the effectiveness ratings um with validators um can you guys speak which can result in like penalties or loss reduction in rewards um can you guys speak to how this would help with effective Effectiveness ratings um if it would um which I I guess the first one is kind of what I introde with my my personal uh attestation Effectiveness drops when someone watches Netflix so uh I imagine for a lot more homesteakers that are kind of on the bandwidth limit or their upload speed is quite low like in like in Australia and we reduce the bandwidth requirements a lot Diva said that when you have these peaks in bandwidth you you can miss out our stations you don't you don't publish them in time and it's not just missed but you also get penalized if they're late uh so even if you even if they still get included in a block but later you get you get less so that that lowers the attestation Effectiveness um from a lot of the people that use lighthouse we find that uh that there's a number of Main that that impact the attestation efficiency and that's one is bandwidth and the other one is CPU limitation flight processing so if you're running a a node that's overburdened and and this the topic subscription is another thing that can overburden a node because if you're subscribed to a lot of notes let's say you have five validators attached to you and you have five Long Live subnets you have to get all the messages and you have to process them so the processing also kills you and lowers your average Effectiveness so if we get that in there that all should in principle improve the effectiveness because of the bandwidth and and because it lowers the CPU usage of your node would it be possible for a node to detect that it's been choked if so um could it somehow combine that uh with being dishonest about its subnet account and other grief or otherwise stall its local node graph sorry I missed the last part could it either if it could combine that with uh being dishonest about its subnet account connection could it just stall or otherwise grief it's local node graph oh wow stream is explicit so I know this is actually going to know it's going to be showed we're asking him to stop sending messages to us it's not yeah you know it's I know it sounds similar to what is happening for example in in strategies for example in I don't know follow sharing but it's different in the sense that we're the ones asking the peer to just stop sending the messages so it's kind of a benefit for them so when uh when we connect to appear uh let's say let's say in this new regime where we have every node has to subscribe to atopic based on its node ID we can look at its its peer ID but let's say it's node ID we know that it's supposed to be subscribed to let's say topic three um the way that we when we connect via gossip sub uh they should send our subscriptions about what it's subscribed to and and if it's not if it's not subscribed to that we know that it's being malicious or faulty straight away and we can just disconnect it uh there's technicalities where I can say that it's subscribed but then won't let us uh so the next the next phase of the thing is we try and create a mesh mesh Network so if we're if we're connected to let's say 100 pairs we only really form a mesh with with the mesh degree with like three of them so in principle they could always just say no no you can't go on my Mash you can't call my mesh I'm full I'm full I'm full so that's one way to kind of grief Us in that scenario they still have to forward us gossip sub messages there's a there's a small mechanism inside gossip sub that is called gossip subscribing as you just in 1.1 which somehow mitigates uh attempts to mitigate censoring so where where you just say that you're subscribe you do nothing and and the whole network essentially tries to kick you out of the mesh in terms of while being choked we only choke people that are in our mesh so only people that are subscribed and then once they've grafted with us so we form this mesh connection which means they have to be subscribed they have to be sending us messages if they're not sending us messages 1.

1 scoring we'll kick them out and then we then we choke or unchoke them in order for them to be malicious and try and cheat that system they're choking abstract the choking strategy is is abstract and can be implemented independently on each node so you don't know on each of your peers what what is specifically their choking strategy they could just pick random nodes on their mash and choke people so I think there is an Avenue of security to to look in there that hasn't been done yet but I imagine we can probably solve that with some of the scoring parameters in gossips up thank you very much Diva thank you very much Adrian thanks