# Eth2 Networking Call 4 (2020/3/25)

- Channel: [Devcon](https://streameth.org/devcon)
- Date: 2020-10-09
- Duration: 1:17:35
- Topics: blockchain, smart contracts, distributed ledger, ethereum, Science & Technology
- Watch: https://streameth.org/watch/yt-F6c6BJ285Mc
- YouTube: https://www.youtube.com/watch?v=F6c6BJ285Mc

## Description

Agenda: https://github.com/ethereum/eth2.0-pm/issues/137
Notes: https://hackmd.io/@benjaminion/rkEn7C_88

## Transcript

okay we are recording cool so networking there's been a little activity on the spec Revo a minor adjustment to the in our requirements there's a discussion on the a ping which we can kind of like share vital information or at least let the other know that there's vital information on has changed well at the same time thing I'm still alive so that's still a work in progress PR for discussion we might talk about that a little bit today and it has certainly come to my attention after a few conversations that aggregation strategy although look relatively straightforward implement there's a lot of kind of edge cases and things to think through to make sure you get this right so I want to kind of talk about where when's that talk about explicitly some of these problems and see if there's anything else that we want to make it into the stack even if it's just guidance cool but we will start with updates and probable protocol abs is here well you want to get started yeah is safe and sound for us we've been working really hard on gossip sub in the last few weeks if you have been following the Liberty specs repo the goal would be to be pops our propose you will have seen a bunch of changes and a bunch of discussion and PRS and so on around gossips of hardening we are evolving the gossip sub protocol although there are no substantial protocol level changes in fact gossips 1.0 and also some 1.1 which is are we calling this evolution 1.1 is a set of security extensions and it's basically 1.0 in one point compatible with each other so it's perfectly feasible for implementations across the languages that are supporting different levels of the protocol to to cooperate basically what is happening the last few weeks is that we assembled a cross-functional team here at protocol labs between b2b rights lab who is represented here today by daily deals we gave let's go file coin and test ground this was an internal cross-functional team and basically what we did was in the mairead and recent for a number of attacks and their feasibility and the cost of those attacks and the potential impact in concepts of 1.0 we designed a number of mitigation mechanisms and basically we hired a but go implementation taking this one as a reference implementation by introducing some of those mechanisms and they are things like peer scoring great listing back offs and other elements to the security protocol we'll be posting the links to the spec itself which has already been merged to master okay there you go so there's a link to the spec which is a document in itself so it's an is conceived as an extension to the baseline spec and there's a PR that implements these changes on the reference implementation it's a work in progress PR and it's been active by a bunch of us and I told me invite everybody on this call to go into this back to read this back to comment on this back and and also on the PR itself um what else what else what else yeah okay next steps basically right now we have a team of three people then I dedicated entirely to testing the effect of our changes against the baseline of 1.0 so we are building we're using test ground for this and we building a number of attacks and coding a number of attacks modeling a number of attacks and testing them against the 1.0 implementation and the 1.1 implementation once the right moment comes we'll release all of this information and the numbers and the improvements to the public right now it's sensitive India over the next few days there isn't red team here at protocol labs that we've assembled that would be auditing the spec and the implementation and I think that once that checkpoint is over that would be the safe moment through to basically throw these changes over the fence for other language implementations to adopt so JVM named Jes rust and basically you know we are here at the disposal of everybody else to support this work to support the implementation the upgrade work in whatever manner is needed right knowledge funding whatever so we can we can talk about that also in parallel we're going to be looking at potentially hiring a an external audit term as well so open to to suggestions from from you guys if you have any that's all for me I think I extended myself tonight awesome thanks ro would you estimate this is in terms of development side two days of work a week and work two weeks of work to get up to spec to get up to spec I think it shouldn't it definitely should and be given and given the fact that so basically what we're doing is we have the reference implementation in the next few days we're gonna be annotating enhancing and even refactoring some parts of the presentation to make to make them more dynamic so that you know it's very easy to like go through and actually understand how things relate to the spec there's a bunch of like algorithmic stuff here that needs to like very clearly be you know cross-linked to the spec itself so hopefully it would be super digestible the spec and the reference implementation themselves the the pair of those I would expect magnitude of days it's not a huge it's not a refactor is not a rewrite it's a bunch of and if you're implemented with the right hooks which is the way that we've done it we've implemented with a set of like callbacks and a number of things you can very easily plug into the right places within gossips of itself to inform the score which then can run kind of like an in decoupled fashion so yeah what's a magnitude of days cool thank you any questions ro great thank you yeah appreciate all the work there exciting feel like if you have anything you don't have this on every five or otherwise yeah so the goal implementation is now up as a pull request on go etherium repo I've been I've implemented a new LRU based session cash for it so it doesn't in no longer stores keys and on this and in general we've reviewed it with coherent team I do need to make some further changes to reduce the time the unit istic because right now the unit tests take about 60 seconds and previously discovery unit tests took about 10 seconds so I need to reduce the test time and yeah other than that we are positive it can go in very soon we have also identified a couple of things that should definitely go into the next spec version so looking at the issues right now Dimitri from the one of the Java team I keep forgetting what this client is called sorry harmony is has proposed that we should change the definition of the final packet to be yeah to not require so many individual requests so we will convert it to have some kind of like to basically accept multiple distances in a single request and doing that will basically reduce the number of requests required during a lookup and we have also identified the problem with the tag construction that is in every packet because there is a type confusion there so notes can under some circumstances miss identify Paquette which they receive and this basically leads to a situation where you get a lot of ugly messages in the log I mean there's basically there's no there's no big danger there it's just something that is easily avoided if we change the format a little bit and we had some ideas and discuss it so these are the two most pressing things I think for the next iteration of the spec would be true oh yeah and then there's one final thing that Yannick suggested which is to potentially use compressed compressed elliptic curve keys in the handshake so this would save something like 32 bytes for every handshake in the in the packet size so I think it's worth doing but it's backwards incompatible change so it will have it will go in the next spec version yeah so I'm gonna be busy updating my implementation with these things to see if everything's still working and I'll create the next big version over yeah probably this weekend and I'll notify everyone and then we can just yeah keep improving Thank You Felix Russians you okay any other updates not really yeah I think and otherwise from anywhere else any other fits so for this thing well an order I served on the earlier dogmatic the lower level testing meanwhile Lakshman and me are looking into this network testing somewhere between like your application before and this third of the testing so we have rumor and my room I'd show these things earlier and those are if often and the passion rapper is very like integrated system a stack that he can produce all this consensus messages to send to clients from these network test and then start from initial test routed consults so this will be worked on Lakshman is new is helping us part-time and yes this repository up called stethoscope if anyone is interested in helping getting feedback to like proof of concept testing for CTS for example then these results can discuss these last it's still early and I don't think I want to bother if alliance with early phase testing right now you cool thanks Berta her adults are now using rumor pi room and a modified version of the Python spec to pull down to think the lighthouse fitment pretty cool alright so within leave it less than ten minutes I think nine minutes something and B was disabled Norse is still dominated by them and the spec can think that thousands lots of letters does not which is kind of fun to check that actually works and you can load your own configuration and this one versus your upward then bring it is on so I think everyone on that version and try it out for themselves saved our does not check if transitions March and then really useful to have some network do they need to be able to interact if you're about that also can produce consensus messages rather HD so if you want to scripture and test the scripture on health monitoring or something you can do this you cool okay other updates you okay we will move on okay so as I mentioned at the beginning some challenges and edge cases in the aggregation strategy and every and subnets I think it's been on a few teams of mine specifically speaking with age he wanted to make sure that everyone was on the same page and that we were kind of hitting making sure they're secure and hitting all these educators to that end I will give the floor to age if he wants to kind of discuss his experience thus far and some of the issues that he's running to it yes sure so I'm not sure how far all the teams are into implementing their naive attestation aggregation strategy but we've we've hit a few issues I've talked with various teams about so maybe other people have some answers or I've kind of made progress in each of these areas so might be a good chance to have a chat about them I guess so some of the main things that we've had to think about is discovery we need discovery to be relatively quickly relatively fast so we've spent a bit of time trying to I guess tweak our disk our disk v5 implementation so that we can find peers quickly we need to find peers because when we obviously when we need to subscribe to particular subnets we need to have peers today us subscribe to we spend a bit of time looking at the validated client to beacon node API because with the new spec updates there's you know there's obviously changes in that API specifically around getting the validated client to know who's an aggregator as well as the beacon know to know who is an aggregator so we've made some changes there so I'm curious to know if any other teams have actually modified this API to see because we've kind of gone off on our own little tangent and potentially you can still conform there may be some people have some good design kind of decisions there there's difficulty we have I guess it's not difficult you just kind of have to sit down and solve the problem that you have to deal with the timing quite quite thoroughly with being able to firstly discover peers and then subscribe to as particular particular subnets for a given future slot and committee and if you have many validators attached to you you can't have to deal with that one of the other issues is peer management I've kind of been this is one of the one of the bigger issues I think that I've kind of been bringing up with everyone when we have when we need to be able to find appears to be able to connect to subnets we need to know firstly the peers that we're connected to what longley up subnets they can they have this this involved knowing the ENR of the piers that were connected to so we don't actually have a mechanism for finding that out if a pier is connected to us through the live p2p stack so that's that's one of the issues that we came across another one I just I don't want to just kind of throw all these things out there but another one I've just recently kind of run into is that if we are an aggregator for a particular slot or subnet rather if we're not an aggregator should we subscribe to the subnet if we're not an aggregator I don't think we actually need to actually receive any of the auto stations on the sudden that we can just we can just publish across the fan-out so I'm curious if we do do that then there's less people overall subscribe but we should have the backbone from the long live subnets I guess that's just a smaller stuff to talking my head of some of the things we've run into I'm curious to hear if anyone else has thought about these or run into any of these issues for the API I can command out that so in the old respect is basically just one trip right validator since the selection prove to the beacon node and we can know verifies that this is we can know verifies that the validator is actually the aggregator at the right slot and then package all the other station together Barack has the aggregated selection object by in the latest spec there's the in the latest fad the aggregator has to sign over the aggregated object so so then it becomes a two-way trip well did her sends the selection proof we can know the response back with the aggregated object and then the validator signs it over and then send it back to the bigger node and then the big you know broadcast the object over to the net network at this that's my experience so far sure so if you guys have an API be interested to have a look at it some of the the small I guess edge cases are that we're signing a future slot and this line is based on that the signings based on a fork so in principle if you're assigning an epoch in advance you could have a fork change in between in between actually having to do any kind of attestation and this the slot signature that you have another thing we ran into is that I think Maron brought this up is that potentially you want to give the option for different beacon nodes to actually do the aggregation so if you're connected to three beacon nodes you probably only want to use one of them to actually send the aggregation off so we engineered I guess a way for when you do the subscription that the beacon node tells you whether you're an aggregator and not so that it's up to the validator client to actually ping the the beacon I don't get into the technicalities I guess probably better offline but if you have an API to be good to have a look at and I can have a chat with you about how it's designed as bustard I will share that with you awesome cheese so the other one which probably is a just a quick one for Danny thoughts on if we're not an aggregator should we subscribe to the subnet apart from being long-lived that's a good question so there's there's there's certainly immediate value in there so our subnets if we don't do this have grown much larger than we expected them to because of the cross-linking at every slot that change that came around DEFCON so normally you would have previously you would have subscribed to a subnet maybe in a pocket van but you don't have a bunch of other people in the symbol or in similar slots subscribing as well so what this does is 1/64 of all approximately 164th of all validators are going to be subscribed to a subnet in the given time and so these subnets are much larger than we originally intended to be so if you're not an aggregator that's actually very nice because you can just send to the cult of fan-out right and not worry about actually receiving those messages and just the aggregators and the person those persistent will receive message - the one downside that's worth considering is that these validators don't then immediately kind of get weight to their fork choice through the individual adaptations they would have seen so there's a little bit of latency and a higher dependency on the aggregators to show up and do their work to add to that like subcommittees fork choice like group fork choice that's application committee is going to be updating their pork choice and the aggregates are going to be send the aggregates are randomly selected and they're going to be sent on a public channel very soon you know within four seconds so my intuition is I actually had a great change and I will think of that a little bit more of it open to cut other comments now actually something that has been in the back of my mind is you know can we make these subscriptions more granular so that you're and modifying the protocol such that you're only participating in a subnet on a particular slot but this actually I think removes any of that complexity that I've been totally chewing on so we defined by slot instead of epoch now it is that the plan no no so the the idea is if you are an aggregator you actually join you subscribe to the subnet so you receive messages from that subnet if you're not an aggregator instead you just find peers on that subnet and publish them so that if you're part of a committee you generally just publish your message the persistent committee is going to see it very quickly and the aggregators are going to see it very quickly and then publish aggregates on the main on the main channel if you're a navigator you actually are going to be seeing all those scheduler messages and for me to do so this stays a little bit on bandwidth and also just reduces the size of these leaf net okay thanks that makes sense you yes so one of the other problems that I mentioned is that we don't know the long-lived bit fields or just a bit field from the ANR for ears that connect to us and I think we should have a mechanism that gets us that this as I mentioned this in the last networking call has anyone had any thoughts on this I made a made a PR which has some comments on it which introduces also a ping protocol which is a jerk just a draft my show people have got thoughts about that right yeah I'll agree with that so cuz if you don't have like any sort of timeout oh we know when to connect for how long to particular subnet so you cannot differentiate between and you know all the in our or when it is new and so on so I think it helped a lot in HDC if only aggregators are joining subnets you still this it's still worthwhile having the persistent how to stay suspicious instead that's passed along because you can still make slightly better decisions on your peers correct yeah yeah I think that I think we need the persistent ones so my intuition our reading the thing protocol is that instead of sending the ER sequence number sending some sort of local identifier with respect to just your your note in general and the payload if that changes is the E and R plus potentially other local information that might have changed so it's a kind of a meta sequence number so that this ping protocol becomes useful beyond just the NR if we want it to be Pokemon yeah sure the small details I expected it kind of changed it I'm happy to you know to some iterations on the PR is kind of manly fishing for weather firstly do we agree do we all agree that we kind of need all we want a Bing RPC method and do we want to be able to might have an dedicated Eni method or do you want to try and just take the bit field out and trying to chuck in a status message or something like that I think they're the two main kind of ways we could go I mean in the ticket I argued for separately in our message so - because it's big blob of data and technically we don't need in order to run or would be nice if 52 protocol remain separate from a hard requirement on in are even if it's kind of in the spectra London would be nice if you could run the rest in the protocol without you know it's really there they bring in peripheral stuff that is not really needed for for it to itself I guess this is something a bit of a design decision right I mean we were there last time in the last call I'm not sure if it was the networking call or the general is to implement his call but we just did discuss it a little bit this this question of like should the note metadata be relayed as you know or something else and I think we just need to make a decision well there's two levels right so when one thing is whether we relate some data over in our for the pure lookup and then there are like in protocol needs when you're already connected yeah so I I think that like for real-time in protocol information exchange you know is not really like the best format also because it is always has to be signed so it's something that like the signature in the e NR is very important when the record is relayed because it protects the record in transit and also when it is stored in some other location but so it cannot be modified in transit but since if you are connected to appear directly you already have an authenticated session with that peer so any updates that we're sending our first of all more recent than whatever could be in the you know that you found earlier and also since it's authenticated you can trust appear to tell you the right things about itself so it's kind of okay not to sign this information so I think you know itself is nice but it's really more for relaying information about the node outside of the real-time communication session that you have with the note that said in the discovery is a bit different of course but it's kind of like in general I agree with you that using e NR as a way to like relay some application specific metadata is in protocol is that it's not the best way I think it would be better to define some other protocol message for that and not have the signature and things like that yeah so there's they're as Asian point out the we have one thing that we want it we know we have one item that's persisted that we want to relay which we can drop with the agitation subnets which we can drop in to be in the message or assuming that there might be other semi persistent information that we want to really it relay every once a while we can do essentially a note sequence number that is dropped into a ping let's just kind of bounce back and forth every once in a while and I notice it changes then I can request for the larger payload right now the larger payload would be the application subnet it could also be other other information that was in the arm or other information it's in general so once we so one thing that would be quite nice is to make this the format of whatever this message is to make that like yeah to basically define that format in a way that allows it to also be transported in through other means so one thing that will definitely come to this p5 in the near future I think it will be optional is this idea of having the like pre connection negotiation so kind of basically like a way to exchange arbitrary application specific information through the Discovery Network session so this is not the same as doing you know like you know doing everything over the discovery but it's more like if you want to connect to a certain pair it's good sometimes to be able to figure out in real time if this connection is really gonna be worth it and this is what the pre connection negotiation is for and if you have a way to encode additional metadata beyond what goes into the INR then this format could also be ideal for the pre connection of negotiation so I would really appreciate if it could be defined in a sort of like separate section of respect to just have this like pure metadata object right and this would also be an ideal place to relay things like the full list of like hashes for something were like lists of gossip subtopics or whatever you need I don't know like basically things too big to fit into a single you know got it I guess another question the do we all want a ping protocol correct like some very lightweight message just saying you know with I think I usually because of at the moment we don't have a way that if peers kind of just drop out to identify that shouldn't touch me at the Lupita P level yeah yeah what do you think sorry goodness emphasize this at this point hit you're not permit so what I've seen anyway in a Rustler poppy there's that there's an IP FS ping which pretty much does a very similar ping to what we're suggesting here which can check for peer liveness we haven't specified any other litte bit of your protocols apart from cut from the RPC that we've built and gossips up which have like these long lasting streams so identifying when peers kind of drop out I think we kind of need a ping variable so if the underlying transport is TCP which is the case right now because we haven't really adopted quick I'm not aware of implementations that have adopted quick yet then this should be part of the so if a peer is connected to you it means that the TCP connection is alive and we do run Kiba labs on that connection anyway so there are TCP keeper lives that are happening so basically when a peer drops out you would be getting if if the rest of levitation is behaving properly you should be getting a disconnected notification immediately when the connection does interesting so one thing that I can say to that is that it's not always a good idea to rely on the TCP level keep alive because TCP keep alive is something it you cannot influence and it might also get you know like there can be issues with that so it's like the TCP Kiba life is kind of a last resort option and you cannot really influence the timing of it so in in in in like you can but it might not be respected think it's kind of these people I was so can can sometimes also fail and and it's something that is it is it's much easier to do it at the application level so in the def p2p transport protocol the way we do we do it is basically that we do have this like ping in for messages which are sent if no other message is sent for 20 seconds or something and it's basically it's much easier to just put it in application protocol because if you do it first of all you can use it to relay other information like the you know reference number or the metadata stick number that you were talking about but also basically it will give you it will assure your your note that basically the pure connection is indeed alive and you nd you can have and basically authenticated live in a signal that is way way stronger than just relying on sort of like the underlying Internet infrastructure to do this for you like I would recommend putting into the application protocol because then you can really be sure that you know like your your key people life mechanism is working and not you know like mangled by some middle box or whatever yeah I think that you do make a valid point Felix like personally I'm not 100% sure of what the assurances in terms of like you know the wider internet impressed or in terms of the cheaper life actually getting delivered and what so there are other layers and there let me to be start particularly the multiplexer the yeah max multiplexer it does run things and it is an easy thing so it's you know and my best thing essentially just like you described if there is a message that comes out anyway in application payload then the thing you know reset and basically the every application level message resets the the ping timer so it's only it's only used when when there is a period of inactivity that it but this could be an even across multiplexer it's because I think that the mBLAQ's multiplexer does not run keep alive so yes if you do see a benefit of using a ping payload at the application level to transmit in the future like some other control message could be useful yeah but basically the two points are I don't know about TCP keeper lives in general I think we have seen them work well but it is also true that most of like I BFS for example and I'm going to use the AMEX which also had that that ping mechanism so I'm not sure if it's just the keep alive or the combination of both that you know makes this mechanism work well for reporting connection you know the connections that I killed immediately so the thing about the TCP keeper life is that really the the reason for TCP kill of people like the why it exists in the first place is that usually when the connection is broken then they're like when you close the TCP socket in the in your user space code or if your process exits the OS will take care of like sending a TCP level control message that will notify the other side that the connection is now over and then basically the other side will in general we receive and it will basically relay it back into users based on the others on the other end of the connection so you will basically see the connection drop but the TCP keeper live is sort of the intention for TCP keeper live is to take care of the case where one side basically gets firewalled off or just drops out because you know the battery runs out or whatever and then in that case no control message will be sent by the OS because the OS is dead so it's kind of this is what the Kiba life is for because if the keeper life extension is enabled on the TCP connection and it's something that is negotiated during TCP handshake the the both sides will be waiting for you know like any message or you know like this like keep a live notification if nothing happens then basically the connection is also considered dead but the timeouts for this can be really long and it's it's all a bit messy so if you do it in the application it's just safer and I think this this thing that you were describing is that having it doing this in the multiplexer is like it totally ok solution because it's really only needed to take care of this case where you know like the connection drops due to a power outage or something yeah so are you are you guys using in rust and the rest of the implementations are you are you guys using networks or boots we have yeah mechs and it falls back to imply X I I've seen some interesting things on the test net which is why I'm kind of proposing this way we just get like protocol negotiation timeouts I have to probably to track down these a little bit more detail as to the source of the problems but on it yeah I think in these examples I'm using Y Amex and I still find some weird things so potentially it's not just a being potentially I shouldn't see disconnects as opposed to what I'm actually saying but I mean now look at that it's probably implementation specific mm-hm if I may like a oh that's like likely recommending an experiment we're using the IP FS ping protocol and it's a protocol beyond just checking if the connection is still open actually we'll use peer routing to find new addresses for the peer if a disconnect happened beforehand and so for the case it was being either like someone changing the IP address because it changed the location somewhere like changing from like regular eyes Internet to somewhat 4G hotspot because the ISP went down or something that if s pin protocol will ask the network again and like real real establish the connection so it's beyond checking a connection open it actually checks if the peer is still alive and running so maybe yeah like checking that out and see if that's like surprises for what you are looking for would be good go take a look I'm currently drawn to an application layer ping with a metadata sequence number and an RPC method forget metadata which would be just longer persisted data like attestation sadness and things that we want to share like that is do people want to agree with exploring that yeah so there's a template PR we can have a discussion on or remove there anyone cool we'll take it there other further comments on this and I will take a look at the IP of us hanging protocol which might the resources there's one thing like we're talking about long lived data mmm some of the data that's been mentioned as long lived might change at Forks so if we specify something that's called a low message we can't specify that it's just a Hello message like that it's only a lot percent um connection basically right I think that we'd want to just call this like I get metadata message and specify under which conditions you might ask for it and even leading up to a port you might ask for it yeah exactly and like not required that the clients call it at a specific point in time either because if they need the data they will naturally do so when they need it right so there's no like I think in the hello message it said something like you know you have to send it on connections or something so something like this yeah probably not and I know there's a little bit of desire to include like client information in version I don't want to include that in the core protocol but maybe even metadata we can leave a like as a requirement but maybe a metadata we can have yeah 32 bytes crap somebody can reference right your favorite if they want to okay cool Hadrian Jers there anything else you wanted to bring up with respect to some of these other items like discovery subnets do you mention that you're doing some things to quicken being able to find peers with certain things in there in ours is there anything we're sharing on them I can just tell you kind of some of the strategies that we're doing in case other people ability to maybe people have already got this by default but so firstly we we had a system where like United peers could kind of get into the tht we remove that pretty much by taking a larger majority before you update like Yoanna there was a there was a PR I also kind of made to the to the spec which means that in ours don't have to specify an IP address so when you first create an IP when you first create an e NR and you're not sure that you have an externally contactable address then you can kind of just leave the IP address out so it's it's clear to say that that your NIR is kind of either behind a NAT or hasn't found a way through the net there's a thing we we come in a bird from live p2p CAD which is in the queries defined node queries so in discovery b5 there's there's a timeout on your on your UDP packets for us we also implement because UDP is lossy we have the option to send packets multiple times after a particular time at originally in the one of the earlier versions of the implementation the the timeouts were kind of quite large and when you're doing the query query has usually a parallel parallelism parameter which allows you to you know send multiple queries find no queries to peers at any given time and for peers that were they disconnected or not not available on the DHT we kind of were waiting for this timeout before we classified the appears being non contactable and going to the next one for the query depart to kind of progress so instead we we've added a like a smaller timeout for okay it's kind of like a query time at so let's say that your your UDP request has a timeout of four seconds then there's a query timeout of like one second or two seconds which after that period of time you you don't consume your power parallelism parameter you keep asking more and more peers but you still accept a response from other piece of that kind of speeds up some of the final queries in terms of in terms of looking for particular in our fields we decided that well I guess it's still implementation specific but in one of our earlier versions there's a particular that we need to I find no query to find peers you the query finishes when either you found the maximum number of peers or you have exhausted the crawl in the particular case for subnets we probably want to have queries that finish earlier if we only need three peers or two peers for example you can you can specify the query to end when it's found a certain number of peers and also we kind of filter the peer list during the query for in our fields as opposed to getting a chunk of peers finding out whether they match or in our field and then you know reprocessing the redoing queries so there I guess some of the main strategies we're going to try that'll hopefully speed up our queries to find ps4 subnets 6h any questions for age four move on maybe I can just share one more thing which can be like nice trick so the way I'm considering to do the that in Goa term what the way we do it with with the discovery is kind of like the design is basically like iterator based so you can kind of just get something like a never-ending iterator of like nodes that you can pull from and then when you pull from this iterator it will just keep advancing they look up and one thing I'm considering to add very soon now is basically since the lookup implementation is event-driven and I think the one in Russ's as well it's kind of actually there's not a big cost to just having many lookups concurrently so basically one thing you can just do is you can just run like 50 random lookups concurrently because they're all just state machines and then if you want to limit the resources you can just put a rate limit on the number of outgoing packets or something per second and doing that can give you a pretty broad scan of the network at a configurable rate which is something that I can really recommend doing to like get many candidates as quickly as possible and also I think that the I hope everyone understood that and the other thing is that I think that the change that was suggested by the Harmony team to improve the final to take multiple distances as the parameter will also speed up these queries because you won't will no longer have to send like up to three requests to find all the nodes that you might need so the goal is really to like get find as many nodes as fast as possible like its purpose of the protocol and basically anything we can put in the spec even to accelerate that is I'm happy about more suggestions got it Thanks okay cool so when you're working on this consider some of these strategies to make discovery particular peers in particular in ours quicker there is in that eqo api is there's been some discussion on a updated protocol between browser client beacon node I know that prismatic has their own set of API would certainly cross pollinate those picture on and using the best ideas peer management we're likely to do a ping with metadata and defined per sort of metadata request and then on the aggregators my intuition is that are for non aggregators my intuition is that the nan I rivers actually don't need to subscribe to subjects and that's just publish I think that's going to be a game for us so I will work on those two potential TRS with respect to ping and redefining how you subscribe or not and otherwise I think it's mainly working through some of the details I think some teams have worked through more the details so please talk to each other okay anything else on the item yeah sorry you two go first okay hopefully this is just really small it got brought up in there India gender as well as that for beacon nodes that have valid a is attached I think my my current thinking is that we for the peers that are connected to us we go into prioritized connections with peers that have greater the larger number of long live subnets so anyone have any concerns with doing that you so the spiritual bootstrap the subnet is it so let's say we've got like 10 10 or 15 peers connected to us 5 of them are just normal beaker notes as in that they're not validating so therefore they don't really have any long live subnets attached to them if I'm a validator I want to find is that have longley of subnets so that when I have to perform an action on one of those subnets I've got a kind of a collection of peers that I know already count of that stuff that which saves me having to go and find more of them so I if I do go and find more and I have a peer limit or I only have a finite number of resources I'm gonna kick probably peers that don't didn't subscribe 20 subnets in favor of ones that are so that I don't have to do discoveries in the future so it's pretty pretty good to me yeah sounds like certainly the viable strategy as a validator I think the one concern which I think was brought up was if you have these like super nodes that are running tons and validators and subscribe to all subnets at all times they might become kind of naturally become central nodes from the network which is something to consider I think we definitely want you definitely want to have a diversity of peer with respect to persistent subnet I'm concerned about having finding peers that have the maximal number of something that even though I know that it's kind of like the best way to optimize that problem think about a little warm I also think is kind of important not to be like have the peer set be too static before the reason that like if you get locked into a particular set of peers and you keep optimizing that then it can lead to a situation where it's very hard to join this club because I'm red so it's also something that like but you can this is something like I guess in every implementation that you have to find like the good policy like if you like you only have so many spots available for connecting to other nodes so you might as well just you know like try to keep some of them available for you know like I don't know like more sort of like dynamic and experimental peer connection so you you actually do notice if there's like new good peers from the network so what my question was my question was that from from the this v5 implementers how are people handling the caching of nodes do you guys in terms of the sessions not sessions but I mean the like discovered nodes so do I do are you keeping like persistent cache case or in memory cache is only or yeah so we for us anyway we the discovery we fire limitation itself just puts them in the not in a in the adding table and then lighthouse when it shuts down stores it on disk and loads it back up on this market but this is only for the for the table that's right okay so one thing that you can totally doing is something that you know is also it like optimize for is that it's actually totally fine for you to keep an additional sort of cash on disk of like an old records that you found because you can always fall back on those records in the future I mean they might be stale but it's something that is very very valuable in Goya theorem for us to spike on the e theorem one minute to have this like notes database where we basically store all of the notes we find and leveldb and basically means that when you start up you already have a pretty complete view of the entire network locally and all you kind of really have to do is just you know and I keep adding to that or keep rivera fiying those notes you do kind of have to expire notes from this cache from time to time like you can't just you know and I keep accumulating note information forever because of churn but it's definitely very helpful to like keep persistent caches off the network for a longer time because it will really really help with rejoining the network but also yeah just it just means that if you need to do queries for like you know like if you've already discovered a note on a particular subnet like half an hour ago then probably if that node is really like interested in ease to it is still alive so you might just you know connect back to it again so if you just I don't know if you guys are using something like sequel Lite or whatever or level DB with a custom like index or something you could totally just keep you know like an on disk index of like notes which are on a particular subnet and and then just try those and you don't even need to hit the DHT for that you basically because you already found them while ago is really good idea will implement that thanks I think any other questions for Felix or age or otherwise this is a great way to find out which validators are on which beacon oh yeah yeah a question on validator privacy so currently how the subnets are like structured it would be pretty easy to like map IP address to like a value this public key because because like the requirement is that each at the station is uh Negra gated so identifying like one validated wrote an IP would be easy across like a preview box so is that something that we're expecting it's certainly a known problem and secretly election and other things on the horizon or to address some of these things and the decoupling of validator from node allows for others to create more sophisticated setups where for example I might publish on two stations to one node whereas I only publish my blocks to another node assuming I might get to us so certainly a bit hand-wavy it is but very very expected to this point to be able to do not imagine our Thanks okay moving on Nicola I took a look at your table I don't know if I fully parsed it let's tickle your table yeah I sent into a chat so it's something that we did it Cece just after last call we've put along the Dimitri McLaren the wealth and the idea was let's be clear about the requirements so that everybody can agree at the meet those requirements and so it's Patil um diamond live at opposed bees with the idea for example typically on privacy is it possible to the anonymize when at work or not we've the initial target what could be well let's see what if if it is actually possible and we expect that it would be possible with a plan so when for phase two we can decide on what to do so there just a few lines at the end of it there which it's about formalizing very very high level things like how many nodes do we expect to have in phase zero and I will happily with one thousand nodes and what's the target for Phase two could be twenty thousand nodes so so the idea of this is really can we find a place to have this kind of high-level requirements in the specification and a well from circuit are we happy with those requirements and do we think that we can actually be met by the implementation and unparticles today so it is two questions I mean do you feel that it's reasonable in people in the corn I mean I mean where's requirements one for some not so you think that it can be done for phase zero and are you okay with the approach should we put this somewhere in the specification we can certainly take it to an issue for discussion and decide where we want it to live my intuition is that will certainly have on the order of six hundred to two thousand nodes an early phase one and that's certainly one of the most under tested items at this point is scaling though it's probably beyond the approximate hundred range yeah under your finger to shoot target one-on-one or do you feel that we should target 1000 so certainly 1100 is the only thing we've seen I think the maximum approximate accident we've seen on a test net the prismatic just as before so that's why I mention that number I think 1000 is probably in the range of what we'll see but I wouldn't be surprised if we saw 2 3 times more than that or in early days off people things but it's we can achieve 3000 not today for example at regulations or discovery whatever what's your intuition on that client team so the I don't think any anyone has seen bandwidth or finding peers or any of the networking components that we might be concerned about be the bottleneck or main failure points at this point which is good but we've also only had 100 100 known networks so I think there's there's optimism but a bit of an unknown quantity there so do you feel that you should we could put this somewhere in the specification so we are known and then we can establish test plans around this and this kind of things ya wanna report the cable to an issue on the secretary go from there okay I actually drafted up sort of like a test spec on how we could sweep the different parameters to figure out sort of like given the current configuration kind of where things start to break down I can add a link you look at it it's basically like define some stability and stress tests where you kind of suite number of nodes validators per node total number evaluators obviously codes and we're also working on a test net plan on the types of things that we need to see in testan F type of stresses the number of nodes distribution of clients that kind of stuff it should also feed into validating some of these requirement any opener well for the table so end of the table also is to face the requirements but just maintenance test met with try and define steps in between so that actually does make sense to work towards it and but they do have action points we do hit some limits you right you all right anything else in those anything else for Nicola okay discussion we hit this a little bit on item three we definitely talked about the ping we have a potential route forward and we're gonna take a conversation to that PR maybe if you are that'll supersede it are there other spec related items specifically say zero stuff that's gonna affect us in the next few weeks so we won't talk about modifications questions issues comment yeah I had some questions regarding sort of the assumptions on beacon nodes should know on he can notation you're breaking up really bad I'm having trouble understanding what you're saying because of latency okay maybe it was just me stuttering okay I can hear you now there okay so so basically I just had some questions regarding what the expectation is on how on how beacon nodes are going to treat like light clients or other non beacon node I'm sorry a couple of places in the spec where it mentions reputation in and I was curious specifically like I like client not to be able to in all so should be connotes gonna disconnect from no doesn't that doesn't implement all the are they going to you know our are we gonna say that if a if a if a client can't adhere to all the musts she just connecting us oh like have we thought I look like right so I felt some of that discussion online and I think it's a valid discussion the must certainly have not been thought about with respect to like clients in the past month of we as we've defined this spec are you still talking I still hear like some crunching and there's no no okay sorry the latency keeps going in and out so I think it's working a path and see what is not viable I imagine that a lot of them us might be with respect to some of these gossip sub validation on some of these subnets especially the attestation said that the light client doesn't need to be a part of them at all so those those are out the window the block the block validations and how light clients are getting blocks I think it's probably one of the biggest concerns as you brought up and I don't have the answer immediately talking to Mikael about this and would polynomial commitments be a solution to that a zero knowledge proof would be yeah but there's the solution everything yeah but certainly a associated zero knowledge truth could tell you hey this actually is solved with respect to things that you do know but that that's not necessarily the path that we're going to go so I you could certainly listen to these subnets and pick up blocks and just not forward them because you don't know if they're worth forwarding you could also another pass is actually not with the names of these beacon blocks being broadcast in real time because you can actually do anything with them the light client sync protocol actually probably a separate protocol where I'm talking to some sort of light client server they provide me with the proof that advances me from point A to point B safe skipping a day's worth or skipping an hours or the epochs or something at which point I have a finalized route and I just get data about the state I actually don't know if there's any value in a light client doing a block sync and actually getting all the blocks instead they're kind of just skipping ahead so the answer might be that a light client actually wouldn't participate in many of these protocols and is more just getting to advancing to check points and at querying state with respect to those checkpoints that's kind of how the like Lampe protocol imagined today and I don't know I don't know the value beyond that to actually be participating in some of these of them it actually would be nice because it reduces the bandwidth requirement and doesn't scatter like clamps throughout your report gossip at least for the core consensus here oh we concerned about being able to tell the difference between a light client and like a regular beacon mode we're certainly concerned about the protocols that a node speaks with respect to what I want to do with them and so if a node shows up and doesn't can't handle all all the things that I want that I might disconnect to them either via the the en are fired the advertisement of some sort of protocol or a component of a status message that I imagine once we do have a light clients and protocols like that information will be readily communicated such that you're pretty aware of who you're talking to and if you're not a light client server and then you're not serving these these proofs about how to advance to some point in the future you very well might disconnect my client very well that might just connect to you from you as well because they see that you're not serving that protocol and so you're not of any use to them are all things that are being chewed on but I certainly have made their way to the speck I think it's definitely worth doing a pass and respect to see if there are any components that are we estimate are going to be requisite for a light plant that have any mus that would make them non viable and I will do that okay um do you mind if I open up an issue that maybe we can just kind of yeah cool we can try to enumerate any problems there something cool yeah and thank you I've it you know these two the designing of the networking spec and the consideration of my clients conveys one and I haven't a lot of those items to cross pollinate and my head yeah totally you you you other immediate sauce on this item my clients issues with PTP spec but I mean another like sort of random thought is that if ultimately like clients and beacon if we can tell the difference between them I guess in my head which is probably flawed I figured that you know the end state of the network there there would be you know swarms of light clients um you know that help kind of anonymize which nodes are actually validating nodes you know to kind of make it more difficult to find them but but I guess I guess that's not really I guess that's sort of a problem already right now with each one right so it's sort of just business as usual a little bit business as usual um ideally we can find like clients like a service center and that we start this thing sooner and there was a good protocol the in terms of swarms of my clients is kind of being embedded because they're almost certainly not going to take place and we kind of like consensus layer gossip protocols they probably wouldn't be too embedded deep in there because the like lands but nodes in general just a node that wants access to the network that is running a full node can and would be embedded in some of these consensus layer control layer gossip topics but even then if I'm just a full note and I'm not validating I very well might not join any agitation sub that and would instead just listen to I might listen to aggregates a cetacean aggregates I might just listen to block because blocks are going to include a lot of information that I care about and as long as there's some majority validators successfully participating it's very safe and I get the reduction of bandwidth so some of the sudden that's might only be composed of primarily validators gotcha okay thanks Danny okay we're three minutes over any clothes in common you cool okay mate stack meetings this week we'll talk a little bit about the easier 11:1 release that needs to drop ASAP the re-release of a of our phase zero bug banning program and other fun stuff dr. Ellen thanks everyone yeah thanks everyone Thanks