Web3 vs Web2 Security: Same or Different? | Devcon Bogotá

foreign [Music] we love you all right guys hey everyone so uh we're here to talk uh what web3 projects need to know about web 2 cyber security uh and really I think this is going to be a great kind of follow-on talk to the bridge security framework talk that Patrick just gave um this is kind of the compliment right so he went over really some of the smart contract and protocol layer aspects of bridge security uh unfortunately a lot of bridges have been hacked though not from you know the hardcore cryptography but from Basics from cyber security Basics okay the cyber security Basics matter they matter because they have geopolitical significance so before we get started on the panel take a look on the screen there was a great article by CNET that came out a couple days ago and it really talked about how Lazarus Lazarus Group which is affiliated with North Korea is seeking to infiltrate this ecosystem by setting up front companies and then having applicants job applicants apply to these front companies and then the engineers get fished right that's actually how the Ronin Bridge attack happened okay so the fundamentals matter and it doesn't just impact people in this room the people at this conference it impacts geopolitics because then it funds things like North Korea's new program or you know transnational criminal organizations and stuff of that nature Okay so what do we need to take away from this panel so a lot of fundamentals from the last two to three decades of web 2 cyber security there's Cloud security risks that we have to understand and mitigate a couple with that there's front-end security risks right there's operational security risks there's also corporate security risks and then obviously big topic with some of these Bridge hacks has been Key Management endpoint detection and response and again management of individual individual nodes and validators um doesn't mean we need to Discount smart contract security risk what it means though is we need to shift the security conversation in this space from very siled you know individual audits to a holistic end-to-end scope of the Project's entire architecture right and not just their technical architecture but the people architecture and that's something that you know we have some newer entrepreneurs in the space and that's something that as we kind of get some Battle Scars as an ecosystem I think we're going to learn time and time again okay so who are the panelists um we'll start left to right we got uh I guess we'll start over there uh Naseem from a16z crypto we have Taylor from from metamask we have Corey from openc special guest Hudson from crypto Twitter and then another special guest Moody from also from crypto Twitter oops no polygon polygon says cool all right guys so we'll start um with kind of a basic question and I'll ask it to NAS first and then the rest of the panel now as if you could kind of describe you know from the context of just normal web3 or like a you know representative web project in the ecosystem what are the web two parts of other Tech stack and then what are the web three parts of the of the tech stack and then what are the security relevant parts of of each yeah I actually feel like it's a it's a fairly uh long answer that is required for that but you know we we do spend kind of quite a bit of time going from the early days of the company up until kind of like the ongoing updates to um you know decentralized protocols I would say that um you know it kind of like starts with the people it's essentially just like a set of people who just want to you know design a component uh build it uh deploy it make it available and just like maintain it over time so there is kind of like the everything that it that it encompasses so that means you know you essentially start introducing devices to build the components on you start in involving kind of like the entire kind of like you know secure software development lifecycle that is relevant to every single person kind of like Building Technology in general um and I'm trying to cover cover like anything that is you know kind of common to what to enter through here um so you're going to have like this part then you know this these people need to build an organization right and like with that column you know kind of like corporate security um as you as you mentioned earlier kind of like anything you know from like a endpoint uh you know security standpoint device security and and so on I would say that like the appsec is also like very uh very similar you know always kind of like you know analysis of your code as you kind of like develop it and and obviously you want to iterate uh pretty fast so you're gonna build your CI you were going to integrate you're gonna run you know security testing and bond scanning and all the the stuff that we all know in love um doesn't really matter whether you're building for web 2003 yet at this point right um I think that yeah like in my in my mind everything that that spans over you know network security um essentially like network security uh corporate security appsec security engineering key management and by the way like the key management is the part that is like highly overlooked like a lot of people think like okay it's okay I can deal with this later you know I'm just gonna have like you know like a single key on like a either letter or like some you know a fairly not robust you know key key storage and then kind of like push it out to uh to later but you know Key Management is like a fairly complex thing that like we have to deal with in in web 2. um yeah I would think I would say that like these are probably like the the the majority of the things that we see and obviously you know comes with like you know physical security I think that like it kind of like ties into you know key management and the security of the people as well um because it's not just about the phones it's also on that note we've heard stories of people walking around uh certain places with you know crypto swag and exactly having their phones out and so on so that probably ties into this exactly as well right um yeah and we had like quite a few conversations actually with uh you know various teams of our portfolio that were coming here and you know because decentralization is a spectrum and like you try to kind of like start obviously centralized with like just very few people and kind of like you know centralize over time uh there is like a point in time and we're probably at this point in time like right now where uh any person that kind of uh gets hacked attacked like physically can kind of like result in either assets getting in trouble or even like reputational you know damages that can be created as well or potentially physical access physical actions company or exact protocol and so I think this but yeah the repetitional damage is actually something that like is is highly overlooked as well it's not just about your protocol it's like you know everything that comes around that do you guys have your stuff squared away or not effectively yeah pretty much anyone else on the panel um good so so when I think about web2 versus web3 security especially right now how early we are with web3 and ethereum and blockchain there is going to be a need for web to stuff in your stack uh not even consumer facing necessarily but like who here actually like doesn't run their project on GitHub and uses something like matter most instead of Discord or slack or other non-hosted you're awesome whoever that one guy in the back is uh but yeah uh yeah good for you but if you're unless you're doing that like you really need to be very very careful about where your data is going and keep up with that something else is that I think is overlooked a lot especially when with web3 and how open it is and how open it is to contribute are two things uh watching your GitHub uh to make sure that the you know PRS that are being put in uh are actually from your team members that you trust and that you've vetted uh Sushi swap had a bad incident where someone joined their join their team and then put something in that like I think stole keys or something on their web page Insider thread yeah Insider threat basically and then secondly um this is something I worked at the ethereum foundation as the org security lead for three years and and something I learned especially as I was leaving and hadn't reflected back on there were just a handful maybe two people in the organization that had Master access to just about everything and that's incredibly bad uh it's even it was even worse earlier so just bear with me yeah like what happens if those people get run over by a bus right not even that they could literally take down the ethereum website post on the ethereum official Twitter post on the YouTube pretend that vitalik died crashed the market like there's there's like a lot of incentive there and there's a montage everyone has a price right yeah yeah so the thing is if it wasn't that we just really liked and trusted the people doing it like we got lucky but really there should be different ways and there's you can look up ways that like people have theorized how to do this best for years separate the people who have access to the most critical parts of your organization and have an audit Trail have like a foolproof audit Trail they cannot wipe uh and so yeah that's that's kind of what I think of because it's like web 2 security seemingly has mastered this in some levels but web3 like forgot about it so that's what I think oh so on the sushi incident first of all glad we are not talking about the horse incident but off limits yeah it happened a while back um so what happened actually was one of the contractors had push access to the code base uh there was no PR review approval process and it just got uh they like got angry with something post malicious code nobody noticed it got pushed out and the key takeaway from that is you should always manage access people should only have access to things that they really need in a production environment no single person should have Push access uh it should always be approval based have at least one or two reviews required and expanding that further uh really like just minimize access to only things people really need for example like uh access should not be considered as uh like a pride or something so uh co-founders for example or founders of a company shouldn't really have access to critical things like your emails servers or whatever unless they are actually doing something of it just because they have a higher precedence or whatever doesn't mean they should have access the thing is like people in People Are People no matter what people will get compromised one way or another so you have to have that second layer of security where even if someone gets compromised the impact is minimum so a practical example in the Iranian case uh the bridge hack their employee got compromised that that I can understand like if someone a phishing attack he was like it was a very well orchestrated attack so uh I'll give it to him like okay sure he got caught but the issue there main issue was that that one single employee had access to everything like that one single employees access could have trained 600 million which should not have happened in production that was that that was the key problem in that heart not that deficient happened that is gonna happen you just have to protect from the impacts cool go ahead all right um so I'll just add that um I think uh if you want to look at the different uh what's missing in the web three on versus web 2 is just look at the web 2 security Award chart uh you know you have your instant response teams you have your uh you have a corpsec group you have a production set group you have an app set group you have your uh secured reviewers you have your vendor security audits you have your uh you know all those roles are there because that's how you achieve security you just have Samsung yeah yeah well not right now but Sam doesn't see it yeah yeah Sam doesn't scale though right yeah yeah yeah and I and so I think like if you want to know what you're missing like look at the Delta between your org chart and you know any any well-established web 2 company so that's an interesting question then so in the various life's lifestyle like life phases of a crypto project what needs to be internal versus external right so obviously everyone goes out gets an audit it's kind of a commonplace thing um I still have audits right I'm the co-founder of an auditing Marketplace so we love audits but it's not the the pansia so I guess core to you first and the rest of the panel like when do you need to hire an internal Sizzle uh we're starting to see later stage projects higher Sizzle but is this something where you know you raise Venture money and then on your use of funds like employee three is an internal ciso uh well uh I have a bias that yes this place maybe playing two actually but uh you know I think I think an under underappreciated property of security is that it's uh it is not something that's added on later it is an emergent property of your design and so if you don't start from the beginning thinking about security you're not going to have security at the end uh and I think that like so yes I would I would argue that you should have them early sorry yes I would like 100 agree with that because um security is really about like the whole system and it's like the social system it's the people system it's it's how you interact with your community uh it's how you hire it's who you hire it's how you vet them um it literally touches like everything that you do um and if any one piece is like not secure that's an entry point and the thing about specifically like these crazy hacking groups that are coming out of North Korea and other locations is that uh let's say that they just get access to your slack right so it's like your internal slack you got like 20 employees right they will then use that just that access and they will sit there for like literally months and they will read everything that you do and they will understand how you operate not creepy At All by the way it's super crazy uh and they will use the information they glean there to then like escalate to fish you Spearfish you via email or Spirit issue via telegram or LinkedIn DMS or whatever they whatever you use they're gonna get you and and they just over the course of like months and even years they'll just like keep going and going going um and so it's like these little things that we don't take seriously because you don't have anyone on your team early that's thinking about security that is the problem right and the solution is not to get a freaking security audit I'm sorry I love security audience no no to be clear like it's a it's a secondary check yeah that's for just a secondary check yeah audits are not going to solve your internal security problems for you projects have to solve that yeah and if you especially if you hire like a good security person early on they will uh indoctrinate your team as it grows and they will uh design your team right your org chart your organization your products how you work right they will help design that from the get-go and therefore every layer of your sack as it gets bigger um as the responsibility gets spread out across more people as you scale up as the financial incentives get greater you're just gonna be in a way better position and I I really cannot emphasize enough like what Corey said like you cannot add Security on after like it's not a thing you can't be like oh yeah we'll just be lazy now and then we'll like fix it later that's not how it works like security is a core part of your entire organizational design your product design and your system design make this a little spicy so oh let's go I think like these guys are just securing their jobs right now but in terms of I mean to be fair we all are but you know yeah but like just being practical if you are just starting out I don't think a second employee needs to be the security guy but what needs to happen is your first developer needs to have a security mindset your first it hire that will probably happen when you are 10 to 25 people that needs to have some security background how do you qualify that like the security mindset when you're high so that's a great question yes it's not something I can like uh like Define qualities or something but if you look for me if I look at someone's code the way they are writing it the way they are documenting it I can tell like it's just intuitive that this guy is thinking about security uh if you just talk to someone like look uh ask them their priorities uh see how well tested their code is uh even the code syntax and everything it will tell you uh how serious they are about security uh so yeah your first developer have someone with a security mindset uh let them train others then your it guy probably will be your next time get someone with price security experience but does it need to be a dedicated security people and then will you scale Beyond maybe 20 people that's when you should start looking into a dedicated security person for your own before that just have internal people with security mindset plus hiring or not hiring but like just Contracting external service companies to provide you with security help external companies they don't only do audits they will help you with internal security as well because the thing is if your org is below 20 people the security guy won't really be utilized fully unless you are someone like FTX with like 10 uh 10x Engineers that's pushing the whole code out then it's a different story but talking about normal logs yeah security people can come in a bit later maybe the 20 metric is there for projects at the application layer yes okay obviously l2s and I just want to chime in on that that the point that you made about like um the security mindset like what is the security mindset um I think one reason why web3 sort of struggles with this security mindset a lot of times is that we are like so generally optimistic and like Visionary and forward thinking and we are thinking about the potential and we are like thinking that nothing is impossible and so so much of our time is spent in this sort of idealistic future world that we're trying to build and that's necessary like we have to be that in order to build these crazy things that we're building uh however um like I would say that I have a security mindset but I would also say that I don't spend most of my time thinking about like the potential of the future I spend a majority of my time thinking about like how we're gonna it up so that's like the security mindset on that note how do you ensure compliance then right I think it's something we talked about before is like getting people to actually do the fundamentals even simple things like MFA being enabled right or um Corey would talk about this you know endpoint detection and response actually being used and if people aren't using that then you kick them off the network how do you how do you get people to actually do the basics so like when we were starting up right like and I I Grew From Me and My co-founder to 20 and now we've got uh like sort of over 100 on the core team uh we got like there's a thousand at consensus right now um it's a when we had like two people three people and started bringing people on the thing that we did from the gecko was we uh we had good documentation about like what was required and what was expected of our employees uh it talked about like what they needed to do and it need they needed to do it across like all of their accounts like there's no personal professional split in crypto so like you have your personal Twitter like that needs to be security if you're gonna be a part of our company um the other thing is a lot of these services like GitHub you can force like if you want to be a contributor to my repo like you have to have your MFA on uh we bought everyone uh multiple like all the Little Hardware doodads right everyone has a ledger everyone has a tresler everyone has uh all all of them the Google ones the the the yeah the Yuba keys and the Google one the Titan one that one was that one was pretty slick uh still go back to my ubiky all the time though um and these things right from the get-go like like the first employees no matter what um then as you grow you like your procedures are sort of in place um but then your early employees also like establish this like floor of like this is what's expected right and and that's the culture like that's how you establish the culture like nobody is going to step out of line one is already there right and and you call each other out like when you see someone being a little bit lazy when you see someone doing something that's a little bit you know even the founders right I mean it starts oh it wonders honestly I feel like the like the the First Security people that need to be hired are like literally the founders because like I mean they need to have like this adversarial mindset like from the get-go as Madrid was saying it's like I feel like almost like web 2 versus web 3 is kind of like the wrong kind of like comparison because like we're really and just the the business of um security critical software we had this conversation yesterday with Corey it's like you know it's like everyone here is writing code like whenever we you know unchain code is essentially like ring zero in a kernel like there is no safety net you know besides like whatever you're gonna put in place essentially no one is going to do essentially like the security for you and so it's just like coming with this adversarial mindset as a founder and just like thinking through like everything that is going to go wrong just like I was saying is like the most important thing so probably like starts from the top the the culture and the DNA of the company needs to be built around that like there is no way for your company to be successful if they don't have that because like the the Founders need to kind of be behind every single one of these decisions where like obviously security always comes at a cost of other things right like it's always the the security it doesn't live kind of like on its own right like in a corner it actually always is a balance between you know convenience or like you know product features and like user experience versus you know the safety of of doing these actions so I think that like just uh just having the founders uh kind of like come with this mindset like right away is the most important thing is this something that VCS in their due diligence process should include like asking the founders about their views on security um I think that like we don't want to be uh too prescriptive about like the mindset that people have before uh before initially like investment but it's more like as soon as like we meet with them we kind of like do the rundown of everything and we kind of like provide you know insight into all the threats that they're exposed to and kind of like try to really if they don't have it already like this mindset uh just make sure that like after a discount do the first conversation they're already kind of like think about like everything that that is going to go wrong because it's only a matter of time right you were saying it's like it's a matter of price and time essentially so that's that's like how we how we approach this interesting kind of switching from the project perspective to the user perspective you brought this up before what the projects need to do to educate end users on their security hygiene exactly so that's one thing this ecosystem needs to understand that uh at the end of the day security is really about users even if your protocol is secure but nobody knows how to properly use it and everyone just keeps getting compromised then you have failed like you create uh like you use the public private key architecture but you never tell anyone how to secure their private key everyone just copy paste that on Google everyone gets compromised you have failed your mission so user education is super important metamask here is doing a great job at it but we need a lot more of it and to get yeah and to get that like we need to really understand like in this space we need to understand one thing that it is uh not always about getting the uh like getting your mindset uh very crypto focused or whatnot you can be a crypto Anarchist but you have to be practical in the sense stop telling like 60 or 60 year old grandmas to manage their own keys they are much better off using a custodial Service uh not your keys not your coins everyone's favorite logo but if you don't know how to manage your private Keys it doesn't matter if your key is yours or not it will just get compromised so unless you know what you're doing it's often better to just go with a service product just a custodial product uh rather than trying to learn what you don't know uh this is why even in traditional Finance like earlier in the days you saw like bonds and everything were paper based but eventually everything got digitized and everyone just uses a custodian nobody really ever buys shares on anything directly anymore wherever you go you are buying them through a platform through a custodial platform because traffic has understand that it's it's not ex like if you want to scale you can't ask everyone to just manage their own keys or secure their whole things you have to make it much easier but what we in web 3 what we need to do is give users the choice so if someone is tech savvy enough if they want to run their own node if they want to manage their own Keys uh they should be able to do that but we shouldn't be like shaming others for using infuro or other products or like using custodial products we just have to change that mindset and educate people on both ends I uh uh so one thing I would add to uh the ux which 100 it's it's ux is such a big uh part of uh security and helping people do the right thing uh but I think an unappreciated part of it ux is um uh is that it's actually a um you know it's a team game it's it's every one of us in this room every product every project uh that works on web3 uh the uh the the ux is not just how good your ux is it's how consistent is your ux with everyone else uh you know the way users uh the way they learn what is safe what is unsafe is consistency uh when they uh otherwise they have no prayer of going on uh to a website and knowing if this is going to scam them or not because they're gonna have a whole new ux and every single time and maybe this website just does a little bit differently and there's no way of telling uh just you know using your um you know your lizard brain on in fact right when you're on that website uh what your what you should or shouldn't be doing um you know I um uh my favorite uh example of this is uh logging in uh you know on a d app uh you connect your wallet and you know some websites change and like start start treating giving you data about like what's associated with your wallet uh some don't yet or they'll immediately challenge you some will let you click around and then you then challenge you um and you don't you don't have this concept of like am I authenticated or am I not like how like where where is that what is it what is expected of me and all of a sudden you click something you just pop up asking you to sign something and you know who knows what that something is hopefully it's science ethereum but you know um and you know compare that Tool uh passwords uh you know users have three decades of experience with passwords and knowing how to log into a website it's consider every website's exactly the same uh they they know exactly what the experience is they know where they know what authenticated state is versus an unsending estate like that is you you we're con we're in Ground Zero with user education and it's not just how good your product is it's how good we can be as an ecosystem at being consistent and showing the same experience over and over again teach users the right moments um and I think that's something we really need to internalize and start pushing more as standards yeah I was just like I was exactly actually what I was gonna say but I was also going to show Taylor's project because they might not uh if you want to look at good user documentation and education around every little thing go to my cryptos facts and their like documentation it's so good because it like will walk you through like the mempool it'll walk you through what a private key is and all these other websites that are built on ethereum assume prior D5 knowledge assume prior Key Management knowledge and if you're really building for people outside of the ecosystem and including other crypto ecosystems that have dissimilar key structures and things like that you should really spend just a little bit of time invest in a tech Rider and or even just point to other people's pages that give that education for the users and like if that sign like sign this message box pops up I mean there's so many people who don't realize that doesn't cost money that is not a transaction that is signing a message from your public from your keys keep here so like like we're just so far behind in that because the protocol layer can't be the ones to teach that users aren't running CLI Geth and learning that from scratch anymore that's where I learned it but that was way long ago where it's very different now uh so yeah just look at really good documentation and try to emulate that in your project yeah I want to emphasize like the amount of docs that I've written but uh it's worth it because um you will be surprised at like how much people want to learn and so like there's sort of like this track where like we can make the ux better and we could give users more choices that better serve them like we should definitely do all of those things but if something is like the current way of doing it and it's not clear and it's not obvious take that opportunity to tell the user just like literally tell them don't treat them like they're idiots right treat them like they're smart capable adults and tell them what's happening because they will carry that knowledge and then they will actually pass that knowledge along to other people and it's uh it's quite remarkable you would think like the docs I've written like what what what comes with that nobody reads docs like it's weird when my words like show up in other people's like Tweets in other people's documents right people reference myself like I'm like wow there's people really reading this um it's it's amazing I also want to say like back to what Corey was talking about we do need to work together better uh I'm told nadav is in the audience love you nadab nadab called us out recently like now we're getting spicy finally Open Sea called metamaskow because there is so much use or loss happening right now um and a lot of the user loss is actually literally with users of openc who are using metamask and we should be working together better to uh make sure that every single thing that we're doing and showing the users is uh way clearer and we should be in lockstep so that when openc releases a new sighting mechanism or a new way of handling something metamask can actually like you know not show gibberish literally not show gibberish please anything but gibberish um and so we have some like exciting things in the works that like hopefully we'll uh we'll get into production soon that that will ultimately serve not just metamask users not just openc users right it'll serve everyone in the ecosystem but specifically right now it will prevent the loss that we're seeing and we are seeing an immense amount of loss what's the way to solve it is it just point to point interactions between like the metamask and Open Season do we need like industry working groups I know I know that's a can of worms a lot of people don't like that we have industry working groups they fall off time and time again because the new shiny thing comes out but now tokenizer man just like that's just so meditative economically tokenized the incentives to economically there's the incentives repositories of this layered protocol so but yeah no I I you might be able to answer a better tail but all the groups I've seen some of them have worked really great some of them floundered because they're the teams especially the ones starting out but they have a big impact are small they don't have the time to commit to going to a bi-weekly Zoom meeting to look into to that and then there's a lot of competing um what am I trying to say there was like for a while there was even competing ERC standards you know a few years ago up until now with even the basics of how we're doing messaging and signing of transactions across a theory that's a very interesting history to look back on but uh to fully answer it I guess yeah just pretty much direct communication with the teams like if something's happening just telegram them and then don't just leave them for a few days like literally get back to them soon especially if it's like money loss security related that kind of that's where you need to have a dedicated person who's doing at least security infra or ciso or whoever who could be like oh let me grab my lead devil think this is in 10 minutes it all comes back to the point of someone on the team hopefully from the founders side to NASA's Point earlier has to own security right yeah one thing yeah oh good okay one thing I'd like to add is kind of like on this conversation I think that like we should learn from a lot of the the great kind of like uh ux efficient security that like that has been put out I think that like apple is probably like top you know like the the kind of like top company in in this space that like has been not inspections like in the tech space in general uh in terms of like uh ux security and you know if you take something as simple you know I I used to work there and um in the security team and kind of like saw a bit like the mindset from like internally and you can see for example like you know um logging in for example like used to be like unlocking your phone used to be pin-based and you know we all know that like people just like hate to remember things so what did people do you know when they were allowed to bypass it they bypassed it right it was kind of like I think for only 14 of the people used to have uh depend on and everyone else had just like the fully or even before the passwords right password you can exactly one two three ABC or whatever exactly yeah and then the moment they switched to face ID this that actually Rose to over 90 percent so you know it it like if you're thinking about kind of like the mechanism to just like make security easy and just like flow with the the ux people will just like come along with you in the process and so I think that like again we're discussing this with because yesterday I think that like the the you know the the user interface can do a lot more for us as of today you know when we're thinking about um when we're thinking about making obviously like educated decision you know metamask and and other wallets are starting to to put like great features in order to kind of like show you you know what are you essentially achieving by submitting this transaction to the network what is what is going to happen essentially uh ahead of time but something that is super interesting is like there is more to be done you know like you have uh approvals for your nfts these are permissions right similarly for like your uh fungible token allowances that could be managed actually directly within the wallet the wallet should be able to do that just like your phone actually manages your permissions you know like giving access to you know cameras and like Bluetooth and like location services and so on like all of these things are things that we can learn from just you know great user experiences on like you know mobile devices and so on uh and and I'm really hopeful that we're going to see more of these things as part of wallets and similarly for like you know fishing when we see like a lot of people that kind of like get scammed sign the wrong like set approval for all and get kind of like all of their wallet trained even kind of like wallet management should be possible like within the within the wallet itself like you know think about uh you know your wallet essentially just like spinning off like you know generating a new address on the Fly uh you know providing essentially really like allowing you to just like send funds like minimal amount of funds to just like pay for the gas fees for the operation and not have access to any other asset that is part of kind of like your main address right like just trying to think about like all of these things that don't necessarily need to show any complexity to the user which is like help them in the journey essentially of like there's a startup doing that right do you all know about this there's like a thing where when you launch a metamask transaction there's a second window and it takes the byte code and puts it into human English so there's uh hexagate that's working on that there's a second uh redefine we Define yep yep yep so something I would add um so 100 agree uh I think all that stuff's amazing and I think um you know we're going back to you know this whole thing sort of web two right uh I think one of the things we can also take away from the web 2 world is like the depth of security controls that have been implemented um you know no one is um you know going back to right now we're kind of running and bring zero um and uh when you when you sign you sign with your wallet um the equivalent of an operating system like um you know adding those kind of controls would be awesome ways to put some sandboxing some capability based access controls and extra reduction mechanisms um but you know there's more to um there's more layers you can add right um I think one of the biggest areas in my mind right now is um uh and I've learned this from talking to victims of phishing attacks is they don't they often don't even know where they signed the malicious transaction at um they they're they're at a loss um and um there's no introspectability uh we're missing um you know we're missing logs we're missing uh and I think that we're also missing like a really like like key linkage for like on-chain and off chain uh transactions so not only do we need monitoring login on the project side yeah we need to log in on the on the user side as the user side yeah we if we're going to trust you we're going to say your key your your uh your Authority you also owe we also owe them every tool they need to be able to do that correctly and that logging and monitoring is a big part of that yes um specifically one thing that we've like experienced um at metamask that openc has called us out on and continues to call us out on is um for the longest time there were transactions and the transactions have like Financial value and it's pretty easy to like you know like teach users like this is a thing and like there might be some bad things that happen depending on if you click this button and then um now a lot more stuff is just off chain right or it's not on chain until after a certain period of time if the order is matched or whatever right um and because of this like super early assumption that like everything is there's a key and then you have transactions and the transactions go on a chain because of that super early assumption that we made uh we did not see this coming we're now a like just a signature right just like an off chain signature can actually lose you like all of your nfts or all of your tokens or allow permission somewhere else right and so now we're like playing catch up while open sees like sprinting ahead like making sure that it's usable making things scalable we're playing catch-up to try to like map back and like in reality we should have never ever treated a transaction like an on-chain transaction signature differently than a message signature and because of that very early design Choice like at the core architecture level we are now like remapping everything and it's like uh it's just a it's a stupid amount of work to be honest and it takes way longer than it should and it's especially painful when users are are losing money because of this every day right like when we are hearing about celebrities losing their nfts because of this thing that like we know should be better that is that's horrible um I also want to comment that like um my talk tomorrow I'm gonna be diving into like some of the early early choices of like wallets but also of the protocol itself uh and diving into like a lot of the stuff we're talking about here because I think that the the biggest sin of it all is that it all traces back to this like private key and this private key gives full authority over everything right like that's the world that we've like like started with and like fundamentally I'm so hopeless like it is so broken like why is the single thing that you can't change you can't do anything with the thing that grants like so much permission and I think that like um we are now seeing that like at the token level we're trying to like Implement permissions and restrictions but ideally like that would actually live like out the very core layer right where like the first thing the thing that holds your stuff is actually like a default like you don't do anything and the first thing you have to do is Grant the permission and then you can take the action um so yeah come to my talk tomorrow uh we'll we'll dive into this but you know I think that this is why I really want to emphasize like have conversations with like the people that are around you learn from your users listen to them like the most valuable thing that I've ever done is have literal one-on-one conversations with people who've lost all their money and like just ask questions and let them talk because you will learn so much about where your product is failing um and you will be able to serve them so much better if you carry those stories with you it's not super fun but it is like the most valuable thing that I think I've ever done uh to this day you agree with that uh on on the bed dot private key search I completely agree that refers back to the point I was saying like uh theoretical security is one thing uh people have built products that are secure in the best case but if they are not practically secure if nobody knows how to use their product if nobody knows how to manage their keys and so on by just failing at your goal and then the other thing on monitoring and alerting that's another super important bit so not just about users but about your product as well if your Bridge loses 600 million you shouldn't have to wait six days before you realize that so definitely always have and that was because of no monitoring yes yeah someone literally message them on desk or hey something's not right so yeah anyway you should you should have proper alerting and monitoring but yeah also don't go too overboard don't start logging private keys of users you have to find the right balance there uh but all of that aside I want to like take a step back and like say we just like never talk really talked about the title which is web 2 versus web 3 same or different because I think we all just assume they are the same web3 just sprinkles a few things on top of webto security but if it wasn't clear like at least in my opinion they are basically the same things Spencer here has been trying to guide us towards a bunch of I think 20 questions or something he prepared if we have managed to get through maybe three no just wait and turn on random spree yeah but yes great job yeah speaking of a curveball question so uh we were sharing more stories outside um yeah there's a reason why we're not going with the prep questions uh so Taylor you're you were sharing one about response because you know obviously after you do monitoring to logging oh crap something happened right maybe you were uh you know what do you do next right so what are the Lessons Learned for that so um this piece is decentralized and a lot of the teams are there for fully remote uh we use telegram we use slack we use Discord we use signal we use like there's like 8 000 chat apps I don't know about you but like for me uh my notifications are basically like off because otherwise I'll just be like constantly inundated with information and I wouldn't be able to think um however if something happens like how do you let that person know that something happened so for example this one time Hudson was giving a talk and you got Sim swap while on stage uh and The Sims offers actually talked to his wife uh and I think told her that he was kidnapped because they're because they're Sim sloppers um and because me and husband have like a relationship that goes back and because Twitter like immediately alerted that Hudson was uh was Sim swapped and was like damning asking for money because of that like I had the actual sort of like in real life Network and the actual phone numbers in place to be able to like contact right same thing has happened with my own team like I don't know having each of those phone numbers and making like old school phone calls that break through like the Do Not Disturb mode is like critical almost every single like incident response includes like tracking down the person who has the person's phone number who has the critical access to stop the thing yeah do two things number one I think both of you are on a list so like if I call you I go through Do Not Disturb mode right exactly same here so like I have that set up yeah we have that set up yeah yeah so like if one of us because another time someone posted something on my Twitter because I had like an API key out there anyways the second thing is when I got Sim swapped and as soon as it got on there I I walked I went back home when I was everything was fixed there were 10 very critical group chats that just said Hudson Sim swap don't listen anything he said Hudson swim stock don't listen to anything he said and my startup at the time open Innovations immediately shut out all my access cleared it cleared it completely the ethereum foundation in less than 10 minutes after the Sim swap happened clear had a round table discussion a live War Room Zoom call and got all my access taken away in minutes did you guys have a plan for that or just kind of uh we did have so by that time so I talked earlier about how not great the EF stuff was by that time we'd improved so much we had a whole incident response plan ready for this for the most part and we had a dedicated CSO role type deal that we have kept over the years and years and years and we have a great one right now who replaced me uh after I left or not before I before I left the EF and I was doing other things but either way yeah that was one of the scariest things because like the cops were called my spouse is crying one more thing and this is something just not related to this exactly but um something I'm really proud of little it for my spouse wouldn't I finally got a hold of her the first person after resetting my passwords and I went to the like you know the phone store I called them and I said hey this is Hudson and they were crying and they were like I'm here with a police officer I know this sounds like Hudson I'm going to give you two questions that only you would know and I was like thank you I've taught you upset oh my God so uh so the day to do it was me and everything worked out from there yeah yeah I I just wanted uh like really high at one point uh you had there which uh well is like having the plan uh but I think a key is practicing the plan uh rehearsing doing tabletop simulations making sure that uh uh what you know documenting where the failures were white what was hard and smoothing it over before you actually need to do it in uh in real life uh actually one of my favorite stories of this was uh we had a simulation where we had to disconnect our Corp from our prod Network previous company um and we had one room that we could get into but that room kept getting smaller and smaller because you know we never need that room uh and then when we in our simulation we did uh we realized there's only one ethernet Jack and we had no way of getting all of us into that room to be able to connect to prod and to keep it up keep everything up and then like so we're like yeah so it's just one of those like there's it's always the thing you don't think about that fails you um in an incident and you want to know that before you actually before it counts something uh you know sweating training versus bleeding and War yeah another another thing I think that like is important to to take into account is that um a lot about um incidents response and like Incident Management is about time right like time works against you the person is you know someone or like some entity some group already kind of like perform like found the issue and kind of like try to execute it so your role is also to kind of like expand as much as possible the time window that you have where time can kind of work for you right it's kind of like and and you know we kind of like always discuss these things well like you know uh governance protocols and like you know uh Bridge security it's it's about like you know time locks and just like freezing funds for like a specific amount of time so that like you actually give yourself some kind of like breathing rooms and kind of like your teams to kind of like go over the logs go over like everything that happened understand exactly like is like is this something legit or not if this is not legit you know obviously if you have other mechanism in place great otherwise kind of like get ready for you know kind of like freezing assets across you know chains and so on so I think that just like making time work for you is also one of the most critical things that you that you can do and uh a lot of people have been kind of like theorizing about this uh it's um how much time essentially like the uh bad guys need to spend in your trenches um in order to kind of like you know get the assets and then get away with it so that's that's something that is very very important and obviously you know uh will never uh emphasize enough that the need to kind of like you know baking time whenever you have like a new governance proposal and just like uh whenever you have like specific thresholds or like limits that are hit you know making sure that um these things like take time and that you have like some some freezing periods where you can actually like cancel operations and just want to hit on this point again uh this is a this all the stuff layers together right you hold on your systems you may uh you you make sure you patch systems and you find them uh and you've audit them and then but then whenever things do happen you time box along the attacker has in your systems that we have you can quickly respond like these all the stuff layers and that's how you get a great security defense and depth on that one little bit so with incident response I agree time is very critical so what everyone needs to do is have an a year plan have a disaster recovery plans literally have playbooks of every scenario you can think of of anything that can go wrong have playbooks of what to do so that even if like your out sake someday and some other security or someone else in the article doesn't really know as much about the product can just see your playbook follow the exact steps and be all good and like you need to practice these playbooks as well it's not just about having the content out there you need to practice all of these playbooks and everyone should be aware like what needs to happen when needs to happen how to follow the Playbook and only then you will be able to properly respond in a timely fashion uh and another bit I would say like coming back to the swim swap thing so there I wanted to add a little thing so switching swaps in fact not just Sims wife but any product where the support department of the company can uh like do whatever with your access or wherever like you don't have control on your product which is most of the webto products don't depend on either get something like highly secure or don't depend on the cheapest one so for Sims almost all of the public providers like they can just same swap any of their support engineer or uh whatever can same swap you so never use mobile numbers for 2fa or recovery or anything permission what do you think of a funny that's the one uh supposedly non-sen swappable cell phone provider I'm not Lumen into them too much but they're like a regular I mean they're like legal requirements for them to like allow specific operations and like porting you know like you're lying to something else so like so long as like this operation is like required by law from from these providers there is an entry point right and it's like this is just like a matter of like how much Social Engineering you can do like people are saying that like Google 5 was actually not sim swappable because there is no one to talk to on the customer service but you know it it could happen you know like no one is kind of safe despite because even the more secure cell phone providers yeah exactly like you know you want to aim for obviously you know the regular kind of like non-sms to fa base but like if you're if it is mandatory go for like you know Google Voice number for example like there's no sim to swap essentially so you know go for for services that do not have like the same associated associated to it yeah and back to like the whole stack right and like layering this on top right get the most secure Sim and then have nothing like to have nothing on that Sim right or like only the the two services that like actually have to be on that Sim and like have that video number that isn't really your primary number right like all of these things layer and like that's and then you're not signing up for like marketing databases yeah uh for this conference and this is like this is why we talk about like the culture right we talk about the organization we talk about the people and like how important that is from the ground up it's like every little decision that you make has to be the most secure one right and so that you have all the different layers on top of each other so that it's less likely that things happen that are really bad but even if those things happen you can mitigate the loss you can respond to loss quickly uh we are being we're getting ushered off the stage thank you for coming love you all thanks everyone