Tales of a sleepy dev: Lessons learned building DeFi for 3+ years - Nenad Palinkasevic | DeFi Saver

[Applause] hello everyone I hope you can hear me correctly so basically this is going to be oh let's stand it up yeah this is going to be my first non-technical presentation so I'll try to do my best to to actually um present to you that the correct information and something interesting so the the the name of the talk is a little bit weird was what does 3p does mean and are we going to talk about how trip is an important important thing in development kind of and kind of not so basically the the term Devi sleepy or or sleepy Dev is one of my favorite memes in the the ethereum ecosystem it started a few years ago when one of the team members from one project I won't name them it was a legit project but one of the team members got scammed and hacked and uh and the kind of excuse of the team was that the dev was creepy so that's kind of my favorite excuse every time something's not really working or I make a mistake that was sleepy right okay so basically I wanted to share a little bit of my experience as someone who started in 2017 how the how the ecosystem looked back then and how kind of we managed to evolve to what's right now in the in the D5 ecosystem so when I first started developing on ethereum obviously there there weren't much going on you as a regular user couldn't really use ethereum for uh day-to-day Financial stuff you have to go to XX to trade and stuff like that mainly there were icos nft and crypto kit is starting to pop off but as a developer as someone who wanted to use the daily ethereum there there weren't really much opportunity for me to do that now I'm gonna share here uh basically this is one of my uh favorite chats because it actually is how the the term D5 originated so in the spirit of D5 the the name defy actually comes from a telegram group chat where a few early members who were starting to develop Financial applications on ethereum wanted to kind of interim named uh named movement of financial ethereum applications there were a few different names proposed but D5 was kind of Chosen and the community accepted it and our application is now named if I server so like hopefully it will not change in the future okay so when we look at the the start of D5 and once a couple of different applications we're starting to pop off one of the major ones was a maker Dao as that was the first decentralized stablecoin project on ethereum and obviously for trading and for people to to maintain stability and to actually to maintain the value and still keep their funds having a decentralized stable coin what was one of the first and one of the most powerful Primitives for everything else obviously Landing protocols like compound were the first ones to to actually combine Port Landing which was much more easier and much more ethereum friendly before that there was each land which is a precursor to Ave but they did peer-to-peer landing and because of that they didn't really catch on one also application I wanted to mention uh pulled together was also one of the first apps that actually built on top of one of those protocol Primitives and it's something that didn't just recreate traditional Finance but kind of was a nice little show case to to actually see that we can do even better than that uh the the ecosystem back there was simple they were dye as a decentralized stable coin there was a few Landing protocols there was a few exchanges that were decentralized that were starting to gain some sort of liquidity but things were very simple like interoperability and building on top of each other wasn't really a concept yet and so while there were a few projects it was still kind of in an early stage like I explained uh basically on this slide die as a stable coin actually worked quite well there were periods where it would went off back a little bit but the system actually worked directly and that was I guess a surprise to most as it was one of those things that people were always predicting that it will fail uh obviously as a few protocols like compounding maker were starting to gain traction as a few dashboards are starting to pop off and actually enable users to have better UI and so on the total very locked in D5 was starting to rise at first it was basically laughable in comparison to traditional Finance but even then it was starting to to actually seem like a like a real thing so now I want to mention like the bad things that started to happening um every time you actually start to have users you start to have problems right so one of the reasons I actually chose the bzx or the fulcrum hack as it's probably one of the first hacks that happened that involved a flash loan and it actually involved price manipulation that's something that further down the line we we saw a lot happening in D5 what's really interesting about this is actually another important point is that it happened twice right so they they managed to to fix the bug that they exploit when it happened but then they kind of uh too soon they didn't check other cases and two days later they got hacked again on a similar exploit that's kind of one really important lesson is that you have to take your time in response to to any sort of exploit or trouble one also think that it's important to mention if we actually analyze a little bit about this hack why it actually happened one of the reasons is that the that the user actually could grab a flashlight that was their new feature that they just released and the other the other reason was that they were using a decentralized price Oracle which at that time was uh kind of their selling point why it's better than compound which obviously the centralization is in in itself a really good concept but at that stage D5 was really early and we didn't know a lot about how to properly grab prices especially from on chain so the attacker could kind of use this to these two features to manipulate the protocol and hack it if we look at compound which didn't have both of these features it's something that we can kind of conclude that Simplicity and not kind of rushing things so not really occurring with a feature set you know Financial applications is really important from security and Longevity wise one one of the protocols I would like to mention which in my mind is always a kind of an example of protocol that did it right is uniswap we probably all know and have used uni swap they never really had any serious security issues and obviously they are now one of the most powerful D5 Primitives we have a couple of core ideas that units will be implemented was that idea that in the first version it was the simplest possible code it was maybe a hundred lines of wiper code the developer who wrote it wasn't really even an experienced developer but it was one simple idea it didn't have any extra extra things in it right a couple of other reasons why that turned out to be a really powerful it was really gas efficient and it was permissionless so at the time as Eunice grew up those kind of Concepts really made it possible for for them to to explode as one of the the biggest dexes out there after that after they actually had uh usages and after they kind of verified idea obviously they started iterating and made the version 2 and made version three both versions uh really really good versions but they kind of listened to the users and made made kind of a minimal subset of changes iteratively and that's why I think they're lasting so long and generally don't have a lot of security and Bug issues as we now continue in the in our journey through like early stages of D5 we see now like basically a Cambrian explosion of protocols we we now essentially have recreated a big chunk of a traditional Finance finally the axes are starting to gain real liquidity and even larger Traders can now use dexes maker is just an example a lot of different protocols but maker also did their upgrade where at first there was PSI single collateral die then they switched over to die multicolateral die so that was also like one powerful thing that's happening as the the stable coin was starting to to be more stable and gain more traction right now as you might might notice in this talk I always kind of give the kind of the positive and Good Vibes Then I then I talk about some hacks basically what happened for instance in maker it was a so-called black Thursday event it was one of the largest stress tests that defy experienced why this is for me a pivotal moment as I actually think like this is something where after that event people actually started to believe that this system isn't only working when things are going well this is a resilient system which can kind of operate in different markets obviously there were some issues for instance in Black Thursday maker Dao had some bad reputations happening that basically means that because of the network congestion because of all the different uh we have on the side here yeah okay okay if somebody watched Chernobyl they'll get the reference but we had the network congestion we we had a lot of different problems and at the end of the day there weren't enough liquidity and and bought auctioners to to actually properly uh go through the the maker Dow auction and what happened is a lot of bad that occurred around five million die of bad debt was a credit maker at this time that will be you know a laughable amount which is kind of easily filled with something but at that time that was kind of a pivotal moment whether or not the the ethereum D5 ecosystem would survive and how it handled and there there is a few of kind of interesting stories around it because at the time we were one of the main integrators of maker so as as well maker was experiencing a lot of issues we have our automation system which obviously we had all the the issues with our API fairings or the issues with gas prices we never expect at that time I guess gas price which was like 200 gray was unheard of I actually in the smartest contract I had like a sanity check where it wouldn't go over 200 way as I thought that the user messed something up as we know now it can go over a thousand of ways a ways and and so on so why the shout out to some drunk maker whale he's actually kind of a legend early on in the the maker Community Forum he was a person who called who did hold a lot of maker tokens which you have like a government's power so with the government's power he was one of the the main voters and kind of uh he was controlling in a way with with other voters how the system would work he would also a lot of times send messages drunk so the drunk maker whale kind of things sticking while the black Thursday happened uh we were kind of trying to recuperate we were trying kind of to to fix all the things that we saw that were broken but we also saw that the kind of the danger is not over yet that bad liquidations were still happening at maker and we wanted to participate in in those Liquidations in order to actually stop the protocol from from losing more money we didn't really have time to write like a full-on bought for it so I basically on etherscan went through all the maker contracts and figure out how to participate in the liquidation myself basically by eaters can one thing that's that was kind of a broker for me there is that I was poor so that's where the shout out to drunk maker whale goes because I've talked to that person a few times on in a DM and I asked him if he could borrow me some money so I participate in the auctions and he sent me half a million die on my metamask account so that was kind of a big shocker for me that somebody I talked to a few times on a message board would actually entrust me with a half a million dollars so obviously I did return the money and things things went well so shout out to that guy whoever he is he he really helped us a lot there in general that's one thing I realized there even though the the community is more it really is a community of individuals who who are real Believers there and who kind of have really good values behind them as we now kind of continue to look what happened in D5 obviously compound actually released their own tokens and they did like one revolutionary thing they didn't do in Ico or or some some sort of mechanism there actually distributed tokens to the actual users so you could actually mine the compound token which I thought was the stupidest idea ever but it wasn't uh obviously a lot of different protocols kind of continued and there we have like a period That's commonly named the defy summer as there were a bunch of different food tokens offering like 20 000 API people exiting in a few days obviously it got a bit uh kind of scammy and very fast but uh what kind of stock was that new projects could release their own tokens to the actual users who are using them and that way the distribution will be fair and they will have like an institution like they will kind of incentivize the users to to use the protocol also one thing that we got during the default summer or after that is that Protocols are worth protocols and apps are starting to get a lot more complicated that that's one of the most powerful features of D5 is that interoperability like we can stack the the protocols on top of one another and create some some real powerful things but it's also starting to look like something that the the regular user cannot follow and it's something that's maybe too complicated okay so now to to jump over the the whole bull market where I do think uh some interesting things happened but if we look at the ecosystem now just as a snapshot of a few years and then now we can see that there the protocols here are much more mature uh most commonly used protocols we had a few iterations uh had a few security scares they kind of went over the best practices and so on so we can trust them much more in terms of security size size but also just in terms of UI definitely it was much much better we also moved to proof of stake right that was also a really powerful thing because now we have a way a way to bring staking staking eater tokens so lsds liquid staking derivatives in DFI we have a source of apy we have a source of taking which is kind of totally legit because it generated from from ethereum and that allows us also to kind of for leveraging and create powerful things also as I mentioned a few times like network is congested and the network is congested at the worst time possible when you need to send that transaction through so as the whole Community kind of started working on scalability Solutions we now have a couple of really good l2s that are going in the right direction of decentralization and being actually useful now I have talked a lot about how different people get hacked let's talk about the time I was the Sleepy Dev so basically because you never expected it will happen to you but then it does we had a problem where hope like gratefully the the hack was reported by a whitehead hacker so nobody really lost the money but obviously it was really stressful it happened at our time zone it was night so I was sleeping and my colleagues called me because I was still very much asleep I just hung up the phone and just I didn't really remember anybody calling me really the good thing is I have good colleagues so they managed to kind of identify that the problem is real and how we could fix it so in the morning once we woke up we realized that we would need to uh actually white hack Rescue all the funds it was a really stressful situation but we we managed to to pull it through one thing that we did is we did learn a lot of from from that hack uh one surprising thing that happened is I actually thought like this is probably over for us like why would anybody trust us this is such a stupid mistake I can believe this happened and so on so obviously I was thinking like this is the worst thing ever one thing that we did do is we did not say that that was repeat we were actually really honest what happened that we made a mistake and that we would take like concrete steps in in improving our security and we actually did right after that a lot of architecture change a lot of tests a lot of audits a lot of things needed to be changed so things like this don't happen in the future that's basically something I was mentioning earlier so at the end of the day there isn't any magic bullet in order to write a better code in in order to kind of survive in DFI it's just to kind of do the basic things right right so you don't hurry you you move slow you iterate you you try and kind of test out the process before you go out in general you do have to be a bit paranoid in order to survive years and years as a DFA developer as a developer for Finance in general right I I don't want to be too paranoid but like the North Koreans really are out together there are a couple of examples where people were specifically targeted and so you have to always kind of be careful and if you're sleepy or you're drunk or anything your your concentration is not there it's better not even to to just touch the computer if you're in D5 obviously uh burnout looking at the prices things are going to fail tears and things are not going to fail the dynamic environment is great but it's also really stressful so you have to pace yourself you you have to kind of conversate concentrate on the important things and not worry so much about uh price and everything else so uh when we talk about sleep and I mentioned what is the the three p Dev and so on this presentation in general was pretty negative because I talked a lot about hacks mainly because you learn the most from it so when you have a stressful situation like black Thursday or when you have a event where you have an expert and so on those are kind of the pivotal moments where you learn it's kind of uh that saying it's like either you win or you learn and we've learned a lot um the slip trade off I'm matching here so as a developer there's gonna be some sleepless sleepless nights there's going to be a lot of stress or sources with it but if we look actually what we're building I've put here a few quotes where people are actually referencing how they do sleep better which is great because I don't but they do so better so when we look at it as a global as a global system what we're building from three four years back to now we have improved a lot the the original vision of D5 which is a open decentralized system that doesn't discriminate which is a system that works for everybody equally which is open source a system which is much more fair than the regular system that's something we're we're building here and that's why a few sleepless nights is kind of an okay trade-off at least if you ask me so that's it for me if you have any general questions obviously this wasn't really a technical talk but if you wanna do Throwbacks to good old moments a few years back I'm all for it this is like microphone so I know this talk isn't technical but can you can you maybe explain a little bit about what the hack was uh our own hack we had yes yeah uh well it was actually a it did involve flash ones but basically our system is kind of we don't really pull together people's money like in my mind it was always that we don't have a large kind of uh surface of area where somebody could hack us and that's why we started like writing a lot of contracts moving really fast because I always thought okay the way the system was organized people have their own smart wallet like an account abstraction every operation is in the context of the smart wallet there's as aside from like three pages or locking some of this money somewhere there's not a lot of things that can go wrong here one thing that did went wrong was that we had essentially an approval bug because the user gave approval to a contract that contract actually did something with flash loans so it had kind of a function anybody could enter Because The Flash one should enter from that function so the attacker then kind of tricked the contract to pull money from the users who gave approval to the contract that's kind of a simplified version it actually like happened to us it was hard to explain because a lot of people had approvals that they didn't remove so it took a lot of time for us to actually kind of try and find the users who had approvals to to remove it so they're not accidentally hacked a year later I found out we weren't the only ones this is kind of uh a hack I saw because then I was looking at it I saw it like five six times in in different protocols is that that people generally don't think of approvals as like storing value but basically if some some country has approvals to it you can count that contract as like holding money you have to be really careful with it I I thank you for the amazing question do we have any more questions in the audience yes what's the biggest obstacle you encountered during making the devisaver yeah so I would say the biggest obstacle would definitely be I think kind of be living in it during times where things weren't going so well so obviously we started off in kind of a bear Market D5 wasn't really a sure thing we were running low on funds uh VCS weren't so prevalent back then so I think the biggest obstacle was kind of having that kind of core belief like hey things we're doing here other people will find out soon we will kind of that the industry will come alive and that was kind of our major bet with Device ever so we are a kind of an advanced dashboard that combine different protocols and so on so the bet there was there will be a lot of protocols there will be users and so on so that was kind of I guess the biggest obstacle was realizing that it will take time and we kind of need to survive until the adoption comes we have another question we have time for one more question I would personally use this opportunity because OG is like nanads are really hard to find and definitely yes so going back to the hack um you guys how did you like fix it did you did you actually white hack your own contracts yeah and then just put them back can I swear um I don't know this that was really stressful um yeah we basically basically had to everybody who cared approvals and that token uh we had to like white hack it because every I mean every everybody doesn't know but they're constantly Bots monitoring ethereum mempools so we had to be really careful how we're gonna perform the white hack so Bots don't intercept the transaction and kind of steal the the money before we steal it we did make a kind of a really good uh set up there where because I set steal the money it was we because we use a Smart contract as um wallets we actually just for each user we made a separate wallet where once we white hacked we kind of put the money there and only they had access to it so technically we never actually touched the money ourselves but we did perform the white hack so so others don't perform the black yet yeah please is there any future plans for defy saver regarding upgrades that you could share with us okay if I understand you correctly like is there any future features okay yeah so we're I have 40 seconds uh we're really excited about the new strategy system marketing uh but yeah we're really excited about uh offering more and more automated strategies that's a system we developed where we first had only like one two hard-coded operations we can do for the user now we have a really more generalized system so we can build any sort of device strategy specifically for a subset of users so we're really hoping that that's going to be kind of the next wave once uh more and more users come they will want and we'll have a lot more kind of automated features to help users manage so unfortunately we are out of time huge thank you for nanad for being the real OG