# Penumbra: Building a Private DEX with ZKPs and Threshold Cryptography by Henry de Valence

- Channel: [Devcon](https://streameth.org/devcon)
- Date: 2023-10-07
- Duration: 26:25
- Watch: https://streameth.org/watch/yt-JjmOelgfqNo
- YouTube: https://www.youtube.com/watch?v=JjmOelgfqNo

## Description

Visit the https://archive.devcon.org/ to gain access to the entire library of Devcon talks with the ease of filtering, playlists, personalized suggestions, decentralized access on Swarm, IPFS and more.
https://archive.devcon.org/archive/watch/6/penumbra-building-a-private-dex-with-zkps-and-threshold-cryptography/

ZKPs allow transactions to prove their state transition was valid, without revealing anything about what it was.  But this isn't enough, because useful applications require shared state, which ZKPs can't provide.  In this talk, we'll describe how to break this barrier by combining ZKPs with flow encryption, a new threshold crypto primitive, to allow private interaction with public shared state, and how Penumbra uses this technique to build a private DEX.

Speaker(s): Henry de Valence
Skill level: Intermediate
Track: ZKPs: Privacy, Identity, Infrastructure, & More
Keywords: ZKP,threshold crypto,DEX

Follow us: https://twitter.com/efdevcon, https://twitter.com/ethereum
Learn more about devcon: https://www.devcon.org/
Learn more about ethereum: https://ethereum.org/ 

Devcon is the Ethereum conference for developers, researchers, thinkers, and makers. 
Devcon 6 was held in Bogotá, Colombia on Oct 11 - 14, 2022.
Devcon is organized and presented by the Ethereum Foundation, with the support of our sponsors. To find out more, please visit https://ethereum.foundation/

## Transcript

foreign [Music] so looking forward to the next talk uh who is who is here for for zkps who thinks who thinks zkps are the greatest things that blockchains uh ever enabled in the space in terms of getting it funded all right all right that's just my personal opinion though but I think zkps are actually the the nice thing that nobody expected coming from cryptocurrencies but they still happened uh so next up uh we have Henry de valence uh the founder of penumbra and he's going to uh to be talking about um their experience of building uh a privacy oh they just disappeared in front of my eyes because the talk is starting now but uh Henry's building uh privacy preserving decks uh using zkp so please put chance together for Henry all right um yeah well uh great to see everyone um so I'm gonna be giving a talk about how to build a private decks however just as a kind of a pre-warning um if you're really interested in all of the sort of details of ZK proofs that actually won't be covered in the talk and the reason is that I think part of the problem with deploying zkps for uh privacy rather than just for scalability is that there's some interesting and fundamental challenges in the state model so that's actually sort of the whole summary of the talk so to get started for context so I'm working on a project called penumbra what that is is a private proof of stake L1 inside of that L1 we have an interchange shielded pool so you can take any IBC compatible asset moving into this chain and that's the Privacy boundary so as it moves in it's recorded in this shielded pool and the cool feature that we have is this private Dax that's integrated with that shielded pool on top of that privacy preserving base layer however um I'm not sort of here to just you know chill the project I'm here to talk about the kind of technical aspects and specifically what are the challenges to actually building this system so the first challenge is oh sorry uh so just as as a bit of motivation right like why do we actually care about having um a private decks in the first place um the the thought is that every Market is also a marketing information when we talk about price Discovery for instance you can think of that as being a kind of decentralized computational process that as a side effect produces some information about prices but that means that information leaks are value leaks when people talk about Mev and have these you know dashboards of here's how much value is extracted from people who are trying to do trading really that's just the sort of first order symptom of this problem it's not even if you fixed Mev in some way um you still have the problem that you're you're leaking information about what trades are doing what trades your account has done your entire history of your whole trading strategy all of that information is valuable too it's just harder to make a dashboard of it so the the thought here is that privacy is a really interesting thing for uh trading because trading is a context where having privacy allows you to unlock greater Capital efficiency and as somebody who's really interested in privacy you know for its own sake as a value as a human right Etc um this is a really interesting and exciting uh opportunity because it means that there's the potential to build a private product that can actually out compete the non-private ones because it's private so we we get out of this mode of like oh well we'll you know have people accept all these trade-offs to use the private thing no we're going to build something that's that's better because it's private so right I said I was going to talk about the challenges and the first challenge which is the most fundamental is actually the state model so if we consider a transparent chain like ethereum the the Paradigm of that state is global mutable State there's like one big ethereum state and every transaction is applied sequentially and it can do whatever mutation to that state and as it's doing that mutation effectively it's taking like a global lock over the whole state doing some changes then you do the next transaction and so on but in order to have a shielded chain where the user state is not public information we have to change the state model to a model of immutable composable State this is actually kind of like historically speaking this is sort of rewinding to a Bitcoin style sort of utxo model but I think utxo is a little bit of a loaded like baggage Laden term so I prefer just having this idea of a state fragment and instead of having one big Global State we're going to have a tree of State fragments which are immutable and each transaction is going to consume some State fragments produce some new ones and put those you know append those into this tree the reason that we want to do this is that that way we can make the state transitions private and the move to do that is you replace all of those State fragments with cryptographic commitments to States and then in your transaction rather than explicitly declaring the plaintext values of those State fragments you can just have a ZK proof that the state transition was valid and here are the commitments to the new states that I produced okay so this is all cool and that's why everybody's like using like private blockchains all the time no why is that because that diagram isn't quite right in fact what you're submitting to the chain is this ZK proof pi that represents the state transition but what we've actually done is we've moved all of the execution off chain along with moving all the user data off chain right but now we don't really have any way to access any shared State because when somebody's submitting their transaction effectively every transaction is is its own little micro Roll-Up and what I would claim is that the key problem here is is one of early versus late binding so what we have in this sort of ZK model is early binding every transaction has to have this fully sealed complete State transition but that's not actually what we generally want right if you make a trade on uniswap you're not going to sign over here's the exact State of the Union reserves of this contract here's the exact outputs that I'll get because the only way that you could do that is by sort of having the entire world stop while you you know submit your transaction right what we want is the ability to have a partially filled transaction where we can have some placeholder values where when the transaction is executed the the shared state is going to go in that spot and then it'll determine some outputs um so in order to make this work with the sort of ZK model where we're keeping all of the user data off chain we need to have some better concurrency model for shared state right fundamentally the problem is that if you're doing this off chain execution you need to have some kind of like exclusive control over the the data that you're executing on and you can start with the kind of oh well we'll take a big Global lock and you can make the locks you know smaller but the the more interesting idea I think is what if we try to model concurrency with message passing rather than locking if you look outside of blockchains that just kind of like normal you know high performance High concurrency software systems um you can do better by having finer locking but a cleaner and more scalable approach is to have different parts of the system executing independently with their own State and passing messages back and forth to each other so what would that look like in the context of trying to build a privacy preserving chain well we could try to think of this kind of actor model for chains where a transaction rather than synchronously doing a contract call where it locks some state does some execution and then finishes and releases the look we could say that the transaction is going to pass a message to a contract and now we're going to have each contract execute once per block but as it executes it's going to receive the set of all of the messages that any transaction passed to it and it can process those all in a batch and depending on you know the specifics of that contract logic in some cases you know you could combine all of those messages into one right like you could do a batch swap which I'll get to later on in the talk but also you could imagine say like an auction protocol that orders these messages by the the bid amount um and the key thing is that you're not doing any execution at a higher time resolution than is actually provided by the underlying consensus right when people talk about transaction ordering a lot of the problems arise from this mismatch between the economic mechanism and the contract which is trying to execute at this sort of faster than consensus speed and the actual you know consensus ordering guarantees that are provided by the blockchain which happen block by block right like if you're agreeing on transactions in in batches like maybe you should also be executing them in batches um and on the other hand our our per user State we can still have this sort of off-chain uh execution but now the execution is going to be asynchronous right when I consume my state and I pass a message to a contract I can't compute my outputs yet because I haven't got a message back so I need to somehow record this sort of intermediate future that represents the eventual result of the computation that I'm going to be doing but I can do all of the uh all of the computation that is only touching a particular user State can happen off chain uh privately in ZK so the really interesting thing about this model is that it actually simultaneously unlocks both scalability and privacy because all of the data that doesn't need to be having any shared execution is happening that execution is is happening on each individual user's client and the on-chain data is limited to the minimum uh shared computation and it's executing in a batch so you can potentially have a much more efficient implementation so to see kind of how this works in a in a diagram uh which I think it might be a little bit more clear we're going to start off with our first uh you know somebody's preparing a transaction they're going to perform a state transition and produce a ZK proof of it with their private inputs and that transaction is going to send a message out to this contract what it's going to record as the output of that initial transaction is actually a privately minted state nft which commits to the Future that this computation is supposed to be uh representing so now this user has this in this diagram this is the the Diamond is this nft that records the kind of intermediate state of their per user computation after they send that all out to the chain it's included in a block the chain is going to send a message back by publishing data to the the chain or the contract's going to send a message back by publishing the data to the chain and we take that message from the contract combine it with our intermediate execution state we're going to burn that state and inside of this ZK proof we can now prove that we've you know modeled the correct future with the correct uh you know same input values that we had committed to in the first transaction and now we can privately mint some outputs so this is this sort of big picture diagram of how this sort of async ZK execution can work um in principle um and we'll get to a concrete example of how that works out for the case of doing batch swaps but before that I just want to get into this second challenge which is um once you have an idea of okay we have maybe a workable State model but what is the Privacy model right and the the claim here is that basically every useful blockchain revolves around public shared State that's why you know for instance ethereum is valuable because it has everybody's state in it and it's all there um so the idea of like oh well we're gonna just like make everything private and like everything will be completely private doesn't actually work because in fact like the reason that we're we're using the coordination tool is so that we can coordinate nobody wants to use a market where they have no idea what the liquidity is or what prices they might get or you know what the volume is um and all of the the sort of big breakout successes uh of of useful contracts have the contracts uh public shared State acting as this kind of universal counterparty so you don't need to go and find some some person to interact with you can just interact with the chain so we need to have public shared state but you know maybe we want it to be private so what what does that mean exactly right this question the question is like how do you allow private interaction with public shared State um we want to have uh some kind of notion where the individual users transactions have privacy but all of the aggregate state of the chain like you know all of the aggregate Market data is public but the individual users trades are not so here's two basic strategies of how you could have a private interaction with public State one is splitting flows and the other is batching flows when I say flow the reason I use that word is that I think of um you know value being in in different sort of pieces of the chain State and as people do things you're going to have flows of value between different portions of the chain State and we want to have some kind of system where people can have privacy about what their contribution to that specific flow is but still have a sort of overall um a transparent aggregate so looking at these two in turn for splitting flows the idea is that we're going to start off with some users value that you know this maybe this is an amount that they want to trade they can split that into randomized sub amounts and then reveal each of those sub amounts in distinct transactions and then reunify them later so this method only works if you have a shielded base layer if you have a transparent based layer then this like is trivially broken because you just like look at what the account was and then you can see what it did but if everything else is private and you're only revealing like here is a specific amount that I'm sending to this part of the system like this is a a public delegation amount to some validator this is like an amount in in one sort of randomized sub trade this might be okay and it's a it's a fairly practical and simple thing however it's it's not really ideal so the other strategy which is more interesting and more cool is this idea of batching flows so if we know that we're already going to be processing uh uh the contract state in batches it would be nice to have a way to batch up flows um we do this using a construction that we we're calling flow encryption which is effectively additively homomorphic threshold encryption uh with sort of end-to-end verifiability once you've tacked on you know six different properties you might as well give it a distinct name but the idea is that each individual user with their you know distinct contribution to this flow can encrypt an integer amount uh include that in their transaction and then the validators can sum up those encryptions and decrypt only a batch total um so that there's the batch total and the idea is that now you can use that as the input to some public on-chain computation but still have long-term privacy for individual transactions you know as long as the the batch size is is big enough as an example if you're trying to have um you know a a delegated proof of stake system you could make it so that oh the validator weights will change you know in each Epoch and we'll just sum up all of the the delegations or undelegations in each Epoch and only reveal the net change um so these are sort of the the two pieces let's see how this kind of works out uh as a as a concrete worked example um which is doing sealed input batch swaps on penumbra um so looking first at the private State part right remember that we have this model where we explicitly segment away the per user private state from the shared public state we're going to start off with okay here's the the private input for this trade that a user wants to make they're going to encrypt that input include that in their transaction the rest of the transaction right is is totally shielded because we have this shielded base layer so there's no Public Accounts or any other metadata in the the transaction and what though meant in that swap transaction is a swap nft where the the asset ID of that uh nft is a snark friendly hash of their input trade amounts the trading pair their you know the address that they're going to Mint the funds to some prepaid fees uh and as this transaction that they submitted is included in a block it gets batched together with the others decrypted you send that into the Dex engine which resolves all of the trade intent and publishes the output data into the chain State and now that user can consume their swap nft which remember is modeling this sort of future that resolves eventually to their swap outputs and produce a proof that uses these uh public back swap data as a public input and their swap nft is a private input to privately mint their correct prorata share of the batch Swap and so that that means that that individual user's output can be minted completely privately and as long as you have sufficient uh uh sufficient volume in in the batch um you can get long-term privacy for individual users trade amounts now if you're on say like a thinly traded uh pair maybe you end up in a badge size of one but in that case you can use this uh sort of randomized splitting technique that I mentioned earlier to make effectively a kind of like randomized t-wop of your trade so zooming out of the the private State part to how this works on the public State Side um we have all of these uh uh all of these swap transactions that have been included in the the block we have the Dex engine group all of those inputs by pair we do this batching and decryption but now at this point we we don't just have you know a single swap we don't just have a single Swap transaction on a single pair we have a complete picture of here's all the trading intent uh for the entire chain in this block and we also know that we only have to run the Dex engine once per block because we're in this batched computation model so that means that we can actually be you know considerably more sophisticated in how we resolve these trades we can load up the entire liquidity graph of all of the assets on the Chain sort of put the trade intent into the correct spots on that graph and then compute uh some optimal resolution of all of these trades simultaneously so you can do some kind of global resolution of the trading intent do optimal Arbitrage and this is computationally tractable to do just like in consensus because you know that you're only doing this once per block so you you end up in a model where rather than sort of having to call each contract you know potentially like thousands of times a block you just know that the chain is going to step in discrete time from one set of consistent prices to a new set of consistent prices you can handle all of the uh newly added or removed liquidity also in a batched way right so you first add all of the newly created liquidity then you resolve all of the trading tent then you are up all the prices to be consistent with each other then you close all the you know closed positions and this also lets you as a side effect have like single block uh jit liquidity uh with the kind of like batch semantics so there's a lot of really interesting pieces of Market design that get unlocked by this model um so that's basically the content of the talk if you want to play with this we have some weekly test Nets that you can you know play around with here's a bunch of links um so far uh the the big last sort of piece to build is the the actual Dex engine but we have a toy uh amm that you can do these shielded swaps against uh on our test net so that's a talk if anybody has any questions happy to get into it [Applause] uh just wait for the mic please I'll run to you with the mic thanks for the talk uh just curious so have you found a tight Bound for how much of a loss you have when you provide batching because if it's the content function Market maker there's a loss in what sense uh so you're going to get an average you're smoothing it out in that sense of speaking so you can actually have a spike in a block you're just averaging so have you done an announcements I did some analysis so I'm curious if you have done that so I guess the question that I have is like a loss relative to what right assuming no private so two volts same block private versus non-private uh uh I'm no okay do we have more questions yeah they're not sure who was first sorry just a quick one thanks very much for the talk this is fantastic to hear and to get uh this content and this all right quick question on reverts the whole batch reverts all right so I missed that on revert the whole batch reverts if a transaction reverts oh objection is not valid yeah so um it's possible for instance that someone could submit a transaction that say you know they submit a trade for a trading pair that doesn't have any um liquidity on it right so one of the outcomes of the the batch swap is that the the trade can fail and in that case rather than having people privately meant their Pro rata share of the outputs they privately meant their Pro rata share of the inputs they get their their funds back uh yeah so the whole batch goes goes at once so like technically I guess you could have some kind of griefing attack where someone is like continually locking up capital in trying to like dot prevent other people from Trading by submitting a giant whale trade but like they can't actually do that without running the risk that like someone will provide liquidity at a bad price so I think it's probably not a big deal right the time is off well one more round of applause forehand really please thank you Henry