# Improving Oracle Infrastructure on Ethereum by Lorenz Breidenbach | Devcon Bogotá

- Channel: [Devcon](https://streameth.org/devcon)
- Date: 2023-10-07
- Duration: 33:15
- Watch: https://streameth.org/watch/yt-9c0BubOWhDs
- YouTube: https://www.youtube.com/watch?v=9c0BubOWhDs

## Description

Visit the https://archive.devcon.org/ to gain access to the entire library of Devcon talks with the ease of filtering, playlists, personalized suggestions, decentralized access on Swarm, IPFS and more.
https://archive.devcon.org/archive/watch/6/improving-oracle-infrastructure-on-ethereum/

Oracles continue to fuel the growth of the Ethereum and layer-2 ecosystem. To build dApps that serve real-world use cases, from DeFi and DAOs to NFTs and beyond, Ethereum developers need access to secure oracle networks. In this session, we will dive into the latest oracle innovations and showcase new capabilities that enable Ethereum developers to build the scalable, secure, and feature-rich applications that can achieve widespread adoption.

Speaker(s): Lorenz Breidenbach
Skill level: Intermediate
Track: Developer Infrastructure
Keywords: Smart contracts,oracles

Follow us: https://twitter.com/efdevcon, https://twitter.com/ethereum
Learn more about devcon: https://www.devcon.org/
Learn more about ethereum: https://ethereum.org/ 

Devcon is the Ethereum conference for developers, researchers, thinkers, and makers. 
Devcon 6 was held in Bogotá, Colombia on Oct 11 - 14, 2022.
Devcon is organized and presented by the Ethereum Foundation, with the support of our sponsors. To find out more, please visit https://ethereum.foundation/

## Transcript

foreign [Music] my name is Lawrence I'm head of research and development at chain link labs and yeah today I'm excited to tell you about improving the Oracle infrastructure on ethereum for those of you who don't know oracle infrastructure connects smart contracts with the rest of the world with the world outside the blockchain and so let me start by talking about the most prominent use case of oracles data feeds I'll start by reading a quote we'll play a little game you guys can rather read the quote guess where the quote is from I'll give you a few more seconds after I'm done reading the quote and then I'll reveal the answer so the quote is for example one powerful use case of an oracle contract would be a hedging contract where A and B put in a thousand dollars worth of bitcoin and after 30 days the script sends a thousand dollars worth of bitcoin to a and the rest to B this would require an oracle to determine the value of one Bitcoin in US Dollars sounds a little bit like defy the terminology is a bit different from what we would say today so the solution is it comes from the ethereum white paper all the way back in 2014. so already back then it was clear that oracles are required for certain types of applications and indeed here we have an example of a you know price feed or data feed Oracle and if we think a little bit about the properties that this implies that we need our Oracle to have there's two things that stand out so first of all we need reliability we need to make sure that when the 30 days are up here indeed the Oracle supplies the exchange rate between Bitcoin and dollars so we can fairly settle the hedging contract and then we also need security and integrity right the price that is provided here by the Oracle will be used to settle the contract and so if the price is incorrect then we won't be able to settle the contract fairly so it's essential that we we have a Oracle that provides these properties and I think if we if we think about the history since then billions of dollars have been stolen in Oracle manipulation attacks so Oracle security is Paramount for D5 so let's have a look at how one might construct such a data feed Oracle and I'll start with a straw man construction and then I'll evolve that construction into what I would say is the state of the art today and is what chain link does today so here in this Roman example we have a data source that Pro you know the Oracle observes in this case the data source says Hey ten dollars is the price the Oracle will sign that price and it will then provide it to a contract running on chain and then adapt contract can read price from our data feed contract but if we think a little bit about the two properties I had just mentioned reliability and security then this system does not achieve them because we have a single point of failure if this Oracle goes down or fails to provide the price then the system on chain won't work and similarly if the single Oracle here is compromised or malicious then it can feed a false price to the contract on chain and cause the contract to be settled incorrectly unfairly so how can we solve this problem well there's a few different approaches that people commonly take but I think the most common one today is to say let's decentralize the Oracle so let's have a bunch of independent oracles all provide the system together and so what we've done here is we now have multiple oracles that each fetch a price from a data source and then send their individual signed observations to our contract running on chain and that contract on chain stores the observations until there's efficient number of them then Aggregates them into a single price using a robust aggregation for for these data feeds the typical aggregation one would use for numerical data is to take the median because that's a robust statistic that doesn't respond to for example false outlier values or things like that and then you know once we have medianized our value we can expose it to the dab contract Downstream and this is nice because we can now handle subsets of oracles being faulty right so if some of the oracles here do not send a price or if some of them send false data that's okay if they if they don't send a price our data feed contract can just handle that by ignoring a certain number of missing submissions and if they send a false price as I had mentioned we use the median which is a robust statistic and which ensures that as long as no more than half of the oracles here are faulty the aggregate price still lies within the interval of prices reported by correct oracles so this works well in terms of giving us the security properties and reliability properties we want but there's a problem here right if you're familiar with smart contract notice that we're sending one transaction to Oracle here we store the submissions in state and we compute an aggregate on chain all of which are expensive things to do and indeed this is how chain link used to work in the past with a system called flux Monitor and indeed that became prohibitively expensive and so we moved on to the next Generation which I'll describe now so I would say that the state of the art then is to say let's move as much work as possible off chain the less we need to touch the chain the more efficient our system is going to be but we'll want to do this in a way that preserves the security and reliability properties that we desire and so so roughly speaking what we do here is we again have our oracles observing prices from from their respective data sources but now we have the oracles communicate with one another using a peer-to-peer Network and we for the distributed systems folks out here have them run a Byzantine fault tolerant protocol Byzantine fault tolerant means that they can handle arbitrary faults so any of these oracles could be you know malicious or faulty or down or whatever else and as long as you know some subset of them is affected by this depending on exactly how you set up the assumptions either less than a third or less than half of the oracles the overall system will keep functioning correctly chain Link's version of this is called the off-chain reporting protocol and indeed it's far more efficient than the predecessor I had just showed you so we realized something like 90 on chain cost savings and this is good for security because it means that with the same security budget we can have a lot more oracles so we can tolerate more faulty Oracles in the system and it's also good for liability because in this system at the end of the off-chain protocol every Oracle ends up with a identical copy of the report which is signed by a quorum of the oracles and the contract will then validate that the report is indeed signed by a quorum of the oracles and what this means is that as long as a single Oracle is able to get a transaction on chain the the overall system will keep functioning correctly and so for example during times of chain congestion this is very desirable indeed as I mentioned right this is how how the chain link off chain reporting protocol works and it's been it's been very successful so we we designed it for powering data feeds on evm Chains It's been running reliably and securely for at this point one and a half years in production it secures roughly 20 billion dollars of value across all the various chains that chain link is active on 14 billion just on ethereum there's more than a thousand feeds across all the chains we're active on and more than 290 of those are on ethereum and I think the the reliability and secure of the system are also shown by the fact that we have major major D5 protocols relying on this system uh like names such as other compound and synthetics all use this cool but okay so I told you about data feeds but now I want to tell you that oracles are about a lot more than just data feeds they enable in the general sense interactions of smart contracts with the outside world that is to say you know things that are outside the blockchain or contract executes on and so they Empower smart contracts to reach their full potential because now I can do a lot more interesting things with my contract um then if I'm just isolated on the blockchain and cannot interact with the outside world so here some of the capabilities that are enabled by by oracles data feeds I just talked about proof of Reserve I can ensure that on-chain assets are sufficiently collateralized by some off-chain reserve of funds by having independent oracles monitor that reserve of funds and Report how how large the collateral is secure Randomness contract Automation and cross-chain interoperability I'm going to talk about more later in the talks I won't go into them now Fair sequencing you know I can use an oracle Network to protect users from toxic Mev extraction by fairly sequencing transactions I can Outsource computation and storage so that I can enable contracts to scale beyond what the evm provides as much as we all love the evm and as much fun as gas golfing is there are some things that are just a little bit hard to do on the evm and then finally I can also enable oracles to perform real world actions so we actually had a hackathon a couple of months ago inside chain link Labs where we had contracts ordering pizza via the Pizza Hut API to people's houses so I think that's a that's a pretty neat use case that is enabled here and that would not be available to contracts natively and then finally we can also provide privacy preserving access to almost any Web 2.0 data and I'll cover that also towards the end of my talk so this gives rise to this notion of hybrid smart contracts and and when I say hybrid smart contract what I mean here is a hybrid system that partly operates on chain in the form of a smart contract running say on ethereum and partly off-chain operated by a network of decentralized oracles and so this overall hybrid smart contract is now able to interact with the real world and so so this enables me to do all sorts of interesting things and let me you know give you some more specific details on some of these and I'll start by talking a bit about secure Randomness and what that enables contracts to do as you're probably familiar there's kind of two on-chain native ways on ethereum these days to get random values one of them is the block hash up code and that sort of you know audit findings 101 do not rely on the Block hash op code for Randomness any block producer can can easily tamper with that and now since the merge there's also a second op code prevao which is a little bit better than block hash but which is still manipulable by block producers and so so the output of both of these can be biased so if your contract needs secure Randomness for example for running a lottery or for doing a nft mint where you assign valuable random attributes to your nfts these are not good enough and so on chain there is no native capability to get good Randomness and this is where oracles come in and oracles can provide that functionality um so we we have a cryptographic construct here that's called a verifiable random function and that enables an oracle to provide tamper-proof Randomness and let me explain how that works so we start here with our dap contract that if it wants some Randomness can request from another contract on chain that's part of this drf service some random words the contract will emit an event with you know just telling the oracles that Randomness has been requested the oracles will evaluate the verifiable random function which is based on deterministic public key cryptography that is to say the oracles actually have no ability to change the output It's deterministic So based on a given keep here and a given blockchain state which is used as the input for the vrf we will always get the same output value the same randomly distributed output value and the oracles will then provide a cryptographic proof that they correctly evaluated the vrf will send that cryptographic proof to our coordinator contract here which can cryptographically verify the proof and if indeed the verification succeeds then finally the coordinator can send the the random output here to the DAP contract which can then for example run its lottery or whatever else and this is live today in production if you are writing a contract and you need this functionality you can use it and I think there's over 1 400 unique contracts that have used this in the past 30 days so okay Randomness is cool uh now let me talk to you about yet another use case that is live today and that you know enables contracts to do more interesting things and that is contract automation once again I'll return to the quote from the original ethereum white paper in 2014 but this time I'll highlight a different part so here we have after 30 days the script sends a thousand dollars worth of bitcoin uh to a and the rest to be so how does the script do this how can the script or the smart contract send those funds to a and b contracts do not have an ability to initiate their own execution right and of course I could say well you know I'll just have my users A or B here you know call the contract after the 30 days are up and then I'll do a payout and maybe if I just have two users that's fine but now let's say I have a lot of users or I want to you know sweep a bunch of tokens from different contracts into a single pool or whatever other expensive maintenance action my system requires to run then that will lead to a very poor ux if I put the onus of Performing those maintenance actions on the users interacting with my dap um because if I'm a user I now have a non-deterministic gas price right once in a while when I interact with adapt it's going to cost me a hundred thousand gas another time I need to maintain some expensive maintenance on the side and suddenly it's going to cost me 500 000 gas that's not great it's also unreliable because I can't really rely on my users calling the contract at a specific point in time and so this is where where contract automation comes in to play so once again here we have a set of oracles or automation nodes that are monitoring the chain and the upkeep contract here is the the dab contract and the app keep contract is just any regular smart contract written in solidity or Viper or whatever programming language you prefer and it has a special function called check upkeep and that function will be evaluated by our Oracle Network here whenever a new block is mined but it will be evaluated in a simulated environment using the say F call RPC endpoint of an ethereum node so we never hit the chain with with these evaluations no transaction lands on chain and so we can call this on every block without incurring any cost um and so then the you know the programmer of this upkeep contract can basically come up with any arbitrary predicate to decide whether their contract ought to be called or not um and you know if if the contract says no I don't want to be called no problem we just check again in the next block um but if indeed the contract wants to be called It Can indicate so to the Oracle Network which will then send a second or will then actually send the first real transaction to the chain that's not simulated um with a perform upkeep call for upkeep contract here and now this contract on chain can you know take whatever logic it wants to so for example in the previous case at this point it could have paid out our two users A and B uh depending on the outcome of this hatching contract and again this functionality is live today there's over 250 unique upkeep contracts that have been active in the past 30 days and if that's useful for your for your smart contracts then I would encourage you to check that out okay so I've been talking a bunch about data feeds and about other Oracle capabilities that are live today but I'm from research right so I'm particularly excited about the future uh and and what future capabilities we're actively working on to bring to Market and I'm going to present two exciting capabilities and both of them are in an alpha state with Partners so both of them are are already working in some form even though they're not quite ready for for mainnet production yet and the the first of these is cross chain interoperability um so the problem here is that smart contracts cannot natively interact with smart contracts running on other blockchains um but we live in a multi-chan world now right there's dozens of chains out there it's a very heterogeneous environment some memorable ones some are l2s some use the EDM some don't and so on and so forth and so indeed there's also a lot of demand for bridging between contracts on different chains so today it's something like 168 billion dollars of value have been bridged just in the past 12 months we've seen over two-thirds of the total volume ever bridge for something like 115 billion dollars bridged um but also we have a challenge with security in this cross-chain space so to date 2.5 billion dollars have been hacked from cross-train protocols um and just this year over 70 percent of all values stolen uh in in kind of the blockchain ecosystem is from cross-chain hacks uh so I think once again this indicates that security is Paramount but that unfortunately the the current Solutions are not yet up to the task so we at chain link are also working on a solution to this problem which we call the cross chain interoperability protocol ccip and so here we're working on a programmable bridge that enables a smart contract that contract here say on ethereum to send a message to a contract running on some other chain and you know it could be talking to multiple other chains so here I have it sending a message from ethereum uh to this other chain B um and the message can carry both data so some kind of arbitrary binary payload as well as value so say tokens and again I need oracles for this because inherently the other chain b or ethereum have no way of knowing what happened on their counterpart chain that they want to talk to here and so oracles can can bridge that Gap and they can transport information about what happened on one chain to another chain to enable contracts on different chains to interoperate um this is again you know as I mentioned it's very important that this done that this is done securely because if for some reason here I were to get uh you know false information about what message was uh created on ethereum and then that could for example lead to to funds being moved uh on this other chain here when they shouldn't be and I would have a security breach um if you want to learn more details about this we just had a talk at smart con roughly two weeks ago where we went in more detail about the architecture of ccip and it's on YouTube so I'd encourage you to check that out if that's interesting to you the second Oracle enabled capability for the future that we're working on is about providing privacy preserving access to Web 2.0 data to almost any Web 2.0 data to Smart contracts and this is I think a super cool capability at first it sounds a bit unreal like how does this work how could this be so let me let me show you what I what I mean here so the the fundamental problem is that the the vast majority of data is private and is therefore not publicly accessible in the Web 2.0 ecosystem so for example my online bank account my Twitter profile my Facebook profile my Amazon account some government website none of these are accessible by smart contracts on chain because all of them are behind some kind of authentication wall right and I would argue that's a good thing I don't want my bank account information to be just visible to any contract on chain but still it would be kind of cool if there were a way to expose in a privacy preserving information just the information required to Smart contracts unchain because that would unlock I think many interesting use cases things like decentralized identity risk scoring where I can use information about my off-chain behavior to you know score how risky of a borrower say I am consequent consequently under collateralized lending where you know maybe in order to borrow a hundred dollars worth of assets I don't have to put up 200 worth of collateral on chain or private defy instruments where I can have a D5 instrument where only the parties that are participating in that instrument need to learn you know what exactly the conditions of it are on what based on what condition it settled and so on to improve privacy of on-chain trading and more things that I probably can't even imagine yet this is a very general technology and so the technology we're working on here to to address this is called Deco Deco was originally researched at Cornell University and ic3 the initiative for cryptocurrencies and contracts and since then chain link has has pushed the project forward is actively developing it and further researching it to make it efficient and practical and so I'm briefly going to describe to you how Decor works and what properties it affords first in the abstract and then by going through an example so we have three off-chain parties here we have a web server we have a prover or a user and we have a bunch of verifiers or oracles and so now what what Deco enables the user to do is to prove to these verifiers and then by extension to the smart contract that some data indeed comes from a certain web server so for example I can show that indeed bank.com made some statement we assume that this web server runs TLS which is the the standard protocol for securing Communications on the web today uh you know whenever you go to some website you'll probably see a little lock icon at your address bar indicating that the connection is secured with TLS so I can prove provenance so I can show that the data indeed comes from a certain Source but I can do this in a privacy preserving way where I can run a zero knowledge proof protocol to only reveal the information the minimal information that I want to reveal to the verifiers and then by extension again to the contract um and right so so the web server obviously if it's for example my bank knows my balance and my password I as the user also know my my password and my balance but the the verifiers or oracles for example should definitely not learn my password right and Deco enables me to do this in this privacy preserving way um and then finally and I think this is actually the coolest property of Deco it's compatibility so it's compatible with the existing TLS web ecosystem and that means that there's no need to get for example the the data holder here the web server to change anything about their API or their stack they don't even need to know that Deco is being used to prove something and you know if you for example look at how hard it is to get the internet migrated to IPv6 you know that protocol upgrades are very difficult so it's I think a game changer here that this works with the existing Web 2.0 ecosystem without requiring modifications on the part of the web server or the user so let me let me show you an example so I'm the I'm the approver I'm the user here my name is Lawrence and I want to show to a contract on chain that my bank account balance is greater than a thousand dollars without revealing anything on top of that um so what I'm going to do is I'm gonna Connect using TLS indicated by the green locks here to the API of my bank using TLS so I'm going to say hey I'm Lawrence my password is Hunter 2 what's my balance um and now the the bank API you know will respond hey Lawrence your balance is say five thousand dollars which remember I do not want to reveal uh to the verifiers or to the contract as part of the way the the handshake works here between me and the banking API I'm also able to just as part of this TLS exchange ensure that the verifiers are oracles here get a cryptographic commitment uh to the data that is being exchanged and that commitment as as we say in cryptography binding and hiding which is to say that if I so so you can think about it as an envelope where I put some data inside if you just have the envelope you cannot see what's inside but you also cannot switch the contents without somebody noticing because you kind of tore the envelope and this is basically a similar thing here so the verifiers hold a cryptographic commitment about any data that was exchanged within the session so I as the user cannot sort of alter the data after the fact but the verifiers do not see what's inside the commitment the commitment is hiding so they have not learned at this point any private information any information at all about what was in the TLs session and so now we get to the final step which here I'm abbreviating with zkp magic because I have like two minutes left on this talk but the the way this works is that we now run an interactive zero knowledge proof protocol between the prover user so myself and these oracles and as part of that zkp protocol I can convince the oracles cryptographically that indeed for example this statement holds so that my balance is greater than a thousand dollars but that is the only thing they will learn they will not learn my credentials and they will not learn my precise balance the oracles can then cryptographically sign an attestation that they have successfully verified this zero knowledge proof and this attestation can be forwarded to our smart contract on chain here again you know signed by a chrome of oracles and then we can verify in the contract that indeed those those signatures match and so now I have convinced the contract on chain that I have you know at least a thousand dollars in my contract and I've done that without having revealed any additional information um and and you know you can think about this in a far more General sense because this is your knowledge proof I could basically prove almost any statement about the data that was exchanged in almost any TLS session so this is a very general and very technology I believe and I'm yeah I'm excited to see what people are actually going to build with it because I think due to its generality I personally have a hard time imagining what's going to be enabled by it um so yeah um thanks a lot for for listening to my talk um there is there is three things I'd like you to to take away so first of all security and reliability are Paramount for oracles um second of all oracles are about a lot more than just data feeds I think I've shown you a bunch of exciting applications of what oracles can can do yeah and third of all I think we we yeah have a opportunity to make contracts more exciting more interesting to empower them to reach their full potential Beyond just doing things within the closed on-chain ecosystem that they would naturally operate in without the native oracles so yeah thanks a lot and if you have any questions I think I'm I'm over time so just come and find me after the talk you have time actually you have time there's time allowed for your q a so there's room for questions right now in contract automation how are the gas fees paid ah I see I see I see so basically what happens is that the the depth that relies on contract automation um oh yeah we'll we'll maintain um a balance I don't know whether it can go back to that slide no no no no more anyway so so it maintains a subscription balance with an automation registry contract that sits on chain and then when the DAP is invoked by the Oracle Network um that automation registry will keep track of how much gas is used as part of that transaction the perform upkeep transaction and will you know Bill the user's subscription for that so um and when I say the user here I mean the upkeep contract um so so in that sense as long as the subscription is funded we will just automatically you know have the contract deduct the the price of the upkeep from that does that answer your question yeah yes so you showed us that the gram where the user and the bank API change that made those messages right so this is about Deco right yeah too bad we can't show the slide yeah okay so the bank says that the balance is five thousand and uh Oracle sees that but without seeing the balance right and then so the Oracle does not see the the five thousand dollars yeah the Oracle just has a cryptographic commitment against which we can then perform an interact with zero knowledge yeah exactly I'm just wondering how the Oracle then can verify that the balance is over one thousand without like showing that the balance is five thousand right because the exchange was only like containing information about five thousand and not about like one thousand right right so so my my short answer and forgive me is just gonna be Magics your knowledge proofs um basically that's what dkps are about right I can I can prove statements about data and sort of very counter-intuitive ways without revealing anything beyond the statement um but I think we probably don't have time to go into the details of how that works unfortunately but come find me afterwards and I'm happy to talk more uh so I have a question about the OCR the off-chan uh Oracle reporting protocol so with uh with a blockchain is easy for the different note Rich consensus right because you can check the uh the transaction signature they can't really run rerun all the byte code of a smart contract by the way the Oracle uh how does the different node ritual consensus because uh if it's a if it's a data phase then you cannot compute right you cannot compute the weather this feed is great or not so so I believe that the answer to your question is OCR is a Byzantine fault tolerant protocol that is actually the same family of protocols that consensus Protocols are from right it uses similar techniques it just has a different goal its goal is to provide you know reliable and secure reports about some you know data observed from off-chain to contracts on chain but it's it's similar techniques and so um you you kind of as part of how that protocol Works get that property that you know the oracles communicate with each other over the peer-to-peer Network and that some of them can be faulty and so on so you you know you you broadcast things you converge cast things you use digital signatures the usual type of things you would do in bft protocols yeah uh I mean if uh let's say if it's easier and or if the other blockchain uh different nodes just have to have the same similar result from the trans executing the transaction have a similar result if a two-thirds two-third of node have a similar result then can reach a consensus right but but with uh I understand with uh you can use that for the for the consensus layer but uh how could you have buy something for tolerance for the for the data field right data phase for example one node say the the price of a series 1000 another node say the the price of ethereum is 2000 uh you you cannot uh how can how can reach the consensus I I think what you're asking is how we reliably aggregate the data in inside OCR is that correct no uh for like for computation for computation tasks it's very easy to reach a consensus between different computers because as long as your as I run on the result or computation is the same if two two-thirds of node uh computer agreed like the result of the commutation result of one transaction is the same then you have a consensus you have you're fine you're good but uh with the data fade like let's say if if two if one sort of a server or node or Channing knows I think the the price of ethereum is uh one thousand I see I think I think now I get the question okay so so basically inside the OCR protocol which I did not have time to explain in detail here uh there is a phase where where the different Oracles um converge cast their individual observations to each other and at the end of that phase every Oracle will have a signed observation from every other Oracle so if you know we have a bunch of Oracles in here every Oracle will know what all the other oracles think the price should be and at that point I can then you know have everybody compute the same kind of deterministic aggregation function and I will end up with the same outputs and so I can reach agreement between the different Oracles thanks everyone