Technical Details of the Solidity Compiler by Daniel | Devcon Bogotá

foreign [Music] for yeah over four years now and yeah originally the plan was for the store to be held by Chris who couldn't make it to Bogota so I have to improvise a bit I brought a lot of code Snippets which will explain what we have currently in the compiler currently the state of the language and where we are headed with it yeah I guess you if you are familiar with Trinity which I hope most of you are we probably know new doubles by now it's been around for for a while and it's a very highly appreciated feature from uh our impression which yeah I still will quickly explain how they work that's the owner of arrival here is looks like a state variable but it's not a really proper State variable in storage but a mutable variable which can only be assigned to Once In The Constructor and then can use this many times as one wants read only in the runtime code of the contract but there it's the access is very cheap it doesn't require an S load but it's basically in line in the code as if it was a literal so yeah you'll probably know this and people liked it a lot and an apparent and a very obvious extension they asked for is why do we only support that for Value types that's the Restriction you had so far that it's only integers on the addresses but not arrays of things for example for example so this was for example it's taken from a GitHub issue that I think opens apple and open with us to yeah ask for an array of immutables which would then yeah be initialized by index accessing and otherwise work just as as one is used with immutables and we're working towards that end but there's some issues with that so I mean we have these assignment here where randomly assigned uh into uh elements of this array and in general I cannot check that this is really assigned only once I mean the question is is there still an immutable thing in that sense also if I start having a race as a mutables I will want to have local references to them which I could then reassign so if I keep the name immutable I just change what this points to so this is really it doesn't look immutable immutable it doesn't fit this concept anymore probably so let's yeah let me explain a bit further how Universe actually work in the Constructor owner is actually a position in memory so we store whatever you write to this variable in memory and then when the actual runtime code of the of the contract part of deploy is copied to memory to be returned by The Constructor we fill in from this network location into the byte code the value that is there which in the end results in the runtime code to actually have it as a literal in the bytecode so that's the Rival in the Constructor actually is a memory reliable in runtime it's actually something that lives in code but filling in literal values in some so basically the push argument into the bytecode won't work anymore for dynamic types if you have statically sized arrays we could still do that and we probably will for efficiency reasons but at least for dynamic types to go full way we cannot do that anymore because we don't know the length of the thing so we can't reserve space in bytecode for that so instead we need to rely on code copy and yeah I already mentioned we will probably want to pass the immutables by around by reference we will want to slice them which if you think about all of that together makes immutables not an annotation for a staple arrival anymore but it will become a proper whole data location so this will probably look like this in the end as I said the drive in the end will live in code so code and the op code that will access it is code copy so code is a natural name for these things so here I have a data arrival that bites the dynamic array in code which I can then in the Constructor just treat like any old memory variable assigned to it modify it freely you can drop the requirement that it's only written to Once In The Constructor because that was always an artificial requirement so if we actually yeah called it but it is a code varietal that will actually be inserted into used as in the code in the end we can freely modify it and then in the runtime code it basically behaves like a call data reference a read-only reference only that it doesn't come from call data but from code so I can slice it have local references to it and pass it around the functions all with very little cost this will be a bit tricky to write to type check in the end because yeah in the Constructor it is a memory reliable in the runtime code it vs differently so if the Constructor calls functions we need to type check everything twice but we have different costs for a refract factor in the type Checker to actually do that as well so we will do that and there's still some considerations of gas it would be very handy if I had a code load opcode similar to a call data output to actually read single words from code but we don't so we will whenever possible probably still use the same mechanism as a mutable to use now to actually fill it into a push argument invite code if not we will probably yeah code copy either in an entire memory or if you just need a snippet of a dynamic array called copy to scratch space and M load from there that will still be significantly cheaper than anything in storage okay yeah that was the first topic I wanted to talk about the second thing that you may have noticed in the recent versions is our move to uh to allow user-defined value types go to test case for this concept was always fixed Point types which we decided to not Implement as a first citizen type in the in the type system but to build the infrastructure to Define it as a user-defined type yeah I'm assuming that you're familiar with those uh so I won't go into that much more detail in the basic concept which what we recently added or more recently added is This Global keyword you see in the first using statement which means that currently a using statement is only locally effective whereas a using statement with a global at the end will only be allowed in the very same source file in which you define the type but we'll have that's the effect that wherever the type is available the functions you bound to the type will also be available so this is what you can do then in the end practically to write type libraries when you're in that Library Define the type equipped with some functions and then anyone importing the type only can still use all the functions on that still of course one major inconvenience with that for defining for example fixed Point times that are actually nice and usable is that we have only functions in that can be defined and equipped to the type but the using statements which is what we will soon change so this is open PRS currently that are under review which will probably yeah in finite amount of time we actually merge and come through which will allow user-defined operators and literals in uh in the first versions in a very limited setting so the addition for example can only be between the two between two expressions of the type that you define such that it's not getting messy and you know no longer know what uh function will actually be executed if you have an uh an operator expression but it will still already be yeah very useful I hope uh in making these user event types more yeah convenient to use literal similarly we will introduce a way to define functions that can uh take up as argument a mantissa in that sense and an exponent which is basically yeah here for example for the example below it would be 115 and the exponent is 10 to the Tower of minus two which will allow to yeah then deal with for example a fixed Point setting uh rather gracefully and have literals without yeah especially code constructs eventually we will have a proper irrational rational literal type instead of splitting it up in a exponent here I think at least but for now this is the fastest path to get this working but yeah so far these user-defined types we have only are value types this might be equal cause to want to encapsulate and Abstract Dynamic types arrays structs and so on and there's also of course uh yeah the desire to have proper aggregate algebraic data types in general so the more user-defined data types we have the easier it is to uh like a usable code and yeah there's questions about that for example where data locations would go current if data locations are associated with variables and not types which makes that a hassle which we will also probably change for independent reasons but yeah I mean you may uh see where I'm going with this if we have all these kinds of types and maybe also user-defined containers and all that it will no longer be feasible to uh to write functions to cover all these types and there's several functions for all these types so all this increases the need for having a construct of generics in the language which has been long an idea to introduce with the language and which yeah we're finally now making moves towards but yeah this uh if we have that uh the natural question is if even the built-in tires we have now can be user defined instead which leads me to another topic that we already started which is a standard Library uh the idea here is to move whatever manually hard-coded constructs we have in the compiler now or as many of them as possible to actual implementations in validity written user code and then to ship that as an standard Library that's integrated into the compiler so yeah this is what this is in the first instance going to look like that you can enable the standard Library version of the compiler which in the end we will move towards and then import uh built-in constructs from the standard library and it will be possible to export the standard Library as a set of files out of the compiler and there you can then inspect what this function actually does for example add mode is a simple example which can then actually be implemented in inline assembly with additional checks that works to reduce the footprint of the compiler and to yeah make it easier for people to inspect what's actually happened and to extend it even but the problem there again is this is of course Very restricted in the scope in which we what we couldn't move into the standard Library as long as our functions facility of monomorphic and so means we don't have generic functions that can handle multiple types so the full potential of the standard Library will only be Unleashed if we have generics uh yeah and the question is also whether we can not only move built-in functions but also built-in types which would end then as we had earlier as user defined types in the standard Library so the end go over the standard library is reduce the language to a small simple core language and have most of the current features that we have in the language defined in a standard library that is written in solidity can be quickly iterated on can be extended by the community and can be audited and so on and so forth but to get to that point in full we need generics so the idea is to yeah get inspired by a nice system of generics which is based on a logically grounded type system we have products I have some types function types which if you're a mathematician or category theorist the type system will probably form a partition closed category which is how these things work usually in properly designed to know languages like Haskell or like the trade system in rust or the general system rust uh and yeah we will have polymorphic functions that can take arbitrary types and yeah ad hoc polymorphism using type glasses or that's in Rust maybe if people are more familiar with that great and yeah we will also for this to work out you kind of need to go all in on it and or go for the code the whole way so we will also have algebraic data types and yeah all of this will also be only really useful if you have compile time constant expression evaluation because you will in generic functions have yeah constructed parameters that should be evaluated at compile time so yeah that's also something we will have which is as I said earlier for the code data location we will need to refactor the type Checker to have VMware flexible for this we will need that as well and yeah maybe we will also have linear types which is how rust's follow Checker is constructed but yeah I will now have some examples with some made up syntax and some made up constructs but I'm going to say this is all early research stage so we have nothing fixed here yet I'm just telling you where we headed and what we'd want to do in the next however long it will take so yeah the go-to down to earth example of a use of generics is something like a resizable array some container which yeah in this case it's just an array that if you want to append something to it and it see the array is already full you just reallocate with twice the size copy things over and yeah otherwise you can just add the element to the end of the array and yeah then we can also Define an index access as a user-defined operator which will then have this thing actually work nicely yeah one would need an allocation function and so on equipped with that but we can then build a library to actually build these things without having special casing for each type each kind of Base type but I have years a bit cheated though because in reality things are more complicated of course because we have data locations again maybe I need to restrict the base type as by some trade mechanism to something that can live in memory which may be a value type or another memory array something like that so in the end it will get complicated but it will be worthwhile and yeah if one wants to take this upper notion all the way one can actually then think about even defining the most basic types in solidity itself this would be the definition of uh the current representation of memory arrays in solidity written as user-defined type with the customly defined index access to it so in the end for any bass type of memory array is just one stack slot you refer to memory arrays just as one stack slot be index access by it by fetching I mean this desktop points to some memory area where there is the size of the array and followed by the data there's the current layout at least and yeah in index access we can just fetch the size uh through the bounce checking and fetch whatever result we have at the offset that the index pointed to and again we could Define then that index accesses work as expected and yeah a length field could be added to that all defined in language which has the advantage we had often the request to allow slicing for memory types which we can't because the representation I'll show here doesn't allow it since we expect the size to be at the first memory offset where the memory pointer that is the representation type of this points to is the size we can just slice away from the first element for that to work memory arrays would have to work different similarly to call data types there you have uh offset and size on stack it would be a huge effort for us currently to change the entire compiler to come to change the representation of memory types if we had things defined like this it's minimal changes I can just say now my memory array is defined as a tuple of Stack slots one of them is pointer to data area the other so it's the size and yeah have a similar definition as before slide changes but uh a few Source changes in our standard Library would then be the same as changing the entire layout of uh of memory types which would be yeah month of month of work why we maintained uh everything hardcoded in the compiler of course it's all the disadvantages if we actually keep this extremely generic like that we will lose semantic information that will actually make memory optimization harder because yeah but the compiler sees it's just a bunch of Stack slots there's no idea that that's actually memory areas we're talking about that are allocated may only be allocated temporarily and stuff like that so I'm not sure whether we will actually go this far and if so we would maybe probably do this in a yeah compiler internal manner where we will still assume certain semantic properties about these functions without supporting similar optimizations that we can do if we know what these things do uh in their random user code but yeah why did I say stack slot all the time that's yeah maybe obvious but uh just to mention the stack slot would be the one primitive type maybe apart from uh from product types and even the basic uh integer types size initial types we have right now can be defined generically just yeah like the other cases we had so I mean this will really reduce the footprint you can hear could also then distinguish between types that are checked arithmetic and unchecked arithmetic by having very few functions that are generally written and yeah so yeah but not to give to make you expect this to happen too soon we are still in early design phase for generics there's a lot of questions this is a very complex thing to do and a very dangerous thing to do because yeah all of this needs to be biologically well defined to not bite in the back in the end so I mean we will take some time to design this properly and syntax is also a question that's very much differing opinions on how the Syntax for these kinds of things will end up being I'm not that concerned by that now I would first want to get the semantics right but eventually we also need a good Syntax for it and yeah what I just said we need to decide what to do with this trade-off between making this the language really self-defined in the in the very deepest sense or to have some fixed functions which are fixed in the compiler which means we can assume that semantics or there are compromises between that but we'll see so yeah to summarize uh what I was talking about and what I wasn't talking about we will still do hopefully we will in the future try to allow more pre-competation either in the Constructor by the coated location or in compile time by compiler constant expression evaluation which is something yeah a lot of people have asked for and which obviously makes it easier to write things in that you don't need for example magic constants embedded in a contract or whatever because you can uh compute them on the fly without it costing and he had a huge a huge topic for the future will be to make the language extensible and self-defining by means of improving user-defined data types pushing the standard library and making a move for generics but we also of course can't just ignore is that we're still wasting a lot of memory I mean whoever has used memory and solidity will know that yeah we basically don't free memory which for a long time wasn't the main concern but in the meantime is very huge pain point in uh yeah for cost of contracts there were several uh approaches we discussed so far for improving the situation there we a long time we wanted to deal with this on the Dual level it turns out that may not be as simple so maybe we will move actually to analyze the solidity which has the properties right there we shied our way of doing that for facility being the more complex thing to analyze but maybe it's fine we will see and yeah we will also of course try to move completely towards via our core generation but we have some burdens there to overcome still like the performance of the optimizer better tooling support you still need to Define got the background data for the tooling tooling to consume to actually make the experience as nice as with the Legacy code generation the details there would be that the tooling expects certain patterns to remain in the in the bytecloud in the end whereas the new optimization pipeline will mess them up by optimizing better but tooling needs to understand that we need to Output data for it being able to understand that but yeah that's hopefully you've ever had it and yeah I'll close with that if you want to give us any feedback help us with designing generics or criticize what we are planning and saying we are crazy to do with any of that reach out to us that's the channels we've just said and yeah thank you yeah it used to be the case that the compiler has two back end paths at the moment so I mean it used to be the case that solidity was directly translated to evm bytecode and then only of the Omni optimizations that took place were uh on the bytecode level for the past years we have uh moved away from that and have a different new code generation pipeline that translates solidity first into Yule into an immediate language which preserves some structure and which allows for more complex optimizations for inlining more analysis uh and then only to uh translate Yule to evm bytecode as a second step which can reduce gas costs significantly in some cases in some cases it's the same as before but yeah and yeah the new pipeline the Via IRS via the intermediate representation so compilation via you thank you for the talk you mentioned generics I'm wondering if you could speak to how you're planning on implementing that whether you're going through monomorphization because I worry that the code contract size will balloon if you start you know doing the C plus plus style duplication of implementations or if there's some you know uniform representation you can do a la la ocamel or you know Java I think there's not much you can do actually I mean uh the generics of C plus are different in the sense that they are analyzed differently and you only get errors in on instantiations but we will still need to instantiate uh and generate code for each specific case but that's not worth a worse than what you get now what you get now is writing by hand for different types different functions that would end up separately in by code so yeah it's not nothing is worse than that that's a duplication in code and in byte code if you have generics we at least only have it in source in only in the bytecode so my question is related to that question regarding genetics but from a different aspect so type system I understand but once we get generics into solidity wouldn't the developer have to focus on 10 more things instead of focusing on writing business logic uh uh good question uh I would I would think that the uh yeah down to earth go to smart contract writer will not bother with this it's mainly something for us for defining a senate library and for people writing libraries to support smart conflict developer first so I mean uh the language supporting generics and having generic types doesn't force you to use them and it doesn't mean that anybody has to use them but it will make a language in the the evolution of a language much uh faster and more streamlined we can in the future ask people if they propose a feature to just implement it in a standard Library way and then standardize it in the end if it works out which will yeah have all the advantages this one can think about that but for user code for the in smart or contract code the difference is not that large probably all right are you going to implement Lambda functions just with this equation uh definitely eventually I would say I'm not sure whether I mean getting the basic type system uh going and all that will already take some time but yeah eventually this is of course something that will make things easier to read easier to write and are beautiful so and maybe not I mean as long as you don't want these things to capture reliables it's easy capturing even maybe something at some point we can also consider but yeah not not the first thing we will do but eventually have you been thinking about integrating on trying computation and off-tracking computation in the one source file this is of course very problematic but in some moment in future we might need it not necessarily in solidity but maybe you should try to think about that yeah I mean on the solidity level we don't yet uh interact that much with yeah off-term computations Layer Two stuff or whatever but yeah we are aware that we need to interact with that and support that where we can I think you mentioned something about the performance of via IR uh did you mean the how long it takes to actually use it yeah I mean the compilation time yeah the computation time is kind of like 10 times or even worse in some complex contracts exactly anyone that's used via AR today has experienced that so how much of an improvement do you think we are going to see I mean uh so far the Via our compilation pipeline has not been written with any performance considerations in mind at all if you've written it for correctness first and only now are starting to realize how bad that got and uh that we need to do something about performance there so I could imagine that we can get quite a way but yeah it's hard to tell before we're actually doing it um regarding generics how much thought was put into the auditability for external code Auditors will it improve the story make it worse more training can they forget stuff I think it will actually improve things I mean uh we will be able to have the standard Library definitions of all the built-in functionality which can be exported which can be analyzed I think at the point where we have generics going in the standard Library going we will actually at some point not promising that happening soon either but at some point be able to Define the form of semantics for the core language that remains which can actually help homologation a lot and things like that so I think reducing the language core that is built in and hidden in the compiler is actually a good thing for a formal vacation and audit yeah uh you were you mentioned that the data location is going to be coming part of the type instead of being associated with the variable does that mean that we are going to be able to start writing things like an in-memory array of storage pointers or a in-storage array of code data pointers because those are all well-formed but something like a storage array of call data pointers makes no sense right so what does the well-formedness look like how would that how does that sort of you see that restricting that I mean uh first part of the question yes this is what this means this is what will be allowed second part of course there are invalid combinations there are combinations that don't make sense which then the type of stability checked okay then thank you again