Stratified Design: A Lisp Tradition

I spoke at re:Clojure 2020 about Stratified Design. I was lucky enough to have Gerald and Julie Sussman in the audience, some of the documenters of the tradition.

Transcript

(upbeat music) So, here we are. This is my talk, stratified design, a list tradition. I talk about this in my book and I did a lot of research for this for my book. I thought I'd just mention that if you're interested in maybe a more practical view of how to apply stratified design, then I'm gonna go into today. You should check it out. There's two chapters dedicated to this. All right, so I have some questions that I, it's like a project of mine to answer these questions over my lifetime. And here, it's hard to, this is like something that I'm just very curious about and I wanna understand better. And so it's sometimes hard to express the question, the full breadth of the question in English, because when I say stuff like how do computers even work, it sounds like I'm asking like, how do NAND gates fit together to form a Turing complete circuit? And I'm kind of asking that, but something a lot broader. Like how is it that this circuit that's nothing but electrons flowing around in this very complex way is able to do useful work, useful for us as humans, as people with needs, in biological needs, we have to eat, we have to breathe. And we can get these computers to help us do a lot of this stuff. How is that possible, it's just electrons, right? And likewise, what is programming? We write code and it's just bytes on a disk at some point when you save your code to disk, but then somehow it's instructions for a machine to help us do this stuff that we needed to do. What is computation? I'm trying to pinpoint at this bigger, broader question of what is it that we actually do as programmers, as software engineers building real systems. This is a slide from Alan Kay, one of his talks, a recent talk where he was interviewed by Joe Armstrong. And he has this thing, his talks are always very, very much about this, but he has this thing like right in the middle here, this little oval, a computer is a universal atom of meaning. So this idea of a universal atom of meaning is really interesting, but it's very abstract. And so we need to make this a little bit more practical and look at one way that computers can add meaning to essentially meaningless signals. Okay, so let's look at an example of a web server listening on port 80 and some other machine, a client connects to the web socket at port 80 and starts sending in some bytes. Okay, so those bytes aren't very meaningful by themselves. Well, they're not meaningful without interpretation. So all the meaning is in there. We know that all the information is in those bytes, but we need to start going through different levels of interpretation to make that meaning available. So the first step we'll do is we'll parse it or we'll decode it as UTF-8. And turn those bytes into characters. That's one level of interpretation. We're accessing a deeper meaning. And of course, it's gonna look like an HTTP request. So we'll wanna parse that HTTP request. And so now we'll have some headers and a body 'cause that's what HTTP requests are made of. And but still now we need to understand what is this HTTP request all about. So we do what is often called routing and that will give us this one happens to be because we're doing a post to slash users. It happens to be a user creation request. And so the body is gonna have some JSON. Of course, JSON is just text. So we need to parse that. Another level of interpretation and that gives us data structures. Okay, and we also know what to do with it. We know what we're kind of expecting to find in that data structure. So we're gonna interpret it again. We'll call this step validate. And it's gonna validate that we have a user creation request record. Good job, Renzo on those slides. Now, notice we haven't added the meaning. That's not the right word, but we've uncovered the meaning. The meaning was always there. Even in the bytes, it was there. It just wasn't accessible in a convenient way. And so we've, one thing the computer is doing to add meaning to uncover meaning, to make the meaning available is it is interpreting in multiple levels, the bytes that are coming in. Right? And we can go down further. Bites is not the lowest level. You know, down further, it's packets, then below that, it's pulses of electromagnetic radiation going through the wires. It's, you know, we can go further and further down. And the TCPI stack is made, TCPIP stack is made to interpret the, you know, bytes into packets. And then below that, there's hardware that's like doing analog to digital conversion to figure out what the individual bits are. All of this stuff is happening to add or uncover, however you wanna say it, meaning. And we're just aggregating it, aggregating it, digging deeper and deeper until we've got something that we know how to handle. Let's look at the top two levels of this. The top two levels were at the top had this user creation request. I've abbreviated it as user. And underneath that, we had hash map, right? Which is something that closure gives us. There are operations defined in each of these layers. So one operation we'll wanna do on this user is a get user name. And we usually abbreviate that in closure just using the keyword itself. But notice it is a specific keyword that we know is useful at this layer of meaning. There's a lot of keywords that won't be useful at this layer of meaning. Another thing we'll do is get password, which we also abbreviate. And then on the lower level, we have the hash map which has all these things that are relevant to that layer of meaning of hash map, right? So we have get, which will get a value given the key and associate a value with the key. We can get all the keys out. There's a bunch of these. But these are useful at this hash map layer and not really as useful at the user layer. Now, of course, in closure, we kind of play fast and loose. We go back and forth between them a lot. But we can clearly delineate these things. And notice that the two keywords are actually defined in terms of get. So this hash map that we have, it actually does represent a user creation request. And we can operate at that layer of meaning. But it's defined in terms of things below it. That get, you know, those keywords have a method in them. That makes them act like a function, but it's actually that method is defined in terms of get. We can define new layers on top of this user layer. So we can define a layer that's sort of how we store the user in the database. And it's gonna have to know about the keywords that are expected in that user. And so we'll point arrows to them. And we can also define another layer that's going to handle some login routines. And that's, you know, one of those routines will be called authenticate and it'll take a user and it'll compare the passwords, et cetera. And notice we can not only add layers of meaning just through interpretation, but we can kind of go further and sort of the intent of that we're programming at. The intent might be to log in a user. It has to, it's a higher, more specific intention than just a user record, right? So we're adding meaning both through this, like interpreting data and what our code's purpose is. Okay, so I drew this graph and when I was writing my book, I came up with a set of three rules for how this graph could actually be drawn, like how you can go do archeology in your own code and figure out what these layers are. So there's three rules, really simple. You take a function, you look at its definition and you draw a box for every function and then you draw arrows between function A and B if A calls B, right? Then you're gonna have a big nested, like big bowl of spaghetti. So you need to straighten it out so that all functions, all arrows point down. So things only call things that are lower than them, right? And then once you've got that, that kind of starts the stratification process, but of course, some functions are still gonna be able to move up and down because you could stretch the arrows or shrink the arrows. And without violating the rule of a function has to be higher than the function that it calls, you can still, there's still a lot of room. So you still have to go through and kind of rearrange things, looking for the layers of meaning. So let's go through an example. All right, so here's a kind of standard way to define filter. It's the filter that we're used to. It's lazy, it's got two cases, it's checking the empty list. And then otherwise, it's the normal case where it's gonna destructure the first and the rest. And if it's gonna check the first element against the predicate, at the predicate returns true, it's gonna cons on that first element and recurse. Otherwise, it's gonna skip that first element and just recurse, right? And this is very standard definition of filter. And so what we can do is we can kind of go through and pick out all these function calls and not just function calls. We can even use stuff like if and let their macros or built-ins and we can just underline them and then we draw the arrows between them, right? So we'll have filter pointing to all these things. But then also we could see that if we look in the definition of lazy seek, it's gonna point to if. And if we look at the definition of empty, it's also pointing to if. So there's gonna be arrows sometimes in between the ones we call as well. Okay, so now that we've got this kind of mess of arrows and stuff, we're gonna straighten it out by making all the arrows point down. So we're gonna pull boxes up and down so that they straighten out. And so we'll get something that looks kind of like this, right? We had to pull the if down because it was pointed to by things higher up and so it just came down. But it's still not easy to see the layers. So we can still have some room notice to move things up and down. For instance, let is above cons. And I would not say that let is a, is it like has higher meaning than cons? Cons seems a lot more specific. It's got a lot more meaning in it. It's about a certain kind of data structure, whereas let is a general purpose like variable defining a mechanism defined by the language. And now when I look at this, this is this is design, right? There's a lot of intuition and interpretation and iterativeness that has to go on here, right? You might not get it right the first time. You might have to look and try different ways. But you can kind of see here that we're starting to see layers. Like let and if built language built-ins, those are at the bottom. And then there's this middle layer with these four things where it's kind of sequence primitives. And then at the top, it's this higher level tool for dealing with sequences. So we can label these different layers of meaning and draw a box around so to identify them. And that's what we did here. We had this mess of arrows and functions and we just arranged them so that we could see these clear layers. And of course, your system, your software is gonna be really big. It's gonna have all sorts of functions that call each other and no system looks so perfect as this. But we can understand that there's like gonna be from the top, from the main function all the way down to the lower language primitives. You can have this huge tree or it's not really a tree, it's a graph of these functions and we can start to find layers in them. All right, so we talked before about this function called authenticate. And this is satisfying to me that we can do this, but I wanna understand a little bit better what, how does this add meaning? Are we really at the top? Are we really getting the very specific understandings of our stuff and how does that work? So we have this function authenticate. It fetches from the database based on the username and then it compares the password. Now, please, this is not an example of like a secure way of authenticating someone. Don't store your passwords in plain text, for instance. It's just an example. Okay, and we see where it fits in way up at the top. We've already drawn the arrows, right? So this login layer has a thing that's pointing to something in the DB layer and also to the user layer and it fits somewhere in down here. Okay, so how do we know where it goes besides just the things it points to? Because we've got that body that kind of defines like it has to be above the DB layer because it's pointing down. But what puts it above? And I would say it's the name of it. The name is what gives it the meaning. Because we've got something, the things it calls that it's pointing to, those are kind of anchoring the definition of this in lower layers. But the name is going to be anchoring it above or elevating it above. So for instance, if we called it fetch user compare password which is technically correct, that's what it's doing, you're kind of just restating what the definition says, right? So there's no actual, there's very little meaning added to this. And you see this sometimes, this is considered a code smell in a lot of like software design books where you're just kind of restating the implementation. But it's a thing that functions can do which is to give you a shorthand, a name for a large piece of code, right? It's something we can do, but it does not actually elevate it to a new level. And so I think that this is an important idea that this function does exactly the same thing, the computer doesn't care. But as humans, we want to somehow elevate the meaning of this beyond its implementation, right? And so that's what's happening with these layers. We're adding meaning at a human level. So for the readers and writers of this program, we are imbuing more meaning than what the code can, the definition of it, the body of the function can imbue. All right, so let's review before we move on. There's three rules, you draw every function and you draw the arrows between them based on what calls what, then you arrange them. So all the arrows point down. This kind of starts stratifying, but then you have to do a little bit more intuitional, moving stuff around based on what you feel and what the meaning you're looking for. The function body anchors the definition in lower layers and the function name elevates it to a higher layer. This next video is Dr. Alan Kay in an interview with Joe Armstrong talking about why he liked what the AI lab was doing. - I went batshit over in AI is they were the only people in computing that were worrying about meaning. Everybody else was worrying about how to make simple little machines and debug those simple little machines where the AI people were interested in systems that not only could hold larger meanings, but also could reflect on those meanings. - He was blown away because they were the only people looking for meaning. They were the only people out there in software, in computer science looking to understand how we make meaning as humans and to try to use that to make computers do things. Other people were just trying to make simple machines kind of mechanical like we need to do X. So let's figure out how to program it. And I believe that this is key to why LISP has had such an influence on the industry. Even though it's not very popular to actually use it in industry, it relative to other languages. It has looked for a real purpose and stepped back and said what language do we need to achieve that purpose? And the purpose is a big purpose, like a huge vision of like representing human knowledge, human meaning, something like that. And then stepping back and realizing what we need to be able to encode these meaning capturing engines. Which is a thing that LISP is really good at. Other people are saying stuff like, well, we need to be able to code a mathematical function, do a mathematical calculation. And so we get Fortran, right? Which doesn't have the magic that we think of when we think of LISP. Okay, so I think that this idea of different layers is cool. There's something missing. You know, we're just adding a new like a name which gives something. There's often things going on that year may get more powerful. It doesn't quite explain everything. And so I look to Jerry, to Dr. Sussman, and his work on this. Because I think that they've really done a good job of trying to interpret what's going on and explaining it. So I need to give a little background. In structure and interpretation of computer programs, there's a list of requirements for being a language, right? So you have primitives, you have means of combination. So how do you compose those primitives up into something bigger? And then means of naming. In the book, they call it means of abstraction, but I feel like I'm gonna confuse abstraction with generality and meaning. So I decided just to be more direct and call it naming. So that's just how you name something so that you can refer to it later. Okay, so here's our filter diagram again. And we're gonna look at the bottom two, right? So here we have language primitives like let and if. And then the layer above it, we have combinations of those primitives. So a combination meaning we're defining a function that uses let and if, and then we're giving it a name. And then in the next layer, we're using those sequence primitives as primitives. We're combining them, we're giving them names, but what we're building is actually new means of combination. Okay, so we had these means of combination, the how to defining a new function and giving it a name. But now map, filter, take, et cetera, those are actually new ways of combining things. So map lets you combine a function with a list, filter, function with a list, take, it's a number with a list, but it's new means of building lists. So let's look at this layer, it's got primitives, the primitives are sequences, those it gets from the layer below next. It has means of combination, new means, not just the ones given to it by the language, map, filter, and take are new. And then it has means of naming which it gets from closure. So you can still name stuff using the lower level closure stuff. So this is actually a language, the sequence tools form a language. And that is actually pretty cool. Now this is at least according to these requirements of structure and interpretation of computer programs, right? That this is a language that we have made at this new level, this new layer. We have a new language for expressing ideas at this layer of meaning. What I wanted to say is like each of these layers could be a language by those requirements, right? They all have these things, ways of combining and ways of naming that it gets from the closure language. But that doesn't explain everything 'cause a couple of keywords, username and password, that doesn't really sound like a very rich language. It's very, I mean, sure, yes, it's like saying, well, my dog knows a couple of commands. And so that's a language. And it is, it's a way I can communicate with the dog, but it's not a very meaningful way of encoding meaning. And there's, you know, you can't encode new things. You can't teach a dog, new tricks by combining those commands together into something new and giving it a new name, right? So that's kind of what we're at with those keywords, right? So some layers actually have this other property called the closure property. There's a closure with an S, to not confuse it. So here we have, we can see some examples of using map, filter and take. Map takes a list and returns a list, filter takes a list and returns a list, and take takes a list and returns a list. Okay, so this is the closure property that if we're, if we squint, we can see that, hey, I can take that list that filter returns and pass it directly to map because it also, it takes the same type. And then the thing that map returns, I can pass that to take. So, you know, we can start with a map and then we can wrap it in a filter and then wrap it in a take. Here's this other thing that we know of, but haven't mentioned yet, but concat actually takes two lists and returns a list. So we can nest another thing, and look, are all these other tools that this layer gives us, drop map, cat, distinct, reverse, sort. We can arbitrarily nest these expressions because of the closure property, because the return of one function, it can be used as the argument for another. So we can arbitrarily nest it. And so we have this, you know, before we could compose things sort of by, you know, by chance sometimes something would take the return value of another thing, but also we could compose things in sequence. But now we've got this arbitrary nesting, which is a property of human language that we can have dependent clauses that have the similar structure as the bigger sentence that they're part of. And this gives us a new dimension of complexity that we can express with these, with our expressions. And it lets us feel more comfortable when we're expressing stuff. And I think that's important because we, like in closure, we tend to get comfortable with just hash maps and regular data structures. In other languages, they might define new types, but we just kind of stick in the hash maps. And I think this explains it a lot, that it's just so comfortable to be there because we have all these tools and they're so expressive on their own that we sometimes will just get kind of stuck there. Like, why abstract anymore above this? Why add more meaning? And we can see that hash maps do this. They have the same property. So if Soash takes a map, returns a map, update takes a map, returns a map, merge takes two maps, returns a map. Oh, there's a typo there. But that doesn't explain everything that we do in list. Sometimes we'll nest these function calls and it just gets awkward. It doesn't really seem to give us the expressive power we need. In the list, a language for stratified design, the authors talk about using something like a logic language or a rules engine and how these are not easily expressed in nested calls to functions. And so we have this ability, like for instance, also I think outputting HTML within your code, the common thing we do as web programmers. But if it were nested function calls that we were trying to do, it wouldn't be as convenient as what we do with hiccup. Or for instance, expressing like a data log query as nested function calls would be kind of hard. What do we do? We can actually introduce new means of combination and means of naming and build a whole new language with an interpreter. So we can be at some layer of meaning and say, we need a giant leap. We can't just add a few functions and call it an expressive language. We want to kind of make this discontinuity and define a new language that will let us express what the meaning we want to express because we need totally new means of combination and the means of name. We've already got the primitives. We've got the symbols and strings and whatever other primitives we need. But we want to be able to express it in a totally different semantics, a totally different meaning. So we write an interpreter or a compiler and that compiler is going to be defined at some layer of meaning. And it creates this-- I mean, I think of it as a discontinuity because you've kind of made a new realm where it's hard to share things between. And you've got-- because the semantics are so different. But you can make this new layer where you've got this totally different meaning with new means of combination and naming. OK. So we've got these requirements for a language primitives means of combination means of naming. Those show that we can actually consider most layers to be new languages. Then some of those layers are going to have what's called the closure property, which gives us this nesting that is a new dimension for composition. And we can make leaps in any language, but list makes it really easy to define a totally new kind of language that has different ways of combining and naming. And let's just express new meaning. OK. So I've explained all this stuff about meaning and how we do it as programmers, how we build it up. And one of the things we are often doing is uncovering that meaning in a domain. And usually we're working for a business that the software we're writing is basically running the business now. And this is maybe a lofty idea I have, but I have found in my experience that because the programmers are having to wrestle with corner cases and details of representation and definitions in a very precise language, meaning the programming language, they often have a better, more subtle understanding of the business than the non-programmers. Even say the visionaries who started the business. And it's a different kind of understanding, but it's something that programming gives us. And I somehow believe that we will come to understand this more deeply in the future as an industry that one of the values of the programmer is not just to make the software that does what the business needs doing, but who is also a kind of scientist, a kind of theory builder who develops a competitive advantage for the business through a deeper, more precise, and nuanced understanding of the domain. And if a business person could rely on that person as a kind of not consultant, but like a business person could ask questions of this programmer. Like, does it make sense to consider, let's say, the user to be XYZ? And the programmer could say, well, I don't know how we could express that in our software, but we could express this, which is kind of adjacent. And so then the-- and similar. And so the boss could think, oh, wow, that's actually what I meant. And that's a very precise way of explaining it. And this has happened to me in conversations in the business where understanding the domain, at the level of what do we store in the database, how do we encode what the user's intentions are as changes in the database, this understanding has allowed for new ideas of what the business should actually do. This next video is Dr. Gerald Sussman, who's actually in the audience, which is kind of cool. He's talking about using programming to precisely define physics. Because in physics, often, they don't use mathematical notations in a very precise way. You kind of have to learn what the symbols mean culturally. They're not used the same way that they're used in math. And kind of depending on the formula and everything, they use the same symbols, but in different ways. And this is kind of what Sam Ritchie was talking about earlier, which is also kind of cool. It was not planned. So I'll let the video play. But I'm going to talk a little bit about that. How programming helps you understand things that are usually written very poorly. No matter how good the authors are, it turns out that the traditional methods of writing these things are terrible. Let's see, what is the problem, really? The problem learning things like electrical engineering and physics is that we have to learn several things. They're expressed in mathematical language, mathematical notation, and mathematicians and physicists have shared culture. The reason why I can talk to you, and you can understand me at all, is because we're almost identical. We have a enormous amount. We're almost identical physically, and we're almost identical in our experiences. I mean, if you talk about the really grand scale experience in which, as a consequence, it takes only a tiny little increment of information to produce a big effect in transferring some information, a piece of the data structure in my mind, to a piece of the data structure in your mind. And as a way-- but anybody who has shares cultures know that it's hard to talk across cultures. And what happens is that mathematics has shared culture, physics has a shared culture. And the difficulty is that they talk to each other in ways that depend upon the sharing of that culture. Trying to get an introductory student to understand that is very hard, because they have to simultaneously learn the culture and the language, as well as the content. As I like to say, it's like trying to read Les Misérables while you're learning French grammar. It's that kind of problem. OK, so in this talk, Dr. Sussman was explaining that using programming, encoding the physics-- how should I put this? Encoding the formulas of physics as programming would clarify what they are mean, what they mean. Because physicists will use a mathematic notation, but not be precise enough that you can just, as a mathematician, just read it and know you have to have this shared culture. You have to be brought up and like, well, what do physicists mean when they do this symbol? And by having a precise language like a programming language that either compiles or it doesn't, and it runs and gets the right answer or doesn't, it forces you to clarify exactly and precisely what you mean. And so I really feel like this is one of those-- it might take 1,000 years, but we will one day look back and having encoded lots of stuff as programming and feel like, wow, now we have a much clearer and correct understanding of those ideas. OK, so yeah, let's get back down to Earth, look at things a little bit more practically. How can we apply this stuff to our own code? OK, so one thing that I think that these layers show us is that this idea of layers of abstraction is very confused. I was confused until I started drawing this out. Because we'll talk about, oh, I'm making a higher layer of abstraction, or a lower layer, we need a lower layer. What does that mean higher and lower? And I felt like I was not really clear on that. So I wanted to share that maybe why this is confusing. So we have this function called emails of users. And it's going to recurse through this list and basically transform a user into the user's email. If you're paying close attention, you might see what this is the graph for it. And if you're paying even closer attention, you would notice that, hey, that looks a lot like a map. That's just map. So we make map-- I haven't changed the graph yet on the right, but the code has changed. We can just abstract out this function where instead of having it hard-coded that we're looking for email, we'll pass that in as a function to map. And so now we have this nice general-purpose sequence tool called map that we can use everywhere. And then emails of users, the definition becomes very one-liner. And in fact, we probably wouldn't even name this because it's already very clear with just that one line. So let's do the graph. And notice, because emails of users calls map, the arrow has to point down. So map has to go between emails of users and the stuff that map calls. And I point this out because I've often heard people say-- and I think I say it still-- we're going to extract out this higher layer of abstraction map. But higher than what? It actually came down out of emails of users. It's still higher than lazy, secant, empty, and all those. But we actually pulled it down. So this shared thing, this more generic thing, got pulled down. I-- this to me is just saying like, man, we shouldn't even use the word abstraction because we don't really-- it's too confusing. Because of that, I feel like we should be able to maybe clarify our thinking about maintainability and other aspects of our code from this. So here's this original layer system that we had. And I just want to pull the-- pull everyone here, which-- say, we have this spectrum arrow, double arrow on the right hand side. So we've oriented this top down, up down. What side has of this spectrum? Where would we find the functions that are more specific? And where would we find the functions that are more general? I'll just-- I'll let everybody maybe think about that for a moment, maybe write down an answer, share it in the chat. All right, so I think this is pretty clear that general is at the bottom and specific is at the top. So when we're making-- when we're pulling functions apart into more general purpose, utilities, and then the functions that call those, we're actually making more general stuff. So it goes down. And the language is the more general stuff, like the closure language. It's used by lots of companies and for totally different purposes. And so all that stuff is necessarily more general. And then as you go up, things get more specific. The meanings that it can address are more specific and fewer. So for instance, at the top, we just have authenticate. It's like all it can do. It's one very special purpose. OK, so likewise, for maintenance, we can put other spectrum. Like, what's easier to change at the top or the bottom? I'll let y'all think about that for a second. Like, if I had to make a change somewhere, like, change the definition of one of these functions, what would be the one that's easiest to change? And what would be the hardest? So I would contend that the stuff at the top is easier to change. There's fewer things that depend on it that point to it. Going all the way to the top, where you have main, it's easiest to change your main function. It's not going to have an effect on the rest of the code. I mean, obviously, it's going to change your software the most. But it's going to have the least repercussions on the other bits of code. Likewise, if you change stuff at the bottom, let's say you redefine a social to do something different, all the code above it that points to a social is now going to break. And so even if you try to change it to maintain the behavior, maybe you will make a mistake. So it's harder to change. OK, so what is more valuable to test? This might be harder. Stuff at the top or the stuff at the bottom. So I'm going to argue that the stuff at the bottom is more valuable to test. The reason being, the stuff at the bottom is going to change less often, because it's harder to change. And so the tests are going to last longer. That particular test that you're writing for that behavior is going to last longer. And so that test is going to be more valuable. So you write that test for that-- basically, the stuff at the bottom is unchangeable all the way at the bottom. If you test that properly, you'll never have to test it again. You'll never have to change those tests. Well, likewise, the stuff at the top, let's say main, very top, it is less testable, because you're going to change that a lot. And so now you're going to have to change your tests all the time. So you're writing a test for it is not as valuable. You still might want to do it, but it's not as valuable. Likewise, the stuff that everything depends on, like a social, like everybody's program depends on a social at some point. And so having a good test for a social is going to be valuable to more people, to more of your code, right? OK, I think that's clear now. Now, what is easier to reuse? This one might be an easier one. What is easier to reuse? Top or bottom? I will argue that the stuff at the bottom is easier to reuse. It's more general. And just by virtue of having more arrows pointing to it, the stuff at the bottom is going to have more arrows pointing to it. It is just by an existence proof that it is being reused more, and therefore, it is easier to reuse. It's going to be hard to reuse this authenticate method that you've written in somebody else's software. They might have a different representation of users, but if they're using closure, they're probably using hash maps, and all those things are going to be reusable. And then, of course, going lower level down to the language primitives and things, those are obviously being reused by many, many people, many companies all around the world. Oh, OK, my book again, plugging my book again. You can get 50% off with this code. It's coming out soon. I've already sent it off to production to finish off, so it'll be there. I want to open it up for questions if we have a little bit of time, because I feel like we covered quite a lot, and we have some excellent resources in the Zoom window. So thank you so much for this. Thank you, Eric. We managed. We managed. Yeah, it worked. Somehow. All right. Yeah, it seems like an easy job. Like clicking slides, it turns out it's wrong. They do whatever they want. They're just randomly switching. Sorry for that, so I tried my best. But, yeah. No, you did a great job. Thank you. So, virtual uploads, I know you can hear it, but I think that was the time of asking to do it. Yes, let's open it up for questions. I think, so I'm not sure if the questions I were having on the channel, because I was unable, being full-screen, to follow that. But I'll be quickly doing it. Anyone that wants to raise their hands, while I'm collecting the questions from? Yeah, please go ahead, Jerry. Oh, OK. Jerry's asking, I was, can everybody hear me, by the way? Yes. Because I had trouble yesterday. OK. I was, if you go back to the slide, which has a bunch of very complex trees, very large tree diagram, I don't know if you can do that. But it doesn't matter if you can't do it. I'll just say that the interesting thing is there are some other decompositions that have not been talked about, which is sometimes the whole branch of the tree is separate from another branch. OK? I can go down. The big chunk that may not be connected to other things. And that's actually very good to say that really what we're talking about is not just one tree of tower of languages, but in fact, a multiple tower of languages, because the branches that are distinct can be thought of as being separate, rather separate languages. Right? Right. [INTERPOSING VOICES] From left to right. Yes. Yeah, no, that's a really good point. Thanks, Jerry. We're not just creating one concept and then getting a more specific version of it and a more specific-- we're creating a cluster of concepts. Each one might have its own language for expressing the nuances of those concepts. And yeah, there's going to be definitely different towers. A great program is usually a forest of languages. Great, thank you. For example, emacs. Right. Beautiful language, a beautiful system. One of the things that Alan Kay talks about is how the objects in small talk were sort of-- I call it a nexus or a locus of access to multiple algebras. So there's definitely multiple algebras in real numbers, right? Like, there's addition and multiplication. There's all these things that real numbers can do. But he was saying, like, this object is part of a language for drawing it to the screen. And it's also part of a language for saving it to tape. And it's also part of a language for user interface actions. And it's got all these geometric operations that are possible. So this one value is part of multiple language trees in this forest. And it was sort of the nexus of where they all connected up. Elena, you want to ask your question? Sure. So I wonder what you think about test-driven development and sort of little implementations of it, such as writing, figuring out how a function is supposed to work. And I find it helpful in introductory classes when students are writing their functions. And I say, if you can't try to test for it, you don't know what it's supposed to do. But I also found that it helps in naming, because then naming becomes separate from implementation. And I'm wondering if that's a real experience or sort of an observer bias, because that's what I'd like to see. So I was wondering what you can comment on that. Wow. Well, I don't know if I have an answer specifically, but what it makes me think of is that implementation and naming are definitely different. And over time, your sense of the layers is going to evolve as your understanding of the domain evolves. And so you might implement-- you might call something F just to have a name for now, but then realize, oh, this is actually about fetching a user and checking the password. So you name it that. And then you're going, no, actually, it's part of a higher layer of meaning. We've got to call it authenticate. And so to separate them, yes, they're definitely separate. There's an anchor. And then there's the name which places it higher. And so in theory, you could change each of those independently. And I think this is one form of technical debt. As we write software, we are writing it with our current understanding of the domain. And often, even if we're perfectly capturing that current understanding, in a month, we might have a different understanding. And often, we're not even doing that. We're not writing it perfectly. We're under a deadline, and we just get it working. And then we think, well, my understanding is it's A is above B and above C. But I just got it working, and it's really like A and B and C, and they're all kind of like in a jumble. And so then you have to-- when you read that code again later, you read this ABC jumble, but you have to remember, oh, yes, it's really supposed to act like A is above B and C. And so now you have to do this translation. And you feel like you're just making all these work arounds to make it work in this way that you understand it's supposed to work. And you never get around to cleaning it up. And it's a form of technical debt. And it's not just technical debt. It's like psychological debt, because that takes brain cycles that you have to do. Every time you read it, it's not just like a cost to-- maybe there's bugs or something that are also considered technical debt. It's actually like a burden on every programmer who uses that software. Choosing good names is the hardest job within the programmer. Yeah. Yeah, we heard that a few times. I think caching and good names, I think, are the two. Maybe there are more than that. So we have a few questions that I've pulled in from this quote in the Zoom chat. The Zoom chat came first, so I'm going with that for you, Eric. So does this lead to the idea that programmers will eventually just be working in DSLs for whatever domain they happen to be working in? I mean, that sounds like a good goal to me. You know, I'm serious about the business competitive advantage idea that I mentioned, that you're actually able to have a better encoding of a domain than another company. And as time goes on, that will compound, because you're able to express stuff better than they are, faster or however. And it's much more than just copying a feature, right? It's like we have a better understanding of our user and what is important to our domain, what information to capture, and how to represent it. Likewise, whatever other concepts are in your domain. And a DSL, to do that, let's say you have some DSL for representing accounting concepts, you're writing accounting software, like you might actually be able to express more of accounting than they can. They're doing stuff with four loops, and you've got a DSL, you know, you're going to win. And I have somebody with the raised hand. Jesus, you want to make your question? Yeah, hello. Thank you, I want to know if this practice in functional programming, where there is this big dissection between difficult things and easy things and difficult being things like input output, errors, and another side, pure functional stuff. That's like a pattern you see in the programming, functional programming world. Is something that can be explained as a stratified design or is something totally different? I am having trouble understanding that first part about slides on one side and functional. Let me see if I excuse my English, please. No, no, no, no, no problem. Use a functional programming, or in the carpet paper or the ehrlamp paper. They always talk about a big dissection, right? You divide your system, too, like you have the difficult things, typically input output, right? And on the other side, you have the pure work, the functional work. It's kind of an abstraction, but I don't know if this practice is something that you can describe a stratified design or is something probably totally different. I think it's totally different. A language that makes that distinction-- I think a pretty strong distinction between the sort of impure procedural things that have to happen in a specific order and with specific timings and stuff, versus this purely mathematical world where you're just calculating things. I think the language that does that the most clearly is Haskell with its I/O type. So it uses the type system to distinguish between the two. And so they have an entire language. They call it monad. And then it has syntactic sugar to compose these effects together into sequences and alternatives and things like that. And whatever other parallel things happen. That's all done in I/O. And so I don't know if you would find a-- OK, no, I doubt you would find a layer where all this imperative stuff happens. And then another layer where it becomes functional. But you could have-- like Dr. Sussman was saying, different branches, different trees of different languages to describe the two different sides of that. Yeah, yeah, small vertical is true. Yeah, it's true. Thank you. I think I'm trying to understand it better all days. Thank you very much. Yeah, you're welcome. There's a few other questions coming from Discord. So Ian from the Discord is asking, do you think that semantic web tried to solve part of this problem on naming and related stuff? How does it relate to this problem? So OK, I don't-- I haven't thought that much about that. But I know in semantic web, one of the ideas of naming is you're creating a universal name using a URL of a domain you control. So there's some consensus about the meaning of a name. But it's an arbitrary name. It's not a human level name. And as I argued in the talk, the meaning that a name imbues on a function is mostly just human. It's because it's in English. And we understand English and the meanings of the words or whatever language you're using. That's why it has meaning. The idea of a URL, which is not like a human level understanding of meaning, it's nice. It's convenient to be able to kind of universally refer to something. But it doesn't really give it meaning. Hope that answered the question from the chat. Another one from Discord. Given your thoughts about the developer programmer working in the business, not just being the person who writes the software, but a kind of scientist who discovers stuff in the domain, maybe with more precision, do we need a better job title? What that might be. Yeah, so I think about programmers now are similar to scribes back in the middle ages before the invention of the printing press, where you were kind of like this cherished, highly valued person who could read and write. So a king might use you as an advisor. Or they were valued more than just as a human type writer. Like they weren't just secretaries. Take dictation and just write stuff down. They were seen as access to a world that most people didn't have access to. World of knowledge, a world of record keeping, and things like that. And so yes, maybe we need to be considered more like that. Like, hey, you're hiring me not just to make your computers send the right bits, but also to discover your domain. And we have to start doing it. I don't think we do it enough. And if we do start doing it, then maybe people will. People will start to appreciate us for that ability. And also, we need more people to be programming, just like now we don't have scribes, because we have near universal literacy. And so we don't need this special class of person. So if more people learn to program, perhaps they would not-- they would have access to that cult as well. Thank you, Eric. I see a hand up, Eric, do you want to mute yourself? Yeah, Eric. Hello. You mentioned a couple times that in your conception of this, each function, as you're stratifying the design, each function always calls functions lower than itself in that design. There are certainly cases where that's not true. I mean, every time we use a declare enclosure of necessity, that's a time when it's not strictly true. Often, I feel like that's a code smell, right? But there are times when it seems like it isn't, right? And I think, for example, of the meta-circular evaluator. And I'm interested in hearing thoughts from you and/or from Professor Sussman on how to distinguish. When is it the right thing to not strictly do that? That's a really good question. Basically, you're pointing out that I skipped over recursion and mutual recursion. And I did that mostly to simplify the idea. I do think that there are exceptions. They're not as common as functions that are defined in terms of other things. But I think that it does point to something really interesting where you can have cycles in this graph. And somehow, there's possibilities there. I think that it's really dangerous when you're trying to define meanings and stuff, because we don't like recursive definitions, right? When you're defining a word in English, you don't want to just use another word and then you look up that word and it uses the word you're just at. And you have no idea. It doesn't help you understand the thing. So there's some sense that it's not great for creating meaning. You can always remove recursion, right? And so if you have mutual recursion, you can turn it into regular recursion. And if you've got recursion, I'm thinking of like the recursion in map, right? Like, you can just say, OK, this is a special case. You have two cases. One case is the empty case. And the other case is this recursive case. And it's only recursive because the data structure it's operating on is recursive. And so we're defining something in terms of itself, but somehow magic, you know, it's a cool little magical thing that we're able to make something that is defined in terms of itself. I don't know. I don't have an answer to that. So that's just my little rambling, trying to find something to hook onto. Maybe Jerry has something more to say about that. Yeah, I think that there is a bit tiny bit missing that you gave a very nice talk. But there is a tiny bit missing which has to do with, again, the horizontal structure of those trees, OK? At any particular level, you can think of that there are locally, there are interactions between adjacent members of that level, that cooperation. And that's essentially being talked about here, and that that's pretty essential to making things work correctly. So you know, but of course, it might be that some piece of it is local in the sense that it does not stretch across the entire width of your big tree. It's a chunk, OK? And those little sub pieces are units that are in part of a language level that's being used at a higher level, and defined in terms of a lower level. So one way to think about that is that tightly coupled subsystems that are loosely coupled to their neighbors are in fact what's going on, right? It's OK to have such tightly coupled subsystems as long as they are well defined. And that's a major portion of the design process in building a forest to languages. Right. Think about this. Yeah, go ahead. That was the same, thank you. Yeah, I really like this idea of things having to be well defined. They're often not at first while we're still exploring. And sometimes as we build up, we realize that we've kind of cemented in because we've built on top of it, it's harder to change this top below, that we want to change this thing at this layer because we realize there's a better definition, there's a better way to define this. But then that means we're going to have to change everything that comes above it. And I feel like it's a challenge that we have in our industry because business changes fast. We need the business to make enough money to survive. And so we often come out with a very-- we release our prototypes basically. And we have to correct them as we go, as we learn. And that's hard because the stuff that is deep in there, that is the core of the meaning, there's this huge layer of UI stuff that's important for a business. But it's assuming that the stuff below it is well defined and is meaningful. And a lot of times, we want to change that. And we'd have to change everything above it. And it's a big challenge. It's a big challenge. That's why we have to have good dependency structures. Yeah, if I can apply like an idea, that remembers me of-- and I'm asking maybe the audience to see if there are other systems designed like that. But it remembers me of a project from Reechiki that is still in his repo. I just forgot his name at the moment. I'm going to try to look it up, codec with a queue, where essentially, he can draw a list like in memory, a map of the dependencies of all the namespaces into the functions. And this is done on top of Git. But it could be done on top of any other sources, I guess. And I was thinking that I think there was an idea for this system to be part of the compiler, the closure compiler itself. So when this list loads, the map of all the dependencies of these DSLs that are forming is kept somewhere on a database on the side. And this database can be, of course, like a very small system. And then this point, when you change something and walk back to the system, you have a way to track all the places where this little DSL system was involved. I don't know if anything was done like other than experimental, some experimental language that includes this concept of internal dependencies tracking. You know what it reminds me of? There's a language called unison. And unison is a language built with the versioning at the function level. So when we use Git, we version includes a bunch of changes. And it's kind of like looking at the files. So it's these diffs of files. But what unison does is-- let's say you redefine a function. It does a hash of the code tree that defines that function, of the body of the function, and refers to that hash. So that hash becomes a universal pointer to that code and to that version of the function for all time, because it's probably not going to collide. And so the other code that refers to it can refer to the old version while you change it to this new version, which is going to get a new hash, because it has a different code. I think that's a really cool idea, is to semantically encode in the language a semantics for versioning that's safe. Because your old code can still run, because it's pointing to the old version of the function. And now you can redefine it and then define new stuff in terms of the new one. But of course, that doesn't help you with the fact that you want to adjust the high-level code to fit the newer diversion. There's so many dependency structures saying that allows you to track down the root depends upon lists. That's something we need in languages and systems. How much per-- what percent of our lives is spent making systems you go to a version scale? A lot of the-- I was asked to do it. I longed to ask you to track it. The dependencies are not tracked well. Also, you mentioned that you'd be able to see that it was a different version, but the different version might not affect any code that uses it. It might just be that you notice the little inefficiency and fixed it. That's right. And the other-- the code that uses it doesn't really have to know about that. That changes the respect. Doesn't change the respect, right? Right. And all that stuff is really an open problem, right? How do you say that these two functions have the same outward behavior when their code is different? That's like-- that's a hard problem. Sure. OK. So I'm thinking if-- there are-- there's probably another question or two. Do we want to take them? How do you feel? Yeah, I could go for a few more minutes. Few more minutes. And then-- because we are like beyond time, but this is kind of an open meeting now. OK, so this is-- the genesis of this code-- these ideas in your talk come from taming your own code or while trying to grok someone else code or none. Well, I feel like picking apart the question. It's both. So it's working with systems that I'm experimenting with and how to encode meaning, what kinds of meanings I want to be able to express, but also working at companies where it's a shared code base, right? So I'm reading other people's code and modifying building on top myself. And so, yeah, it's other people's code as well. And one last, I think, is there a presentation trap when creating a domain model where overloading terms forces context overload on the future maintainers? Yeah, and I think we see this cost enclosure quite often. If I understand the question correctly. So I mentioned how enclosure, it's very comfortable to stay at the level of hash maps. We kind of venture up a little, and we do some stuff. We know at this point in the code, I've got a user, I can do certain things to it, certain-- I can expect certain keys. But then we associate and we merge it and do some other stuff to it. Like, it's just a data structure. And this is normally fine. The problem is there's a common complaint enclosure where people say, I don't know what keys I have. What kind of data can I expect, or what kind of data should I pass to this function? What does it expect to have? And if I'm in this function, what can I expect to be called with? And what I think the problem is is this jumping between layers. At some point, so when you're first starting your system, it's really convenient because this layer of what the user is is poorly defined, it's changing all the time. And so you want to be able to use it as a hash map, no problem. If you had to define a type at that point where you're not sure yet, you'd be constantly changing the type. And it would be very inconvenient. But then at a certain point, you do know what you need. And the problem reverses, where you know what you need in a user. But you've treated it like a hash map, or anything could be in there at any point. And so now the problem is I wish I had it more strict. I wish I had defined it somewhere and could rely on that. Because I've got all this other code relying on it. And there's so much code, I forget what needs what. And so I think that this is one of those classic problems where we've made it easy at the beginning. But it's easy to make a mess that is hard to clean up later. And I think some people would disagree with me. They would say, well, it's all just data. Just leave it as hash maps and never go to the next layer. Never go, never define a user entity. Just always think about this is just a collection of keys and values, certain subset of possible keys, and certain subset of possible values. And it could also have more keys that you're not expecting. But I've seen the problem firsthand many times, where people start writing long comments about this is what-- this is the kinds of keys we're going to pass to this function. And it has to return a key. And they start using spec to try to figure out what's what. And so I think it's a real issue. And I think that this explains it. It's the two freely jumping up and down. And at some point, you want to lock it down and say, we're going to stick at this level and stop treating it like a hash map. All right. Well, Erica, thank you very much. And we did our best to get your keynote running anyway. So the recording might have a couple of discontinuities that you let me know if we're going to talk about it, so not depending on how good it is. I just want to thank you, everyone who attended the keynote, Jerry and Julie for being so nice and commenting and asking questions yesterday and today. [MUSIC PLAYING]