Functional Programming and Grokking Simplicity

In this episode of The Virtual World, I talked about my new book "Grokking Simplicity - Taming complex software with functional thinking".

Transcript

(upbeat music) (upbeat music) (upbeat music) - Hey folks, welcome back to The Virtual World. I'm your host and software engineer, Ty. Today is a big episode. This will be the eighth episode of The Virtual World. Today's episode is all about Eric Normand and functional programming. Eric has become somewhat of a functional programming guru over the last decade. He strongly believes in the paradigm of functional programming and in the power of functional thinking and design. He draws a clear line between functional language features and the power of functional thinking. This was a great conversation and one that I really appreciate being able to have. Eric is working on a book called Grocking Simplicity, a book designed to break functional design down to its core principles and introduce it into the lives of the everyday programmer who might not be down to read a few textbooks on category theory first. As someone who has tried that approach, his book is wonderful. His book is called Grocking Simplicity and is published through Manning Publications and available through their Early Access Program while he continues working on parts two and three. Manning was gracious enough to give me four codes for free copies of his book. If you are interested, reach out to me on Twitter @tytr_dev or on Reddit at u/tytrdv. No underscore. I am also officially a Manning Publications Affiliate. This is crazy exciting for me. You can now use my discount code podvworld20 for a 35% discount on all formats of all of their products. 35%, that's actually nuts. Big shout outs to Manning for giving me this opportunity. I'm very grateful and looking forward to working with them more in the future. Lastly, I would like to note that the intro music I'm using is the same as the outro music from the last episode composed by the wonderful Plasmariel. You can check out their stuff on soundcloud.com/plasmariel. That's P-L-A-S-M-A-R-I-E-L. All right, let's get to the good stuff. As always, please enjoy the conversation. (upbeat music) (upbeat music) All right, everybody. Today is August 6th, 2020, and it's about 2 p.m. EST. I think Eric here is about an hour behind and I am sitting down with Eric Normand, sort of one of the, I would say most prolific, functional programming gurus out there right now. How's it going? - Hi, Tyler. I'm good. Yeah, everything's good here. - Yeah, I'm really excited that you came on the podcast. - Thanks for having me. - To any listeners that are interested, so first things first, Eric recently released. I say recently, how long a winded Grocking Simplicity come out? - So it was officially released about a year ago but it's still in early access, meaning I'm still working on it. So it's published at Manning and they try to get books out as quickly as possible just because of the pace of technology. You never know when something's gonna change. So you might as well, as the chapters are done, just publish them. So you can buy it now, but it's not complete. There's about one third of it released so far, but I'm, you know, manically writing chapters and they're coming out at a regular pace, so. - Very cool. So when you say there's a third of it released, the copy that I have is parts one, two and three. Does that mean parts two and three have not been released yet? - That's right, yep. - Aw, that's really sad because I feel like they're gonna be really great. - Well, I hope so, I hope so. Everyone likes that one, everyone. I get a lot of good feedback on the first one. The thing is though, you know, if you're writing a book the way I'm writing it, it's not like, it does get, it does get deeper and harder, but the most important stuff is front loaded, right? Like when you're getting into a new topic like functional programming, getting those initial concepts down is actually the most beneficial thing, right? Getting down the idea of a difference between a function that has a side effect and one that doesn't, that is super fundamental and you can't really do FP without that. So it's this paradox because as someone who's been doing FP for a long time, I find that stuff kind of boring in a way, right? I've been doing that for so long. And what's really interesting to me is the deeper stuff. And I think that you see that in what other people talk about too, but it's actually less useful, right? The deeper stuff, it's less applicable because it starts getting more and more specialized. And so it's kind of one of the curses of expertise is you think the cool new thing that's only useful in this one thing, but it works really well is actually more important than the fundamentals. - I think that's a really great point. And actually that leads me to something I think that's really important and something that your book talks about really quickly and like you said, it's very front loaded. And that is the difference between language features and actual mindset. And that's the biggest issue for me. So some background about myself just so you know where I'm coming from. I would say I am a very traditional software engineer now that is in their mid to late 20s. I say that, but I think most people for some reason kind of peg themselves as a very traditional software engineer, whether they do like embedded systems or Haskell programming or whatever everybody seems to think that they have like a very traditional software engineering background. And so I think we're all kind of a product of the times, but that being said, I'm a very typical JavaScript guy and I'm just trying to like figure out how to do better and how to improve my software and how to sort of like take these other approaches to creating things. And I think the conversation gets a little weird whenever you start talking about functional programming because everybody who's not, everybody who's involved in functional programming is very like the grasses greener on our side. And everybody who's not a functional programmer, for some reason they just kind of end a conversation at like, oh, we have higher order functions in my language. So like I'm already a functional programmer. - All right, right. And I mean, I think that that's a problem with the way that functional programming has traditionally been taught and talked about. This curse I was talking about where you feel like this really hard stuff is more important. You know, it makes you write a book about monads because you just figured out monads and you see like, oh, this is really useful stuff. But you forget that it took you 10 years of functional programming to get there. And so it's not really going to help someone else figure out like, what is this all about? I think also we've been kind of, I mean, we as in functional programmers, we've talked about like you were saying the language features, the interesting tips and tricks that we do and the cool things that we can do with map filter and reduce and stuff like that. And then when they get imported into a language and people start using them in a non-functional language like JavaScript or even Java has them now, they just don't seem quite right. And we don't know why they're not being used the way we think they should, right? Why, you know, they're still cool. But it just points that we haven't done the kind of soul searching as a sub-segment of the industry to figure out really, what is it that functional programming, what unites us as functional programmers? What makes it distinct from just having map filter and reduce in your language? - For sure. And I think one thing that's really cool is that your book has kind of given me a little bit of validation in that I've been feeling this void for a while where I'm doing a lot of these things that I face value like you're supposed to do if you're doing functional design where it's like I'm using higher order functions, I'm doing the map filter reduce thing, I'm using immutable variables wherever I can, I'm doing things like making sure I'm not mutating mutating state unnecessarily and things like that. But it just still feels the same. It doesn't feel like there's business epiphany yet. And so I think, yeah, I think the key is functional design. And right away, your book starts talking about that. We'll get to the specifics of that in a minute because I have been reading it like since the other day when you emailed me and it is really wonderful. To anyone that's listening, I actually have four codes for free copies to the book, which I assume includes feature updates as well. - Oh yeah, yeah, so that's the ebook version, all formats of ebook. Yeah, Manning was gracious enough to give those for the listeners. - Yeah, they seem really cool. They seem really involved and I think that's really awesome. - Definitely. Yeah, their marketing team is really cool, really on point, really big on going around and sharing the love with people. - I'm even more excited that I have you on now because there's one thing that is kind of like the biggest thing for me when it comes to software design and I still think is kind of like the most black box and that is state and state design and modeling change over time. Let's not dive right into that 'cause I think that's a meteor one, but since those parts of the book aren't out yet, I'm really excited to get your opinions on those. To start simple, let's talk about functional language choices. I know this is something they're probably not super interested anymore, but I just wanna get your take on it. - Sure. - Do you wanna talk about this versus kind of everything else? - Hmm. - Okay, sure. This is a really big topic, so I'm just trying to figure out how to approach it. My first thing is, a lot of people ask me, like you were asking, why can't we just do functional programming in JavaScript? Do I really need to use a different language? And the answer, the simple answer is no. You can do functional programming in JavaScript. I do it. I know other people who do it. And you know, insert any language there for JavaScript. But the trouble is when you try to learn functional programming using JavaScript, it is so easy to cheat that you'll often fall back into old habits or using a variable with a for loop or something like that. It's just so easy to do. And the language also isn't supporting the functional idioms as much as a functional language would. And so all that is to say that if you want the immersive experience, if you want, just like learning a spoken language is easier in immersion, it is much easier to learn functional programming by going into a language that is gonna push you to solve it in a functional way. So then we get into what language to choose, right? - One really quick note on there too. I think a big problem as well as the ecosystem and sort of the mindset of the ecosystem when it comes to language choice as well, because even though you can do, and this is something I'm really starting to understand from your book, even though you can do really great functional design inside of JavaScript, you're especially in the JavaScript world, I think, you're gonna be consuming libraries and modules and things that other people have done. And it's actually incredible how many different libraries, especially huge ones from big companies and ones that have millions of downloads a month from NPM, require that you are constantly mutating state of a mutable variable or something like that. - Right, right. Yeah, and it's just sort of, they have to allow it because it's just what people do. If you look at something like React, where the props could be like this really deep nested, you know, JavaScript objects within JavaScript objects, and some value deep in there is changing. Like it has to walk it to figure out, are these have something changed, right? And so it almost doesn't, I mean, I'm not exactly sure exactly how React works, but I think it doesn't walk it. It just re-renders it every time. And you get, if you could assume that everything's immutable, you get an enormous benefit of just checking the pointers because if it's a different pointer, it has changed. And if it's the same pointer, it hasn't changed. I mean, that's basically what immutability means. So there's a lot of like workarounds that you wind up having to do when you don't actually have a functional, you know, functional features in the language. Well, I'll talk about the selection of language. You phrased it as like Lisp versus everything else. The most functional language I've ever used is Haskell. And it was designed with a very strict, functional approach in mind. And I feel like that one has taught me a lot about the functional side of functional programming, you know, doing a lot more with calculations and data, you know, pure functions and immutable values than other languages I've used, even the other functional languages. But I use closure and I prefer closure right now. It is a Lisp. And when you talk about Lisp, yes, closure does, it supports functional programming. It has immutable values and higher order functions and things like that. But in a Lisp, there's kind of a more prominent thing, which is that it's very dynamic, meaning you are interacting with a live system. So I can start up my web server. You know, closure runs on the JVM. So it's a web server is on the JVM and then incrementally compile functions that the web server is using and have it reflect in the next request that's made, right? So I'm running, I'm compiling individual pieces of my software as I change them and can immediately see the result without having to reload everything. So I could have a piece of state, right? I could have some like, I'm trying to distinguish this from a system where like all the code gets reread every time a request comes through, right? This is a persistent server on a virtual machine and I could have a variable that stores like, let's say just the request count. And I can change how that count is calculated by modifying a function, recompiling it and now the web server will use that. And so it gives this very interactive dynamic feel to it that you don't get in any other system. - Yeah, that sounds really cool. I'm curious, what are, you know, from your perspective, of course, what are the sort of benefits and drawbacks of something like closure versus, you know, common list, scheme, bracket? If someone like myself were to try to get into LISP nowadays, kind of how would they choose? And like, what are your honest feelings about the JVM? Is that really a good thing for closure long term? - Oh, wow. Okay, good questions. So the closure is a LISP and it is coming from a long line of LISPs, right? Including common list and scheme. There's a lot that was, you know, borrowed from them. And common list, of course, has a bunch of ancestors too. It's kind of an amalgamation of different LISPs that existed before it was created. And so what I see is that closure is a kind of modern take. It is, you know, common list was written in the 80s. And this was before everything was on the web, right? This was before file systems had basically kind of converged on the slash as the separator of the directories. And so closure seems to me kind of like a reboot. It's like, well, let's learn from them. And but we're starting over. We're not trying to be compatible backwards compatible with common list. So I used common list before I learned about closure. And when I saw closure, I was like, this is weird. Why don't they want to be backwards compatible? But then once I started using it, it just felt way more modern. Like, it was able to leapfrog a lot of the old, jettison a lot of the old baggage, but take a lot of the wisdom with it. Also, the fact that it was on the JVM, I think it was a lot more important back in 2007 and 2008. 2008 is when I learned about closure. It was more important back then because JavaScript hadn't exploded. So just having access to the libraries and infrastructure that the JVM had access to was huge. Common Lisp, one of the problems with it. Now, I haven't, this was 10 years ago, right? So I haven't used common Lisp since then. But one of the big problems was the small community and it being an isolated system. It couldn't share libraries that easily. It meant that you couldn't always find a library for what you wanted to do. And you would find libraries for stuff that other people had wanted to do and took the time to write. But you wouldn't find the plethora of libraries that the Java world was used to. Just as an example, if you wanted to do soap web services, which requires a ton of work to get right. I mean, not recommending soap web services, but sometimes you have to do them. You have to read in a wisdom file and make an XML container for your soap message. - All of this usually proceeds jumping off of something really tall. (laughing) - Yeah, there's, like, you have to have a plan for exiting the situation. - Or at least some safety nets. - Yeah, yeah. Like all this stuff was not even possible in the common Lisp world. And it was just a separate world where they didn't do that stuff. Like, why would you do it, right? What you kind of have to if you need to talk to a soap service and, you know, by being on the JVM, closure could just do it just like automatically. And I think that that was super important. Some other benefits that closure brought were a more abstract view of the list. So Lisp stands for list processing. It is a like fundamental idea this can't sell the pair. It's a way of, it's a very powerful and useful way of structuring memory, right? There's two pointers and now you can use those two pointers however you want, but usually you make a list with them. That's very powerful, but it's very concrete, right? It's very like, you know, you're walking pointers all the time and what Rich Hickey did was say, well, that could actually be just an interface, right? Like in the Java world, you have this idea of an interface and the interface could be implemented by different classes, right? Different concrete classes could implement that same interface and now you're opening the world up to different ways of structuring memory but that all have a very similar powerful way to access them. And so this is in the closure world is known as the sequence abstraction and there's multiple abstractions in the closure world that are similar, but that's kind of the most powerful and most groundbreaking one, it's the most commonly used one. And so I feel like that kind of sums it up. It's like taking a fresh look with more modern eyes at the Lisp experience, taking what wisdom was baked in and learned over time, but then jettisoning stuff that didn't really work anymore and tapping into the resources that the JVM ecosystem provided. - And so do you think closure script is maybe the next iteration of that process? - That's, I mean, that is certainly a possibility. I'm not gonna predict that. So closure script compiles to JavaScript and so that lets it run on the browser or a node or any of the smaller JavaScript run times. - And I think more importantly, let's have access to that ecosystem really easily. - That's right, that's right. And when it came out, there wasn't much of an ecosystem in JavaScript. This was 2011, the most common way to write your front end application was with jQuery, right? And so NPM wasn't that big. It was just getting started. Webpack wasn't around. You know, all these things that we take for granted today weren't there, but it was a good bet. You know, JavaScript exploded. There's millions of libraries now that closure script can take advantage of and more places than ever that JavaScript runs. So yeah, I think it was a good bet. I don't know if I would bet whether closure script becomes like the main focus and closure becomes like an old thing that nobody uses anymore. - Gotcha, okay. Well, so I guess lastly for UI, how do you build your user interfaces now, your websites and whatnot? Do you, are you using closure script? Or I think earlier, you mentioned that you are using JavaScript to a degree, so. - Yeah, I do use closure script. That is my preferred way. Closure script with React wrappers. So React works really well with closure script. Like I was talking about before, the immutable data makes it really nice. The fact that we define most of our components as just functions, you know, we don't make classes and it's interesting because I see a lot of, I guess I'd call it churn and like features, new features of React and stuff like that. Like we've basically been using React mostly the same way with, you know, minor tweaks since like 2013 and we like, you know, poke our head up to see what the JavaScript people are doing and we're like, oh my God, another thing? Like what, how many names do you have to learn? Like just to use, you know, names of libraries and frameworks and stuff, just to use this thing. We've been using the same one for, you know, since 2013 and it's been working fine. That said, it probably is showing its age now, but it still works really well. Still very comfortable. You know, we have a thing called Fig Wheel that gives you a live coding experience in the browser. So I know a lot of-- - I have used Fig Wheel just a little bit and first time that I used it, I think I followed through with like the little flappy bird clone from some article or something and oh man, when like the first time I recompile the function and it just started like working immediately in the browser with like very, very little round trip time there, just my eyes were just kind of glowing. I felt really good about it. - Yeah, yeah. And it's amazing because, you know, if you ask JavaScript programmers, like this has been my experience, nothing against JavaScript programmers, right? But in my experience, I've asked people like, do you have live reloading? Like if you type something into your file and you save it, you see that change in your browser right away, right? And they're like, oh yeah, we've had that, yeah, we have that, yeah. And then you ask them, yeah, but if you type something, like you change like let's say the class of an HTML element and you hit save, like it just immediately, you see the difference in your HTML like output. And they're like, well, not really, because when you hit save, then it recompiles and then it reloads. And if you're not on that few, like because you had to like open in accordion or something, you know, and there's like all these, then you have to click around and get back to where you were. And now you're, now you can see the change. Like, ah, I see, that's not, that's not live. - You know what's funny is that this has like an inverse Morse law to it almost. It's getting worse somehow. Nowadays, there's a lot of like movement back towards like, I don't know how to phrase it, but I've been saying the client is now full stack. So like you'll still have your web services be separate. Like you would expect in sort of a full stack thing in modern days, but your front end is now like full stack again in the JavaScript world in a lot of ways. - You have to have like a server on it. - Yeah, and you have a state and you have a router and you have everything that you have on the backend. - Yeah, and then you like, you change a server route or something like, oh, you add a console log to debug something on the server and it refreshes your entire front end. And like, you have to like fully implement state and sessions for users in your application before you can even like reload yourself onto the same page, whenever, you know, it's just, it's getting worse and worse. And nowadays, like I've been using this thing called Sapper for my own personal like side project stuff. And I swear on my really awesome beefy desktop, the reload time for like a console log or something like a log statement is probably 11 seconds. - Oh man. - It's crazy. - Sorry, I didn't mean to taint it so hard, but somehow it's getting worse. - I mean, I don't doubt it. It reminds me that back in, back when closure script first came out, you know, new and people were using like the only compile to JS language that people were really using was coffee script. And that was super fast to compile. And so people, you know, when something is fast to compile, you kind of don't mind like, let's say it takes 200 milliseconds, 300 milliseconds, less than half a second. You don't mind hitting the refresh button on your browser. You're on your dev machine. That's gonna take less than, less than 100 milliseconds. And so the feedback loop really feels fast, right? But as you add features and you add transpilation steps and things start getting slower because you wanna do like type analysis and you're doing more and more stuff, that loop gets slower and slower. So people were complaining about closure script that it takes so long to compile. But it only, it takes a long time to compile the first time. And so people are used to like, I'm gonna compile from source every time because it doesn't take long. Well, if it takes a long time, you don't wanna do that. And so we don't do that in the closure script world. We compile it once, it takes, you know, it can take a long time, 60 seconds. But then once it's compiled, you only have to compile individual files. And so that's really fast. And plus the JVM is loaded and hot, right? So you don't have that wait time anymore. And you have a watcher on your file system, you're hitting save. And then you've just got a web socket between your browser and the JVM server. And so it's sending up the code really quickly. You can't really beat that, you know? Like there's something architecturally that that that liveness has, that even if the steps get longer and longer, we can just compile less and less, you know? We can play with that a lot more, I think. And I don't know, I feel like I'm going off the rails here, but there's like a, this division between the live coding and the like compile or the edit compile run loop that most languages have. This was done in like the 50s, right? This is that same difference between like a mainframe where you sent your punch cards to and you had to wait a week for the result to come back versus the time sharing where you had a terminal into the computer and you would type something and using threads, right? It would give you some time, like a little slice of time, it'd run it and then give you the answer right back, right? It goes way back to then. And Lisp was on that time sharing side. And the edit compile run loop was on the other side, the kind of mainframe side. And the edit compile run loop like looked like it was losing except then with the personal computer revolution, everyone now has a full mainframe on their desk, right? And so it's fast, you don't have to share it, you don't have to get in line, but you still have that same idea of I'm going to compile a program into an executable, I'm going to run that executable and I'm going to check the output. Whereas the live system, the time sharing system, always had this idea of your program is running, you send it a line of code, it will run that line of code and show you the output. And that's that architectural difference lets you do so much. It makes up for so much like the lack of tooling because you can write your own tool right there or you can write just the tool that you need, like you don't have to write a console log, rerun your program and see what it outputs. You can save that value instead of logging it, save it to a variable and at any time just check the value of the variable, right? You can have it printed every five seconds as it's changing, you know? So there's all sorts of benefits to having this live system that, I don't know, this isn't about functional programming anymore, this is about, you know, live coding, but it's the kind of thing that I think has to be experienced to really appreciate. - Yeah, and I think it's just as important as some of the other things, but with that, let's get back to sort of functional thinking and functional design. I am the type of person that has sort of repeatedly, like I don't believe in either one of the two camps that I find over in the JavaScript world, one of which is the, oh, we've already got functional language features, so we don't need anything else. And the other that's like just kind of oblivious, like functional, you know, we're already making stuff happen. And I do kind of feel that way to an extent, but I still like sort of know that there's something else out there. And I've, I'm the type of person that has read Bartos Miluski's category theory for programmers twice and both times have just come away with just like this glazed, very dazed feeling of confusion. Okay, like, and so I, this is where your book comes into play for me because just the first three chapters, I feel like it's already, like it's already got my wheels moving in the right direction, which is incredible given how many times I've tried to like get to the same level of like a movement in the thinking and how many directions that I've come at it from. So with that said, let's talk about sort of like the core, or one of the core principles of the early stages of the book, which is data calculations and actions. I'll just kind of let you go off on that in brain dump. Yeah, so I try to kind of redefine functional programming to put it like that, to put it kind of bluntly. Functional programming is often talked about as using mathematical functions, stuff like that. Like a very, I would call it a reductionist view, but it's not how I or my functional programming friends program, like we're using mutable state and we're using impure functions all the time. So there must be something that, but we still understand like what the definition is trying to get at, but it's simply not hitting it right. And so that was the opportunity that I saw that there was, this definition was very academic and no one had written anything for the industry, for the, you know, working programmer. And so what I tried to do was figure out like what is before that? And that's what I came up with, this distinction between actions, calculations and data. So actions are in the bad guy. Well, yeah, they're the bad guy in the story. Yeah, but actions are the, what you would normally call an impure function, right? They're not the side effect. See, this is why I had to come up with my own term action. Like a lot of libraries are called like effect libraries, but it's not the effect, right? The effect is the email got sent, right? The cause was your code, right? So I'm talking about the code, so it's the cause, but cause doesn't make sense as a term. So I came up with the term action and this is anything that has an effect or is affected by the outside world, right? The world is changing. Things are signals are coming into your software are being sent to your computer. You have to interpret them. Usually there's some time component to them. So when you run them is important. You know, for instance, sending the email today is different from sending it tomorrow, but then there's also a number of times component. So if I send the email zero times, it's very different from sending it one time and it's very different from sending it 10 times. So anything that depends on when it is run or how many times it is run is an action. Now, if it doesn't depend on how many times it's run or when it's run, it's a calculation. It's still code. It still runs, but it doesn't matter when you call it. It doesn't matter how many times you call it. I'm calling that a calculation. You would hear other programmers call it a pure function, right? And this, the nice thing about calculations is they're much easier to test because notice if you have an action and it necessarily depends on when it is run, that means if it's run in production versus run in development, it's different, right? Or how many times it's run? You want to test a function. You might test it a hundred times, a thousand times in the course of a day while you're developing. Whereas a calculation, it doesn't matter. You can test it as many times as you want. It's not gonna affect the outside world. And it's always gonna give you the same answer if you give it the same input. So the same inputs, same outputs. And so usually in, like a language like JavaScript, an action or a calculation is represented by the same thing, a function, right? So that's where you get the impure functions and the pure functions. The functions is a language feature if you were using Java, it might be a class or an object represents that action or that calculation. And then, okay, so that's action and calculation. The last thing is data. And data is distinguished from the other two because it doesn't run. Data is, if you look it up in the dictionary, the definition is facts about events. So when you think about a fact, doesn't change. Like the fact that I received this message at this time or I read the temperature, the temperature changes, right? But when I read the temperature, the fact was it was at 90 degrees at that time, right? When I read it. And so that is a piece of data. It's immutable and it can't be run. So there's like a double edged sword with that. One, it's really useful. You can like just write it down. You can put in a database. It can be interpreted in many ways. So you can look at that temperature data and interpret it as a time series of like the temperature, right? Or you can look at it like what is the comparing it to another thermometer. You can say what is the discrepancy between these two, right? Or you know, it's sort of like the example I give is when back before the existence of writing where they would tie knots and strings to count heads of cattle that were done in a trade, back then they were looking at it like, I just want to know that I got the money that I was owed, right? 'Cause I needed to count all these cattle. But now we find one of these, we dig one of these things up, this record of this immutable record of that transaction. We don't care if they got their money's worth, but we can now do archeology and figure out like what was the economy like and what was in this year, was it a good harvest? Or you know, we can interpret it in different ways. The downside, the other edge of that sword is that data needs to be interpreted, right? It doesn't mean anything on its own, 90 degrees. What does that mean? Well, you gotta think, you know, you gotta interpret it. Oh, it's hot today, right? You have to apply some code to it to make a decision or a judgment on it. And so data is super useful, super has a long tradition behind it and it's sort of the holy grail as you want all your stuff and data, but then you need to write calculations to make decisions on that data to interpret it and then you need the actions to carry out the decisions. What actions do I take given this judgment? - Yeah, and I think this idea, this is one of the main things I've already taken away from your book. I think the idea of offloading stuff that otherwise could be very long-winded procedural efforts in your code into some sort of like data or objects that acts as not just a definition for something, but also kind of configuration for things that consume it as processes later. The pizza example in your book being a really great one, I find that to be really enticing. And I think it's something that I've accidentally done a couple of times and I've always really enjoyed it. I've always thought like, oh, this is way better than, you know, doing something like very procedural in order to kind of build these data structures up or something. So I think that's an extremely compelling idea. With that, I think you kind of touched a little bit on this, but can you talk about what atomic really means and sort of like what atomic transactions are? - Yeah, sure. So an atomic transaction is a way of doing multiple actions. You know, I'll use the nomenclature we just defined. It's a way of doing multiple actions in a way that if any of them fail, the whole thing fails. And if they succeed, then they all succeed. And from the outside, you can't see the kind of in between states. So does that make sense? So let me give a concrete example. - Yeah, let's do that. - Let's say I have some mutable variable and I want to just keep in track of account 'cause that's always the easiest thing to do. So let's say I wanna increment this number. So when I increment it, I have to read in the current value, add one to that number and then store the value back. It's a three-step process. And unfortunately, languages like JavaScript make it seem like it's a one-step process because they have the plus-plus operator, right? But there's really three things going on. Now, imagine a language with threads. We could, let's say I'm a thread and you're a thread. We're both incrementing this number at the same time. So you read it, then I read it, you add one to your number, I add one to my number, and then we both write our numbers back. We're gonna lose one of the increments, right? Because I did the read first and then you did the read before I got to store my increment into that, right? So it's at 10, then I read 10 and you read 10 and then I add one so it's 11 and then you add one, it's 11. And then I write the 11 and then you write the 11. We did two increments, but now it's only at 11. Should be at 12. So that's not atomic. We need some way of making this three-step process into appear like a one-step process. And so that's what atomic does. It lets you do the read and the write without losing, like if two threads are doing it, it lets you do them in either a certain order, right? So you could queue up and you say, well, we're not gonna read at the same time, reading right at the same time. We're just gonna queue up and I'll go first and then you go or whoever, whatever order you figure out, as long as they're in some order that will work. Another thing you could do is you can use a lock. Just like on a bathroom, you could say, "Hey, I'm in here, don't open this door," right? And so I am now incrementing. Nobody else can come in and increment. So you can't read what's in there right now. You're not allowed. And so that would let me do my business, read the value, add, and then store it back in. And then you just have to wait, just like in a bathroom, a lock on a bathroom. And so these are different ways of achieving the same end, which is to avoid situations of interleaving, right? So if you have threads, the operating system or the hardware, depending on how it works, how the threads work, are interleaving the operations. And you can't control that. So at any point in your program, you could be halfway through some function. And the system's gonna say, okay, your turn is up and start running another thread that might be trying to access that same variable. And you're not done, right? So this is called interleaving, where the instructions from one thread, you know, the machine instructions from one thread are interleaved with the machine instructions of another thread. And then you're totally out of control of that process. So you wanna get some control back, some control over that ordering again. Now, I know JavaScript has one thread. So in theory, this wouldn't be a problem, because if I have one, let's say one callback in JavaScript is doing an increment, and then another callback is doing an increment, only one of them is gonna run at a time. So there's some queue, there's like the event queue, that's making sure that these things happen in some order, right? - Well, there's no guarantee, sadly. It's up to the programmer to, I guess, to make sure that that is actually how it's happening, because you can still program your code in such a way that callback one takes a break for a little bit, and callback two then starts going, you know, so there are still ways to make that same mistake. - Right, and that comes up because you've got a synchrony, right? So if I'm doing, let's say, a multi-step callback. So my callback makes an Ajax request, and then in the callback for that Ajax request, it makes another Ajax request, and then it does another one, and another one so there's like this callback hell chain of callbacks, you know? Each one of those has to go through the queue, and it's started at some later time that's totally chaotic and unknowable, and that reintroduces this idea of interleaving, because now between the request and the response of that Ajax request, other things will run, and you're out of control of that. So even though in the small, like if you're doing synchronous things in JavaScript, nothing can interleave, once you start doing asynchronous things, which is the way you're supposed to do it in JavaScript, you start having this problem again. And so-- - Yeah, I think, sorry, go ahead. - Yeah, in the book, I call these, 'cause I'm trying to unify the idea of threads and these callback chains, and also stuff is happening on the server, right? So you might say, well, in the front end, I wanna do something like increment. I'm gonna make an Ajax request, get a number from the server, add one to it, and then make another Ajax request to tell the server to save this new number. Well, that might as well take a million years. If you got a million clients accessing the server all doing that, they're gonna overwrite each other. And so you also have to now think about the server having its own interleaving thing, right? Interleaving instructions. And that's why I'm calling these timelines to unify this idea of interleavable actions that you have to worry about. - Yeah, I think that's a really great way to go about it as well. So is this kind of how Datomic works? I know that I've looked into it a little bit, and it just seems kind of magical, and everybody that uses it sort of praises it very highly. So do you wanna talk a bit about that? Maybe, have you used it? Do you have any experience with it? - I have used it, yeah. So Datomic is a database written by the same folks who write and steward the closure language. It's a commercial database, I should mention that, whereas closure is open source. And the key thing about the database is that it is an immutable database. That means that instead of updating a row in a table, like you would in a SQL database, or updating a field in a JSON object like you would in like a Mongo style thing, you never update anything. You only add to the database. And that means, the records are permanent, but by a trick of the way you're adding, you can actually, like let's say you want to record a different value for someone's first name. You've already got John's first name in the database, but now you realize you misspelled it, so you wanna change it. You, instead of updating the space on the disk that held his name, you add a new entry that says, well, from now on, here's the new spelling, right? And so this gives you some cool abilities. First of all, you can treat it like a mutable database, right? You can just store stuff and change records and stuff. Without too much concern about it feeling weird, right? So you can have entities with attributes and you're just updating those attributes. But you can also query the database at a different point in the past. So you can say, what data did we have in the database one year ago, right? Or you can say like something that you might wanna do for like accounting purposes, you could say, what was the state of the accounts, you know, the bank accounts on December 31st before the new year, right? Or what was, if you have some, let's say, a bug report, the bug report comes in on January 20th, you can say, what was the exact state of the system when this bug report was filed? Because you might've been, you know, you've been developing and changing this database since then, but you could go back in time and look at it. You can also make it like an audit log, you can say, well, just tell me all the changes that have occurred to this one entity since, you know, the beginning of time. And you can see them all. Instead of basically losing all that information every time you make an update. - The fact that Datomic is proprietary is kind of very sad, actually. I think there's something that I saw that was really cool a while back and this is a library for the closure script ecosystem that sort of mirrors the experience of using Datomic, but on the front end. So instead of having some sort of like global state construct, you just have this sort of immutable database structure, which I think is a really great idea. It almost sounds kind of like event sourcing to me in a way. - Yeah, it's very much like event sourcing. Doing event sourcing in Datomic would be pretty easy because event sourcing, just to let the listeners know, is you have a log of events that occur and you never erase the events. The events are just added to and you can reconstruct the state of the system at any time just by replaying those events through some function that calculates, well, what does the state need to be after this happens and this happens and this happens and this happens? - It's very similar. - Yeah, I think another thing that's kind of interesting too, especially if you take that approach to sort of like this immutable data structure for the database and you use it on the front end, you could actually push that to your web server at some point during maybe like a bug report and then you could pull the entire history of a person's interaction at the app and sort of replay through it. - That's right, that's right. - Which is a, that's sort of, that's one of the use cases that I saw for it and I was like, oh, that's actually incredible. - You can replay it and you know, I've seen where people have, you know, if they're using React, so it's very functional, functional front end, you can actually have a scrubber, you know, and you can go back and forth and see what the UI looked like at different points in time, which is pretty cool. - So with all of that said, I know that we kind of got off into the weeds, maybe a tad bit there, but the, I think the main question that I have here is when moving to, I think state is inarguably probably the most important part of software design and I say, and it's also probably like the hardest to get, right? And I think it's something that a lot of the industry in my experience, of course, my own frame of reference is a little bit skewed due to my experiences, but I feel like we're just getting it wrong still somehow. So my question is in the functional programming world, you know, coming from something very typical, like JavaScript, what does it look like to get state management right for your application without maybe pulling off into the day-tomic world, which sounds like a proprietary, but maybe very easy way to get it right. - That's a really good question and I don't know if I have a functional programming answer. You know, I can't speak for all functional programmers. I do think that functional programmers tend to use a lot less mutable state. So any way that you can do stuff without having mutable state is good. The other thing is that we tend to use more local state, right? So instead of leaking out the state, I guess, you know, there's an old adage that like you shouldn't use global variables, right? And the reason is that if you share these global variables all over your code, like it's impossible to figure out like what actually made the value the way it is right now. And so you're supposed to use like local variables and as much as possible. And functional programming tends to do that. We're not, we're not, what's the right word? We use the variables if we need them, right? We know they're there. In the closure script world, we do tend to use mutable state. I mean, it's our global mutable state. But, so for instance, there's a library called reframe, it's a framework and it has, it stores a global mutable database. That's what it's called. And the structure of the database, it leaves up to you. But it is a global mutable variable. And what we have found in the closure script world is that one global mutable variable is not that bad, especially considering that we also have, in most apps, even if you're not using Datomic, you're just using Postgres or something. Like that's a global mutable variable too, the database. Anything can just read from the database and write to the database. So you have, you know, you have the global mutable data. You're just kind of, you know, pretending it's not there in your front end app if you're trying to be super pure about it. I think the fact that there is only one global mutable variable and then all of the data is immutable. So you can't just change the value of some object or some hash map willy-nilly. You have to change it always through that one global variable. That really helps. And so I actually came up with kind of a scheme, taking these things, this into account. This idea of, okay, let me back up. So I've been trying to like figure out what people are doing in the React world to manage state. And what I see a lot of is people jumping between libraries, thinking that this library is gonna solve the problem. And this library, oh no, let's all switch to this one because this one has a better model of state. Now let's go over here because, you know, that was wrong, we need this other thing. And there's actually the state management ecosystem in the JavaScript world is probably my absolute least favorite thing about it. I don't like any of the libraries. I think they're all terrible. And especially there are some, like there's this one called NGRX in the, sorry, I get heated about this particular topic. There's one called NGRX in the Angular world. And my God, the amount of boilerplate, like to say that you have an array of things that you need to update and read from is just a ridiculous amount of boilerplate code. And the amount of, I would say, mental complexity that adds, or what is the word that people use? I forget, but basically the amount of stuff you have to keep in your head at any one time to understand what's happening with these things, I think is like excessively high. - Yeah, and so it's really, I mean, at some point it feels like a vindication, but at another point it's not really because it's not, we didn't wanna have, we didn't wanna have revenge or anything. But that's what people have always said about functional programming. Like, oh, it's so much overhead. I just wanna make it work. I don't wanna deal with immutable values. I don't wanna deal with like, writing higher order functions. I just wanna for loop and I just wanna make it work. But when you do that, that's what happens is like, you wanna write something more abstract, like a library has to be. By abstract, I mean more general, right? Like you wanna organize this best practice into a library and now you have to deal with all of this, the mess that is created by the mutable state. And to make just one array, you have to, like you said, set up all this stuff because it's mutable and the thing needs to know when any of it changes. Whereas if you just use an immutable value, it can't change. You can just not deal with that. So my, what I realized was when people are jumping from one thing to another, they're kind of thinking like, we want the one silver bullet state library that's just gonna make everything easy. And that's the wrong level to look at things, right? You don't wanna look at, you need a deeper understanding of the problem of state as a programmer. Maybe this is even like architecture level thinking, but you need a deeper level to choose the library. And so what I came up with was a scheme, like a spectrum of locality of state. So you have, some state is super global, meaning like every user of your system needs to have access to this. And then there's some data that only one user needs to have, but of course it's shared between different devices. And then there's some state that is local to one device, but all your tabs will share it, right? And then there's some state that only one tab needs. And then there's other state that just one component needs, right? So like the tab is, you know, when it's at the tab level, maybe it's all the components in the tab might need to access it, right? And then you have the stuff that's just the component, the stuff that's that's super local, like is this drop-down box open, right? You're making a drop-down box component. You just want to have a little bit of state in there just to know open or closed, right? You want that state to be as local as possible. You don't need to share that with the server. You don't need other users certainly to know that. And so by looking at it this way of locality, you're able to say, well, this library works for this particular locality of state. So you look at something like Redux, something I'm somewhat familiar with. I don't know all the state libraries out there, but I do know Redux. And it is-- - They're kind of all essentially the same thing. - Well, but so, but now there's what, recoil and context and there's all sorts of stuff. Well, Redux is local to the tab, but shared by all the components, you know, potentially shared. Anything in that Redux, what do they call it? The thing that holds all the data. - The store? - Yeah, the store. Anything in the Redux store could be read in by one of the components, right? And similarly with recoil, it was designed so that different components could share a piece of state. And so by looking at it like, and then of course, the component state itself, the like, you know, set state, get state thing that the components have, that's very local to the component. So by looking at it like this, where, you know, people have been kind of targeting a certain locality and then they'll give a few examples when they're introducing the library and they're like, look how great this is compared to what you're doing because it solves all these problems. Well, they're only just solving that one problem, that one problem of what if you have some value on the server that you need to share with all the devices that this user is logged in with, right? Or what, you know, we have all this state that we need to share between components and like, how do we do that? But no one is looking at the deeper level and saying, well, actually in some cases we need this library and in other cases we need this library, and in other cases we need another library, but that doesn't even exist. So we're gonna have to write our thing ourselves. They're basically looking for someone to fix all their problems with a single stroke, right? Some single library. - On that last note, there's a couple, there's actually a funny story. I think, so NGRX, this is my least favorite state management library, so I kind of, I talk about it a lot actually. Me and an old podcast guest, we talk a lot about NGRX and the way that they sort of like, I guess market themselves when you're first reading about it. They do that same thing where they're like, look how much better this is, but they take it a step further, which is they actually start to like, mansplain to you that everything you've ever known about, state is wrong and they start to shame you. Like there are little bits of the documentation that's like, you know, how could you ever think this is right? How dare you? - Oh God. Yeah, that's bad, that's bad. I mean, the real trouble is, I mean, they probably do know a lot, right? The problem is- - Oh, for sure. - That our industry is continually being reborn because it's growing so fast. So at any point in time, it's not fair to say, like why didn't you know this? You should have known this. Like no, like half the people have less than five years of experience at any time. And so- - Ah, yes. The immaturity factor, that is an interesting topic in and of itself. - It's not fair. It's like, you know, when you hear someone talking to a kid that way, you're like, hey, they're only three. Like no, they didn't know that. And it's not unreasonable that they didn't know that. So yeah, talking down- - I think that's a, yeah, that's a really great way to put it. And I think I'm probably over-exaggerating a little bit, but yeah, that's how it feels when you're reading the documentation. And it's really interesting because it's something where they're tackling specifically JavaScript client-side state. And state is like the core problem in software. And it's also something that's somehow, in the last 80 years, we haven't solved this problem. But still, there's a group of people out there that will like shame you because you haven't mastered the art of mastering state in JavaScript clients. - It's hard, it's really hard. I feel like that is, I mean, that's one of the things that we do in functional programming is we're like, it's too hard, right? Let's do something easier, which is not a beautiful state. - Right, and I think that's a step in the right direction. The problem as well, I talked with a recent podcast guest about this, and that's like when you change tooling, you legitimately put yourself, like you take a step back in your skill level, like if you change to a new type of keyboard, or if you're instrumentalist, you're a musician and you switch to a new instrument, or anything like that, you switch to a new programming language, like sometimes you just feel like a child. And it's not because you should feel this like incredible imposter syndrome, or you know nothing, it's because you're unfamiliar with the territory. And I think that is probably the biggest problem in the JavaScript ecosystem, because JavaScript right now just happens to be where the money is. Most people that are doing software, they're just trying to pay their bills, be their family. And so they're, you know, they go to X company, and X company doesn't say, hey, let's sit down and like talk about the core problems of software engineering, they start with like, we need to pick a library for this, we need to pick a library for that. And so every six months, you're, you know, in some new world where you're mostly just trying to get some deliverables done, and you're trying to just glue the pieces together the best that they'll fit. - Right. - And you're not really dealing with like core fundamental issues. - Right, and as you gain experience as a programmer, you do develop a kind of skepticism about new languages and new libraries. I had the good fortune of working at a company where they kind of called me in as like the grown up. You know, I learned that everyone was in their early 20s. And this was like their first piece of software they were writing. And I could tell that one of the things that I could really provide value on was just like, hey, don't upgrade that library. You know, don't switch. Just stick with this one because that other one is gonna have like it, the library you have has problems, but the other one is gonna have problems too. And they would get so, you know, kind of grasses greener and believe the hype of the library without really understanding how it worked. I mean, you know, if they're under tight deadlines, they don't have time to read the code and figure out what it's actually doing. They're using, I mean, it's smart in a way, they're trying to outsource all of that to a library. I don't, not just outsource the code, but outsource the understanding. And sometimes that doesn't work. And it was really interesting because I would, you know, there's like the not invented here syndrome, but some of these libraries are so new and so untested that you kind of should write it yourself sometimes. Like, 'cause then at least you know it and you control it and you understand how it works. I don't know, it's a new and complicated world. - Yeah, and it's layers upon layers as well because it's not as simple as being stuck in library hell because in the end, once you get used to being in library hell, that becomes part of your tool set. And so breaking out of that becomes unfamiliar and starts to feel kind of scary, just like you don't have libraries for everything. - Yeah, exactly. - Which library do I use? I don't know, oh, well, we just write it ourselves in this language, it's three lines of code. - There's a lot of pressure too, I feel, to constantly be upgrading and whatnot. Here's like a concrete example of this is, and something I've constantly criticized React about is the ever-changing API with like super minimal gains. In one case, this was before, I don't know, do you know what React hooks are? - Yes, I've read about it. - So basically, back in the day you would just, you'd have this like state object in your component and you would use this function called the set state in order to update that state and then it would like decide what it needs to re-render and re-render those things. And then they switched to like this idea where you'd have one call to this state mechanism per thing that you want to be stateful about and then you would de-structure the return from that function into like the variable itself and then a method to, or I guess I should say a function to update that data. And like six days after the API update that came out with hooks, I put a question on Stack Overflow about some state issue that I was experiencing, some bug that I had found in the React code base actually. And the responses were largely not like, how do I solve this problem? It was mostly just like, wow, you're not using React hooks yet. - Nah. - What a nerd. (laughing) - Right, right, like I'm gonna read like that. - And so there's a million line code base. (laughing) - Right. - Yeah. - Because React decided that they didn't like the syntax for state today. - I mean, I honestly don't know like, are those people who are just like, they just started like, you know, a week ago? (laughing) Or they're, you know, it's possible. This thing is, we're growing so fast. Like, there could be people who don't have any legacy code yet. They're not, they don't have a job yet. They haven't, or they're in a startup where they're making all the decisions. You know, all these things are possible. - Yeah, I think it's weird too, because on the internet, the voice is sort of whitewash, like everybody's the same entity, they're the not you. So it's hard to know who's got wisdom and who doesn't. - Right, and of course, the people with wisdom probably also don't have the time to-- - Yeah. - And train stack overflow question. - For sure, the people with wisdom are out doing something with it. - Right. - And so it's really, yeah. It's kind of a weird grind, but I think I've officially exhausted everything that I can use to sort of appear insightful in this conversation in particular, but I'm curious. Do you have, are there any other topics that you wanna discuss or things that you wanna talk about? - Not really, no. I mean, I could talk all day, but I don't want the podcast to be forever and I will have to go soon, no. - Yeah, no worries. So I guess to wrap up, where can people, like where and how can people support you beyond checking out your Twitter and LISPCAST? Do you wanna talk a little bit about LISPCAST for anybody that's interested? - Yeah, yeah. I'll talk about my various things. So the book, you can find the book, it's called Grocking Simplicity. It's at lispcast.com/gs for Grocking Simplicity. That'll take you-- - Which I have just found out is actually like part of a, it's not part of a series necessarily, but there are a bunch of books called Grocking X in the Manning-- - And they all have very similar covers. - Yeah, they all seem really cool. - Yeah, yeah, it's a cool collection there. So I'm on Twitter, I'm Eric Normand on Twitter, like to talk about functional programming there. My blog is lispcast.com. That's where I talk about, you know, my kind of high level ideas. And then I teach closure at purelyfunctional.tv as video descriptions. - Which I'll be subscribed to. - Oh, cool, are you interested in learning closure? - I've been interested in learning closure for like about a year and a half now, and it's just something that has been, it's been pushed off only by my sort of daily responsibilities. So, yes. - Cool, well, I want to totally revamp my site, but COVID and the kids being home kind of put an end to that. So I need to get back into that. But yeah, it's a collection of like long form guides. Those are free written guides and then videos that you can buy. That go, you know, if you like watching someone code over their shoulder to understand the issues, that's what those videos are. And yeah, so that's where I, how I split my time, writing this book and making and teaching closure. - Yeah, don't forget to mention your YouTube channel because that's where I found out. - Wow, right. - There's a lot of great stuff. - Yeah, yeah. So I have a podcast, I publish it to YouTube. I record the video of it just because, why not? So it's, you can find the podcast and all the links to the YouTube and everything at lispcast.com/podcast. And it's a podcast about functional programming. And it's very informal, it's just me and an idea. And they can run from, you know, 15 minutes to an hour and a half, me ranting and rambling about some idea. And sometimes I will read a paper and read excerpt, like an important paper in the history of computer science. And I'll read excerpts and just comment on them. - Yeah, I did catch your out of the tarpid analysis. - Yeah, how'd you like it? - I, it kinda like was doing some other stuff at the same time, so I need to go back and watch it again. But I think, it's one of those things where really, I think I need to start by reading the paper myself. - Yeah, yeah, probably. - Which I've tried a couple times and it's not the easiest read. - No, it's a very wordy paper. I'm experimenting with the format. Like part of the thing is, I think people need to read more papers. And I don't really expect, yeah, and books, and I don't expect that they are going to. So people are listening to podcasts. So maybe if I highlight some of the cool bits from the paper on the podcast, and one of the reasons that they're hard to read is you kinda need some background information and context. Like some of these papers are so dense and tight that even after reading them three times, I'm still finding new stuff. And so I kinda think that that's part of the commentary is to say, "Huh, in this phrase that they're using here, we gotta unpack that." Because I can't expect someone to read it three times if they're not gonna read it one time. So that's kinda the value I see there. It's just really getting, we just don't know enough about the history of computing as an industry. And it's great to surface these old papers. - For sure. I think, I definitely will be creating a t-shirt probably that has someone with a cup of coffee or something that says drink coffee in an increment, as kind of like a joke about the bathroom scenario that you put together earlier. - Cool, cool. - So I'll probably gotta create some sort of joke t-shirt from that 'cause I thought it was hilarious. But yeah, I definitely appreciate you coming on the podcast. I think maybe in a month or two, once I've learned some new things and have some more interesting questions, maybe you can come back on and we can chat again. - Yeah, yeah, no problem, this was totally fun. Thanks, Tyler. - Thank you. All right, folks, that wraps it up for this episode of The Virtual World. I hope you enjoyed getting a solid introduction into Eric's thoughts and approach to functional design. There's a ton more great information in his book, Grocking Simplicity from Manning Publications. Don't forget I have four free copies of his book to give away. If you are interested, please reach out to me on Twitter or Reddit @tytr_dev. No underscore on Reddit. Also keep in mind that I now have a discount code and affiliate link with Manning. You can use my discount code podvworld20 to get a 35% discount on all of their products. It will still be a day or two until the affiliate link is ready to go. I will be including an affiliate link to Eric's book and all of my posts about it as well as the podcast episode description once it's ready. Thanks again to Manning for all of their generosity. Check them out on Manning.com. I will also be interviewing another one of their authors, Tim McNamara, the author of Rust in Action very soon. Stay tuned for more great content coming your way from The Virtual World. This is Ty signing off. And don't forget to enjoy this cover of The Great Fairy Fountain's theme from the Legend of Zelda series, By Yours Truly. (upbeat music) (upbeat music) (upbeat music) (upbeat music) (upbeat music) (upbeat music) (upbeat music) (upbeat music) (upbeat music) (upbeat music) (upbeat music) [BLANK_AUDIO]