Why does stratified design work?

Stratified design is one where you build more specific things on top of more general things, typically with many layers. But why is this powerful? In this episode, we explore why it's sometimes easier to solve a more general problem than a specific one.

Transcript

Eric Normand: Why does stratified design work? In this episode, we will explore the idea that some problems are easier to solve in the general case, which is pretty weird when you think about it.

My name is Eric Normand, and I help people thrive with functional programming. Stratified design is a way to structure your code so that more specific code is built on top of more general code.

Just as an example, we all do it. Our programming language has a bunch of general constructs. It has functions, for loops, variables. They're not specific to any particular domain, but then we write our software in terms of those things. It's obviously going to be more specific, unless we're writing a more general language on top.

Usually we're writing an accounting system or something right on top of all these for loops and stuff. It's much more specific about what it's doing. Specific code on top of less specific code. That's just two layers.

You could break it up into smaller and smaller layers, but you always have this property that the more specific stuff has to be built on top of the less specific stuff.

I was reading this book called "Elixir in Action," trying to learn Elixir, trying to broaden my mind, find some new stuff to wrap my head around. In one of the earlier chapters, they build a to-do list.

To do that, you build a multimap, which is just a hash map that can have multiple elements at a key. It's like a list. There is a key, and instead of a value, there is a list of values. It's not hard to write this little extension to a map. You just have a multimap ad or multimap put that puts a new key and value. It checks, is there a list there already?

If there is, then add this to the list and store that as the value in the map. If it doesn't exist, then do a singletonMap where the value should go. You could see how to write this pretty simply. On top of that, they build the to-do list. The to-do list, the key is using a multimap. The key was the day, the date, the day that the to-do item was due or something like that.

You would put in a date, and you would get out a list of to-dos. This is a great example of stratified design. I was very happy to see that this was in the book. I don't know if this is true because I'm not really into the Elixir community that much, but it's a good sign that they understand this idea of building the layers.

You could imagine writing the to-do list directly without writing this intermediate layer of the multimap. You just try to write the to-do list itself in terms of a map. It's doable. It's not that hard to do but it's going to be harder than writing the multimap separate.

Why? Why is it harder to write it in one step than to write it in two steps? This is a weird thing to me and I don't think I'm the first to identify this, that there are a lot of problems that are easier to solve in the general case.

The general case is, "Hey, we are storing a list of values per key. Let's just do that. Instead of making a new type that stores a date and a list of to-do elements, we're going to make this more general thing that stores any key and any list of values." Seems weird that this happens and I just want to explore this a little bit because I think it's at the key to why stratified design works at all.

The stratified design, when you're making the layers, it actually is easier to do it with layers than to try to write it from the ground up with just the basics that your language gives you. In the "Structure and Interpretation of Computer Programs" book, there is a system that...It's not completely developed, but it's developed enough that you get the idea of how it works.

It's how to draw pictures in a certain way and how to compose them up. You can see that it's a pretty powerful system. It lets you draw MC Escher-esque pictures where you start with one picture, and now you duplicate it over here, but one-quarter of the size and then, you see, you can make a fractal pattern. You repeat that over and over and it makes cool patterns.

If the image you start with is cool enough, it makes a cool MC Escher style repeating, tiling pattern. If I were to try to write that myself using the primitives that I have, like draw an image, loop, add certain width to the image, then draw another image over here, I don't know if I would ever solve that problem. I don't know if I could generate cool MC Escher style art.

I could draw something. I could get something working but it would not be where I felt like, "Oh, I have all this liberty and expressivity to make all this whole style of art." In the book, they do it.

How do they do it? It's by building layers. The first layer is just how do you draw an image and then how do you compose two images up comes up from that. The next layer up is I can draw two pictures next to each other, or I can flip the one image, or I could scale an image.

When I put two images next to each other, that gives me an image. I can put two images next to each other, scale it down, flip it, meaning make a copy that is now flipped above it, and flip it, and flip it. You're building these small operations that are each easy to write. That's important. The general operations are often much easier to write.

Now, by the time you build up layer after layer of things you can do, at the top, you see how to build all this whole style of art. It's not just one picture. It's a whole style that's now available to you through these operations. Like I said, I don't think I could start from the beginning and try to implement the top layer directly.

I couldn't do that out of the primitives of the language, but by building up stuff that you do know how to do, one layer at a time, where each layer is built out of the previous layer, I can get to a height that I couldn't get to before. That'd be really hard, let's say.

It has something to do with when you go general, you're actually going more abstract, which means you're ignoring more stuff, which means it's easier to fit in your head.

I don't have to worry about what the pixels look like. I don't have to worry about how this is going to be used. What it's going to look like when I draw it. I'm just making a simple operation. It's pretty dumb. It just flips an image. That's all I'm doing. That flipping operation, I understand how to do it.

I just reverse. I'd just zero minus the x coordinate, and that flips it. Flip it vertically is zero minus the y coordinate. I understand how to do that. Putting two things side by side, also easy, translate the second one by the width of the first one or translate it by the height of the first one. That's it. It's very simple. Very easy. I understand. It's just addition.

I just add the height of image A to the y coordinate of image B. Done. Easy. I can do that, and because I have a simple brain like all humans, we have a limited capacity. We can't think of how to make the whole thing from the top, but by building up these things that we do know how to build at the top, we can get to the top like a step ladder.

I'll just build one little more piece of my staircase, and then boom at the top. You can look down and see the whole thing. OK. I just wanted to explore this idea. I think it's one of the reasons why stratified design works so well.

It's basically because we have limited mental capacity as limited beings. We need to have some way of ignoring stuff so that we can focus on a small piece of it at a time. When you go deeper in the layers, you're actually ignoring more. Remember the stuff at the bottom is more general. Stuff at the top is more specific.

At the top, you've got a whole accounting system, but at the bottom you've got for loops. On that, on top of that, you have account records. You're not even doing anything with them yet. This idea of being able to move down and ignore more and more stuff, it makes it easier to program at that level. I don't think I need a recap in this one.

It's just been a freeform exploration. Maybe I will recap. I started with this idea of stratified design comes from structure and interpretation of computer programs, that name does, and it's just a way of layering your code so that at the bottom, you have the more general code like your language constructs. At the top, you have the more specific.

You can have multiple layers building up to that most specific part. I was reading Elixir in Action and they built a multimap. The language gave them maps, hash maps. They build a multimap, and then on top of the multi map, they built a to-do list. Very cool. I wanted to know, why not do that in one step? What is the use of building this multimap first?

I know that the multimap is conceivably reusable, but I also think that it's easier to build this general purpose multimap than it is to build the single purpose one. It's easier because we don't have to think about to-dos and dates and that kind of thing. We're just thinking, I just need to store values in a list. Some problems are just easier to solve with that general layer.

We can ignore more things. It fits in our brain more easily. If you've enjoyed this episode, you should go to lispcast.com/podcast. There you'll find all the old episodes, including audio, video, and text transcripts. You'll also find links to subscribe and to find me on social media. Get in touch, if you have questions, comments, love to hear from you.

I always love to answer questions about my ideas. I'm very open. I hope that it comes across that I don't know all of this stuff. This is just an exploration.

If you have a better explanation than I do about why this works, why general problems are easier to solve, let me know. I really want to get into this and understand. All right. This has been my thought on functional programming.

My name is Eric Normand. Thank you for listening, and rock on.