Sources of complexity in software

There are two sources of complexity in software: the complexity inherent in the domain (essential complexity) and the complexity we add as programmers due to the platform or due to bad programming practices (accidental complexity).

Transcript

Eric Normand: What is the main obstacle to writing correct software on time and on budget? Hi, my name is Eric Normand, and these are my thoughts on functional programming.

There is a book called "The Mythical Man-Month" that talks about one of the main challenges of software is its complexity. That software is just complex. It's just going to take time and energy to make it correct, to make it work for us. I think that's fair to say. In the book, Fred Brooks, the author, talks about two kinds of complexity, essential complexity and accidental complexity.

Essential complexity is a complexity you cannot avoid and accidental complexity is complexity that you add, that you didn't need to. There might have been a way that you could've gotten along without it.

For what it's worth, that's a fair way to divide it up. There is a paper about 14 years ago called "Out of the Tar Pit." This is a very influential paper in the functional programming world, because it does a little bit better at dividing up the complexity, of defining the two types of complexity than Fred Brooks did.

For instance, Fred Brooks said, "Oh, there's just a lot of state in software and that's unavoidable." As functional programmers, talking about state as unavoidable, we know that's not true. You can avoid it.

This paper defines essential complexity, I think, in a more useful way where essential complexity is complexity that is due to the domain. If you are going to be making a piece of software about rocket science — rocket science is just complex by itself — your software has to have at least that much complexity, as much complexity as rocket science.

All of that is due to the domain. It's essential. You wouldn't be doing rocket science correctly if you didn't have that complexity in it. However, there is another kind of complexity, which is all the complexity that we add as software developers. This is everything that has nothing to do with rocket science.

This is threads, the file system, exceptions, syntax errors, types. All that stuff is complexity that is added on to the system if it adds complexity. Some things you could argue. These are actually removing complexity.

In general, if there is some complexity and it's due to something that when you name it, it is not naming a rocket-science thing it's naming a computing-thing, that is actually accidental complexity.

What this gives us is a minimum floor for our complexity. Everything else above that is all accidental. We can work with it to reduce the complexity or at least manage it better. Know that we're introducing complexity but we're managing it.

I say we're introducing complexity. For instance, the platform you're running on is going to have complexity. It's going to have accidental complexity by itself. That means you're making Web-based rocket science software.

Now you have to deal with browsers, the DOM, the AJAX requests, HTTP. All this other stuff that has nothing to do with rocket science, that just has its own semantics and some of those semantics have complexity to them.

You have to deal with timeouts of your network requests. You have to deal with all the different response codes when maybe you only needed two. These are the kinds of complexities that start adding up.

In general, we see in most software that the accidental complexity swamps out. It's way bigger than the essential complexity. That means that in your rocket science software, most of the complexity is due to the implementation and not to rocket science itself, which is a sad fact, but that's also optimistic.

It means there's a lot of room for improvement. That we're doing this to ourselves as programmers. We're adding in...Some things are maybe functional or operational requirements that are not part of rocket science.

You might have a functional requirement that says, "We have to implement all of rocket science." That's great. Then you might have a requirement that says, "It has to run in the Web browser." This has to be Web-based software. Now, it's a requirement.

You're going to have to take on all of this complexity. It's something that the business has to decide. That the deliverability, the availability of the Web as a platform has its benefits versus the cost of all that complexity, because complexity cost money, cost time, developer time dealing with.

The real reason it costs time is because when you have something that's complex, it's hard to hold in your head. We have limited space if you will. It's a metaphor. We have limited capacity in our heads, in our minds to understand complex things.

The more complex it is, the harder it is to hold in our head. That's going to cause bugs. We're going to forget things. As the system gets more complex, things slip through the cracks, and so we need help. We need to do something about that complexity.

The Tar Paper book out of the Tar...Sorry, Tar Pit not Tar Paper. The Tar Pit paper is all about using functional programming to reduce complexity. It does a good job of beginning that search. It's laying the groundwork for how functional programming can reduce complexity.

I feel like there is a lot left out that we still need to deal with. For instance, a kind of hand waves around state, about mutable state and storing things durably over time. That's fine. It is a paper, it had its scope. It's still a very great paper. I suggest you all read it.

One thing that we can do besides just eliminating unnecessary stuff that we're adding, like do we really need mutable state here? Maybe not. Do we really need extra threads here? Maybe not. Those things are obvious, eliminate them. Sure.

There's got to be some stuff for managing it. Maybe for making it easier to work with. These are like developmental concerns. For instance, if you could encapsulate the complexity behind something that is a good abstraction, let's call it that.

If you could somehow take all of the complexity of reading in files and storing them and stuff, and just turning it into a simple interface that you don't have to worry about it. If this complex thing is written correctly, I can stop thinking about it.

That's one way of dealing with complexity is to encapsulate it. Another way is to cut stuff into smaller pieces. If you have two complex pieces...I'm going to get into this more in a future episode.

If you have two complex pieces and you break it up into...Sorry, if you have one big complex piece. Imagine its complexity, and you broke it up. The complexity, not only is it easier to fit in your head because it's smaller, it's two smaller things, but the complexity is actually cut in more than half for each piece.

The reason is that complexity compounds super linearly depending on the type of complexity. Just as an example, the complexity of the number of code paths you have in, say, a routine, a function, that complexity multiplies.

Let's say, it's a simple case. You just branch into two possible...conditional has two branches if it's an IF statement with a THEN and an ELSE. Every time you add one of those to your function, it will multiply the number of possible code path by two. If you get rid of one, you're actually reducing the number of code passed by...You're dividing it by two.

If you multiply them out when they're in a big single unit like a one function with 10 code path, that's 2^10 possible code paths. Sorry, you have 10 conditionals with 2 branches each. That's 2^10 code paths.

If you cut that in half, you have two functions with five branches each. Let's say you could do that. Sorry, five conditionals each, two branches on each conditional, that's 2^5+2^5. That's way less. Before you add 2^5*2^5 that's 2^10, but if you do 2^5+2^5 you have significantly reduced the amount of complexity. With tricks like this we're able tricks...

There are skillful manipulations of the units that we have to reduce the complexity and just make it easier to work with. The code should do the same thing but have fewer...It's less to hold in your head. Let's say it that way.

All right. My name is Eric Normand. This has been a thought about functional programming. You can reach me over email at eric@lispcast.com. I do enjoy getting people's emails and hearing about your thoughts on my thoughts.

You can also reach me on Twitter. I'm @ericnormand. Follow me, message me, and mention me. We'll get into a cool discussion. I really appreciate doing that. Thank you so much for listening. Subscribe where you subscribe to this. You'll hear about the next one when it comes out. Awesome. Thanks so much. Bye.