Is there a silver bullet for software? (part 2)

We continue this exploration with the notion that maybe practices such as Agile, Design Thinking, etc., give us a means for eliminating essential complexity that we thought was necessary. Do we really need to implement everything? Maybe not.

Transcript

Eric Normand: "Is There a Silver Bullet?" part 2. My name is Eric Normand, and I help people thrive with functional programming. In the last episode, I talked about the history of the ideas of essential versus accidental complexity.

It all goes back to "The Mythical Man-Month." This book right here by Fred Brooks. He has an essay called, "No Silver Bullet." He talks a lot about how there is an essential complexity, meaning the conception of what to program, and how to program it is complicated just by itself. Things have changed since he wrote it.

The biggest thing that I want to talk about in this episode is the big idea that's getting more and more traction in conference talks, especially at Agile conferences, is the ability to filter out ideas so that you're building the right stuff. I imagine that 50 years ago, when the events of this book took place in The Mythical Man-Month, and the essays were written, that people sat down and had ideas at meetings and came up with a plan, a big spec, for how things should be written.

He talks a little bit about how waterfall is a bad idea, we need something more iterative. It's evolved more since then, that most of the complexity that we add to software is unnecessary. There are talks online about, for instance, how you can solve 90 percent of the problem, or let's say 80 percent of the problem, but we're using software.

Some accounting department at a company, they process transactions, they process invoices. They have to decide what to do with each invoice. 99 percent of their time goes to the main, easy case invoices. Why spend a lot of engineering effort to get a perfect invoicing system that covers all of the weird cases?

Because they don't even know what those rules look like. They have to take each of those 1 percent cases on a case-by-case basis, but 99 percent of the time is on the easy cases. Why not write software that completely handles the easy cases and frees up the accounting department to do a better job with those one percent?

Then, you could extend the software if you'd need to. If you say, "Oh well, we'd save another 10 percent of our time if we added this part to the software." Sure, add it.

We often over-complicate it because we try to make a complete software solution. When there's people in the loop, anyway, they're already in the loop. It doesn't have to be 100 percent, especially in a department in a company. This isn't a solution that you'll say, "Oh, this is gonna totally relieve you from even having an accounting department."

No, you still have the accounting department. You just have this thing do the easy work, the mechanical wrote work, that's easy to decide. All that is to say that I could imagine 50 years ago, the accounting department sitting down with the development team, and spending two weeks trying to figure out a complete set of business rules to totally solve accounting.

Once they had that, then they would spend the next 12 months implementing it. It's very waterfall, but it's more than just about the waterfall. It's about the idea that you would have to solve the complete solution, that you couldn't just start with a partial solution. That partial solution might be enough.

If you can eliminate 99 percent of the work, you can have a smaller staff, or the staff could be working on more important things. This is a big savings already from a business perspective. I'm also thinking of things like design thinking. Design firms that come in, and you hire them to solve a problem with a product for a market.

They come in, they prototype, they have a method that will find something that will do really well. They have a very good track record. Much better than sitting down with a random group of people and writing down everything that we might want this software to do. Those kinds of things are actually getting at the core problem.

They're prototyping, they're doing all sorts of things that eliminate a lot of the complexity that would be called essential back then. There's also movements like, "I'm not a big fan of the Lean startup movement," but the idea of Lean startup is similar that you probably don't need all the features you think you need, especially to get started.

Don't implement them yet. Implement something smaller, but do a better job at it, so that you can get good customers who you're actually solving a problem on.

Then there's Agile, which is to say...I'm not going to go into what Agile is, totally, with a perfect definition. The relevant part of Agile is, let's ship something all the time.

Whatever we have, whatever features are available, we're going to make them part of the product. We're going to be able to adjust from there.

Because we will never have a complete product. That takes 12 months to get to market. That's too long. Let's just start shipping what we have. The idea is also very similar that you might change your mind about what you want.

Why write up a big spec and then start implementing it? This whole idea of essential complexity can be broken down into...Basically, the stuff that you were writing that was so complicated could actually be eliminated.

It could actually be simplified further that you probably don't need that extra feature, or people don't actually have that problem, or when it does happen, you need a human in the loop anyway.

You might as well just kick it out of the system to the human. You don't have to handle it in your software and making your software more complicated.

I feel like these advances, I don't know if they've given a 10x. That's where I think he might be right. I don't know if you've got a 10x improvement, especially not in the 10 years since he wrote that essay originally.

If you look at the track records of some of these really successful design firms like IDEO, their products do really well compared to the market, compared to all products designed everywhere. They have something going, where they can find something that will delight customers, that works really well for them, that gets a lot of attention, a lot of word of mouth.

The idea of applying that to software could actually eliminate a lot of that complexity. Their process eliminates the complexity before you start writing it. It arrives at a good solution faster than you would get with whatever other process you have.

They have a good process. Now here's the thing, he does address some of this in the essay. This is in his follow-up essay where he says, "Oh, I was wrong."

That these things could actually help a lot, and they are helping. Of course, he claims not 10x. Each of them only contributes a little bit.

I don't know. I just feel like where we could be getting better as an industry, and we might be 10 times where they were 50 years ago. Not sure, what do you think? Are we faster than we were back then?

I love to hear your comments. You can find this episode and all past episodes at lispcast.com/podcast. There are videos, audio and text transcripts of every episode. You can also find links to subscribe and to find me on social media.

I love getting into discussions. Like I said, I'd love to hear your opinions on this cutting out of the essential complexity, like the domain complexity, how I chopped it up in the last episode.

There's a lot of room there that we often over-complicate things as programmers. If we had a better process, maybe we could eliminate some of that domain complexity that we think is so essential.

Awesome. My name is Eric Normand. Rock on.