Clojure Gazette 178: The Biggest Waste in Our Industry

The Biggest Waste in Our Industry

Issue 178 - June 20, 2016

Please read the personal message at the end of this letter.

Hi Clojurists,

I would like to talk about two nasty habits I have been party to working in software. Those two habits are 1) protecting programmer time and 2) measuring programmer productivity. I'm talking from my experience as a programmer to all the managers out there, or any programmer interested in process.

In factory management there's an idea called the Theory of Constraints. If you've got a machine whose throughput is less than the throughput of all of the other machines, that's called a bottleneck. Let's say a particular lathe is a bottleneck in your factory. Well, you could increase the productivity of your whole factory if you could relieve that lathe of unnecessary work. Maybe a different lathe could do it just fine. Or you could even outsource it to another factory.

Your programming team might be the bottleneck in your company. They're working hard, but you have way more ideas for features than they can develop. And they're so expensive! So what do you do? You start to protect their time. You think "I'll make all of these decisions for them. I'll write them up and divide them into little pieces and prioritize them. All they'll have to do is pull one down and start working on it. I can even ask the designers myself for some work. I won't bother the programmers with it until the designers are finished." They can spend more time coding and less time worrying about what to work on.

So then, you think, I've got all of these little tasks for them to do. How quickly are they doing them? I'd love to know when they'll be done. Maybe I can even learn to tune things so they can work even faster. Congratulations. You've turned your creative humans into a compiler. They compile "user stories" into code.

This is not how knowledge work works best. We're not mechanical tools that you can measure the throughput of. Your local "efficiency" has removed 90% of the developers' brainpower from the equation. We don't work better with the end result of a long analysis process. We do better when we're involved in the process and engaged with the problem. We work in gestalts. We require the big picture. We solve problems. We're the foreman, not the lathe.

And your "efficiency" has another problem. There's a peculiar fact of abstraction which is counterintuitive: general problems are often easier to solve than specific ones. It's very weird, but a hand-rolled parser for a language is harder to write and read, and often takes more code, than writing a small general parser and giving it a grammar. It's bizarre but I've seen it many times. Powerful abstractions are like quantum leaps, not incremental improvements.

The tasks are broken up into little bits. Add on top of this the measurement/management of "velocity". We reward programmers for the speed of individual tasks. Their best strategy is to grab one, finish it as quickly as possible, then move on to the next. They can't see the big picture. There's no way to see those general problems that are so easily abstracted. They're stuck hand-rolling bespoke solutions for each one. The software would be better off if the programmer read 100 tickets and went to sit in a hammock to think about them. Tickets would then be just a signal to the programmer, not a TODO list.

So the biggest waste in our industry is this: any efficiency you gain by relieving the programmer of work, you lose an order of magnitude more in lost brainpower and abstraction. So what's the solution? I'm not a management expert. I don't know. But I feel I should propose a solution after presenting the problem.

My solution is first and foremost, bring everyone into decisions as soon as you can. Don't protect their time. Instead, use their time on higher-leverage things like strategic planning, business directions, market analysis, etc. This will pay off down the line when they're in the trenches. This might mean programmers are off solving business problems on their own, doing experiments, using trial and error, faster than you can write the tickets.

Ok, second solution: people write their own "tickets". They plan their own work. Yes, that takes time. But that's good. It means a limit to the evergrowing backlog of feature requests. They'll know what the tickets mean and how they fit together.

Finally, the third solution is going to be controversial, but I'll mention it anyway. Reduce throughput on your programmers. Don't expect so many completed tickets. Let them mull over the design longer. They can explore better abstractions before they have to deliver anything. In effect, give them more hammock time.

All of this takes a lot of bravery. Programmers can be a company's biggest expense. It takes courage to loosen the management of their time. But it's essential to getting them to work at peak performance.

Rock on!

PS Want to get this in your email? Subscribe !
PPS Want to advertise to smart and talented Clojure devs ?

Personal Message

Hi! The Clojure Gazette has recently changed from a list of curated links to an essay-style newsletter. I've gotten nothing but good comments about the change, but I've also noticed the first negative growth of readership since I started. I know these essays aren't for everyone, but I'm sure there are people out there who would like the new format who don't know about it. Would you do me a favor? Please share the Gazette with your friends!