The Surprising Benefits of Automation

Summary: The benefits of automation are not limited to saving time. We go over many advantages, some pitfalls to avoid, and some principles to follow.

Here's an XKCD cartoon about the pitfalls of automation.

Source: https://xkcd.com/1319/

It shows how automation is sometimes a fool's errand. We think we are spending a lot of time on the task. We decide to automate it by writing some software. We wind up spending much more time on the software than we ever did on the task.

I have to admit that I've done this before. I've tried to automate something and found that in the end, the bugs and general unreliability of software made it not worth it from a "time spent" perspective. Automation is not all roses.

However, in my experience, automation does help tremendously, despite sometimes being a wild goose chase. Further, I think the focus on time spent is a very Taylorist view of the way we spend our time. We shouldn't measure productivity or efficiency with a stopwatch. There are many human factors involved. For example, building a robot to mow your lawn may be more fun than mowing your lawn, even though it's unlikely to replace you without lots of work.

It's a silly example, but it shows that there are many benefits besides time savings. I want to explore automation from a system-level perspective---including human factors and other system-level thinking.

Let's start with the benefits.

Time saved

Sometimes you do hit gold and an automation straight up saves you time. Computers are fast. You no longer spend one hour doing a task. Now it takes you three minutes, thanks to software. You could easily imagine a formula to calculate the time saved. The variables are how long it used to take, how long it takes with the automation, how many times you do it, and how long it took to build the automation.

Bottom line: computers are faster than humans at many tasks.

Consistency

I send a newsletter out every week. I have a lot of the process automated, including layout. It does save time, but more importantly, the automation allows me to be more consistent. There are stylistic decisions that I've encoded right into the software. I care a lot about stylistic consistency, so I would spend the time to maintain it. But now that it's automated, I can relax. This frees me up to focus on the content. Likewise, if I find a way to be more consistent, or if I choose to change one of those decisions, I have a place to put it.

Bottom line: computers are more consistent than humans at many things.

Fewer defects

My newsletter often has many links in it. When I lay it out by hand, there are many ways you can get the link wrong. Sure, there are copy-paste errors. But sometimes it's more subtle. You put in a placeholder link, then forget to come back to it. I rarely have those problems these days.

Why? Because having an automated system, I have a place to add in automated checks. These are checks I probably should have done manually before. But look at the process: send myself a test email. Click on each link in the email and see if it goes to the right page. Make sure I'm using an incognito browser so that I'm not logged into my own blog, because if you're logged into the blog, you'll see your drafts, even before they're published. It's a total pain! I just never did it.

Automation lets me do those things I wouldn't have done. And here we see the first bit of alchemy that is missing from the stopwatch measurement of efficiency. The software is adding benefit that the human wouldn't add themselves. You need to have at least the beginnings of an automation to be able to add these kinds of benefits.

Bottom line: computers can avoid certain errors better than a human.

Frequency

I've automated most of my podcast production and publishing as well. I decided early on that I wouldn't do the podcast if it took too much time away from my direct money-making activities. It would have to fit into the unused spaces in my life. For instance, while walking or driving. In short, I needed to spend nearly zero minutes editing and uploading.

I experimented with a few different ways to do it, but in the end, I settled on a pretty stable workflow: film myself with my phone, upload it to YouTube (setting title and description). From there, an automation would download the video from YouTube, extract out the audio, title, and description, then create a post on my blog and attach the audio.

The fact that it was automated made it so I could do it at all. When it was so easy, I was posting 3-5 new episodes per week (that would publish at a twice-a-week cadence).

Programmers will also be familiar with continuous delivery. If we make deploying fast and easy, usually through automation, we do it more frequently. If it is entirely manual, we will do it less frequently. If you just did the stopwatch math, you would conclude that we don't do it enough to justify automating it. The automation unlocks the ability to deliver features in hours instead of weeks.

Bottom line: automations enable us to do a thing more frequently than we would have without it, and frequency might open up new benefits.

Ease

There are many tasks that are harder than they need to be, just because of the user interface. Although a person could do them, and even do them quickly, there's an amount of tedium to them that makes the work stressful. When I use Twitter, I often want to add someone to a list. To do that, the best way I've found is to open that person's profile in a new tab, wait for the profile to load, click the menu, choose "add to list", click the list I'd like to add them to, click "Save", then close the tab. That's a lot of work, especially when you consider how "easy" the task could be.

On the other hand, I could easily script up the same thing. Even though the task might take longer to perform (I'd have to switch to the command line, type in the command, etc), a script that added someone to a list might be easier. And by going to the command line (an easier place to automate in general), I could benefit from other automations, such as text expansion to auto-insert the command.

Bottom line: bad interfaces (especially mouse-driven ones) can make some tasks difficult, even when they shouldn't be. Sometimes it is better to make a task easier, even if it takes more time, because you're reducing coginitive load.

Scalability

Because the newsletter layout and sending is mostly automated, I feel like I could handle another newsletter or two. Better, I would trust someone else to be able to send the newsletters while maintaining the consistency. What that really means is I don't have to find someone with my unique blend of tastes and talents. They could be an expert on the topic of the newsletter, and not give a hoot about how consistent it is. That person would be easier to find.

Bottom line: even if it doesn't save time, automation can make it easier to scale out.

Waiting

You don't know how many processes I have to wait for before I can continue working. That puts me in a dilemma. I can either wait for it (and waste time and increase stress) or work on something else which will be interrupted when it's time to get back to it. Both are evil in human-factor terms. That time when you're expecting an interruption is rarely productive.

Automations can eliminate this waiting time. I transfer a lot of big video files over the internet in order to publish them. These can take forever. Some workflows require you to upload first, then take an action when it's done. Do I really have to wait for it? Too often, the answer is "yes".

Automations can help you move the work around, because they're like a little assistant who does the work for you. For example, to upload my video to YouTube, I have it ask me up front for the title and description. That way, I can type those in when I'm still engaged, hit Enter, and walk away. The automation handles retries and sets the title and description at the right point in the process. I don't have to be there.

Bott om line: waiting is evil and automations can reduce waiting. You might say "do something else while you wait", but that time is usually unproductive.

Queues

Many tasks come in batches. If you got one X per week, and X took 2 hours, it may be something you just do manually. But what if you get ten Xs one week, and none for the other nine? If you devote 20 hours to it, that might eat into other responsibilities you have to get done every week. And if you queue them up, the turnaround time for the last one will be really bad. You could work overtime to finish them quickly, but that's not sustainable.

However, if you automate it, you might be able to get through them faster when they come in all at once. That would avoid the pileup. But what if that automation costs 3 hours per week to maintain? That means 30 hours of regular work vs. 20 hours of rushed work. Is it worth it?

Of course, it depends on what X is and how valuable it is. But it's worth considering that you should trade off time for regularity, so that you can avoid long queues and long turnaround times.

Bottom line: long queues of work are bad, and have secondary consequences. It may be worth it to spend more time automating than you spend on the task to avoid queues.

Simplifying

It has happened to me many times that automating something has made the task simpler. For instance, I would never use `ffmpeg` to edit video. It's a command-line application. To really edit video, I need to see what it will look like. That's why I use a video editor with a nice interface. However, once I start automating my video pipeline, a switch to `ffmpeg` and a little time figuring out how to do the tasks I do in my video editor with it, may remove the video editor---an entire piece of software---from the process.

In general, the fewer moving parts you have, the more reliable your process can be. Automation can often remove the need for whole steps. For instance, my workflow before required me to open Screenflow (my video editor), create a new document, import the video file, trim it to size, etc, etc. The new workflow does not require Screenflow at all. Those steps are now unnecessary.

Bottom line: automation can simplify a process by removing steps that were necessary when done manually.

Learning

Automating a task is a learning process. You get to know the workflow better since you are essentially encoding your tacit knowledge into a program. You are also learning more about the tools available, how they work, and what they are capable of. Those learnings could be valuable for manual processes as well as future automations.

Bottom line: learning about your tools and media can open up the door to better automations or more efficient manual work.

Mental space

I can't switch between lots of different cognitive tasks very quickly. I doubt anyone can. Automating a task often means that I don't have to think about it for it to happen. It could be set up on a timer or something, and it just happens without intervention. That frees up space to get more important work done. And that space is very valuable, more valuable than the time spent automating the task.

Let's look at an example. Let's say I work from 8-12 am. I have to tweet at 10 am every day. It doesn't take long, maybe 10 minutes, but it cuts the morning in half. Anything I have been working on in the early morning is swapped out and it will have to be swapped back in to continue work on it. Swapping out and in can take 20-30 minutes.

However, if I automate the task well, I can forget about it. My morning becomes one continuous four-hour block without interruptions. That is invaluable.

And I have automated many of my tweets. I use a service that sends out tweets from a reservoir. It takes time to fill up the reservoir, but it's paid back in continuity of focus.

Bottom line: automating a task can give you more continuity of focus, which is more valuable than the time it takes to automate the task.

Infrequent tasks

Some tasks are so infrequent that I forget how to do them. If I did it every day, I'd remember the steps and I could to it in 20 minutes. But because I haven't done it in three months, I can't remember where the pieces are that I need. I clumsily search for them, and I sometimes interrupt other people to figure out how to do it. The task that should take 20 minutes takes 45 minutes.

However, if it's automated, there might be just one step. My favorite is when the automation guides you through the steps, like a wizard. You just have to remember how to start it.

Bottom line: it's easy to forget how to do infrequent tasks and have them take longer than they should. Automation can turn them into self-driving workflows.

Compounding

The more you automate, the more ability you have to build higher-order automations. As a programmer, I think this one is pretty cool. Seriously, once I have a script to upload a YouTube video, I can write a script that uses that one to upload 100 videos. Things compound pretty quickly. Even an automation that isn't "worth the time" (because it took longer to write than you save by running it) might become part of a larger automation that does save time. It's non-linear.

Bottom line: compounding is real. The more you automate, the more you can automate. And piece of automation gives more benefit.

Accumulating

Many automations won't actually compound. You'll never use one automation to automate another process. However, even when that doesn't happen, automations do add up. You automate yourself some mental space, that gives you more mental space to use automating. And if you automate half of a process today, in six months you might find the secret to automate the rest. The point is, starting to automate little things can snowball into entire processes taking care of themselves.

Bottom line: start automating now, even if you can't see the end of the process. Every little bit counts.

Now that I've sung the praises of automation, let's talk about the downsides.

More code

Automations typically require code. Code is a liability. It rots. It has to be maintained. And bugs can cause serious problems. If you've got a script to order transcripts for your videos or to email your customers, a poorly written for loop can do serious damage.

Rabbit holes

Some automations don't work. You think it'll save time and you'll use it a lot. But you don't. Or you spend ten hours trying to connect to that API and you can't get it to work. They're mistakes in judgment. I chose to do X because I thought it would be good, but X was not good. I was wrong.

But that's not what I'm talking about with rabbit holes. Rabbit holes are different. Rabbit holes are failures of judgment. You're not making decisions. You're letting the decisions make you.

If you're in a rabbit hole, you think you're making progress. You're perpetually *almost done, *but not quite. You get carried away with features. You spend a lot of time on it. And maybe it works, but it adds no value. This is probably what the xkcd cartoon is referring to.

Rabbit holes are dangerous because it's so easy to focus on the problem ahead without checking back in with the big picture. It's wise to be skeptical of automation because one fall down a rabbit hole could eat you alive.

Almost automated

There's another phenomenon where a process is almost 100% automated, but still requires some human steps in it. That seems like it should be acceptable. After all, you removed a lot of the work. But I find it frustrating that what should be doable from a shell script still requires human intervention. What's more, I've invested money and time into those tools under the false belief that they would be automatable.

For instance, I send email with a service that calls itself an "automation platform", but their API does not work according to their documentation. My newsletter workflow still requires me to copy and paste html and text versions of the newsletter into their web gui.

But instead of thinking "they just don't have that feature", I get frustrated by it in a way that would not frustrate me if I hadn't automated it . Is that frustration a cost? Yeah, probably a minor one. But I can't help but notice every time I run that workflow that there's only one step left that's not automatic. Do I file a feature request? Do I switch services? I think about it a lot.

Now let's talk about some principles that I use to minimize the downside and

Simplicity

The KISS rules applies to automations. It's very easy to overcomplicate a process in order to save work. Processes that are complicated, whether they are automated or not, are more work than they need to be.

In general, removing steps is good. Removing moving parts is good. Removing choices is good. Keep it simple, silly!

Outsourcing

Sometimes you will benefit from an external service or software. For instance, I publish a lot of videos for my membership site. I want them to be available online to watch, and also available to download in different sizes---HD, MD, and Phone. People pay for the content. The video is just the medium for that content. Video codecs is not my area of expertise.

That's why I outsource video transcoding to Vimeo. I upload a video, it gets converted to a web-friendly format for embedding in a web page. And Vimeo also produces the HD, MD, and Phone videos that I can download and host. They do a better job than I can do, and I'm happy to pay for it.

Insourcing

Sometimes, however, you want to bring the service under your control, so you can modify it. You might want half of a process, or you genuinely might have a better way to do it. For instance, I may one day bring my automated tweets back into software I write. I've written Twitter clients before, so I have the code. That will let me automate it a bit more how I like it, instead of how they've done it as a mass-market service. I like the service, it's just not exactly how I would do it.

Human-in-the-loop

As a programmer, I have the temptation to completely automate a thing. But sometimes it's best to stay in the loop. Even Toyota is keeping humans in the process. Even when they can automate something fully with robots, they choose to pay a person to be there. Why?

Well, according to them, humans can spot problems and further remove waste from the system. Humans have general intelligence and can come up with new ideas. Keep a human in the loop and build the process around them. Think about enhancing a person instead of replacing them.

Small pieces

It's a fool's errand to automate away an entire complicated workflow. You likely won't succeed. But if you automate a piece of one, you'll finish quickly and have something to show for it. Add bits to it over time. But start small and let it grow as your time allows.

Lowered expectations

Your software will never paint like Picasso. Hell! It won't paint like my six-year-old daughter. But that's okay. If you really need to automate it, maybe it's okay that it's not hand-crafted and artisanal. If you need human-level artistry, automation is not going to work.

Sometimes, automating a task forces me to really think about what is important about the task. I'm surprised by how much human attention I'm paying to things that nobody cares about---not my reader, not my customers, and not even me, when I am honest with myself.

Different process

Automatic workflows are different from their human-run equivalents. Humans and computers have different ergonomic needs. A computer can do many things at once. A human wants to get immediate feedback for every action. We have different needs.

Because of that, I often find that I have to rearrange parts of the process to make it make sense for a computer. Some steps would be ridiculous for a person to do, but to a computer it's natural.

For instance, if I'm publishing lessons to a course, I know what lessons I just recorded. I can upload them, make pages on my site, and embed the videos to just the ones I am working on. However, my automated software doesn't know what I've done. It has to find out. So the first thing it does is download metadata for all videos, for all lessons, and for all files. It uses that information to figure out what is missing, what has changed, and what is extra. From there, it can take the right actions.

The software runs every 15 minutes, checking for new videos that need publishing. I, as a human, would never do that. I would start the process only after I recorded a new video. We're different, and we have to change the process.

Ongoing process

Automation is an ongoing process. Our automations evolve over time. We will never be done.

That should be relaxing and permissive. We should feel okay to not have everything automated yet. And we should feel okay to start anywhere and automate just a little.

Take your time

I find one of the best ways to make good automations as opposed to bad ones is to let them sit a while. Don't rush to automate something. Do it manually first. Understand it well. Search for tools. Casually read documentation that might help you. Let it simmer. Slowly gather the ingredients. Do tests to see if you can perform a step adequately.

When you've got all the pieces, start assembling them. There is no rush. You can continue doing it the manual way if it doesn't work. Take your time. Comment your code. If it's a good automation, you won't be reading the code very often; on the contrary, you'll let how it works slip from your mind. Time is a good test.

Conclusions

When we do a simple time accounting of a task, it rarely takes into account the real benefits of automation. Yes, there are rabbit holes we can go down. There are automations gone wrong. But you can't look at the modern world and say that automation has been all bad.

I like automation. It has helped me grow my business and avoid tedium. As an experienced programmer, I find it very pleasing to construct a helper that relieves me of work and does a good job of it.

That said, I find it challenging to keep it organized and simple. There is simply a lot that needs to be done, and I'm not that organized of a person. Further, I really dislike most software. Most software offers very little room for automation. A lot of software even requires the mouse for interaction. Dragging and dropping might be okay for moving a few files around. But if you need to move thousands, the mv command works way better. I wish that the features of applications were accessible outside of those applications. I paid for them.

In the end, I'm always on the lookout to make things run smoother and to make processes add more value. Automation is just one technique we have to do that. Don't overuse it. But when it's good, it's real good and it can multiply your effectiveness.