What are macros?

Summary: Macros are one of the most talked about features of Lisp. They are a powerful way to extend the language without modifying the compiler.

You've heard of macros, and you've heard that they're awesome. But what are they?

Let's say you're writing a compiler for a language called FOO. Now, you've got limited time because you've got a deadline. You're designing your language and you really want to get started coding in the language. What do you do?

Well, one thing you could do is write a very small language. A small language has a small compiler, so you can finish faster. And in fact you build it quickly and start writing programs in it.

Here's what your system looks like:

FOO code compiles to Compiled code

After a while, you realize your decision to make a small language came with a serious drawback. You wish that there was a lot more syntactic sugar. You don't need much to make a computer language. But to make it convenient for people, you need some more niceties.

So now you have a choice. Do you open up the compiler again to add more syntax and features? That scares you because compilers are not so easy and maybe you'll break something. You'll be working in the lower-level language you wrote the compiler in, and that is clearly worse than your new, shiny language. Or, you could try to live with the limits of the language you wrote. It's a spartan existence, but, well, you're tough. There's a third option, and it's the macro one.

What if you could leave the compiler alone and extend the language in your new, awesome language? That's macros. So you make one last addition to the FOO compiler and call it FOO-PRIME. Macros are functions that take FOO-PRIME code and return FOO-PRIME code. You can call them just like regular functions. The difference is that they run before the code is compiled.

FOO-PRIME macroexpands to FOO which compiles to Compiled code

Macros return FOO-PRIME code, but the Compiler still only knows FOO code. So the Macroexpander's job is to keep expanding macros until all it has left is FOO code, which the Compiler can handle.

You've solved the problem! You can extend the language to make it sweeter, but you don't have to modify the compiler in the low level language. Yay! What's more, your language is getting better faster, and so your macros are easier to write, too! If done right, it's a good cycle, and can compound over time. A better language to improve your language.

And, in fact, macros really do let the Clojure language be extended without changing the compiler. Much of what we call Clojure is just macros. For instance, the cond form is just a macro. defn is just a macro. Lots of Clojure libraries have macros. And the killer macro library is core.async. In many languages, something like core.async would need to be done as a new release of the language. But in Clojure, it's just a library. It can be iterated separately from the language and imported only when needed. core.async is one of the more extreme examples, but macros are actually really common just to make life easier for the programmer.

Disadvantages

It's not all rosy. Macros do have some disadvantages. First, macros are called like functions but they don't have a runtime value like functions do. You can't pass them to other functions, store them for invocation later, and whatnot. It's actually very hard to even define what it would mean to call a macro at runtime, after the compiler has already run. Lots of research has tried to give semantics for it. So in Clojure, it just throws an exception.

Another disadvantage is that you've got a Turing-complete language generating code. That means you can get into infinite loops and other not-so-recommended things. Be careful.

And, finally, there's a lot of cognitive load writing and reading macro code. You have to distinguish when the code will run---at compile time or at runtime---and teasing all of those things apart can get intricate. Luckily, you can manually macroexpand code (check out macroexpand and macroexpand-1) so you can inspect the output.

Conclusions

Macros make it possible to extend the language without modifying the compiler. By inserting a macroexpansion step just before the compilation step, you can get a compounding expressivity cycle. It means you don't need to wait for new releases of the language to get new features. And if you think a language extension really would save you time, you can build it yourself.

There's a Clojure macros module as part of Beginner Clojure.