What is the Curse of Lisp?

What happens when your language is so powerful that small, independent teams can solve their problems without libraries? Does everyone flock to it? Or do you just get a lack of libraries?

Transcript

Eric Normand: What is the Curse of Lisp? In this episode, we are going to explore this counterintuitive property of powerful languages. My name is Eric Normand, and I help people thrive with functional programming.

The Curse of Lisp is it has a very personal appeal to me because I used to be one of the people that thought about this all the time. The real question is if Lisp is so powerful, if it's such a great language, and it lets you do so much, why isn't it more popular? It's not about the parentheses. People can get over the syntax. It's been a lot of time to get over the syntax and it's not really what it is. We're going to explore that.

You used to hear this more. In recent years, like the last 10 years, there's been a real renaissance of languages. There's so many possible languages to use now. Mostly because of the Web and the rise of startups who can use whatever language they want on the back end, and there's been a blooming, a blossoming of different possible languages to choose from.

But back in the day it was not like that. Big companies standardized on a mainframe and there were certain languages that ran on that, so they would standardize on a language and train their employees in that language and that's what you had.

There were a lot of — I'm talking about 50 years ago — there were a lot of Lisp programmers, mostly at universities and graduate departments, who learned Lisp, got into Lisp, felt very productive in Lisp, and looked around and said, "Look, you've got this hundred-person team working in COBOL and they are producing very little software, you know, just as very slow and here I am with Lisp and I'm like inventing, AI." [laughs]

I'm making stuff happen just all by myself. Maybe I have another person working with me, but like two people doing a lot more work than this team of a hundred people and so it seem to them. This is a perception thing.

The question they would always have is why isn't this more popular? If it's such a powerful language, why aren't businesses investing in it? Why aren't we seeing people realize that they can hire fewer people, a lot fewer people not just 1 or 2 percent, but maybe you slash your team by 80 percent? Why don't we see Lisp being more popular?

Over time, you see this language like C or C++, Java starting to get a lot of attention. A lot of jobs around it. Meanwhile, Lisp is this not growing very much. All this stuff that you could do in Lisp in a few lines of code is now you realize taking men years, person years of work. Sorry. I said men years. I didn't mean to use the sexist version. Person years of work to make a library to do the same thing.

For instance, parsing in a Lisp or any kind of functional language are recursive to send parser, it's very easy to write. Now you need some kind of parser generator system like a Lex and a YACC or an ANTLR in Java. You just need these huge systems, this huge piece of software just to do the same thing that a lisper was able to do.

This is my version of the Curse of Lisp. What happens is because it is hard in Java to do a nice parser, people recognize that it's hard. They work together because the challenge is so great, they need a hundred people to come together and work on it and make it very good and powerful.

It creates even like a cottage industry. Now you need books written about it. You have maybe some conference talks about how to use this new parser generator in Java. It becomes a new thing that you can become an expert at because it's not just like you know how to make parsers. No. You have to know that tool. It's even less general knowledge, but still, it's very powerful because all this work has been put into it.

The question is why is it that the industry seems to prefer this giant effort to make a thing because of a less powerful language and so it takes more effort to do. Why does it prefer that over the more powerful general solution?

I should pause here because it's not about Lisp. I said it at the beginning, it's about powerful languages in general. Any sufficiently powerful, expressive language, like you get with a functional language using higher-order functions and stuff. I think this is going to happen too.

I believe that there's something about the fact that it's so expressive and so powerful that there's...Here's the curse. The curse is it breeds individualism. Because you can be so powerful alone, you work longer alone. You never learn to collaborate as much as you would if you had to work in a language that did require a team of 20 people to get anything done. There's that issue. It breeds individualism.

Here's another thing. Because it's so easy to write the solution...It's easy to write your own parser in Lisp. I'm just going to say it like that. It's very easy. You write the parser that's just for your little language that you need to parse. That's a 60 percent solution to parsing in general.

Whereas, if you go to ANTLR, it is a 100 percent solution to parsing. It's got a really tough API to use. You have to learn the grammar language that it's got. You really have to understand it well, but then, boom. It's a hundred percent. You can do whatever you need to do and that knowledge is transferable. I can go to another project wearing this library with me.

Whereas in Lisp, you just have every individual could write this parser. Everyone thinks, "Oh, my...It's not quite like what they did, so I'm going to do it myself a little bit differently. I'm going to solve my 60 percent." There's hundreds of parser libraries out there.

This is part of the curse. Is that it's so easy to write a new one, you don't think, "Oh. I better jump on the bandwagon of the one big one."

Another aspect of the curse...Let me just tell you the curse and then I'll give my opinions on another. This is another aspect of it. Is that we're obsessed with bootstrapping. We love the power that a small amount of code that has a lot of leverage to get us to the next level of expressivity. We love that. But then, we don't take it to its final point.

What do I mean by that? Here's an example from the Clojure world. Recently there was a lot of buzz about a system for having examples in comments, runnable examples. You put the examples in your dock strings in your code. When you run this tool on it, it will generate an HTML page where those examples are runnable. Totally awesome, right? You can actually run in the browser where you're reading your documentation. Run that code.

That's awesome. Someone bootstrapped this cool system, and it's something that used the power of Lisp, and the fact that we've got in-browser, compilers, and stuff like that. It used all of that stuff to make this cool system. I haven't checked recently but I don't hear anyone talking about it. I don't see a lot of examples popping up.

Like, "Look what I did in this tool, look what I did..." I think the reason is we're so obsessed with that first initial, "Look at all the power I have." We don't go the extra mile and say, "OK, now, let's all get together, let's divvy up this task, and let's write a thousand docstrings with examples." We don't do the boring manual labor work that's required.

Another language community who's not so obsessed with the bootstrapping, they probably could do the same thing in way more code. OK, way more code. When they do it, it's so much work. They're not going to stop because that was a lot of work. They will only do it because they know they have so much work.

They're not going to do it unless they know they have the resolve and the labor to write up all of those examples. This is another aspect of the curse.

Let's go through them again. One is the individualism. We don't have to work together. The language is so powerful that we, as an individual or a tiny team, two or three people, can build a solution for their problem. They never have to work with a big group to get things done.

Two, it's so easy to just solve your problem. You don't get a complete solution. Everyone has their own little solutions that work for their problem, but then they're incompatible with all the others. In Clojure world you see this with the proliferation of routing libraries.

Everyone's like, "Oh, I need a little bit different from this, so I'm gonna..." whereas in another language, they might have two or three because it's a harder problem, and they want to standardize on it.

The third one is this obsession with the first initial bootstrapping. You put this idea out as a high-leverage piece of code, and just expect it to blossom. That's only the initial seed. You still need all the work of getting people to actually do the hard work of it.

The computer can't do all of it. The computer can do that cool first part, but then the rest of it has to be done by people. I see it so much. This is the Curse of Lisp. There's another cynical side to it.

I'd have to put this out that companies...This is the very cynical side but it's got a grain of truth in it. Companies would rather have a lot of replaceable interchangeable programmers then one or two powerful programmers.

They want someone to be able to, for instance, be fired or change jobs or go on maternity leave, or what have you, and just be like, "Oh, that's one percent of our team gone."

If you only have two programmers and one of them goes on leave, that's half your team, that's half your productivity. They'd rather have this big mass of people working on a less expressive language then put all their eggs in two baskets and have this trouble.

That's like an economic argument for why Lisp isn't more popular in business. The Curse of Lisp is something that I've thought a lot about. I was really into Lisp for a while, and I always wondered why it wasn't more popular because I felt way more powerful.

It's not even about like Lisp is absolutely more powerful than other languages because there's always new languages, and they have cool powers. I'm not saying that. Going from a language like...I learned Java going in university. Going from Java to common Lisp was like...

There's no comparison between solving problems in Common Lisp, and having to write all this public class, etc., just to get anything done. It's very easy to compare the two languages and say, yes, Common Lisp has a lot of power in it. I want to make that clear.

I'm not saying Lisp is like the best language. I'm not saying Lisp is more powerful for everything. I do think that when you go from a language to a more powerful language, you often wonder why is that language that I left so much more popular than this one when Lisp is one of the oldest language like, "Why isn't it taking off?"

I think that a lot of it has to do with the power. It's too powerful. What is the antidote? In Clojure, people would often talk about Clojure escaping the Curse of Lisp because it's able to use the JVM libraries. In a lot of ways, that's true. It has escaped the Curse of Lisp.

Empirically, it is really popular compared to Lisp of 20 years ago. So it has escaped a lot of that problem.

We still see the same kinds of problems, a lack of coordination what I would call a very earnest efforts to get a project going and not a lot of uptake on it, a lot of, "Oh, I can do better. I could do that in a couple of macros," and not a lot of follow through from the original person making it. I see the same problems there.

Luckily we do have the java libraries and a lot of people willing to wrap those java libraries in things that make it easy for us to use in Clojure. We're escaping part of it but we still have the legacy of the curse. I think we could figure out a way to solve this.

Another example, just to say it's not strictly about Lisp. I'll just explain it like this. I used WordPress a long time ago before Rails came out. The interface was pretty clunky. When Rails came out everyone started expecting a lot more from Web interfaces, from the gooey that users click around on the Web.

They wanted dynamic stuff to happen. They wanted little menus that were easier to use with your mouse. They wanted loading spinners. WordPress didn't have a good way to do that. Rails did. It did a lot of that for you automatically.

But by and by WordPress had a lot people using it and a lot of developer power and they built it like one little widget at a time making it very much like a default Rails app would be and it was through sheer willpower.

You could ask, "Well, Rails and Ruby are so much better at making this kind of thing than WordPress, why do people still use WordPress?" The same question, WordPress is insanely popular. It's insane. Why? Why do they still use it? It's a very similar question.

You got a more powerful language, Ruby. You've got even a more powerful framework, Rails, — a better framework, a more modern framework — one that does the stuff you needed to do very elegantly versus WordPress where you have to do it all manually.

Yet, there's some company selling a plugin to do what you could do in a few lines of code in Ruby. Of course, that plugin is like thousands of lines of code. What is it? Why is this? It's there everywhere. Luckily, the Ruby and Rails community have a lot more follow through.

They're not so cursed as the Lispers are and a lot of it has to do with the popularity. That's all I'll say about this Curse of Lisp. I don't think we've escaped it. There's a lot more that needs to be done. I'm talking about the Clojure community right now.

We need to put in the boring work. I think that that's one of the key issues. We need to have more collective action. We need to form more projects that people say, "I'm not going to make my own solution. I'm going to use this one." We just need that. I think that that would go a long way toward helping us.

Another thing that saves the Clojure community is we are very practical. Our libraries don't do more than they need to, which is a good thing. We don't build in functionality, bite off more than we can chew and paint ourselves into corners. We want the power of the library that we're wrapping or whatever we're doing without a lot of assumptions about how it's supposed to be done or what it wants to do.

This episode is going kind of long so I'm going to wrap it out now. If you like this episode, you can find past episodes at lispcast.com/podcast. There you're going to find audio, video and text transcripts of all the past episodes. This one is number 141, I think. A lot of episodes. You can go find everything back there.

You'll also find links to subscribe and to get in touch with me on social media. If you're interested in the Curse of Lisp or why some languages ain't as popular as some other language.

I still want to think about this and figure it out. I don't think there's any kind of magic sauce. A lot of it is the market forces, if you will. Still, there's a lot of room for Lisp to succeed. I think about it a lot.

All right. Thank you so much for watching. My name is Eric Normand. This has been my thought of Functional Programming. Thank you for listening and rock on.