What Clojure needs to grow — a boring web framework and boring data science

I think a lot about what Clojure needs. Is there something that is sort of the bottleneck for growth? People talk about different things as their hypotheses for what would make it grow. I think that what Clojure needs — I'll talk about my hypotheses — I think what we really need is to solve all of the boring problems that other languages have already solved.

Transcript

Eric Normand: Hello, my name is Eric Normand and these are my thoughts on functional programming. As you probably know, I am invested in Clojure and I want to see the Clojure community to grow, for reasons which you can probably guess, so I'm not going to go into that.

I think a lot about what Clojure needs. Is there something that is sort of the bottleneck for growth? People talk about different things as their hypotheses for what would make it grow. I think that what Clojure needs — I'll talk about my hypotheses — I think what we really need is to solve all of the boring problems that other languages have already solved.

What do I mean by that? We need a web framework. It needs to be a boring web framework. We need to solve...have a data science story; data science, machine learning, that kind of thing. It's possible to do it in Clojure. A lot of people have a lot of success with it, but in terms of, "use this thing and it just has everything you need," we don't have it.

https://twitter.com/ericnormand/status/1006914392959668225?ref_src=twsrc%5Etfw

The reason I call it boring is that no one wants to just sit down and do the work. It's a lot of boring work to put together the libraries, make sure that they run in all the platforms, make the installation easy.

People get very satisfied with getting it working on their machine and talking about how great the setup is, but then not writing it down anywhere or making it easy for other people so that what took them three hours to set up with a lot of background knowledge, someone could set up in 20 minutes with no knowledge.

We just need boring, boring solutions to these problems. I used to think that we knew that killer app. I thought that we needed something that people would learn Clojure for. Regardless of their biases, or their previous experience with languages they would jump to Clojure for this one killer thing that was going to solve all their problems and make their life easier.

They wouldn't care about the syntax, how bad the error messages are. They wouldn't care about all these things that people complain about, because they just needed that thing. I'm now skeptical of that as a solution. If it hasn't happened already, then I don't think it's just going to happen. I don't have any good ideas for what that would be, this killer app.

I've resigned myself, giving up that idea, "What do we need?" It's obvious if you just sit there. You give up these swing for the fences moves like some web framework that makes Rails obsolete or something. That's what I was thinking before.

If you just stop trying for that thing and you work on the obvious holes in the Clojure ecosystem, it's usually more of the holes are coherent story, good documentation, clear guide for getting started,...easy install on all the platforms, the boring stuff.

https://twitter.com/ericnormand/status/1028657678258720769?ref_src=twsrc%5Etfw

Then stability, something that people can invest in and their code is going to run for a while on that platform without an upgrade cost every time you change something.

I'm suggesting that for a web framework. We should just look to something like WordPress as the example, not to Rails. A lot of people look to Rails as an example of a successful web framework. WordPress, it's born out by data. WordPress is way more popular. It's used in a wider variety of cases and also has a nice ecosystem.

We need a framework. If you look at this stuff that people try to do with frameworks and Clojure, the most popular example is Luminus. What they've done is just giving you a good start at a web app, the same web app you would write yourself. That's fine for what it is.

If you know Clojure and you understand how the web libraries work, using Luminus can save you a lot of time because it's all set up for you. It's this fully integrated solution. I'm thinking of something that allows for much more plugins. It's what it is. It's an ecosystem of plugins.

One of the reasons why WordPress is so successful is that they separated the users into at least two groups. One are the developers, they make plugins and themes. The other is the people who are consuming those plugins and themes, who are just bloggers basically, or making a site for somebody. By separating those two out, you get the best of both if the interface is right.

WordPress has its flaws. We can learn from those flaws, too. I'm not just going to copy WordPress. The idea is to make something where...I'll also talk about where Rails succeeds.

Rails is a good framework as far as it creates a...it lets you program for the web without having to understand the web. It gives you a process where you just write Ruby code. You just learn one language basically, and you can begin to code for the web. You don't have to know Sequel because active record handles that.

You don't really have to know HTML and CSS, although you do have to start learning that pretty soon. You can start because if you use a scaffold, it generates all that for you. There's styles built-in already that you can work with as they are. You don't have to learn them, build the page yourself and that kind of thing.

Then you've got all the security stuff built-in. There are gems to do, users, all the login, password reset, all that stuff. It gives you that ecosystem. What Rails is giving you is a process, a mental framework for developing web apps. That's why it's so popular at boot camps because they need to teach you something. Something that is teachable is going to be their first choice.

https://twitter.com/ericnormand/status/1005827002119618562?ref_src=twsrc%5Etfw

Those are the two requirements. Something that it has a process for how to build web applications and something that allows for reuse of components that are programmers. None of the web frameworks in Clojure...actually, it get much reuse beyond the very basic things.

There's a middleware that work. When you get into anything complicated, like a user-system like, let's say, Buddy — it's a library for authentication — it requires you have to put this middleware here. Then you have to give it a table where you're going to store your users.

The amount of code that you need, the amount of stuff you have to understand before you can actually use Buddy, is a lot. That's great for what it is. I'm not complaining about that, but I'm trying to...I know people are going to ask me, "What is a Buddy? That thing?" and it's not.

If you read the docs, there's a lot of integration that you have to do to get Buddy working, and it's error prone. There's ways to get it wrong that are not obvious. It's basically rolling your own security. Buddy gives you the tools and you can put them together wrong.

We need a single click user system. Maybe it even comes with that, but I have not figured out what the framework would come with and what would be plugins. This is the third thing that good web frameworks do and you don't even notice they're doing it. How do I put this? They wrap the web. They model what web sites do anyway, what you're going to end up doing if you write a website.

https://twitter.com/ericnormand/status/1076206913707220992

What do I mean by that? In the templates in WordPress there's a hook. There's a piece of code you should run in your template before...Let's say you write your own template. Good templates will do this because it's just a standard. Right before you end your head tag, the close of that head tag, right before that, you run this code.

It's like, "Here's where you put your CSF." Then, with that hook, there's a system for registering all the CSF that your plugin needs, and it's going to get inserted there, in the head. There's a similar hook for the body. There's all these little things that are known. I want to call them places where things should go, where things happen.

The web framework should capture that knowledge. These are places where you might want to put something if you're a plugin. We need to capture that knowledge. I would love to have it so that we could have one click installed for, "I need a CMS click. I need a user management system. I need a rest API."

The solutions have integration points that are well-defined and well-understood so that they can work together using those minimal systems. All of these integration points are defined by the framework and well-documented, and you just boom, boom, boom install these things, and boom, you've got yourself a website that does X, Y, Z.

https://twitter.com/ericnormand/status/1069985066422550528

That sounds like magic, but a lot of the magic is simply in doing the work of just writing it so that it does the right thing because all of these things are well-understood, and they happen in other frameworks already. It's magic if you're just coming to it and you don't want to understand, it can just be magic, but it should all be understandable.

Now, where does the web framework...Why would you do this in Clojure and not in Rails or in WordPress? I think that Clojure has a dynamic language with a runtime...You boot up the JBM and now your server is running. Can you give some advantages that you can't see in PHP, and even in Ruby?

https://twitter.com/ericnormand/status/1052575186485211136?ref_src=twsrc%5Etfw

For one thing, you could have all of those hooks. You could have a page in the admin section of your site that just lists all the hooks — "Here are all the hooks and there are three plugins that use this hook, two plugins use this one, and no one is using this one." — Then you could have it show you the code. "Where is the code that is running in this plugin for this hook?"

A lot of these problems of, "No, it's spaghetti with hooks everywhere." A lot of those problems can be solved by good introspection, I believe. I run WordPress on my sites — lispcast.com, clojuresync.com, purelyfunctional.tv. They're all WordPress sites, and I deal with this issue of having a million hooks in this plugin. What is it actually doing?

There are problems with that, but the answer is not, "Oh, let me have more control and know exactly what is in my middleware stack," and that idea that you're going to totally control it and put it together yourself like a kit car or something. It's much better to be able to rely on other people's code and introspect it when necessary.

https://twitter.com/ericnormand/status/1001478345392119810?ref_src=twsrc%5Etfw

"Show me all the hooks that this plugin is putting in." Maybe even have some customizability and say, "Mm, I don't like it when it does that. I like the plugin otherwise, but that little hook I'm not interested in."

Anyway, I think that, as a dynamic system, we can provide niceties on top of this boring model. That's my idea. There's nothing special about Clojure that makes it good for the web, but it's just nice in general, and we should just do some boring solution, and just make it nice. Put in the work and make it feel good.

That's my ideas for what Clojure needs to grow. In terms of data science, I don't have a lot of ideas because I don't do much data science. I know that a one of the reasons a lot of people like R, or MATLAB, or Python is that every statistical calculation is already written. It's just a matter of finding, what is it called? What library is it? What import statement do I need? Then, boom, I got it.

That is one of the things that we don't have. That's all the boring work. The boring work of writing all of these statistical modeling functions. It's interesting work.

I think this is what we do best, is looking at it and saying, "Well, I can make a graphics library." It's a cool DSL. We do that, and we think we're superheroes, but then it doesn't compete with having a graphics library and every single statistical formula ever invented. That's what we need. Someone to just write all those formulas out.

I don't know what other applications we would need. Those are the two that come to mind. We just need more people doing the boring work. I mean, we don't need more people, we just need to do it. That's my opinion.

These robots that are watching us, watching what we click on, watching what we listen to, watching what star rating we give it, you know why they do that?

They don't have emotions. Those poor robots don't have emotions. We need to help them. Use our emotions. Tell them what's good so that they can learn to recommend things to other people.

Please, please for those poor robots, please click on the like button. Please help them. Tell them...Subscribe. Give them all the signals that they need to develop a model of emotion and recommendability.

All right. My name is Eric Normand and these have been my thoughts on functional programming. Check out lispcast.com. That's my website. Subscribe. Like. Do it. Mash the buttons. See you later. Bye.