Issue 266 - February 26, 2018 · Archives · Subscribe

Hi Clojurators,

As far back as we can tell, Lisp has always been an interactive language. It was used in AI research, where you don't know exactly how to solve the problem. You just had some ideas and wanted to test them out and learn from the exploration. And there was nothing like Lisp for moving fast. It's one of its principle benefits.

Other systems had very slow programming cycles. You compiled your program to punchcards, then got an appointment to have them loaded into the computer, then waited for the calculation. It took too much time. The feedback loop was too slow. Any error you had would send you back to the start to debug. Then only after your program ran correctly could you analyze the results to know if your idea might work. You could not explore the ideas quickly enough.

Fast-forward to 2018 and things are different. The descendents of punchcard-compiled languages take very little time to compile and run. It is not uncommon to run entire test suites every time you make a small change to your program. People are used to fast compile or test cycles. But Clojure can still have a faster overall cycle---and maintain its important benefit, if certain conditions are met.

Let's break down an example programming cycle. There is an inner loop and an outer loop. The inner loop is getting one idea to work. The outer loop is testing a bunch of ideas.

OUTER LOOP: (explore ideas)
a) Have idea to solve problem X
INNER LOOP: (get code working)
b) Write/modify code
c) Compile
d) Test
if error then
  e) Analyze error
  GOTO INNER LOOP
(at this point, your code works)
(but does it solve the problem?)
f) if problem X not solved then
  GOTO OUTER LOOP
(at this point, you've solved your problem!)

Does Clojure help you have more ideas? Better ideas? Ideas faster? (a) I'm not going there today! Let's just assume no, for now. In any case, it's outside the inner loop so it doesn't need to be optimized just yet.

It's arguable whether Write/modify code (b) is faster in any given language. Typing in new or modifying existing Clojure code can be very fast if you use an editor with parenthesis management. Keeping parens balanced is very easy if your editor does it for you. It lets you work quickly. Otherwise, programming with parentheses can be a painful exercise in counting parens.

Further, the semantics of Clojure may improve coding time (a), if you already understand immutable data structures, recursion, and higher-order functions. If not, even the simplest algorithms could take time to write. This issue seems to me largely an educational issue, but it could be addressed by an innovative tool.

Compilation (c) is also fast if you have an integrated REPL. For me it's just a keystroke and it's done before I can blink. REPL-integrated IDEs are currently the recommended way to program in Clojure. But there are only a handful of decent REPLy IDEs, so a programmer often has to give up the comfortable environment they are used to just to try the language.

Testing can also be fast (d). You can either run a regression test suite. Or you can do manual testing. Both can be fast. Regression testing can be fast if you already have Clojure running and you've compiled your project's latest code. Manual testing is fast under the same circumstances. If you don't have Clojure running, you have to wait a long time, from 500 ms to 2 minutes just to boot your code. And even if you do have Clojure running, you have to figure out how to reload your code and all of the code that depends on it. Even if you know how to do it, it could take minutes to execute all of the commands.

Sometimes you get an error (e). Sometimes it's a stacktrace from an Exception you recognize. You can jump right to the location of the problem in your code, if you have enough experience. Without the experience, just finding the relevant information in the huge stacktrace can take a long time. A good IDE can help with this. And once you do find the error message, it's often obtuse or doesn't really tell you what's wrong. It requires deep knowledge or lateral thinking---both of which take time and precious cognitive resources.

Finally, how do you know if your problem is solved (f)? Clojure can make this fast because most of its data can be easily printed out for manual inspection if you know how to pretty print or hand-explore the data. Large and nested data structures are hard to analyze when printed on a single line. Note that this is outside the inner loop, so it's not such a priority. We won't address it here.

When you don't have the stuff above (paren management, integrated REPL, hot code reloading, and experience debugging the errors), the cycle can be much slower than the descendents of punchcard languages.

You can eventually get fast in Clojure, but it requires a huge investment of time learning editors and other tools. We can't expect a beginner to Clojure to make that investment. They are probably just trying out the language. They're rightly skeptical about the claims. And they're going to compare it to other languages that they're also trying. Marketing and hype do play into language choices, but there's nothing like a great experience to make the choice easy. We want those beginners to have a deep experience of Clojure quickly. We want them to feel what experienced Clojurists feel. Because we know it's a good feeling and they'll tell their friends.

But what's more, even having invested years, I often feel like I'm walking a tight line while developing in Clojure. If I take one step off of that line, it slows me down tremendously. And even an error that I haven't seen in a while can take an hour to figure out---even though the fix was simple.

What I'm getting at is that improving the Beginner Experience would improve all of our experiences. We're all beginners sometimes, like when our parens somehow do come unbalanced, or our running system doesn't seem to be in sync with the code we just wrote.

I want to address the steps in that cycle and ways we could improve them without requiring a big investment from programmers. I'm particularly interested in tools today. So the links in this issue are organized by step. There are inevitably some tools that I don't know about, and some that I just didn't have room for. I'd love to hear your recommendations.

Finally, I've noticed the topic of Beginner Experience coming up more and more in the Clojuresphere. It gives me renewed hope in our community. The number of projects trying to address this warms my heart.

Rock on!

PS Want to get this in your email? Subscribe!

Parenthesis management

Parinfer

Shaun LeBron created a new paradigm for editing Lisp. I use Paredit myself, but if I had to start over today, I'd use Parinfer. All you need to correctly parenthesize your code are the spacebar and backspace keys.

I would definitely recommend this to beginners. It's available in quite a few editors.

Understanding Semantics

Clojure is hard when you're not a functional programmer. We can do a little better by helping exploration and visualization at that level.

Maria

A cool, beginner-friendly ClojureScript notebook-style editor. It visualizes data structures pretty well.

Compilation (and manual testing)

Figwheel

Figwheel makes coding in ClojureScript so sweet. It keeps the JavaScript VM in your browser always in sync with code in your project. With a tiny bit of easily learned work, you can have instant feedback on your code changes---all without any special IDE. This is a mature project that you can use right away. Check out the [Leiningen template](https://github.com/bhauman/fi %20gwheel-template). It also provides a REPL out of the box.

rebel-readline

I'm very hopeful about this new project from Bruce Hauman (creator of Figwheel). I think of it as Figwheel for the Clojure REPL. The REPL in the terminal sucks in terms of UX. And this project promises to fix that. The hope I have is that people will be able to get 80% of the benefit of REPL-integrated IDEs just with this nicer terminal REPL. Bruce has told me that he is already thinking about code recompiling. If there is one project to watch, it's this.

Automated testing

lein-test-refresh

This plugin is made to automatically run your tests when your code changes---eliminating the need to restart the JVM each time. It's custom made for testing, so it has lots of options and features dedicated to that.

Lein-Auto

This is a more general Leiningen plugin that can watch your code for changes and run a Leiningen task. It's here because it lets you do

lein auto test

and have your tests automatically re-run.

Errors and Stacktraces

Error messages are a big problem in Clojure. They're consistently at the top of the "frustrations" question in the State of Clojure survey.

Pretty

Pretty gives you better stacktraces. It put the most important Exception last, so you don't have to scroll up. And it filters out the noise from the stacktrace, so you see stuff relevant to you. Finally, it formats the errors with ANSI colors in your terminal. I think the only thing missing is custom printers for different Exception types.

Expound

If you're using Spec during development time to catch errors, you can improve the error messages by using Expound. It has a nice, human-readable format.

Pyro

Another stacktrace prettifier, this time with source code for each line in the stacktrace!

StackParam

This project claims to be able to add information about local variables into the stacktraces. That means you could know what arguments are passed to your failing function. I'd love to see someone get this working at a Leiningen plugin.

Humane Test Output

Clojure's test output is pretty hard to read. For instance, if you compare two maps for equality, and they're not equal, it prints out both whole maps. Humane Test Output shows only the differences. And it pretty prints it, to boot.

Ultra

A Leiningen plugin that colorizes the REPL, add syntax highlighting to in-REPL source code listings, cleans up stacktraces (using Pretty), and gives better test output.