Pre-West Interview: Alan Dipert and Micha Niskin

Talk: Boot Can Build It

Alan Dipert and Micha Niskin's talk at Clojure/West is about how Boot can be used as a build system for your projects.

Background

The main project tool in the Clojure community is Leiningen. In Leiningen, you write a project's configuration using a declarative syntax. In order to add build actions, you generally have to add plugins, which also need to be configured.

Boot takes a different approach, which is to code everything in Clojure. Boot is just a library and you script builds similar to how you might do so with shell scripts or Make, but you're in Clojure.

Micha at Boston Clojure Group (Youtube)

Boot, the Fancy New Clojure Build Framework

Boot is like the lisped-up lovechild of Git and Unix in that it provides abstractions that make it much more pleasant to write code that exists at the intersection of your operating system and your application.

Why it matters

Builds and deployments are getting more complex. A tool needs to be able to build a Docker container, push to Heroku, tag a release, spin up a server, etc. Can a tool built around declarative configuration and plugins keep up? The creators of Boot say the answer is "no". Boot's approach is different from Leiningen's, and so is not competing head-on, and yet still fills a need. Also, Boot's branding and developer pedigree seem to put it on firm foundation. There could be room for a second project tool. If there is, it will likely be Boot.

About Alan Dipert

GitHub - Twitter

About Micha Niskin

GitHub - Twitter

Introduction

Alan Dipert and Micha Niskin generously agreed to do an interview about their talk at Clojure/West. The background to their talk is available, if you like.

Interview

PurelyFunctional.tv: How did you get into Clojure?

Micha Niskin: I watched the SICP lectures from MIT and was blown away by the simplicity of Lisp. At the time I was programming in Perl, C, and Java, mostly (I was actually on a programming hiatus building boats in Maine, but you know, you never just stop programming). So I started doing some things in Scheme (Racket) and I really enjoyed it. I was learning awesome new things at a rate I'd never experienced before. Then Alan recommended I try Clojure. I was initially skeptical of the syntax, but persistent collections were a real revelation and I quickly joined the cult.

Alan Dipert: When I learned about arrays in Visual Basic 4, someone told me that some language "LISP" was somehow made out of arrays, which I giggled at because it sounded so silly. Later, in college, my coolest professor challenged my Java class to complete assignments in alternate JVM languages. I took the opportunity to learn some Scala, and for the final assignment, Clojure. Clojure blew my mind - what didn't immediately click was somehow very obviously something I would need to learn and would unlock more awesome things. The Clojure, and the learning, haven't stopped since then.

PF.tv: Last year you all released Hoplon. Everyone else was zigging toward React, while you zagged to a completely different paradigm. I can't help but see a parallel with Boot. Nearly all Clojure projects use Leiningen, and here you are zagging again. I think you've explained why already. What I want to know is the thinking that leads to two very zaggy approaches. Were the existing libraries not scratching your itches? Or do you have a deep desire to develop your own tools from scratch?

AD: Hoplon is a packaging of a vision we have for how to write and maintain Single Page Applications (SPAs). We had been developing this vision for about two years before Hoplon's release, and were driven to see the vision through because we needed practical tools for professional use and found nothing that existed to be suitable.

In the first year (2012) of pre-Hoplon exploration, it became clear we needed to either find or develop a model for state that could work in a browser. Of the technologies we evaluated, we explored FRP most thoroughly 1. We built a series of applications in ClojureScript using Flapjax 2.

Our experience with Flapjax informed our opinion that of the units of composition in FRP - behaviors and event streams - only behaviors are necessary for driving browser UIs. This is because UIs are views into memory/data, not into processes. For anything to appear in a UI, it must first appear in memory somewhere. We think behaviors, which are conceptually similar to cells in a spreadsheet, are the ultimate tool for coordinating change in data (NOT process) over time. We think core.async is probably the ultimate tool for coordinating process (NOT data) over time. FRP in its various formulations tries to do both, and not in a way that we find especially helpful.

We codified these opinions in early 2013 into Javelin 3, a ClojureScript library for modeling data in SPA UIs. What Javelin didn't solve — or even imply — was how to actually display the data being modeled.

If a UI is really just a stylized view into the heap, then any data structure is a kind of UI. If Lisp is the ultimate way to compose and manipulate data structures, then it must also be the ultimate way to compose and manipulate UIs. This is the thinking behind the second big piece of Hoplon, "the HLisp semantics", our answer to the question of display that Javelin leaves totally open. We worked on and around this concept in the 2nd year of pre-Hoplon (2013).

Concretely, HLisp is a set of semantics 5 that map HTML markup into the Lisp evaluation model. That is, the HTML expression <ul><li>one</li></ul> is identical to the ClojureScript expression (ul (li "one")). Both evaluate to a live Node object in heap that can be passed to and returned from functions. Under HLisp, a list might be constructed like this: (apply ul (map li ["one" "two" "three"])). Because these expressions evaluate to native browser objects, they can compose with a vast array of plugins and libraries available in the frontend ecosystem.

A big positive of HLisp is that HTML becomes a syntax for a subset of ClojureScript. We find this especially beneficial because it means any designer or UX expert familiar with HTML can easily contribute. We've never worked on a team that didn't include at least one designer/UX expert, and we believe this demographic is totally underserved by existing tools. Part of our vision is to empower designers to experience the same level of "flow" that we Lispers know comes with simple, clear models that we can infinitely extend.

Realizing fully HLisp pushed the limits of every build tool we tried to make it work in, because it was different than what anyone had ever wanted to do before. It was clear to us that the kind of transformation HLisp required needed to be accessible, and we began to imagine how a build system capable of these kinds of things could work. We dreamed of compiling Markdown, into HAML, into HTML, into ClojureScript, into JavaScript, reliably, simply, and accessibly.

We developed Boot1 to serve these needs, and shipped Hoplon atop it last year. We haven't had much reason to change Javelin, HLisp, or Hoplon over the past couple years, but we have rolled the set of ideas for improvement accumulated over a year of Boot1 use into the nascent Boot2. Today, Boot2 is capable of the Markdown -> JavaScript transformation and much more. We are really happy and excited that it is drawing enthusiasm from Clojure and ClojureScript users outside of Hoplon-world.

To finally answer your question: It's not that we think being different or building our own tools are virtues; we're just lucky to have had a series of opportunities — starting with our friendship — to experience, study deeply, and attempt to solve as definitively as possible the big programming challenges we've faced.

PF.tv: In terms of Boot, what were those challenges that Leiningen did not solve? And how does Boot solve them?

AD:

• Leiningen does not supply a general file system watcher; every plugin must provide its own
• Leiningen does not coordinate filesystem access between plugins; the user must be mindful of every plugin's file use and weary of filesystem contention
• Leiningen plugins cannot easily be composed or invoked from the REPL, which requires the user to start a new JVM for every experiment

For these reasons I would say that Leiningen plugins must all duplicate essential functionality, don't compose well, and are cumbersome to develop and experiment with.

Hoplon existed for a short time as a Leiningen plugin, but like all ambitious Leiningen plugins it became its own build tool and didn't compose especially well with other plugins. Once your Leiningen plugin becomes essentially its own build tool, it's not hard to imagine taking the last step.

This is what we did and Boot was the result; Hoplon dissolved into a half-dozen Boot "tasks". Boot solves the problems we encountered in Leiningen:

• Boot tasks are "stateful middlewares" composed in a pipeline, similar to Ring middleware. Tasks can decide whether the tasks after them should run, the same way a Ring middleware can decide whether or not to invoke the next handler. With a reference to the next task, a task can invoke the next task once, multiple times, or not at all. With this ability, "watch" is no longer special, and it becomes a regular task. Library authors are empowered to write other watch-like tasks that initiate or prohibit builds depending on any condition, not just filesystem state.
• The value passed through the pipeline of tasks represents the file system, and is called a FileSet. It is an immutable snapshot of a filesystem. If a task needs to effect a change, addition, or deletion to the filesystem, it performs an operation on the FileSet — returning a new FileSet — NOT directly on the filesystem as a mutation operation. After the last task returns the final FileSet, Boot synchronizes the FileSet to the target directory. Access to the value of a FileSet doesn't need to be synchronized because it is immutable and the actual filesystem is not in contention. FileSet operations are efficient because Boot uses a content-addressed, hard-link backed structural sharing scheme to minimize actual file operations.
• A Boot task is a function, nothing more. Boot tasks can be created, tested, and composed in the REPL or in any Clojure program just like any Clojure function can. Facilitating task composition is a "unit of classpath isolation", the "pod", which is a Clojure runtime that can have dependencies independent of the rest of the build. So, these task functions can have dependencies independent of the project they might operate on. For instance, a task might supply typechecking using core.typed in a pod, but this doesn't imply the user know or care to add core.typed to their project's dependencies. Because tasks are functions, and because we have a unit of classpath, most builds can be orchestrated in a single JVM. Builds are easier to develop, and more efficient to run.

PF.tv: This sounds great! Is there an efficient path you would recommend for switching a project from Leiningen to Boot?

AD: The best place to go after installing is definitely the wiki.

On the wiki there are many community-contributed tasks and projects to look at for ideas and examples. There is an active and helpful community that you can connect with on IRC (in #hoplon on freenode) and at http://hoplon.discoursehosting.net/

If you have an idea for a script or small program, this is a very nice introductory tutorial that demonstrates making a standalone, boot-powered script that can be called from the command line.

PF.tv: Does Boot have dependency management? Does it work with Clojars?

MN: Sure, Boot uses Pomegranate, the same Aether library that Leiningen uses. Boot can also use Leiningen style wagons (like the S3 private wagon, for instance). The built-in tasks include tasks to install jars in your local Maven repo and to push to remote repos like Clojars. Dependencies can be loaded dynamically at runtime, and Boot has "pods" in which you can load dependencies that conflict with something in the current runtime.

Incidentally, I've been looking at Apache Ivy a bit, it looks really interesting. Maybe it's something we can use in the Clojure world.

PF.tv: Where can people follow you and Boot online? How can they help?

MN: The Boot website is the main entry point into the weird world. We also hang out in the #hoplon channel on Freenode IRC where we provide support for all things tailrecursion but mostly just enjoy the interesting conversations that go on in there :) Help is always welcomed, people can check the open issues or create an issue here or on the associated kanban board here. We try to schedule RC releases weekly every Monday.

PF.tv: Where can people follow you online?

AD: We can be followed at https://twitter.com/alandipert and https://twitter.com/michaniskin.

PF.tv: If Clojure were a food, what food would it be?

AD: We both agree that Clojure is like anchovies: it's weird, relatively unpopular, but an essential ingredient in almost anything that's good. You should only ever need a small amount to make the dreariest meals magnificent. But never tell — nobody needs to know exactly what they're so enjoying.