Clojure Gazette 181: Leaps of Abstraction

Leaps of Abstraction

Issue 181 - July 11, 2016

I want to run a Clojure training in your area. See the message at the end of the letter.

Hi Clojurists,

We've talked about what good abstractions look like and how they eliminate whole classes of problems. And they do so with less code than actually solving the problem. But how can we build abstractions systematically? Is there a process we can follow that will reliably generate good abstractions?

I have to mention the old standby, Structure and Interpretation of Computer Programs . At the end of Chapter 2 , the authors describe a system for building graphics. The end result is a layered abstraction built with lambdas. The takeaway was that lambdas are a great tool for abstracting in themselves.

The advice, though, is pretty shabby. The authors talk about the benefits of a stratified design , but don't really give a useful procedure for how to build those lambdas. They do give a clue, though, in talking about how painters are closed unter the operations given. It's a mathematical property that we can rely on.

Denotational Design , Conal Elliott's design process, has a lot to teach in this regard. It is essentially a process for designing those lambdas that you will compose. The first step is to find a very clear, simple description of the problem. That description should be free from implementation tradeoffs. For instance, an image is abstracted as a mapping from 2D point in continuous space to color. There is no mention of pixels or how color might be represented.

In effect, abstractions should be abstract (surprise!). The other thing he does is to lean heavily on known m athematical abstractions from category theory. Doing so bares a lot of fruit . We don't know how to compute , so it's nice to rely on well-studied mathematical structures with known-properties. These structures are known to compose in elegant and reliable ways.

Unfortunately, even though Denotational Design yields very elegant abstractions, they're expressed as large, structures built of lambdas. It's not very convenient to compose big expressions from lambdas. They get hard to read real fast. The layers help by breaking down the structures and giving you parts to name. Ideally, though, you would compose clearly and readably without naming everything. We communicate all the time without naming intermediate ideas.

For a solution to this, I turn to Alan Kay. The folks at Viewpoints Research Institute use a different method. They have been researching how far parsers can be put to abstraction . For instance, they wrote a new language (called Nile ) for describing how to choose the color of a pixel given polygons you want to draw. The compiler for that language plus the code to desribe every graphical operation is less than 400 lines of code. By contrast, Cairo, which is what most systems use, is over 10,000 lines of code. And VPRI's solution is faster. When people think about declarative programming, they often think of general purpose languages like Prolog. Instead of looking for a general purpose solution, perhaps we should be writing custom declarative languages for each problem. Writing parsers is easy.

The big drawback to this approach is that error messages from parsers are generally pretty bad. One wrong token and you've confused the parser. Making good error messages is probably harder than making the parser in the first place. I've looked fo years for research into this area. Parsing correct input is solved. Helping a person correct incorrect input is not. Showing errors as a person types seems to help a lot, but more study is needed. Solving this will help us all decrease our code sizes by orders of magnitude while making our software more robust.

As software programmers, we deal in abstractions and yet we're not that good at it. We're still trying to figure out how we can build powerful systems without being overwhelmed by huge codebases. I believe the answer lies in declarative DSLs. They tap into our natural linguistic abilities better than lambda expressions do. And being abstract, they allow for automatic optimizations when applied to specific problems. They are powerful inflection points where a small amount of code can do big things.

Rock on!

PS Want to get this in your email? Subscribe !
PPS Want to advertise to smart and talented Clojure devs ?

Clojure Training

As a teacher, I benefit so much from live, in-person trainings. It helps me refine my material and get essential information about what people need to learn. So, I'd love to do a training in your area, if there are enough people to fill it up. Imagine two days of intensive Clojure! You'll learn so much. Let me know where you'd like the training by filling out this easy survey . I need to fill the trainings to justify the cost of travel, so get your friends to complete the survey, too!