Pre-Conj Interview: Paul deGrandis
Talk: Unlocking Data-Driven Systems
Paul deGrandis' talk at the conj is about the design tradeoffs when building your system in a Data-Driven way.
Background
The recommended priority for choosing an implementation strategy is Data > Function > Macro. Macros are made for humans, not machines, so they are the least composable. Functions are composable, but opaque. Their only meaning is their side-effects and their return value. Data, however, can be interpreted in different ways, depending on the context. If one can design it right, a data-driven system can be both easy for a person to read and write and reusable for many problems.
This talk is part of an ongoing discussion about data-driven systems in Clojure. Data-driven design has been around far longer than Clojure has. Paradigms of Artificial Intelligence Programming is the book on the subject, showing its use in Common Lisp. Christophe Grand gave a talk at the Clojure/conj in 2010 that explained in depth why macros should not be the first choice when building DSLs.
Many systems use data as their primary interface. Datalog queries in Datomic are just Clojure data. Prismatic Schemas are data. Leiningen project files are data (with a small macro for human convenience). Data-driven system is related to code-as-data, that all Lisps share. If code is data, why can't data be code? Clojure makes data-driven solutions easy with its variety of literal data structures.
Why it matters
The conversation about how best to design data-driven systems is not over. Active Clojure developers are still experimenting and testing the approach. This talk will show data-driven systems taken to an extreme, where an entire ClojureScript application is represented as data. Many people already agree with the approach and are eagerly awaiting the deeper analysis this talk promises.
About Paul deGrandis
Introduction
Paul deGrandis gave a nice interview about his talk at Clojure/conj about data-driven systems. Read the background to his talk.
Interview
PF.tv: How did you get into Clojure?
Paul deGrandis: I was a very early Clojure adopter. I came from Python (and before that a mix of C/C++/Java/Python), but I had always used Common Lisp to prototype ideas. Once I saw Clojure publicly announced, I gave it a shot and instantly identified with the Clojure Trinity: Simplicity, Power, and Focus. I began using it in all of my proof-of-concept work, and as I became confident in the language, integration with the platform, and my own skill set, I also started putting it into production.
PF.tv: Can you define Data-Driven systems briefly?
PdG: Data-driven systems capture their interactions and operations as values - they are one possible conclusion of programming with values. We see this done in Datomic - queries are expressed as data. This concept is easily mapped to other systems - services, client-side applications, and more. Note that is more than just, "data all the things," this is, "data all the thing's things." Another example: Pedestal routes are represented as data ("data all the things"), but imagine if we captured the expression of an entire service as data. Imagine the properties that naturally fall out from that design decision. My talk walks through a real project that pushed this design decision as far as possible, and the steps that other developers can follow to do the same for their own systems.
PF.tv: At Cognitect, they talk about Data > Functions > Macros. Is that what your talk is about?
PdG: My talk, at a very high level is about "Data > Functions > Macros," but it's mostly about modeling entire systems (not just interfaces) as data, why you'd want to, and why it's hard. It definitely challenges the previous notions (established by existing tooling) of what the conclusions of "programming with values" actually are (or could/should be). Tangentially, I'll talk about DSLs, but only in regards to constraining your domain as a means to increase expressiveness.
PF.tv: I'm a big fan of working with data. But I've also seen its dark side. I've seen it pushed too far, or in the wrong area, to where it would have been better to use code. Have you noticed this? How do you avoid that?
PdG: I have indeed noticed it and it's a topic within my talk. Two crucial pieces of any great design (in software or otherwise) are the external constraints imposed by the problem, and the internal constraints imposed by the designer. Both are important, but it seems common that people overlook self-imposing constraints - and one needs them to achieve a really great design. I don't necessarily agree that capturing a system in data can be pushed too far, but I think it can be done without constraint or design intent - which produces a useless system. The crux is knowing that limitation and working within it. Outside of those limitations and constraints, one should fall back to foundation - regular Clojure code, etc.
PF.tv: What is one thing you want someone to be able to do after they watch your talk?
PdG: I want someone to come away with a set of techniques to envision, constrain, and design data-driven systems. I want to create a spark that encourages Clojure developers to look beyond the initial merits of programming with values, and see further-reaching impact within their own systems - how to turn our ecosystem into a set of super powers.
I hope to provide a compelling example of how this approach plays out in production systems, and the general metrics and project impact it has.
PF.tv: What resources would you recommend to a beginner so they could maximize their understanding of your talk?
PdG: My talk touches upon a lot of themes covered in some inspiring and informative talks: Tim Ewald's keynote from last year's Conj (Programming with Hand Tools), Rich Hickey's GOTO conference Keynote (The value of values), and previous Conj talks. I'm still searching for literature and books that I like on the topic, but I haven't hit any that I'd strongly recommend.
PF.tv: Where can people follow your adventures online?
PdG:
- My Twitter: @OhPauleez
- My Github: https://github.com/ohpauleez
- I sometimes blog: http://www.pauldee.org/blog/
PF.tv: If Clojure rode an animal, what animal would it ride?
PdG: A very serious pony.
PF.tv: Thanks for a great interview. It was informative.
PdG: Thanks a ton! I really appreciate all of this!
