Christian Weilbach :clojureD 2017 Interview

Christian Weilbach will be giving a talk at :clojureD 2017. He will be speaking on how we can replicate state together.

Follow him on GitHub and Twitter.

PurelyFunctional.tv: How did you get into Clojure?

Christian Weilbach: When I started working on new collaborative software, I worked on some BitTorrent integration in the browser. You could serve static content completely over the p2p network including an URL scheme, but it was hacked together in C++ for KDE (a Linux desktop environment) and I was not able to figure out the bugs (race conditions) even after weeks of traditional debugging at some point. I then started working on Votorola, a consensus building platform for discourse-theoretically motivated deliberation (Habermas). It was written in Java and again I was feeling that things were way too complicated and programming was really hard and taking way too long. Since I wanted to succeed in the projects and was used to look for radical ideas by my philosophy studies, Lisp appeared fairly reasonable to me, as did its roots as an intellectual amplifier. I would even say that it has political potential as an individualistic ideology in the sense that its formalism is endorsing the programmer so (too?) much. I had also read some parts of "Java Concurrency in Practice" and my friend working with me on Votorola told me how difficult it was to get it right.

So I was really happy when Clojure came along my way... it was a big boost in expressivitiy and productivity and I do not worry about these problems anymore. I now use it in most of my projects, both non-commercial and commercial, except for numerics where the JVM is not really popular due to its difficult native interface hurdles.

PF.tv: What is your talk about?

CW: It is about how to think beyond the current abstractions of data management from the programmer's perspective. Instead of building highly optimized distributed systems behind REST interfaces as modern service providers and then gluing them wildly together as application developers, we can move towards shared write semantics for open networks, possibly up to internet scale. Instead of developing your logic in the backend and then wiring a dedicated view to it with tons of effort, you can directly change the state wherever you are in the system and it will converge eventually. So you can develop with a REPL directly on your device or in the browser and the backend will work without explicit wiring. We are not completely there yet as we lack public-private key authentication, but I am now confident that it can be achieved and it has significant benefits for participants, as they can build apps composed of the state of other apps and so on. The most important value of modern applications is the data that is in the application and it is insanely complicated to integrate it in other systems today. We want to change that.

The platform I am presenting builds on CRDTs (confluent replicated datatypes) which you can imagine as durable persistent data structures + commutative merge semantics for conflicts.

PF.tv: Who is your talk for?

CW: This talk is for people who worry about state management. Clojure is very good at tackling data and functional state management concepts (STM, core.async), but it has no semantics to share data from different distributed writers. There are solutions out there, but they are usually not open, but only valid in some previously determined closed subsystem (backend). I have regularly have to integrate external state and all these solutions do not really help me then, so I imagine most programmers have similar problems.

PF.tv: What do you hope people will take away from the talk?

CW: I think Clojure programmers could build inherently more decoupled systems in a p2p fashion without losing the flexibility to have total control. In general I think a little bit more formal structure to composition can provide long-term, more flexible abstractions. You will also learn about some of the pitfalls I hit in Clojure(Script) when I was building this cross-platform system.

PF.tv: What concepts do you recommend people be familiar with to maximize their experience with the talk?

CW: You can think about how concurrent write operations are resolved in your applications and what you actually know about different consistency guarantees of your applications. You do not have to know something about CRDTs, but you can read up about them if you like.

PF.tv: What resources are available for people who want to study up before the talk?

CW: I will try to talk about the concepts mostly and give some concrete example application with a CRDT. You can read up the techreport on CRDTs: http://hal.upmc.fr/inria-00555588/document

There is also an interesting talk about propagators from Edward Kmett https://www.youtube.com/watch?v=acZkF6Q2XKs to get an idea what computation can look like on such a platform. You can also read http://tonsky.me/blog/the-web-after-tomorrow/ and https://writings.quilt.org/2014/05/12/distributed-systems-and-the-end-of-the-api/ to understand some of the motivation.

I also recommend the "Call me maybe" blog series of aphyr, these posts give you a sense that a strict formalism is very necessary for solid building blocks in this area.

PF.tv: Where can people follow you online?

CW: @wh1lo on twitter and whilo on github

PF.tv: Are there any projects you'd like people to be aware of? How can people help out?

CW: First of all I want to encourage the use of core.async. People are weary about pushing abstractions on users of libraries, but core.async should be seen as the standard to build decoupled systems, because that is what it was made for and go-lang is really succeeding with these concepts. I think it should be seen as a standard way to build IO interfaces between Clojure(Script) systems. A lot of the JVM libraries for Clojure still use blocking IO out of convenience, but this does not help when you want to build decoupled, composable systems. I would like to propose to define simple general interfaces for side-effects with core.async and then to build cross-runtime and portable code on them. This still allows to keep the logic in pure functions, but the CSP interfaces are really nice to express side-effects in my opinion.

I have done some jak-shaving in this direction with superv.async, konserve and kabel for replikativ and I would be happy to build a toolbox for composable distributed systems together with others interested in this area. I do not think there will be one big solution for everything, but a strong set of libraries can make Clojure as appealing as Erlang for distributed designs, I think.

PF.tv: Where do you see the state of Clojure in 10 years?

CW: Clojure will have more hosts and probably much more portable code, either pure or defined against simple abstractions. Concurrent programming will get more distributed. I hope that the scientific computing community (like Python, Julia) will become more reachable from Clojure.

PF.tv: If Clojure were an animal, what animal would it be?

CW: Fungus simplex, a fungus which breaks down complexity and gets into the weirdest places evolving state-of-the-art formalisms to develop its unstoppable hunger even further :).