Clojure Gazette 104
Issue 104November 30, 2014
Property-based testing (also known as Generative Testing) is gaining a lot of popularity. There are libraries in many major languages.
With Generative Testing, instead of writing a few test cases by hand, you automatically generate the test cases. Since it's automatic, you can generate a huge number of test cases randomly, and so cover a much larger space and find bugs you wouldn't find with a few hand-written cases.
Unit testing (written by hand) a function consists of two parts: the test case (the input to the function) and the expected return value. Well, imagine generating the input randomly. How do you know what the expected return is? That's hard, so let's try an easier problem. Instead of testing an expected value, you could test some known property of the value, for instance that it's a positive integer. Or more complex, like the return value is negative if the input is negative. It can get very much more sophisticated.
It turns out that generative testing finds more bugs than hand-written unit tests, and can even give a minimal input that makes the test fail.I've been researching generative testing and also using it in production code. I though I'd share some of the resources I have found useful.
Enjoy the issue!
Is sponsoring your own newsletter tacky? Well, maybe, but I was so excited to finish the LispCast Clojure core.async videos that I had to tell everyone I know. Have you ever wanted to learn core.async? Then these videos are a great introduction. After watching the animations, doing the exercises, and following the explanations, you should have a solid understanding on which to build more.
This is the paper that introduced QuickCheck. Originally written in Haskell, QuickCheck introduced the world to generative testing.
John Hughes spoke at Clojure/West 2014 about generative testing. Though it is recent chronologically, I think it's the clearest presentation of the idea and potential of generative testing.
test.check is the official Clojure generative testing library. It is written by Reid Draper and it integrates well with clojure.test.
Powerful Testing with test.check Youtube
Reid Draper presented his generative testing library at Clojure/West 2014. test.check found a serious bug in the Clojure implementation of transient data structures.
A fork of test.check by Chas Emerick that uses cljx to bring test.check to Clojurescript.
John Hughes shows how they tested a prominent chat server written in Erlang to reproduce known bugs. What is most interesting is that since the chat server is highly concurrent (it's written in Erlang) and connected to many clients, there are many issues dealing with time, race conditions, and side-effects that had to be accounted for in their
Of very great interest to me is the idea that they looked only for tests that failed every time. It was counter-intuitive to me when I first heard it, but it made sense after a little thought. I also liked how they began describing the properties in terms of intervals of time, to give the system a bit of a margin for getting things straight.
Generating Generators Youtube
Steve Miner at Clojure/conj showed how data-driven schemas (his own schema system called Herbert) can be used to automatically create test.check generators for your data for use in generative testing.
Generative Integration Tests Youtube
Ashton Kemerling (at Clojure/conj 2014) presented how he used test.check in Clojure to test a web frontend for a Rails app.
Free QuickCheck for all Vimeo
John Hughes presents some rules of thumb for coming up with useful properties.
QuickCheck Evolution Youtube
John Hughes gives some guidelines for using generative testing in the longterm for regression testing, continuous integration, and continued development.