Generative Testing

Clojure Gazette

Issue 104November 30, 2014

Editorial

Hi Clojurators,

Property-based testing (also known as Generative Testing) is gaining a lot of popularity. There are libraries in many major languages.

With Generative Testing, instead of writing a few test cases by hand, you automatically generate the test cases. Since it's automatic, you can generate a huge number of test cases randomly, and so cover a much larger space and find bugs you wouldn't find with a few hand-written cases.

Unit testing (written by hand) a function consists of two parts: the test case (the input to the function) and the expected return value. Well, imagine generating the input randomly. How do you know what the expected return is? That's hard, so let's try an easier problem. Instead of testing an expected value, you could test some known property of the value, for instance that it's a positive integer. Or more complex, like the return value is negative if the input is negative. It can get very much more sophisticated.

It turns out that generative testing finds more bugs than hand-written unit tests, and can even give a minimal input that makes the test fail.I've been researching generative testing and also using it in production code. I though I'd share some of the resources I have found useful.

Enjoy the issue!

Rock on!
Eric Normand

PS Learn more about the Clojure Gazette and subscribe. Learn about advertising in the Gazette .

Sponsor: LispCast

Is sponsoring your own newsletter tacky? Well, maybe, but I was so excited to finish the LispCast Clojure core.async videos that I had to tell everyone I know. Have you ever wanted to learn core.async? Then these videos are a great introduction. After watching the animations, doing the exercises, and following the explanations, you should have a solid understanding on which to build more.

QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs PDF

This is the paper that introduced QuickCheck. Originally written in Haskell, QuickCheck introduced the world to generative testing.

Testing the Hard Stuff and Staying Sane Youtube

John Hughes spoke at Clojure/West 2014 about generative testing. Though it is recent chronologically, I think it's the clearest presentation of the idea and potential of generative testing.

test.check Github

test.check is the official Clojure generative testing library. It is written by Reid Draper and it integrates well with clojure.test.

Powerful Testing with test.check Youtube

Reid Draper presented his generative testing library at Clojure/West 2014. test.check found a serious bug in the Clojure implementation of transient data structures.

double-check Github

A fork of test.check by Chas Emerick that uses cljx to bring test.check to Clojurescript.

Testing Asynchronous Behaviour in ejabberd with QuickCheck Vimeo

John Hughes shows how they tested a prominent chat server written in Erlang to reproduce known bugs. What is most interesting is that since the chat server is highly concurrent (it's written in Erlang) and connected to many clients, there are many issues dealing with time, race conditions, and side-effects that had to be accounted for in their

property-based model.

Of very great interest to me is the idea that they looked only for tests that failed every time. It was counter-intuitive to me when I first heard it, but it made sense after a little thought. I also liked how they began describing the properties in terms of intervals of time, to give the system a bit of a margin for getting things straight.

Generating Generators Youtube

Steve Miner at Clojure/conj showed how data-driven schemas (his own schema system called Herbert) can be used to automatically create test.check generators for your data for use in generative testing.

Generative Integration Tests Youtube

Ashton Kemerling (at Clojure/conj 2014) presented how he used test.check in Clojure to test a web frontend for a Rails app.

Free QuickCheck for all Vimeo

John Hughes presents some rules of thumb for coming up with useful properties.

QuickCheck Evolution Youtube

John Hughes gives some guidelines for using generative testing in the longterm for regression testing, continuous integration, and continued development.