Property-Based Testing
In this episode we'll be Test Talking about Functional Programming and Property-Based Testing. As a tester, I feel it's important to understand the latest trends and techniques in software development. Especially if it's a developer-based testing technique that we can educate our programmers about to help create better quality software.
Transcript
**Eric Normand**: [00:00] Actually, functional code is very easy to test as you can imagine because it's all acting in memory, you can give it some inputs and you check the output if it's working, it's working.
[00:16] [background music]
**Announcer**: [00:16] Welcome to the "Test Talks" podcast, the place to go to geek out on software testing. Now your host, whose mission is to help you succeed with test automation, Joe Colantonio.
**Joe Colantonio**: [00:52] Hey, it's Joe, and welcome to episode 150 of Test Talks. I can't believe I've done 150 episodes. Hopefully, I'll do 150 more and beyond. A big thank you to everyone that's been encouraging me all these years, leaving feedback for Test Talks to make the show even better. I really appreciate every one of you. I couldn't do this without the Test Talks' faithful, awesome listeners.
[01:00] In this episode, we're going to be test-talking all about functional programming and property-based testing. As a tester, I think it's really important to understand the latest trends and techniques in software development, especially if it's a developer-based testing technique that we can educate our programmers about to help create better quality software.
[01:21] Eric Normand is the founder of PurelyFunctional.tv, where he develops Clojure-based training material, as well as on-site and in-person training in Clojure and related technologies. He also consults on software architecture, development practices, DevOps, and functional programming.
[01:39] I was really excited to get him on the show today to talk about property-based testing and how we as testers can learn more about functional-based programming, and what that means to our testing efforts. Check it out.
[01:50] [background music]
**Man 1**: [01:53] Test Talks is sponsored by the fantastic folks at Sauce Labs, the cloud-based automated testing platform that eliminates the need to maintain your own Selenium grid and test infrastructure. Try it for free today. Visit testtalks.com, and click on the sign up now link onto the home page sponsor section.
[02:12] [background music]
**Joe**: [02:13] Hey, Eric, welcome to Test Talks.
**Eric**: [02:18] Hey, Joe. I'm glad to be here.
**Joe**: [02:19] Awesome. Today, I'd like to talk about high-level functional programming and property-based testing using Clojure. Before we get into it, could you tell us a little bit more about yourself?
**Eric**: [02:29] I am a functional programmer. I teach Clojure at PurelyFunctional.tv.
**Joe**: [02:36] Very cool. Before we really dive in, I just want to set the stage with some terms that my audience may not be familiar with. What is functional programming is, I guess, what we should start off with.
**Eric**: [02:47] Functional programming, you probably already heard of it. It's trending right now. It's a paradigm of programming that focuses on functions as the main way to make abstractions. You see a lot of use of functions and high-order functions, that is, functions that take other functions as arguments or return a function.
[03:13] You'll also see functional programming or functional programming languages with immutable data structures. There's been a lot of research into type systems in functional programming.
[03:28] It's basically a paradigm where you're dealing with immutable values and transforming them using functions.
**Joe**: [03:37] Awesome. What kind of benefits would someone get out of functional programming that they don't get out of object-oriented programming?
**Eric**: [03:44] Joe, that's a really good question. I think a lot about this and I don't know if I have the best answer. I'm not satisfied with my answer.
[03:59] Because you're dealing with immutable values and functions that don't have side-effects as much as possible, some functions are going to have to have side effects, but because you're dealing with functions that don't have side effects, you're actually able to be more...it's easier to reason about your program.
[04:25] You can say this function only depends on the arguments and the return value is always going to be the same given the same arguments. You can be a lot more relaxed about how often you call it and under what conditions you call it.
[04:41] As opposed to a function that, say, prints out something to the screen or writes to a database or something like that where you don't want to call that twice. You don't want to add two rows when you mean to add one.
[04:54] Actually, functional code is very easy to test as you can imagine because it's all acting in memory. You can now give it some inputs and you check the output. If it's working, it's working.
[05:13] The main benefit to a programmer besides having easier code to reason about, from functional programming, is that it can push you out of that local minimum of a highly optimized object-oriented thinking. If you've been doing object-oriented thinking for a long time, getting a different perspective on the problem is very beneficial.
**Joe**: [05:42] Are there any times when you have an application where you would mix and match an OOP with functional programming?
**Eric**: [05:50] That's a good question. I mean it all depends on the problem you're trying to solve. Some problems lend themselves very well to object-oriented solutions. Clojure itself, a lot of the abstractions are based on Java's class system, methods and instances. You are using it all the time anyway.
[06:24] When would I use functional? Sometimes, it's very useful to look at a problem in terms of, "Here's the data I have and its shape. What values do I have? What's the shape of this data? What values do I need and what shape do I need it in?" It's just a series of transformations to get it from what you got to what you need.
[06:50] Sometimes, it is easier to think in terms of entities with relationships. They call each other methods. For instance, a simulation that's really nice to have.
**Joe**: [07:02] Cool. As I was saying, as I was researching, I came across...you have a bunch of free content on PurelyFunctional.tv and one of the free courses is "Introducing Clojure To The Enterprise."
[07:16] One of those points you made is you could use a JAR file and introduce it almost that way to your code base. That's why I was curious to know when you would do that or why you would do that.
**Eric**: [07:26] One of the great things about Clojure is that it does run on the JVM. It has really seamless interop. You can call methods on any object you have a handle to just very easily. A lot of people will use native Java libraries right from their Clojure code. Also, if you've already got a lot of Java, you can call your classes, instantiate your classes, call the methods.
**Joe**: [08:01] Eric, I am not sure that many testers know that there's actual functional testing language bindings for Selenium. For example, there is one by Haskell. I am not sure if there is one for Clojure.
[08:13] If someone is listening and they really want to learn more about functional testing, that's one way that they actually can get started, grabbing one of those functional testing language bindings and start creating their tests using Selenium would be a quick way to learn.
[08:25] What would you say to a tester who wants to get started with Clojure or functional testing? Why are you so excited about functional testing?
**Eric**: [08:35] I am pretty sure that there are Clojure bindings for the Selenium drivers. I like Clojure myself, to answer your question. I have been using Clojure for nine years now and, I don't know, I really like it. I like that it's on the JVM. I am familiar with the JVM. I like that it's a Lisp. I feel comfortable in it.
**Joe**: [09:03] I am sure the coming question you probably get asked all the time is — I get asked all the time — what language should I learn? Why one is popular? A lot of people when they hear something like functional programming or Clojure, they may not think it's really a skill that would make them more employable. What are your thoughts on that?
**Eric**: [09:22] This is another really good question that I get a lot. It depends on your individual situation. If someone really needs a job and it doesn't matter what language it's in, I would tell them learn JavaScript, because there's a million jobs out there. They are always hiring for it.
[09:44] However, there is also a lot of competition, because there are a lot of JavaScript programmers. If you want to differentiate yourself and find something that is maybe more interesting than a common JavaScript job, you might want to go into something that's a little bit more obscure.
[10:08] If you learn something like Clojure or Elixir or Haskell, you are actually going to be learning something more interesting than JavaScript if that's what you're interesting in. You going to have less competition and there are jobs.
[10:25] You're not going to have the variety of jobs, but it's an illusion anyway. You're not going to get all of those jobs. You only need one job. With remote work now and being able to learn any language and use them all, basically...I was at Clojure/conj in December and I was in the Clojure/west just now in March.
[10:56] There were tons of people looking for programmers, tons of companies sponsoring the thing trying to find programmers to hire. I think that it's an illusion. When you do a search and you see JavaScript programmers, Java programmers, all of them are very in demand. Clojure programmers are in demand, too. There's just not as many.
**Joe**: [11:20] I don't know if it's somewhere on your side. I thought I saw something where I said, "Some of the more newer technologies or newer hot tech areas like machine learning is something that a lot of companies are using Clojure for."
[11:33] Is that true? Is that something that...Now that's one of the benefits of learning Clojure, it's more cutting edge, so therefore you'll be able to maybe work on more cutting-edge projects?
**Eric**: [11:45] I don't know what you read. I do know that for stuff...where Clojure really shines is using all of your course or for doing big data like Hadoop style MapReduce things. It's really great for that. Apache Spark, that kind of thing. It's being used for that for a while. Functional programming is just perfect for that because you have the immutable data.
[12:19] It doesn't matter where it is and how many cores are running at the same time. No one's going to modify it. There is a, I think, bias in the cutting-edge things to use more interesting languages because they want to be able to move very quickly and do a lot of experimentation.
[12:49] Languages like — I'm going to call them out — Java, C# or JavaScript, they might require too much boilerplate to actually be able to experiment quickly on them.
**Joe**: [13:02] Awesome. I mean, definitely that's one of the reasons why I love Python.
**Eric**: [13:05] It's one of the things that Lisp was designed for, was to be able to work on problems that you didn't know how to solve. If you already know the algorithm, it's like a well-known algorithm, sure, just implement it in Java and you're fine.
[13:22] If you don't know how you're going to solve it, you probably want something that's really flexible, that gives you a bunch of tools to try different things, experiment and then through away half of your experiments because they don't really work.
**Joe**: [13:41] Eric, at the beginning of the call, you mentioned that functional testing is easier to test. One of those ways that seems to be pretty popular is property-based testing. Can you just tell us a little bit more about what is property-based testing?
**Eric**: [13:56] Property-based testing is a random testing. Basically, it means you generate random data. Because it's random inputs to your function or whatever you're calling, you don't know what the result is supposed to be. You might know a property of it, like a mathematical property. You might say, "Well, it always should be greater than zero.
[14:24] I don't know what the number's going to be but I know it's a positive number." If you could come up with these properties, you can actually express a whole class of tests. Instead of just saying one or two examples, you can have the computer generate hundreds or thousands of random tests that is just randomly finding bugs in your code.
**Joe**: [14:54] It's almost counter-intuitive usually for a tester because they always say, "Oh, testing need to be deterministic." Now, you're saying, using non-deterministic data. It's a cool concept. It's almost like you're able to find bugs that you may not be able to find if you were using deterministic data. Is that whole truth?
**Eric**: [15:16] Yeah, exactly. Here's an example from the inventor of property-based testing. He's name is John Hughes. He originally developed a system in Haskell called QuickCheck to do it. He also ported that to Erlang. His example is of a database that comes with Erlang. A company was using it and every few months, they would get a corrupted database.
[15:50] They would guess it like, "What it was and why it was being corrupted," but they really didn't have any clues. It was only happening in production, only when it got to be 10 gigabytes big. I'm messing up the details but they're not important. They had these clues but they didn't know what it was. John Hughes wrote property-based tests to just do random stuff to the database.
[16:19] Meaning, store this, read that, delete this, close it, open it again, that kind of thing. He made a property that said, after I do the sequence of random stuff, the database shouldn't be corrupted. He let it run. After a period of time of generating these tests, it figured out how to corrupt the database in the same way that this company had been seeing.
[16:55] The way his system works and the way good property-based testing should work is it shrinks that random input. You could generate a string that's 10,000 bytes long and it's all random bytes in it, all random characters. When you get that and this breaks your system, you're like, "What do I do? There's 10,000. I can't figure out if one of those characters is breaking it or what."
[17:25] What the system does is it will remove a character at random, but systematically. It's going to try to remove all of them eventually. It will also decrease the Unicode character, the Unicode code. It's trying to shrink that string down into the smallest string that still breaks your test, that still fails your test. It does all of that automatically.
[17:56] You could have 10,000 characters in there and then it could shrink it down to the one character that's breaking it.
[18:03] That really helps you debug your system, because you're like, "Oh, this one-character string is breaking my database query." The cool thing is that shrunken version can become an example in your test suite so that you can avoid a regression on that same thing again.
**Joe**: [18:27] It almost sounds like it gives you better coverage or confidence in knowing that you're testing things they may not have thought of. It almost seems like it's able to find those areas that...
**Eric**: [18:36] Exactly. You can sample the random. You can say, "Give me 10 strings at random." You look at them and you're like, "I would never put that in it." [laughs] I don't even know how to type half of those characters, let alone make an example-based test out of these. I would never do this by hand.
[18:58] All those times when you don't test some of the diacritical marks in name fields and stuff, whether they get stored properly and decoded properly. I've been bitten by that. I worked for a Swedish company, and the first time someone had O with two dots over it, our system rejected their username.
[19:21] We were like, "What do we do?" We had to develop tests that had [laughs] Unicode characters in them.
**Joe**: [19:31] It sounds like it's another great solution for...I can't stand UI-based tests normally, especially if it could be done at a lower level. It almost sounds like this is what could help you so you don't have to have a stupid end-to-end test because it's being tested at a lower level and be easier and more maintainable.
**Eric**: [19:50] Yeah. Actually, that brings up a story of someone at pivotal who...Pivotal Tracker, they have a Kanban-board style system for keeping track of work. They wanted to test it. They have UI tests, but what they did was they developed property-based tests for it.
[20:18] Instead of going through the UI and clicking buttons, they actually found the functions that those buttons were bound to. They could actually generate random calls to their back-end that the buttons would have triggered. They could test everything from the program without actually testing button clicks and stuff.
**Joe**: [20:46] Oh my god. That is, to me, super sexy. Can you do a property-based testing with non-functional languages?
**Eric**: [20:55] Yeah, I've seen it in JavaScript. There are property-based testing frameworks and JavaScript. I'm sure in other languages too, like Java, C#.
**Joe**: [21:07] I'm just curious to know, how hard is it to get started? I would think more people would be doing this, but based on my experience, not as many are. Is it a technical hurdle or just an education hurdle?
**Eric**: [21:22] It's education. It's actually not hard to get started. This was the topic of my talk at Clojure/west, which I'll send you a link to so that you can link to it.
[21:37] You can take your example-based tests, where you test the string A. [laughs] Just a little basic test. Instead of using the string with the A character in it, you can turn it into a property-based test that's generating random strings.
[21:57] You think about something like, if you have a key-value store, "I'm going to store with this key in this value, and I should be able to read it back out, and the value should be the same." That's a property that is a high-level property.
[22:14] It's easy to think, "Well, if I give it two and two, when I add them it should be four." That's like, "You know how plus works?" You're doing the plus in your head and writing an example. When your testing something like a key-value store, what you want to know is that when I store something, I can get it back out.
[22:37] You can actually have that key in value be randomly generated, and you know what the value should be because you already have it. You can have this property that, whatever I put in, I can get back out.
**Joe**: [22:50] Very cool. This also would help. I don't know if you've seen this, but a lot of times, we have...when someone checks in code into RCI system, we do code analysis. A lot of times, we'll look at the unit test, and it turns out, sometimes the unit tests aren't doing anything.
[23:07] Is property-based testing something that...like you said, because it's seeding the data almost by itself, you know it's actually doing something. You have confidence in knowing that I have run property-based testing, and I know that my system is functioning as expected.
**Eric**: [23:23] There's are different levels. I talked about this in my talk too. If you just go through and make random inputs, and then check that the answer conforms to a property, that might not really test the whole range of behavior of your system.
[23:45] It does check stuff that you probably wouldn't to do on your own, like funny Unicode characters, empty strings, really long strings. To answer your question, it comes down to the quality of your tests.
[23:59] Where it really starts to test everything, where you're getting confidence that you wouldn't have an example-based tests, is when you...instead of just generating the random arguments to a function, you're generating the, say, method calls on an object.
[24:22] You generate those as data, then you run those methods, you have a random sequence of methods to call, with their arguments. You run those on the object and then you check that all of your invariants still hold.
[24:40] Nothing has crashed, it hasn't done an exception, those kinds of things. That's when you're testing the whole range of behavior of the system that you're testing. With those, I would feel a lot more confident.
**Joe**: [24:56] Also, in your presentations, you kept mentioning generative testing. Is that another name for property-based testing? What is generative testing?
**Eric**: [25:07] I said generative testing. I don't know where I learned that term. I went searching for it. It's something in the Clojure Community. I say generative testing, but I mean property-based testing.
[25:26] Generative testing, I think that some people came up with it in the Clojure Community to try to also be an umbrella term for a different kind of testing, which they are calling simulation testing.
[25:38] Simulation testing is this black box testing, where instead of testing for correctness, you're testing for other things like performance, stability and security, availability, those kinds of things that are not like correctness, per se.
[25:57] You're calculating the right value. It's more like, "This system doesn't crash, or this system can handle 10,000 concurrent requests per second for 10 minutes." Things like that.
**Joe**: [26:10] As a developer, how do you know that you've tested enough of your code? How do feel good that, "I'm going to release this piece of software, and I feel good about that it's been tested and it's going to perform as expected"?
**Eric**: [26:25] Wow. That's always the question. How much do you test? What doesn't need to be tested? How do you test it to feel confident? All I can say is I feel way more confident with property-based testing.
[26:45] I also like to use guarantees from the language. That could be, I'm using immutable data to eliminate problems that I would normally have to test. That said, I don't know if we're ever going to feel like our software is free of bugs.
[27:06] I do know that I like having the test suite, even if it's empty, because I'm going to find bugs. When I'm debugging, I want to be able to reproduce the bug with a test case, so I know I've fixed it.
[27:24] That's another benefit of having a test suite, is a place to put your bugs. [laughs] As opposed to a test suite that...I don't know, everything passes, so you don't have bugs in a way.
**Joe**: [27:41] I definitely agree. Eric, what is PurelyFunctional.tv? What's your website all about for people that may want to learn a little bit more?
**Eric**: [27:51] PurelyFunctional.tv is my company where I teach Clojure. I have about 38 hours of video right now. I'm constantly adding more. By the time you hear this, it might be more. I'm teaching Clojure. There's a monthly subscription, a monthly membership. There's also a yearly membership.
[28:13] You get access to all of those videos. I'm trying to make it the best place to learn Clojure.
**Joe**: [28:22] Very cool. Eric, Before we go, is there one piece of actionable advice you can give someone to improve their property-based testing efforts? Let us know the best way to find or contact you.
**Eric**: [28:34] I would say, find the property-based testing library in your language, and convert some of your example-based tests to use random data instead of the examples that you've hardcoded in your test. As far as where you can reach me, you can go to purelyfunctional.tv and sign up for my newsletter, or one of my other email courses.
[29:07] The newsletter is a weekly newsletter with about 10 links meant to inspire functional programmers and Clojure programmers. It's not all new stuff, it's a lot of old stuff. I really like history and things that have stood the test of time. You can also find me on Twitter @EricNormand.
**Joe**: [29:25] Thank you, Eric, for your functional programming, Clojure property-based testing awesomeness. [inaudible] value we've covered in this episode, head on over to testtalks.com, and in the search, type in 150.
[29:40] While you're there, make sure to click on the free trial button under the exclusive sponsor section to learn all about Sauce Labs' awesome products and services.
[29:52] That's it for this episode of Test Talks. I'm Joe, and my mission is to help you succeed with recreating automation awesomeness. As always, test everything and keep the good. Cheers.
[30:03] [background music]
**Announcer**: [30:03] Thanks for listening to the Test Talks podcast. Head on over to www.testtalks.com for full show notes, amazing blog articles and other automation awesomeness.