17th Jul 2020
Updated on 27th Jul 2020
All experienced frontend developers know one thing to be true: Users are unpredictable. No matter how much user research you conduct or how thick the font-weight is on your input label, you can never be certain how users will interact with your product. That’s why, as the creators of the interface, we put in constraints. And to ensure that those constraints work properly, we write tests.
But there's a problem with traditional unit and integration tests.
They require us to manually think of and write every scenario that our tests will cover. Not only does this take a lot of time, but it also limits the test coverage to our imaginations. Whereas users, as we know, are unpredictable. So we need a way to test our software to withstand an unlimited number of potential user flows.
That’s where property-based testing comes in.
- Differences between property-based and example-based tests
- What your existing test suite probably looks like (and is missing)
- Choosing which properties to test for
- Writing your first property-based test with fast-check
- Available property-based testing frameworks
- A solid understanding of what unit tests are.
- (Optional) NPM or Yarn installed if you want to follow along in your IDE.
🐍 Prefer Python? A similar guide is available to help you learn property-based testing in Python instead.
We've created a GitHub repository to accompany this guide. This repository includes all of the featured tests with instructions for how to execute them. It also provides more resources for learning property-based testing.
Software testing as we know it today requires a lot of time and imagination. When you're writing traditional example-based tests, you're stuck trying to manually reproduce every action that a user might make.
Property-based testing is a different approach to writing tests designed to accomplish more in less time. This is because instead of manually creating the exact values to be tested, it's done automatically by the framework you're using. That way, you can run hundreds or even thousands of test cases in the same amount of time it takes you to write one
As the developer writing the tests, what you have to do is:
- Specify what type of values the framework should generate (i.e. integers or strings).
- Assert those values on guarantees (or properties) that are true regardless of the exact value.
We'll cover how to choose which properties to test for later in this guide. But before going any further, let's talk about why you would want to integrate property-based testing into your workflow.
Nicolas Dubien, the creator of the fast-check framework we're exploring in this guide, wrote a post outlining the primary benefits of property-based testing.
To summarize his words, property-based testing enables developers to:
- Cover the entire scope of possible inputs: Unless you specifically tell it to, property-based testing frameworks don't restrict the generated values. As a result, they test for the full spectrum of possible inputs.
- Shrink the input when tests fail: Shrinking is a fundamental part of property-based testing. Each time a test fails, the framework will continue to reduce the input (i.e. removing characters in a string) to pinpoint the exact cause of the failure.
- Reproduce and replay test runs: Whenever a test case is executed, a seed is created. This allows you to replay the test with the same values and reproduce the failing case.
In this guide, we'll focus on that first benefit: Covering the entire scope of possible inputs.
Even with the limitations mentioned, traditional example-based tests are likely to remain the norm in software testing. And that's ok because property-based tests aren't meant to replace example-based ones. These two test types can, and very likely will, co-exist in the same codebase.
While they may be based on different concepts, property-based and example-based tests have many similarities. This becomes evident when you do a side-by-side comparison of the steps necessary to write a given test:
|1. Define data type matching a specification||1. Set up some example data|
|2. Perform some operations on the data||2. Perform some operations on the data|
|3. Assert properties about the result||3. Assert a prediction about the result|
At its core, property-based testing is meant to provide an additional layer of confidence to your existing test suite and maybe reduce the number of boilerplate tests. So if you're looking to try out property-based testing but don't want to rewrite your entire test suite, don't worry.
Because property-based tests are meant to fill the coverage gaps missed by traditional testing, it's important to understand how these example-based tests work and their downfalls.
Let's start with a definition: Example-based testing is when you test for a given argument and expect to get a known return value. This return value is known because you provided the exact value to the assertion. So when you run the function or test system, it then asserts the actual result against that return value you designated.
Enough theory, let's write a test.
Imagine you have an input where users write in a number indicating an item's price. This input, however, is
type="text" rather than
type="number" (trust me, it happens, I've seen it). So you need to create a function (
It might look like this:
Now that you have your
getNumber function, let's test it.
To test this using example-based testing, you need to provide the test function with manually created input and return values that you know will pass. For example, the string
"35" should return the number
35 after passing through your
Note: To run this test on your IDE, you'll need to have Jest installed and configured.
And with that, you have a passing example-based test 🎉
There are many situations where an example-based test like this would work well and be enough to cover what you need.
But there can be downsides.
When you have to create every test case yourself, you're only able to test as many cases as you're willing to write. The less you write, the more likely it is that your tests will miss catching bugs in your code.
To show how this could be a problem, let's revisit your test for the
getNumber function. It has two of the most common ways to write a price value (whole number and with a decimal):
Both of these test cases pass. So if you only tested these two values, you might believe that the
getNumber function always returns the desired result.
That's not necessarily the case though. For instance, let's say your website with this price input also operates in Germany, where the meaning of commas and decimals in numbers are switched (i.e. $400,456.50 in English would be $400.456,50 in German).
So you add a third test case to address this:
But when you run the test... you hit a Not-A-Number error:
Turns out the
getNumber function doesn't work as expected when the input string contains a value or specific characters that
Number() doesn't recognize. The same error occurs with inputs like
$50. Maybe you already knew that, but maybe you would've never known that without a specific test case.
🐛🚨 This is one example of how property-based testing can be used to find bugs in your software. Once you realize that any string with a character that
Number() doesn't recognize will return
NaN - you might reconsider how you built that input. Adding the attribute
type="number" to the input restricts the possible values that users can enter and, hopefully, helps reduce bugs.
In an attempt to minimize confusion, the rest of this post will focus on testing floating-point numbers. If you want to see a solution for the
NaN error missed in the previous example-based test, check out our follow-up tutorial.
Issues like the one faced with the input type also help you write your property-based tests because then it's more clear what the property you're testing for actually is.
Let's dig into this. In property-based testing, a property is an aspect of the function being tested that's always true, regardless of the exact input.
If you look at the
getNumber function from earlier, one property you'd test would be the string that is passed to
getNumber. Regardless of whether that input value ends up being
"$50" - it will always be a string.
Some other examples of properties:
- List length when testing the
sort()method on an array. The length of the sorted list should always be the same as the original list, regardless of the specific list items.
- Date when testing a method for the Date object like
toDateString(). No matter the specifics entered, it will always be a date.
Let's use the
getNumber function from earlier. As a reminder, here's what that looked like:
Now let's write a property-based test using fast-check. To limit the scope, you'll only generate input strings with floating-point numbers because values with decimals are more common in prices.
When getting started with fast-check, you first have to set up the base structure of your tests.
Initially, it'll look identical to any other Jest test. It starts with the
test global method and its two arguments: A string for describing the test suite and a callback function for wrapping the actual test.
Next, you'll import the framework and introduce your first fast-check function:
assert. This function executes the test and accepts two arguments: The property that you're testing and any optional parameters. In this case, you'll use the
property function to declare the property.
Finally, you'll add the details of the specific values you want to generate. There's an entire list of built-in arbitraries (aka generated datatypes) provided by fast-check. As mentioned previously, this test will cover input strings with floating-point numbers. There are multiple arbitraries for generating floating-point numbers, but this test will use
float arbitrary will be passed as the first argument of the
property function, followed by a callback wrapping the
expect statement and any other logic necessary for executing the test.
In this test,
testFloat represents each floating-point number generated by fast-check and it's then passed as an argument to the callback. The
expect statement indicates that when you pass the
testFloat as a string to your
getNumber function, you expect it to return the same
testFloat value as a number.
And there you have it, your first property-based test 🎉
By default, the property check will be run against 100 generated inputs. For many arbitraries, you can also set a minimum or maximum number of generated inputs. At first, running hundreds of test cases might feel excessive - but these numbers are reasonable (and even considered low) in the property-based testing realm.
Going back to the example test, you can peek at the generated input values using fast-check's
sample function. This function takes in an arbitrary or property and the number of values to extract. It then constructs an array containing the values that would be generated in your test.
If you wrap the previous function in a
console.log() statement, you'll get something like this:
Note: Your numbers will likely be different - and that's ok. You also might not want to log a bunch of random numbers in your terminal - and that's ok too. You can take our word for it.
We opted to use the fast-check framework for this guide, but there are many other options out there to help you write property-based tests in a variety of programming languages.
- Hypothesis: Python (used in our Python guide)
- FsCheck: .NET
- jqwik: Java
- PropCheck: Elixir
- Proptest: Rust
- PropEr: Erlang
- RapidCheck: C++
- QuickCheck: Haskell
- QuickCheck ported to Rust: Rust
- ScalaCheck: Scala
While it won't replace example-based tests, property-based testing can supply additional coverage where traditional tests fall short. One of the benefits of property-based testing is that it helps cover the entire scope of possible inputs for any given function. We explored that benefit throughout this guide by creating a
getNumber function and writing a test that uses a generative floating-point number property.
This guide wasn't intended to be a series, but the possibility of future guides about shrinking, replaying tests, property-based testing in TypeScript, or our favorite fast-check features emerged during our research. If that sounds interesting to you, tweet at us!
At Meeshkan, we're working to improve how people test their products and APIs. So if you made it to the end of this guide and have thoughts, we want to hear from you. Reach out on Gitter or open an issue on GitHub to let us know what you think.
Don’t miss the next post!
Absolutely no spam. Unsubscribe anytime.