ARTICLE

From Elm in Action by Richard Feldman

__________________________________________________________________

Save 37% off Elm in Action. Just enter fccfeldman into the discount code box at checkout at manning.com.
__________________________________________________________________

This article deals with writing fuzz tests in Elm.

Writing Fuzz Tests

When writing tests for business logic, it can be time-consuming to hunt down edge cases -those unusual inputs which trigger bugs that never manifest with more common inputs.

In Elm, fuzz tests help us detect edge case failures by writing one test which verifies a large number of randomly-generated inputs.

DEFINITION: Elm’s fuzz tests are tests which run several times with randomly-generated inputs. Outside of Elm, this testing style is sometimes called fuzzing, generative testing, property-based testing, or QuickCheck-style testing. went with fuzz because it's concise, suggests randomness, and it's fun to say.

Figure 1 shows what we’ll be building towards.

Figure 1 Randomly generating inputs with fuzz tests

A common way to write a fuzz test is to start by writing a unit test and then converting it to a fuzz test to help identify edge cases.

Let’s dip our toes into the world of fuzz testing by converting our existing unit test to a fuzz test. We’ll do this by randomly generating our JSON instead of hardcoding it; this way we can be sure our default title works properly no matter what the other fields are set to!

Converting Unit Tests to Fuzz Tests

Before we can switch to using randomly-generated JSON, first we need to replace our hardcoded JSON string with some code to generate that JSON programmatically.

Building Json programmatically with Json.encode

As we use the module to turn JSON into Elm values, we can use the module to turn Elm values into JSON. Let’s add this to the top of , right after .

import Json.Encode as Encode

Because JSON encoding is the only type of encoding we’ll be doing in this file, that alias lets us write instead of the more verbose . While we’re at it, let’s give our import the same treatment, and change it to this:

import Json.Decode as Decode exposing (decodeString)

Json.encode.value

Whereas the module centers around the abstraction, the module centers around the abstraction. A (short for ) represents a JSON-like structure. In our case we'll use it to represent JSON, but later we'll see how it can be used to represent objects from JavaScript as well.

We’ll use three functions to build our JSON on the fly:

and translate Elm values into their JSON equivalents. takes a list of key-value pairs; each key must be a , and each value must be a .

Table 1 shows how we can use these functions to create a representing the same JSON structure as the one our hardcoded string currently represents.

Table 1 Switching from String to Json.Encode.Value

Json.Decode.decodevalue

Once we have the we want, there are two things we could do with it.

  1. Call to convert the to a , then use our existing call to run our decoder on that JSON string.
  2. Don’t bother calling , but instead swap out our call for a call to .

Let’s start by editing our line to expose instead of . It should end up looking like this:

import Json.Decode as Decode exposing (decodeValue)

Then let’s incorporate our new encoding and decoding logic into our test’s pipeline.

Listing 1 Using programmatically created JSON

decoderTest : Test
decoderTest =
test "title defaults to (untitled)" <|
\_ ->
[ ( "url", Encode.string "fruits.com" )
, ( "size", Encode.int 5 )
]
|> Encode.object
|> decodeValue PhotoGroove.photoDecoder ❶
|> Result.map .title
|> Expect.equal (Ok "(untitled)")

❶ We now call decodeValue instead of decodeString here

from test to fuzz2

Now we’re building our JSON programmatically, but we’re still buiding it out of the hardcoded values and . To help our test cover more edge cases, we'll replace these hardcoded values with randomly generated ones.

The module helps us do this. Add this after :

import Fuzz exposing (Fuzzer, list, int, string)

We want a randomly generated string to replace and a randomly generated integer to replace . To access those we'll make the substitution shown in Table 2.

Table 2 Replacing a Unit Test with a Fuzz Test

We’ve done two things here.

First, we replaced the call to with a call to . The call to says that we want a fuzz test which randomly generates two values. and are fuzzers which specify that we want the first generated value to be a string, and the second to be an integer. Their types are and .

DEFINITION: A fuzzer specifies how to randomly generate values for fuzz tests.

The other change was to our anonymous function. It now accepts two arguments: and . Because we've passed this anonymous function to , will run this function one hundred times, each time randomly generating a fresh value and passing it in as , and a fresh value and passing it in as .

NOTE: doesn't generate strings completely at random. It has a higher probability of generating values which are likely to cause bugs: the empty string, extremely short strings, and extremely long strings. Similarly, prioritizes generating 0, a mix of positive and negative numbers, and a mix of extremely small and extremely large numbers. Other fuzzers tend to be designed with similar priorities.

Using the randomly-generated Values

Listing 2 Our first complete fuzz test

decoderTest : Test
decoderTest =
fuzz2 string int "title defaults to (untitled)" <|
\url size -> ❶
[ ( "url", Encode.string url ) ❶
, ( "size", Encode.int size ) ❶
]
|> Encode.object
|> decodeValue PhotoGroove.photoDecoder
|> Result.map .title
|> Expect.equal (Ok "(untitled)")

❶ url and size come from the string and int fuzzers we passed to fuzz2

TIP: For even greater confidence, we can run to run each fuzz test function five thousand times instead of the default of one hundred times. Specifying a higher value covers more inputs, but it also makes tests take longer to run. Working on a team can get us more runs without any extra effort. Consider that if each member of a five-person team runs the entire test suite ten times per day, the default value of of one hundred gets us five thousand runs by the end of the day!

Next we’ll turn our attention to a more frequently invoked function in our code base: .

Testing update functions

All Elm programs share some useful properties which make them easier to test.

  1. The entire application state is represented by a single value.
  2. only ever changes when receives a and returns a new .
  3. is a plain old function, and we can call it from tests like any other function.

Let’s take a look at the type of :

update : Msg -> Model -> ( Model, Cmd Msg )

Because this one function serves as the gatekeeper for all state changes in our application, all it takes to test any change in application state is to:

  1. Call in a test, passing the and of our choice
  2. Examine the it returns

Testing Clickedphoto

Let’s use this technique to test one of our simplest state changes: when a message comes through the application. For reference, here's the branch of 's case-expression that runs when it receives a message.

SlidHue hue -> 
applyFilters { model | hue = hue }

This might seem like a trivial thing to test. It does little! All it does is update the model’s field, right?

Listing 3 shows a basic implementation, which combines several concepts.

Listing 3 Testing SlidHue

slidHueSetsHue : Test
slidHueSetsHue =
fuzz int "SlidHue sets the hue" <|
\amount ->
initialModel ❶
|> update (SlidHue amount) ❷
|> Tuple.first ❸
|> .hue ❹
|> Expect.equal amount ❺

❶ Begin with the initial model

❷ Call update directly

❸ Discard the Cmd returned by update

❹ Return the model’s hue field

❺ The model’s hue should match the amount we gave SlidHue

takes a tuple and returns the first element in it. Because returns a tuple, calling on that value discards the and returns only the -which is all we care about in this case.

Let’s run the test and… whoops! It didn’t compile!

Exposing variants

 port module PhotoGroove exposing (Model, Msg(..), Photo, initialModel, main, photoDecoder, update)

The in means to expose not only the type itself (for use in type annotations such as ), but also its variants. If we'd only exposed rather than , we still wouldn't be able to use variants like in our test! In contrast, is a type alias, and writing yields an error; has no variants to expose!

NOTE: We could also write instead of separately listing what we want to expose. It's best to avoid declaring modules with , except in the case of test modules such as .

Because we’re using without qualifying it with its module name-which we could have done by writing instead-we'll need to expose some variants in our declaration to bring them into scope. Let's do that for , , , and by changing the line in like this:

import PhotoGroove exposing (Model, Msg(..), Photo, initialModel, update)Unfortunately, elm-test doesn’t currently support testing commands directly. You can work around this if you’re willing to modify your update function. First, make a custom type which represents all the different commands your application can run. In our case this is:type Commands = FetchPhotos Decoder String | SetFilters FilterOptions Then change update to have this type: update : Msg -> Model -> ( Model, Commands )Next, write a function which converts from Commands to Cmd Msg.toCmd : Commands -> Cmd Msg toCmd commands = case commands of FetchPhotos decoder url -> Http.get url decoder |> Http.send GotPhotos Setfilters options -> setFilters optionsFinally, we can use these to assemble the type of update that programWithFlags expects:updateForProgram : Msg -> Model -> ( Model, Cmd Msg ) updateForProgram msg model = let ( newModel, commands ) = update msg model in ( newModel, toCmd commands )Now we can pass updateForProgram to programWithFlags and everything will work as before. The difference is that update returns a value we can examine in as much depth as we like, meaning we can test it in as much depth as we like!This technique is useful, but it’s rarely used in practice. The more popular approach is to hold off on testing commands until elm-test supports it directly.

Excellent! If you re-run , you should see two passing tests instead of one.

Creating multiple tests with one function

One way to add tests for the other two is to copy and paste the test two more times, and tweaking the other two to use and . This is a perfectly fine technique! If a test doesn't have other tests verifying its behavior, the best verification tool is reading the test's code. Sharing code often makes it harder to tell what a test is doing by inspection, which can seriously harm test reliability.

In the case of these sliders, though, we’d like to share code for a different reason than conciseness: they ought to behave the same way! If in the future we changed one test but forgot to change the others, which would almost certainly be a mistake. Sharing code prevents that mistake from happening!

Grouping tests with describe

Listing 4 shows how we can use the function to make a group of slider tests.

Listing 4 Testing SlidHue

sliders : Test
sliders =
describe "Slider sets the desired field in the Model" ❶
[ testSlider "SlidHue" SlidHue .hue ❶
, testSlider "SlidRipple" SlidRipple .ripple ❶
, testSlider "SlidNoise" SlidNoise .noise ❶
]


testSlider : String -> (Int -> Msg) -> (Model -> Int) -> Test
testSlider description toMsg amountFromModel =
fuzz int description <| ❷
\amount ->
initialModel
|> update (toMsg amount) ❸
|> Tuple.first
|> amountFromModel ❹
|> Expect.equal amount

❶ Group this List of Test values under one description

❷ Use testSlider’s description argument as the test’s description

❸ (toMsg : Int -> Msg) will be SetHue, SetRipple, or SetNoise

❹ (amountFromModel : Model -> Int) will be .hue, .ripple, or .noise

The function has this type:

describe : String -> List Test -> Test
Figure 2 Failure output after using describe

Returning tests from a custom function

The function is a generalized version of our test from earlier. Table 3 shows them side by side.

Table 3 Comparing slidHueSetsHue and testSlider

Have a look at the type of :

testSlider : String -> (Int -> Msg) -> (Model -> Int) -> Test testSlider description toMsg amountFromModel =

Its three arguments correspond to what we want to customize about the test:

  1. lets us use descriptions other than
  2. lets us use messages other than
  3. lets us use model fields other than
describe "Slider sets the desired field in the Model" 
[ testSlider "SlidHue" SlidHue .hue
, testSlider "SlidRipple" SlidRipple .ripple
, testSlider "SlidNoise" SlidNoise .noise
]

This compiles because the variant is a function whose type is , which is what the argument expects, and because the shorthand is a function whose type is , which is what the argument expects.

Running the Complete tests

Let’s take them out for a spin! If you re-run , you should still see four happily passing tests.

TIP: Notice how always prints " " and then a big number? That big number's the random number seed used to generate all the fuzz values. If you encounter a fuzz test which is hard to reproduce, you can copy this command and send it to coworker. If they run it on the same set of tests, they'll see the same output as you; fuzz tests are deterministic given the same seed.

We’ve now tested some decoder business logic, confirmed that running a message through sets appropriately, and expanded that test to test the same logic for and by using one function that created multiple tests. Next we'll take the concepts we've learned this far and apply them to testing our rendering logic as well!

That’s all for this article. If you want to learn more about the book, check in out on liveBook here and see this slide deck.

Originally published at https://freecontent.manning.com.

Follow Manning Publications on Medium for free content and exclusive discounts.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store