How much should one control the `Depth` parameter in smallcheck?

Question

I'm doing my first bit of real work with smallcheck, and I'm a little confused about how to use the Depth parameter. Before I go into that, let me state what I'm using smallcheck for.

At work, we're building a simple web service in front of our own in-house database. The web service does some queries, and responds with the query results serialized to JSON. What I'm working on at the moment is the assurance that: given an object that represents the results of a query, this object produces the expected JSON. For example:

data Action
  = Action { actionType :: !ActionType 
           , actionDescription :: !Text
           , actionPerformedAt :: !UTCTime
           , actionAgentName :: !Text
           }

Must produce JSON such as:

{
  "type": "Booking",
  "description": "Whatever",
  "performedAt": "2012-01-04",
  "agent": "Tom"
}

This looked like an ideal task for smallcheck, and I have this formulated as the following:

testAction :: Tasty.TestTree
testAction = Tasty.testGroup "Action"
  [ SmallCheck.testProperty "type" $
      SmallCheck.over actions $ match $
        Aeson.key "type" --> Aeson.toJSON . actionType

  , SmallCheck.testProperty "dateActioned" $
      SmallCheck.over actions $ match $
        Aeson.key "dateActioned" --> expectedUTCTimeEncoding . actionPerformedAt

  -- and so on
  ]

-- (-->) :: Eq a => lens-aeson traversal a -> (b -> a) -> b -> Bool
-- actions :: Monad m => SmallCheck.Series m Action

The default smallcheck depth in the tasty framework is 5, which results in test runs that I've yet to see finish. smallcheck has the changeDepth and changeDepth1 functions, so I could use these as changeDepth (const 3) to ensure that my tests always run in a sensible amount of time. However, by doing this I can't help but feel I'm missing the point somewhere? For example, it is now not possible to run a longer test, perhaps overnight, by just changing the command line options to run the test. On the otherhand, if I used changeDepth (- 2), it still feels as though I am making an assumption of how the tests are ran! Perhaps it's best to assume that a global test depth of 5 runs in n seconds, and it's up to each property to adjust the depth as it sees fit?

Would love to hear some feedback on this more practical side of smallcheck.

Answer 1

When you test using QuickCheck's random testing, the only metric you have is the number of tests, so it's natural to have as many tests as you can afford.

SmallCheck is different in that you can actually reason about what's being tested. Ideally you shouldn't treat depth as just a metric correlated with your confidence in test results, but you should have a good idea about what depth you need.

If we're talking about JSON, then most of the functions processing JSON work with one, or sometimes two layers of structure at a time. So if there's a bug, it can be discovered on a structure of depth 2 or 3, roughly speaking. (You need to find or calculate smallcheck's depth that'll give you the required depth of the structure, based on your Serial instances.)

So, to answer your question, if the depth 3 is the biggest you can afford, then first of all you should decide if that's enough for the kind of code you're testing.

If it happens to be not sufficient, then you could trade breadth for depth (e.g. by reducing the depth for the leaf values), or indeed switch to QuickCheck's random enumeration strategy.

I think you should use QuickCheck only if you feel that the functions you're testing may have bugs due to the size of the structure, as opposed to some local combination of the structure's components. Some examples I can think of are:

• numeric overflows
• undiscovered arbitrary hard-coded limits (perhaps in foreign C code — this is very atypical of Haskell code)

Answer 2

While the "exhaustiveness" of smallcheck (for the small cases anyway) is an intriguing property, I would rather suggest quickcheck in this context. While JSON has a lightweight structure, from the point of view of actual bits of data, it is pretty heavy.

The testing time also quite crucially depend on how you define "size" (depth) for smallcheck in the Series instances for your types. If your types have a lot of branching (many constructors), the number of tests will grow quickly. It is exponential in "depth", while the base of the exponential is the amount of branching in your Series instances relevant to a particular testcase.

In other words, if you have 2 constructors on average, you are looking at something like 32 testcases to run, but if you have 20, it's more like 3200000.

However, your coverage is affected as well -- if you reduce the branching (making depth increase faster) in your testcases, you will get less coverage with given depth. With quickcheck, you trade losing some of the "small" testcases in favour of sampling a number of bigger examples which you can't reach with smallcheck.