very interesting! The test categorization definitions remind me of this blog post via @ScottWlaschin

fsharpforfunandprofit.com/post…

Choosing properties for property-based testing

Or, I want to use PBT, but I can never think of any properties to use

^{fsharpforfunandprofit.com}

As keen as I am on property-based testing, I think the 50x result is fake (even separately from critiques of mutation testing).

1. They're comparing projects with both property-based tests and unit tests, so the unit tests are often ones written assuming the property-based tests do the heavy lifting.
2. They say 55% of properties catch a bug on the first example. That means there's a unit test 55% as effective as that PBT.

@DRMacIver hm, is 2. really a counter-argument? it's unlikely that human unit-test authors would have picked the right good unit test after all. also, does 55% really match your gut feeling? I am sure that I have written a lot of hypothesis tests that are much better at bug finding than any amount of standard unit tests I could ever have written.

I find the argument 1. more convincing. of course it's quite hard to avoid this problem.

I think it's plausible to me that 55% of property based tests in the wild could be replaced with a unit test without much loss.

I think it's hard to really come up with a single number capturing the effectiveness of property based tests. I wouldn't be shocked at there being a reasonable experiment that really shows 50x on metric, I just don't think this one is it.