What are Proof Tests and how are they different to Unit Tests?

I describe two solutions in C++ and C# to the LineSpace Algorithm and I tested using the following pseudo-code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

const int N = 1000; // number of test samples
for (int i = 0; i < N; i ++) {
    double left = rand();  // random left
    double right = rand();  // random right;
    int part = rand();     // number of partitions
    auto result = LineSpace(left, right, part);  // test the function
    // now we want to make sure
    AssertEqual(part + 1, result.size(); // the size of vector should be partitions plus one
    // the distance between results[i] and results[i + 1] 
    // should be the length/partitions
    double plen = abs(right - left) / part;
    for (int j = 0; j < results.size() - 1; j ++) {
       AssertEqual(plen, abs(results[j + 1] - results[j]));
    }
}

const int N = 1000; // number of test samples
for (int i = 0; i < N; i ++) {
    double left = rand();  // random left
    double right = rand();  // random right;
    int part = rand();     // number of partitions
    auto result = LineSpace(left, right, part);  // test the function
    // now we want to make sure
    AssertEqual(part + 1, result.size(); // the size of vector should be partitions plus one
    // the distance between results[i] and results[i + 1] 
    // should be the length/partitions
    double plen = abs(right - left) / part;
    for (int j = 0; j < results.size() - 1; j ++) {
       AssertEqual(plen, abs(results[j + 1] - results[j]));
    }
}

Interestingly, my colleague said this is not good, because unit tests are specific (and the test is random). However, I argued that the above is not unit test, but a proof test. If you want to unit test the LineSpace algorithm, you could write a number of specific different test cases but these test cases are fixed and you can keep adding more later. The advantages of doing so is that you know what you exact test so when things go wrong, you know where to start.

unit-test

The proof tests, on the other hand, generates random test data each time. If you generate as many samples as possible, in some sense, you could ‘prove’ that your algorithm is correct. However, as these are random, there is still a chance that your algorithm fails on some corner cases. Such risks are reduced if you have used a good random generator and the number of random samples is large.

Also, if the random tests are found, you could also output them to console and the next thing is to add them to the unit tests as a specific test case. In this case, I did find some corner test cases using proof tests that it is hard for you to think about on a piece of paper.

–EOF (The Ultimate Computing & Technology Blog) —