Unit Testing with Parameter Value Coverage (PVC)

Parameter Value Coverage (PVC) is the ability to track coverage of a method based on the common possible values for the parameters accepted by the method.

Current code coverage tools fail to take into consideration the possibility that a value for a parameter is not handled resulting in a bug. However, there may not be any code addressing this value in any way, introducing the possibility of obtaining 100% code coverage or line coverage (LC) without detecting the bug.

If the common values for types, especially primitive types and known types, are documented and methods that use them are required to have a test for each possible parameter value, bugs can be avoided.

The following is a table of parameter value requirements for an int or System.Int32 and string or System.String. A unit test should exist for each of the possible values in order to have 100% PVC.

System.Int32 System.String
  1. A positive integer
  2. A negative integer
  3. Zero
  4. int.MaxValue or 2,147,483,647
  5. int.MinValue or -2,147,483,648
  1. A null string
  2. An empty string, String.Empty, or “”
  3. One or more spaces ” “
  4. One or more tabs ” “
  5. A new line or Environment.NewLine
  6. A valid string.
  7. An invalid or junk string
  8. Unicode characters such as Chinese

Were code coverage tools enhanced to take into account Parameter Value Coverage (PVC), better tests would be written and more bugs would be found.

Such PVC lists could be created for each primitive type and a list of required parameter values to test against for each of primitive type could be standard for each language. You could also create a list of parameter values for you own types though you may find that your own type is actually a collection of primitive types and methods, and if they are tested with PVC, your class is automatically tested with PVC as well.

Finding bugs beyond 100% Code Coverage with PVC

100% code coverage or line coverage (LC) can fail to find many bugs. Developers often write a unit test with only the goal to get as close to 100% LC as they can. Unfortunately, they are only covering the written code, but bugs may exist due to code not written.

Here is a quick example. The method below is a very common example of a piece of code that looks so simple, most assume there is no way there is a bug, but after a brief analysis, the bug becomes obvious.


To get 100% LC, the following test could be used.

1
private void Add_Test()
{
    Adder adder = new Adder();
    var actual = adder.Add(1,2);
    var expected = 3;
    Assert.AreEqual(expected, actual)
}

You now have 100% LC, but have you found all the bugs? No you haven’t. Here are two problems:

  1. An int or System.Int32 is a 32-bit integer with a max value of 2,147,483,647.What happens if you add one or more to the max value?
  2. The minimum value is -2,147,483,648. What happens if you subtract one or more from the minimum value?

How should this problem be fixed?

  • Should you enabled “checked” to disallow overflows?
  • Should you return a long so two 32-bit integers can always be added together?
  • Or should you ignore the issue as it isn’t going to matter in your project?

Your answer may be different depending on your project. I am not going to tell you how to fix this bug in this article, that is not the point. The point is to show that 100% LC failed to find this bug, proving that some thing is missing, some value is not being tracked and reported on.

What is the missing value? Parameter Value Coverage (PVC).

If PVC were taken into account, the test above only provides 20% coverage as only one of the five integer value types were tested. Reaching 80% to 100% PVC would have found this bug.

Every bug that is found internally will cost your business far less money overall than if a customer finds the bug.

Return to C# Unit Test Tutorial

One Comment

  1. Larry L. says:

    Great points, but I think your missing a few items,

    Doubles, Floats, bytes.. all have these edge cases ,.. and don't even get me started on dates, and ternary operators....

    and if you're talking doubles and floats you also need to take into consideration the accuracy of the hardware...

Leave a Reply

Powered by sweet Captcha