Stephan Schmidt

How Unit Tests Really Help Preventing Bugs

Write unit tests the right way finds and prevents bugs

Quality and testing is part of my CTO Coaching

Contrary to popular belief, unit tests are useful for quality and eliminate certain classes of bugs. The belief that unit tests don’t prevent bugs, or are not helpful, or a futile exercise arises because people do unit testing wrong.

Not every unit test is the same. Not all code coverage is the same. 80% (line or branch) in one code base can be very different from 80% of another code base. It is in the quality of the unit tests that determines how useful they are. This is the reason I advise all my CTO clients to put QA in charge of reviewing all tests, including unit tests. This way they support developers to write better tests, and educate developers and help with preventing bugs.

Lets get into details.

When you have low code coverage or no measuring of code coverage at all, unit tests are not going to help (contrary to a small number of E2E tests). To benefit from unit tests, you will need to measure and raise code coverage. If you have no code coverage, start with a goal of 10% coverage. Then increase by month or quarter the coverage level by 10%. Reaching 70% is straightforward, so aim for that at first.

There are several levels for unit tests on how they help prevent bugs:

At the lowest level, unit tests only test the happy path. On this level, unit tests are written to make sure the feature works if everything goes well. This level of unit tests (with the right amount of coverage) helps with refactoring. Even more, this level of unit testing is essential to be able to constantly refactor your codebase (I wrote about how developers get refactoring wrong ) before. This is the first level to reach.

A test for an add function might look like this:

func TestAdd(t *testing.T) {
  result := add(1, 2)
  assert.Equal(t, 3, result,
    "1+2=3")
}

The next level: To prevent bugs, you need to check edge cases with unit tests. Testing an add function (as a simplified example), developers will test if 1+2=3 (see above). To get to the next level, you need to add checks for 0+1=1, 0+0=0, very large number + very large number = correct, -1 + 1 = 0 etc. This way you will eliminate bugs in your code base. Sadly adding these unit tests usually does not raise code coverage. Luckily ChatGPT is very good at coming up with these kinds of test cases—especially if it knows your code base.

An edge case test

func TestAddLargeNumbers(t *testing.T) {
  // This should succeed
  result =: add(math.MaxInt-1, 1)
  assert.Equal(t, math.MaxInt, result, "they should be equal")
}

then leads to an implementation of add that adds checks for overflow:

func add(a int, b int) (int, error) {
  if (b > 0 && a > math.MaxInt-b)
    || (b < 0 && a < math.MinInt-b) {
    return 0, errors.New("integer overflow")
  }
  return a + b, nil
}

The final level is error handling. According to studies, critical bugs happen when there is bad error handling, or no error handling at all. Add error handling to your code to prevent these bugs and add unit tests to check for error handling. The good thing: This raises code coverage, and is usually what gets you from 80% to 90%, so you can see progress. The bad thing, writing unit tests for correct error handling is difficult.

For the example we can write a test to see if add returns an error for error conditions (which might also be that add calls another method that returns an error instead of failing).

func TestAddLargeNumbers(t *testing.T) {
  _, err = add(math.MaxInt, 1)
  assert.Error(t, err,
    "should return an error due to overflow")
  assert.EqualError(t, err,
    "integer overflow",
    "error message should be 'integer overflow'")
}

Another technique is Property based testing. If possible let the computer check the edge cases and error handling for you:

func TestAddPropertyBased(t *testing.T) {
  rapid.Check(t, func(t *rapid.T) {
   a := rapid.Int().Draw(t, "a")
   b := rapid.Int().Draw(t, "b")

   result, err := add(a, b)

   if (b > 0 && a > math.MaxInt-b) || (b < 0 && a < math.MinInt-b) {
    assert.Error(t, err,
      "should return an error due to overflow")
    assert.EqualError(t, err,
      "integer overflow",
      "error message should be 'integer overflow'")
   } else {
    assert.NoError(t, err, "error should be nil")
    assert.Equal(t, a+b, result, "they should be equal")
   }
 })
}

This property based test tries to find all edge cases in the input domain and test for them. If something fails, it tries to focus in on the error.

You might also add Fuzzing. Fuzzing is underrated, try it to find bugs (fuzzing has the downside of taking a very long time to run and can’t be part of the build chain but needs to be done on it’s own and could be argued to not be unit testing).

Write Unit Tests the right way to find bug

When writing unit tests the correct way (happy path + edge cases + error handling) they can significantly reduce the number of bugs. Bugs are destructive to your product development process, and flow - on top they make you look bad. Reducing bugs is a good thing and makes product managers and customer support happy and developers proud of their work.