Fixing the false negative problem

False negatives happen when these three things happen in order:

  1. Operations (ops) promotes the software to the next level, or ships it to end-users
  2. Someone (or, some automated process) discovers a bug in the promoted or shipped version of the software
  3. The QA team or role realizes (or, is told) that “Hey, your automation was supposed to check for that bug!”

This is related to “defect escape,” but defect escape is more general because it could be any issue that a customer discovers before the software team knows about it, in addition to any issue the team thought it had fixed or managed. Defect escapes are bad because when a customer finds a significant issue before the team knows about it, the team has not even had a chance to characterize and prioritize it to decide whether it should be fixed. Customer trust takes a hit and the software team has to scramble, which can be expensive in terms of time and interrupted workflows.

A false negative is a defect escape related to the quality automation.

The solution is to apply the Hierarchical Steps pattern of MetaAutomation to your automation: make the automation code self-documenting in a way that preserves context. The code samples here implement this in a platform-independent way. (Although they are packaged in Visual Studio solutions, they can be separated and built to run on multiple platforms.)

The great think about this approach is that every check run documents itself in a hierarchy, with the root node facing the business. Anybody on the team who cares about what the checks do can drill down as far as (s)he wishes towards the fine-grained technology side of driving the SUT, to see exactly what the check did and with milliseconds to completion for each step. It’s very detailed and highly trustworthy.

The QA role or manual testers can see exactly what is done and measured and what is not. Depending on how automation is prioritized and probably depending on value and risk as well, either issues can be covered by people doing exploratory (manual) testing, or the un-measured features can be automated in checks.

Defect escapes will happen because software is increasingly complex, but the problem of false negatives related to automation goes away!


  • Wayne Roseberry said

    A false negative is when a test indicates something did not happen when in fact it did. Given the language of software testing has been to treat finding a failure the goal of a test, we call it a test positive when it fails and negative when it passes. Maybe a bit backward, but there you go.

    Given that, the only thing that gets rid of false negatives is for the test not to have a bug in its assessment of correct state. Hierarchical steps do not fix this problem, they only make the assessment of the place where a found failure occurs more precise. If there is a bug in the test code, or if there is a missing state check, there is no way to realize this without manual inspection or to have the defect escape.

  • MattGriscom said

    Hierarchical Steps will make the problem much less likely by bringing the business logic language in the hierarchy much closer to the code than it can be otherwise, with more detail. This includes having compound verifications where each verification is self-documenting separately, and the code that implements the step carries out exactly what the step says … or it should anyway, to be verified at code inspection.

Add a Comment

Sign up here for emails with bites of wisdom on quality automation and MetaAutomation

Recent Blog Posts

  • The end of test cases

    Test cases have served teams well as a tool to enable measures of quality over time, over many runs of those test cases, so the team knows (or should) roughly whether quality is getting better, worse, … more

  • Example: Fixing the Shopping Cart False Positive

    Last month I posted here about the false positive problem.

    Here is a specific example that I learned from a friend at Microsoft, where the false positive caused a defect escape:

    Imagine a shopping … more