False negatives happen when these three things happen in order:
- Operations (ops) promotes the software to the next level, or ships it to end-users
- Someone (or, some automated process) discovers a bug in the promoted or shipped version of the software
- The QA team or role realizes (or, is told) that “Hey, your automation was supposed to check for that bug!”
This is related to “defect escape,” but defect escape is more general because it could be any issue that a customer discovers before the software team knows about it, in addition to any issue the team thought it had fixed or managed. Defect escapes are bad because when a customer finds a significant issue before the team knows about it, the team has not even had a chance to characterize and prioritize it to decide whether it should be fixed. Customer trust takes a hit and the software team has to scramble, which can be expensive in terms of time and interrupted workflows.
A false negative is a defect escape related to the quality automation.
The solution is to apply the Hierarchical Steps pattern of MetaAutomation to your automation: make the automation code self-documenting in a way that preserves context. The code samples here implement this in a platform-independent way. (Although they are packaged in Visual Studio solutions, they can be separated and built to run on multiple platforms.)
The great think about this approach is that every check run documents itself in a hierarchy, with the root node facing the business. Anybody on the team who cares about what the checks do can drill down as far as (s)he wishes towards the fine-grained technology side of driving the SUT, to see exactly what the check did and with milliseconds to completion for each step. It’s very detailed and highly trustworthy.
The QA role or manual testers can see exactly what is done and measured and what is not. Depending on how automation is prioritized and probably depending on value and risk as well, either issues can be covered by people doing exploratory (manual) testing, or the un-measured features can be automated in checks.
Defect escapes will happen because software is increasingly complex, but the problem of false negatives related to automation goes away!