Traditional test automation includes flaky, slow tests and poor, frequently ambiguous communication on what is measured and what the results are. This misses on speed, scalability, clarity, communication and labor costs, and doesn’t help business leaders on the team or the essential manual test role nearly as much as it could.
MetaAutomation clarifies the “what,” the “why” and the “how” of quality automation. By focusing on and developing what automated verifications – aka checks – do well, and not getting confused by what automated verifications do not do well, MetaAutomation delivers on speed and communication, clarity and effectiveness.
MetaAutomation is a pattern language: language-independent and platform-independent patterns with a defined dependency structure, to show a way to achieve optimal quality automation. It’s not a tool. There are free, working sample implementations available here.
The Seven Patterns of MetaAutomation
The seven patterns here are in dependency order, from least-dependent to most-dependent. The reason for the pattern "language," rather than just a set of patterns, is that the intra-pattern dependencies range from somewhat important to pivotally important.
There is a diagram of the original five patterns on this blog post.
Log statements aren't sufficient to record how a product is driven, or product state or instrumentation; they're inflexible in abstraction level, will inevitably miss information, and are poorly suited to analysis. The common-occurring Hierarchical Steps pattern is a much more powerful solution for recording and expressing procedures.
The Hierarchical Steps pattern is an important glue to hold the other patterns together; for example, it enables detail on root failure cause that enables the Smart Retry pattern.
The least dependent and most basic and revolutionary pattern, Atomic Check, is about creating checks that focus on business requirements; one check per requirement. Check steps are self-documenting. The artifact from a check run is a structured block of pure data that includes, at a minimum, the name of each basic step in the check, the result of the step (pass, fail, or blocked), the time-to-execute for that step, and structured data on check failure for the fail case. Checks are as lean as possible to run fast and scale with resources, and check results have no presentation so they are easy to store and perfect for analysis over the short- and the long-term.
Atomic Check reduces or eliminates the expensive repro-debug sessions of conventional automation, and provide data for unprecedented clarity on quality. Better Sarbanes-Oxley (SOX) compliance for software product assets is an added bonus!
For products or scenarios where requisite information resources can be created asynchronously, Precondition Pool allows the checks to run even faster and simpler.
Running checks in parallel is not a new concept, but this pattern is included in the language because most of the benefit of Parallel Run depends on the checks being atomic and, as Atomic Check requires, independent of each other and available to run in any order. This pattern is about scale for automated verifications.
Parallel Run is an important requisite for the Smart Retry pattern.
The compact, detailed data results that Atomic Check specifies also enables a more powerful, smart retry technique: on check fail, the available data shows exactly what failed, and because it’s structured and compact, it doesn’t need a human to interpret it. Smart Retry includes decision points on what failures to retry and which ones not, and whether a failure was reproduced, or a different failure occurred. The benefit here is that team members need never be notified of external failures, or (depending on configuration) transient failures, and internal failures can be reproduced and reported as such.
The essence of Automated Triage is that, if human action is indicated, the target team member or members is identified by the nature of the issue and root cause of the failure.
The less-dependent patterns above make for automated verifications that run faster and more reliably; Automated Triage takes the next step towards make team members' jobs easier, and ensuring that real quality issues are handled with the correct priority.
Quality over Time
The business value of doing behavioral and perf analyses on the SUT over the SDLC are vastly greater than with conventional automation, because the information is much more granular, accurate and precise. Implementation changes in how the atomic checks are done are self-documenting in pure data, so the transparency of MetaAutomation creates greater trustworthiness in the analyses of product quality that will be done, e.g., by product owners.
This pattern brings greater clarity and detail to stability of software assets under development, which helps SOX compliance as well.