When developing software, the team needs to know that quality is always moving forward.
To do that, they need to know: “Does the system still do what we need it to do?”
The feedback must be fast and as reliable as possible: If developer A makes a change and checks it in, and the changes pass the quality gate, then developer B checks in her change set, the quality gate must not fail based on anything in developer A’s changes; if it does, people are randomized trying to figure out whose changes caused the failure. The quality checks must therefore be (at least mostly) repeatable.
Data on what the automation did to the System Under Test (SUT), and how it measured it, needs to be persisted so it can become knowledge for the team to use. This is important in case the check passed or failed, so the knowledge can be used to guide manual testers (“What did the automation not do or measure?”) and performance data is recorded, too.
But, how to persist that data?
The simplest conventional automation does not save the data at all; all that information is simply dropped on the floor.
For UI automation, there might be screen shots, but these do not show how the system was driven and a list of screen shots could easily miss some SUT behavior, too.
Another approach -- one that I have used many times -- is logs, sometimes long series of log statements to try to persist the runtime information I need before it is lost. Logs are ideal for monitoring where performance and simplicity are important and the relationship between different log statements is unimportant, although some of that relationship can sometimes be cobbled together with timestamps or other identifiers. Unfortunately, logs are poorly suited for code that drives the SUT through some procedure and makes measurements, because almost all context is dropped. (I write more on logs here. Logs drop most or all of their relationship to other log statements, other than the timestamp, so they’re not very useful for automation driving the SUT through a procedure.
I found the solution when I discovered the Hierarchical Steps pattern. The Hierarchy of steps forms an ordered tree to be traversed inorder (that’s a math thing) but conceptually, each step comprises all of its child steps, begins before the first child step begins, and ends after the last child step ends.
For example, see the Ikea assembly manual for a workstation here.
The steps form a natural hierarchy. The root step is the entire assembly procedure and has the numbered steps as children. Each numbered step has a number of drawings and steps as children. For the furniture-assembler who wants more detail, each of those steps has more steps as children that are implied in this diagram, e.g., picking up a screwdriver, placing a screw, driving the screw, etc.
Imagine assembling the workstation from instructions if there was no hierarchy, instead just a very long numbered list of fine-grained steps. Confusing! These steps would each have no context other than an order. Following these steps would require the furniture-assembler to keep careful track of what the context is in order to find the meaning of the step (the context) and not to get lost. The instructions would have become fragile; easy to make mistakes, and harder to correct those mistakes. Instructions would be difficult to extend with more detail, because it is hard to find where to add the extra steps.
Ikea uses an ordered tree hierarchy to make the instructions usable. Many others do it, too, as Bosch did with the instructions I followed to install a dishwasher.
For driving the SUT through some procedure, a hierarchy doesn’t just make the steps readable, accessible to any role, and analyzable, it clarifies root cause of failure for a failed check, and since the context is inherently persisted, that becomes part of the knowledge for the team and for automated analysis of a check over time or in comparison with other checks etc. Logs simply cannot do this.
A hierarchy also gives a place and a context for added data to be placed, e.g., parameters, SUT instrumentation, or other status. The running samples here show how this works, and allow you to modify the code and the data driving it to play around and learn how cool this can be.
I did not invent the Hierarchical Steps pattern; I just discovered it. Everybody uses this pattern, all the time, and I’d argue that animals use it too.
It is a natural fit for managing quality with automation, and it enables the automation to persist all of the data of how it drives and measures the SUT, whether the check passes or fails, and in a robust structure that is robust for analysis and accessible for all roles. I write more on this here and here.