How should we test this? Part 2 - Learning

This blog series explores a central question in software testing: “How should we test this?” (HSWTT?) We'll start with some foundational concepts that shape how we approach testing: learning, modelling, and heuristics. From there, the series dives into practical thinking tools: how to learn about the product (Product Analysis), how to identify important potential problems (Risk Analysis) and how to search for those problems effectively (Test Strategy). Finally, we’ll look at how to communicate what we’ve found in a meaningful way (Reporting).

How should we test this:

• Part 1 - Introduction

Part 2: Learning, heuristics, and models

Learning

Agile is incremental and iterative. And that’s for a good reason. Software is complex, many problems can no longer be solved by one person or within one discipline. Collaboration, critical and creative thinking, communication, being able to see relationships, all these aspects are therefore becoming increasingly important in software development.

Chopping learning in smaller pieces makes things easier to understand. Iterative work is a natural process. We call this evolutionary design: “A strategy to add value quickly by growing something over time from a primitive but complete entity to a higher level of sophistication”.

This is how people learn: we must accept that deep learning is a confusing process at first, but it gets better quickly. By letting go of the work for a while (defocus) we give our brain a chance to process the information learned and integrate it with the models we have in our head. Learning happens in “loops.” So, as in agile, feedback is important: do something, check or discuss the result, move forward based on the result. The shorter and more focused the feedback loop, the faster you get feedback on whether you are doing a good job. The less waste we produce.

Bootstrapping

In RST we teach the bootstrap approach: this means that we need to accept that any deep learning or real innovation is a confusing process at first, but it gets rapidly better.

Booting is the process of starting a computer, specifically with regard to starting its software. The process involves a chain of stages, in which at each stage, a smaller, simpler program loads and then executes the larger, more complicated program of the next stage. It is in this sense that the computer "pulls itself up by its bootstraps"; i.e., it improves itself by its own efforts.

Source: https://en.wikipedia.org/wiki/Bootstrapping#Computing

This is very similar in how we learn: we start with understanding small parts and while we progress these small parts of knowledge or expertise helps us understand larger, more complicated knowledge. Step by step we learn the complex things we need to learn.

In practice this means that you should digest information in (small) parts, take time to learn and accept that you will experience some confusion when learning a product. This confusion is necessary, temporary, and healthy.

Some other interesting facts about learning:

• Learning needs time, your brain needs time to process
• New ideas, understanding often comes when you are NOT working (defocus)
• Build upon ideas from others in your team is an effective way to learn (and innovate) fast

Applied to an agile way of working this means that refining your stories should be done in several steps. So called learning in chunks. Where many teams refine a story only once, I recommend rather spend 3 times 10 minutes on refining a (complex or complicated) story than 1 time 60 minutes. Your brain needs time to process all the information shared. The same can be said about Risk Analysis or Test Strategy. It needs time to unfold and be developed.

Another tip on learning is asking questions. Asking questions is I will write a blogpost on asking good questions later.

Heuristics

Another aspect of learning and understanding better are heuristics. In RST we say: “heuristics are fallible methods for solving problems”. They are like guidelines or rules of thumb. Some examples: “Pull on the handle, push on the plate”, “Don’t drink and drive”, “If it rains, take an umbrella” and “If you are having difficulty understanding a problem, try drawing a picture”. Heuristics are mental shortcuts that help you solve problems or makes decisions fast and with minimal mental effort.

"The engineering method is the use of heuristics to cause the best change in a poorly understood situation within the available resources.”

A heuristic has four definite signatures that make it easy to recognize:
1. A heuristic does not guarantee a solution
2. It may contradict other heuristics
3. It reduces the search time for solving a problem
4. Its acceptance depends on the immediate context instead of on an absolute standard
Source: Billy Vaughan Koen in Discussion of the Method

The most important set of heuristics I use is the Heuristic Test Strategy Model (HTSM). This model is full of guideword heuristics: words or labels that help you access the full spectrum of your knowledge and experience as you analyze something. The HTSM is a model which consists of four sets of heuristics:

1. Project Environment help to understand our context and mission(s)
2. Product Elements help to identify dimensions and factors of the product that could be examined in a test
3. Quality Criteria help to identify value and threats to various criteria of the product
4. General test techniques help to design tests

These heuristics help me think of important aspects when thinking of risks and test strategy.

Heuristics are not followed, they’re applied as means of solving problems in uncertain conditions with uncertain, imperfect, or incomplete information. They can substitute for complete and rigorous analysis when that kind of analysis would be very expensive or impossible.

More on heuristics

• The Essence of Heuristics by James Bach
• Heuristics for understanding heuristics by Michael Bolton
• From Templates to Heuristics: Enhancing Thought Work by Mario Kedemo
• A collection of Testing Mnemonics made by Del Dewar

Models

The last important aspect of learning and understanding better are models. Your mental models of the product control your testing of it. Better (mental) models mean better thinking. Better thinking means better testing.

A model is a representation of an idea, activity, or object such as something complex that you need to work with or study. It helps you understand or manipulate the thing that it represents. For example: a map helps navigate and atmospheric models help predict the weather in your town. Models help you understand by visualizing something that is difficult to see or understand. Mental models help you think better. A mental model is simply a representation of how something works in your head. We cannot keep all details of a product in our brains, so we use models to simplify the complex into understandable and organizable chunks.

Mental models helps understand life. Mental models about testing help you do better testing. For instance, the RST framework is a model of how testing works. Elements of excellent testing is a model of what testing is, what it involves and what a tester does. The heuristic test strategy model helps you think better about test strategy.

Visual models help you think and talk about your testing. There are many, many models you can use to visualize important things in your work: (whiteboard) sketches, a bunch of stickies on the wall, UML, data flow diagrams, data models, activity diagrams, state diagrams, process flow diagrams, architecture diagrams, context diagrams, use cases, wireframes, swim lanes, mind maps, product ecology, product coverage outline, and many more. I use drawings all the time. I cannot explain stuff without drawing or pointing to a visual model of some kind. I like to use all kinds of models in refinements and risk analysis sessions. The models help understanding because we can point to something while talking.

Modeling a hospital

When working in a hospital several years ago, we had to test the 32 most important applications (medical record, drug administration, surgical planning, etc) when they moved from three different locations into a single new building. All department codes had changed and we had to do regression testing to see in these 32 applications still would work. We had 6 months and we did not have a lot of experience with these applications. The models (see picture below) we used saved our lives and gave us a quick insight in how things work. Every time we spoke to people we put them in front of our wall of models to quickly understand what they were talking about.

More on models

• Mental Models. The Best Way to Make Intelligent Decisions
• Many models - better test ideas by Rikard Edgren

---