Here are the sketchnotes I made during DEWT3. Click the images to view them full size.
What Testers Can Learn From Social Sciences
This is the video of my talk at TestBash 2.0 in Brighton March 2013.
What can testing learn from social science?
Why is this important to testers? My conclusion is that testing and the social sciences are very much interconnected and there are many lessons that we could learn from this area. We should see what we can apply to our daily testing jobs. We should be more aware of what we do in testing: for example social research, making observations, doing critical thinking and most importantly continuing to learn. We should start learning from what people have done and are still doing in the social sciences. Testers should not only focus on quantitative analysis like bug counts or test case pass or fail, but also do qualitative research. Test reports should be stories about the product and the testing we did (see Michael Bolton’s article on test reporting and the telling of the story). We should use the numbers to support or backup our story. I often see it the wrong way around: lots of tables full of counts that do not tell us anything without the context. Managers do tend to draw their own conclusions and make decisions based upon this data, if we do not help them by telling the story. Again, testing is about collecting information for people who matter to enable them to make informed decisions.
If a manager comes up to you and asks you:
“So what is the coverage?”
“How many test cases do you have?“
What do you say? It is really hard to talk to managers who are obsessed by numbers and think that testing is about the number of test cases, right and wrong, green and red.
Consider this next time you talk to them. Make a simple calculation of all the possible tests you could do for the project you are working on regardless of how simple or complex it is. Think of all the possible combination both positive and negative.
What number have you ended up with?
- 1 thousand?
- 1 million?
- 1 billion?
The number of possible things that you can test are endless, exhaustive testing is futile. Even a simple requirement has infinite possibilities to test.
So what is the coverage if we do 1,000,000 test cases?
- Coverage: 1,000,000 divided by infinite is very close to zero!
- Coverage: 10,000.000 divided by infinite is still very close to zero!
So no matter how many test cases you have the coverage of all possible test cases you could have done is close to zero. This is why risk, priority and making choices become important for testing but that is a different topic.
Be a scientist
Science is important. It gave us critical thinking and that helps us proving the theories we have about the product. Try to prove yourself wrong instead of proving yourself right.
Ask critical questions:
- Could it be something else?
- Is this what I expected?
- What did I do differently?
- What else can I do?
- How can I explain what I did?
While testing we should practice critical thinking: question things we encounter, make sure that what we see, is true (or not). While thinking we should be aware of fast conclusions, biases and fallacies. We often do it the wrong way around. If we focus too much on the numbers and the averages we miss the outliers: the unique random events that can do the most damage.
The grounded theory method is a research method that operates almost in a reverse fashion from traditional social science research. You start with a view, theory or expectation before you start. However as you gather more data when testing, your theory/expectation becomes more ‘grounded’ upon the information you uncover or discover. Basically as you experience and gather more and more data you change your perspective and viewpoint. This is what social scientist do when they go and live with social groups. They have something they want to find out – an assumption or otherwise – and find out if it is right or not by taking part (compare missions in Exploratory testing).
In qualitative research done by anthropologists for example the context of the research is very important. Here they accept and deal with ambiguity, situational specific results and partial answers. Qualitative data deals with meanings. Use it when you want to understand the underlying thoughts and intentions.
Testers should not observe from the side-line. We should act like anthropologists do: become part of the group you are observing. Let me give you an example from my own experience.
A couple of years ago I worked as a test coordinator for a Media Company selling newspapers and magazines. We were implementing a new CRM system and my assignment was to organize the user acceptance testing. For some days I worked with the people selling the newspapers to learn how they were selling subscriptions and while doing that I learned what was important to them. First I was observing, but after a day I was selling newspapers myself and really learned how the users were working and what it took to make the department successful. We had requirements and designs, but the stuff I found out on processes and user sentiments was also very valuable to do testing. There were important steps not documented. I saw the people use the software in ways I didn’t expect and that wasn’t written down anywhere. I asked them what they did and why and they answered me “that is normally how we do this”. I learned that these people took short cuts to do their work. The team who were designing and building the software had no idea what I was talking about when I told them about my observations.
Humans will always take the shortest quickest route and the one that requires the least amount of thinking. This made clear how important it is to find out what people are thinking. And more important: the reasoning why they do the things they do. The product is a solution. If the problem isn’t solved, the product doesn’t work (5th basic principle of context-driven testing).
Now I know it is called qualitative research and I think every team developing software should do something similar. Try to really understand the users and the environment in which the product will be used. IT is often way to focussed on technical stuff. Testers go out there and meet the people who are using the product. Be part of their world for a while and start asking those critical questions.
Software is build by humans for humans. Social science is about people. Software should solve problems and help humans. To really solve a problem, we need to know more about how the users work, what they think, how they feel, their emptions, their desires. Too often I hear development teams say things like: “The user should not do that”. Or the all time classic: “no real user would ever do that!”. IT is way too technical focussed.
Read John’s blog titled “The Human Element”. It is an awesome story about his wife, a nurse, who explains why the human element is very important in her work. You see the parallel with our work?
Now it is your turn!?
Use the reading list to learn more about what social sciences, biases and other relevant topics. I am curious and I want to learn from you too. So please share your thoughts and experiences with social science.
“Great software is not produced or tested in factories, but in studios
and rehearsal halls.” (Michael Bolton)
- Thinking fast & slow – Daniel Kahneman
- You’re not so smart – David McRaney
- The invisible Gorilla – Christopher Chabris & Daniel Simons
- Qualitative Data Analysis: a user-friendly guide for social scientists – Ian Dey
- Critical Thinking: a user’s manual – Debra Jackson & Paul Newberry
- Thinking in systems: a primer – Donella Meadows
- Are testers ethnographic researchers? – John Stevenson
- What You Believe Might Not Be True (Part 1) – John Stevenson
- Danger: confirmation bias – John Stevenson
- Combining Qualitative & Quantitative Research – Laura Klein
- Research Methods: Participant observation
- Qualitative Evaluation checklist – Michael Quinn Patton
- Critical & creative thinking
- Software Testing a Social Science – Cem Kaner
- Testing Through The Qualitative Lens – Michael Bolton
- Curing Our Binary Disease – Rikard Edgren
- But how many test cases? – James Christie
- Reponse to How many test cases by James Christie – John Stevenson
- The human element – John Stevenson
- What we can learn from Russel L. Ackhoff – Aleksis Tulonen
- Predictably Irrational: The Hidden Forces that Shape Our Decisions – Dan Ariel
- The Upside of Irrationality: The Unexpected Benefits of Defying Logic at Work and at Home – Dan Ariel
Social science: three presentations
Social science is about society, human nature and human interaction. It is an umbrella term to refer to sciences like anthropology, economics, education, linguistics, communication studies, sociology and psychology.
Anthropology teaches us about how people life, interact and something about culture. Education and didactic helps acquire new or modifying existing knowledge, behaviours, skills, values, or preferences. It helps us understand how we learn and how we can teach others. Sociology teaches us empirical investigation and critical analysis and gives insight in human social activity. Psychology is the study of the mind and behaviour and helps testers understand individuals and groups. Now how is this useful in testing? I’ll try to answer that question later. Let me first tell you about three awesome presentations on the subject of social science and testing.
Testing as a social science
Cem Kaner did a talk titled “Testing as a social science first” time in 2006 (slides are here). I haven’t had the pleasure to see the talk myself but the slides drew my interest. Cem made me aware that to test effectively, our theories of error have to be theories about the mistakes people make and when and why they make them. We design and run tests in order to gain useful information about the product’s quality.
Testing is always a search for information. Cem talks about measurement and metrics and the dangers of using metrics wrongly to measure test completeness (new updated article on this can be found here). He argues that bad models are counter productive. Cem also touched the topic of inattentional blindness in which humans often don’t see what they don’t pay attention to. He reminded us that programs never see what they haven’t been told to pay attention to. This is especially valuable when thinking about test automation. When testing we can’t pay attention to all the conditions. The systems under test are simply to complex and there are to many factors that are variable (and uncontrollable). He concludes that thinking in terms of human issues leads us into interesting questions:
- What tests we are running and why?
- What risks are we anticipating and how?
- Why are these risks important?
- What we can do to help our clients gather the information they need?
At EuroStar 2012 in Amsterdam I track chaired two excellent talks, which inspired me to study the subject of social science and qualitative research more.
Curing Our Binary Disease
Rikard Edgren talked about the getting cured from the Binary Disease (slides are here, video is here). The binary disease is when testers don’t provide useful information, because they aren’t allowed by (project) managers. They demand counting passes & fails and insist everything must be verifiable. The binary disease limits our thinking. Testers are addicted to counting passes and fails and don‘t communicate what is most important. When addicted there is no attention to serendipity moments. A model can help testers find important things, but a percentage number might not include things that are important. Therefore a coverage model is useful to get ideas but is not useful as a metric of completion. In his talk he introduces the testing potato to show that there are more things important besides written requirements. More about the potato can be found in his fabulous must read free eBook “The Little Black Book on Test Design”.
Testing Through The Qualitative Lens
Michael Bolton’s (slides of the StarEast version are here) talk elaborated differences between physical and social sciences. In physics, humans are ideally irrelevant and mostly get in the way of the experiment. Use quantitative and qualitative research methods and accept high tolerance for ambiguity, context-specific results and be aware of biases while doing research. We should value “partial answers that might be useful”. You do qualitative research when you want to understand something. You do quantitative research to inform that understanding. Quantitative research put human values first; use participant observation and practice storytelling and narration. Software testing is the investigation of systems composed of people, computer programs, products, and the relationships between them. Excellent testing is more like anthropology: interdisciplinary, systems-focused, investigative, and uses storytelling.
To be continued… part 5: So what can we learn from social science?
People are predictably irrational
You think you are rational, but you are not. People fail to realize the irrationality of their actions and believe they are acting perfectly rational, possibly due to flaws in their reasoning. People’s actual interests differ from what they believe to be their interests. We have mechanisms that have evolved to give optimal behaviour in normal conditions lead to irrational behaviour in abnormal conditions. Many people put on one “mask” for one group of people and another for a different group of people. Many will become confused as to which they really are or which they wish to become (source: wikipedia). The subject of irrational behaviour is huge. I recommend you to read more about it. We can predict irrational behaviour to a degree due to lots of studies and work done in this field.
John gave me two book tips by Dan Ariel on this topic that I haven’t checked myself yet:
- Predictably Irrational: The Hidden Forces that Shape Our Decisions
- The Upside of Irrationality: The Unexpected Benefits of Defying Logic at Work and at Home
Or check this website also by Dan Ariel: Predictably Irrational – Investigating the Hidden Forces that Shape Our Decisions
You are not so smart
A great collection of examples that show people are easily fooled can be found in the book called “You are not so smart” by David McRaney. This book is a dose of psychology research served in tasty anecdotes that will make you better understand both yourself and others. The author describes cognitive biases, logical fallacies and heuristics. For example there is the well known “confirmation bias” where you tend to look for information that confirms your beliefs and ignore the information that challenges them. Another interesting phenomenon is the availability heuristic: a mental short cut that occurs when people make judgements about the probability of events by how easy it is to think of examples. The availability heuristic operates on the notion that, “if you can think of it, it must be important.” Examples are lotteries where you only see the winners so you might think it is easy to win. Or school shootings in the USA. People believe that since Columbine there are more and more school shootings but the opposite is true! Before Columbine there where more, but we don’t know about them. After reading this book an interesting thinking exercise can be to recognize the biases and fallacies in your thinking and testing.
Thinking fast and slow
Daniel Kahneman wrote a fascinating book about how our brain works “Thinking, fast and slow” which has been a bestseller for some time now. This book changed the way I think about thinking. Although it was sometimes hard to read for my as a non native English speaker, I almost read the book in one go. The book is about two different ways our brain works: System 1 is fast, instinctive and emotional. System 2 is slower, more deliberative, and more logical. I encourage testers to read the book and watch this video. In the video the author explains the main points from his book. You also might want to have a look at this shorter video where the same stuff is made more visual. The book will help you understand how your brain works and it will also make you aware how people make judgements and come to conclusions. Read what software tester Andy Patterson writes about on his thoughts of the book here.
There is a great video with Daniel Kahneman and Nassim Taleb (The Black Swan) in which they talk at the New York Public library about how individuals and humans make decisions – a fascinating video to watch – details and access to video can be found here.
An interesting source the read to learn more about inattentional blindness and other illusions of memory and knowledge is the book “the invisible Gorilla” by Christopher Chabris and Daniel Simons. It makes you aware of how you can be fooled by your illusions and perception. More reading on gorillas and inattentional blindness is this article. Alan Page loves the gorilla! Especially the video. Check what he has to say about the gorilla here.
To be continued… part 4: social science
Testers need to do a lot of thinking. To me testing is an investigation, gathering and providing information about things that are important. I like the definition by Jerry Weinberg: “testing is gathering information with the intention of informing a decision”. Rikard Edgren recently wrote an excellent “open letter” to define testing. Testing is much more than finding bugs or checking if requirements are met.
We should not only investigate the “system under test” but also take related products in mind. What about the people using all these products or the organisations and processes in which the products are used? Testers should know more about systems thinking: the process of understanding how things, regarded as systems, influence one another within a whole (source: wikipedia).
A system is not just a collection of things. A system is an interconnected set of elements that is coherently organized in a way that achieves something. It must consist of three things; elements, interconnection and a function or purpose (source: “thinking in systems: a primer – Donnella Meadows”). If you want to learn more about systems thinking, you might want to watch this youtube movie by Russel Ackhoff and read this post by Aleksis Tulonen about what you can learn form Ackhoff.
In one of my projects my client was moving a hospital from several old locations to a huge new building. It was logical that the location codes changed since it was a new building with a very different layout. But initially they forgot to oversee that this location code was actually used as department code in several information systems. And these systems used the codes to book costs (finance), plan staff (HR) and distribution of food and medicine (logistics). Moving to the new building without overseeing the full impact would have paralysed the whole information landscape. Defining a temporary coding and making minor changes to several systems solved the problem.
Testing can be seen as a form of research: investigating the system and finding information about it. In research critical thinking is important. Collecting, analysing and interpreting information requires critical thinking skills. Critical thinking to me is about thinking (critically) about your own personal thinking. Framing your own assumptions and using this to try to remove bias and hopefully clarifying your thoughts with reasoning.
In this video James Bach helps to gain quick understanding of critical thinking by asking three simple questions:
- Huh? What does this mean? What is the point?
- Really? Are you absolutely certain? How can I know?
- So? Where does this lead? So what?
These questions are very helpful for understanding and to think critical about anything. This picture (click to enlarge) is taken from the book “Critical Thinking: a user’s manual” by Debra Jackson and Paul Newberry. This book is a helpful source to learn about critical thinking.
Rule of Three
“If I can’t think of at least three different interpretations of what
I received, I haven’t thought enough about what it might mean.”
At EuroStar in Amsterdam I met John Stevenson who has an excellent blog with the intriguing title: “The expected result was 42. Now what was the test?”. We talked about what testing can learn from social sciences and early this year we had some fantastic conversations via skype. John pointed me to some very interesting readings about qualitative research: “Qualitative Data Analysis: a user-friendly guide for social scientists“ by Ian Dey. On his blog he wrote some very interesting posts related to testing and social science you might want to read:
- Are testers ethnographic researchers?
- What You Believe Might Not Be True (Part 1)
- Danger: confirmation bias
John is currently writing an awesome series of articles about creative and critical thinking. Part 1 of “Creative and Critical Thinking and Testing” can be found here. From there you can find the other parts about the different styles of thinking.
So why is this important?
Systems thinking reminds us to look at the big picture and see systems as a whole. What is the purpose of the organisation we work for? And is the project we are doing contributing to that? Creative thinking helps us to solve problems in a creative way or come up with more things to test and how to do it effectively. Critical thinking helps you to really understand what you are doing. Like in research we have to process large amounts of data and make sense of it. But we also have to recognize, analyse and evaluate (see critical thinking diagram above) information, arguments and problems.
Thinking is an under appreciated subject. Thinking is very important for testers and we should learn from science: doing research, learn to design and perform experiments, collect, organize and analyse data and use the results to decide on the next steps in our work. Critical thinking helps us ask better questions in our projects and identify problems faster. It also helps avoid traps: biases and assumptions. More about that in my next post.
To be continued… part 3: irrationality and biases
Last February and March I have had the privilege to talk at Belgium Testing Days in Brussel and TestBash in Brighton about what testing can learn from social science. In a series of daily blog posts I am going to write about this subject: why I choose this topic, what sources I studied and finally how I have applied this stuff to my work.
Rapid Software Testing
In the Rapid Software Testing Class I took in 2011 Michael Bolton talked about being empirical and a critical thinker as a tester. About collecting data from experiments using a heuristic and exploratory approach. About reporting by telling stories in testing instead of only reporting figures and numbers. Testing is about providing valuable information to inform management decisions. This awesome class empowered me to connect the dots of stuff I had been thinking about for years. It also pointed me towards a lot of books and information “outside” the IT and testing domain. It also triggered me to learn more about social science.
Do you recognize test reports like this? (click the report the enlarge).
I used to write test reports like that. I was counting test cases and issues and advising my clients to take applications in production. But what do these numbers tell us? What if we didn’t test the most important functionality is the software? Numbers don’t mean anything without context!
Another example was an assignment I did at a telecom company years ago. Testing was estimated by numbers of test cases. We have 8 weeks to test so we can do 800 test cases, was a normal way to plan and estimate testing. Somewhere along the project my project manager told me his budget had been cut 10%. He asked me to drop 80 test cases from our 800 test cases scope. What was he thinking? As if all test cases take equally long to create, execute and report?
Exact or social science?
Testing and informatics (the science of information) are often seen as exact or physical science. People perceive that computers always do exactly the same. This gets reflected in the way they think about testing: a bunch of repeatable steps to see if the program is working and the requirements are met, but is that really what testing is all about? I like to think of testing more as a social science. Testing is not only about technical computer stuff, it is also about human aspects and social interaction.
Traditionally the focus in testing is on technical and analytical skills, however testing requires a lot more! Testing is also about communication, human behaviour, collaboration, culture, social interaction and (critical, creative and systems) thinking. The seven basic principles of the Context-Driven School tell us that people, working together, are the most important part of any project’s context. That good software testing is a challenging intellectual process and only through judgement and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.
Can we measure the quality of software? And can we do that objectively? When I ask people about quality they often refer to requirements. “Quality is compliance to functional and non-functional requirements”. In my experience I have never seen a document that contained all requirements for a software product. We can argue that requirement engineers have to do a better job. Are they doing a bad job? Can we solve the problem by writing better requirements? When discussing quality I like to use coffee as example. I like strong, black coffee without any sugar or milk. But what if you do not like coffee? For somebody who doesn’t drink coffee, my cup of coffee has no value at all. But is still the same cup. How can that be? And how about the taste? Why does coffee from an average office machine doesn’t taste very well while it meets the “requirements” I just mentioned. And what if I change my mind? Not so long ago I drank lots of cappuccino, nowadays I don’t like that any more. That is why I like the definition by Jerry Weinberg and the additions made by James Bach and Michael Bolton.
Quality is value to some person (Jerry Weinberg)
Quality is value to some person who matters (James Bach)
Quality is value to some person at some time (Michael Bolton)
I began to believe that there is much more to quality than requirements alone. I also believe that software quality is very subjective and will change over time. To better understand the subjective, human aspects of software quality I started to study social science in general and our thinking and qualitative research in particular.
Qualitative and quantitative research
Quantitative research is about quantities and numbers. The results are based on numeric analysis and statistics. There is nothing wrong with numbers, but we need to understand the story behind these numbers! Like the test report example: what is the story behind these numbers? What did we test? And how good was our testing? That is where the qualitative aspects come in. Qualitative research is focused on differences in quality and is usually for more exploratory purposes. It is more open to different interpretations. Qualitative research accepts and deals with ambiguity, situational specific results and partial answers. When doing this, testers may be more prone to bias and personal subjectivity.
Shmuel Gershon’s tweet pointed me to an article on the scrum alliance website Agile Methodology Is Not All About Exploratory Testing by Dele Oluwole. I share Sigge Birgissons concerns: “the post clearly shows what I mean when having deep concerns about the knowledge of testing in agile community”.
I think Dele doesn’t fully understand what testing is or at least he uses a different definition than I do. And certainly he doesn’t understand exploratory testing. Rikard Edgren wrote an open letter to explain what testing is. Please read his high level summary of software testing to understand what testing means to me.
"It is imperative to state in clear terms why Agile testing
cannot be all about exploratory testing"
(The text in these gray frames are quotes form the article by Dele)
I wrote a post some weeks ago about agile testing titled what makes agile testing different: agile testing isn’t that much different from “other” testing. Why do some people think agile is so different? To me there is no such thing as Agile testing. There is testing in an agile context. And every agile context is probably different. So saying that agile testing cannot be about exploratory testing makes no sense to me.
It is unequivocally the case that: you cannot estimate your time for
exploratory testing, i.e., assign points realistically
Estimating testing is an interesting topic. This blogpost by Morgan Ahlström nicely emphasizes that estimates are guesses. Martin Jansson of the Test Eye writes about “utopic estimations” here. Michael Bolton wrote a lot about estimation here, here and here. He explains that testing is an open-ended task which depends on the quality of the products under test. The decision to ship the product (which includes a decision to stop testing) is a business decision, not a technical one.
Especially the fifth part of the black swan series is interesting in this discussion because Michael writes about the fallacies surrounding “development” and “testing” phases (by James Bach). He also explains why estimating the duration of a “testing project” by counting “test cases” or “test steps” is not a smart thing to do.
"You cannot plan for exploratory testing, as you do not have defined
Why are some people so obsessed by expected results? And why is there a need to have expected results to be able to plan testing? Expected results can be very helpful, but there is much more to quality then doing some tests with an expected result. A definition that I like is by Jerry Weinberg: “Quality is value to some person”. To understand this, you might want to read this blogpost to understand that there is more to quality than the absence of bugs. Also have a look at the excellent free ebook “The Little Black Book on Test Design” by Rikard Edgren. On page 2-6 he uses the “testing potato” to explain that there are more important aspects to the system than the requirements only.
"There is no defined scope for exploratory testing."
In the exploratory testing I do in my projects there is a “scope”. I do targeted and focused testing using charters, sessions and planning my testing using a dashboard that resembles a scrum board. Have a look at the slides of my presentation “Boosting the testing power with exploration“.
Using a coverage outline in a mind map or a simple spreadsheet, I keep track of what I have tested. My charters (a one to three sentence mission for a testing session) help me focus, my wrap-up and/or debriefings help me determine how good my testing was. My notes and the sessions sheets keep track of what I have done in my testing. Like in scrum, I use “standup” meetings to plan my testing. In these meetings we discuss progress, risks, priorities and charters to be executed. This helps us to make sure we do the best testing we can do continuously.
"The tester, product owner, and Scrum team are not in control."
I am not sure if I understand what you are trying to point out here. Are you saying that the team is not in control when doing exploratory testing? My model above shows that you can be in control when doing exploratory testing when done right. Exploratory testing is NOT ad hoc or unstructured when done right. If you do it right you will have control!
"There is no measure of progress, as testers cannot determine when
testing is enough."
How do we determine how much testing is enough? Stopping heuristics might help here. No matter how simple the system is we are building, there are simply to many variables to test everything. So testing is about making choices what to test and what not to test. Even with a huge amount of automated checks, we can not check/test everything (to understand the difference between testing and checking read this). Testing is not about doing X test cases and when they all pass, you are done. Testing is providing information for managers to make good decisions. And when do managers have enough information to inform their decisions?
Still, not many Agile projects will require just two phases, like
integration and regression. But it's definitely not only exploratory
testing that's needed, as is erroneously believed in some quarters.
I am not sure what you mean by two phases. What do you mean by testing in phases? I like to use the agile testing quadrants when I try to explain how I think of testing in an agile context.
A team is developing software and the programmers do testing before checking the software in and making it available to the team. How do we call that sort of testing? Unit testing? But is that /only/ testing done by the programmer? I argue that the programmer might do all kinds of testing before checking his code in, even functional and acceptance tests. They probably will create a lot of automated checks and maybe even do some exploratory testing to see if the software meets their expectations: quickly testing some usability and performance aspects. So before the software is checked in, the programmer has covered testing in all 4 quadrants. This does not mean testing is done. More testing can be done, it depends on the context. What does the product owner wants to know? What are the risks involved? How much time do we have left? When discussing a test strategy I try not to speak about phases, I like to discuss what gets covered and why. What information is needed by the team and it’s stakeholders? When talking about coverage I do not mean code coverage but test coverage: the extent to which we have traveled over some map (model) of the system.
So I don’t think you can say “only exploratory testing is needed or not”.
Dele concludes his article with this statement:
It is the responsibility of the tester (and the Agile/Scrum team) to
ensure that acceptance testing is in line with the expectation of
the product owner. If we agree that there is an expectation, we
therefore have to design test cases (even if minimal) that will verify
the specified acceptance criteria."
Dear Dele please read about testing and exploratory testing. Some good starting points are these lists of resources on this blog or the one made by Michael Bolton: resources on Exploratory Testing, Metrics, and Other Stuff. I am happy to point you to more good sources of information if you are interested. Just let me know.
At EuroStar 2012 in Amsterdam, Henrik Emilsson did a talk about the Software Quality Characteristics poster made by The Test Eye. After his talk he asked if someone was interested in translating this poster into other languages. Me and my DEWT colleagues happily picked up this gaunlet and we proudly present the Dutch translation: Software Kwaliteit Kenmerken. I use this checklist often when preparing my testing. Now available in the Dutch language!
Why use checklists?
The modern world has given us stupendous know-how. Yet avoidable failures continue to plague us in health care, government, the law, the financial industry—in almost every realm of organized activity. And the reason is simple: the volume and complexity of knowledge today has exceeded our ability as individuals to properly deliver it to people—consistently, correctly, safely. We train longer, specialize more, use ever-advancing technologies, and still we fail. Atul Gawande makes a compelling argument that we can do better, using the simplest of methods: the checklist (source: Amazon.com)
Last week Pete Walen asked me the following question via twitter: The Question (read it carefully!): What is it that makes Agile Testing different from “other” testing?
Agile vs agile?
What is agile testing? And what is agile? Let me first say that I do not distinguish agile and Agile. For me it is all the same. Agile is a mindset, a way of looking at the world. For me it is not a process or method. It is more a container than a way of working. This blogpost discusses agile vs Agile and I like it. It describes agile as a mindset and Agile with a capital A as something commercial: “The problem here is that all of these sensible suggestions got formalized into “Agile”, with a Capital A. This set of suggestions for “things that seem to work” got boxed up into a package with a bow on top, to be sold to companies and managers.”
And agile testing? What is agile testing? I rather say testing in an agile context instead of agile testing. Good testing in an agile context is done by looking first to the details of the specific situation. Remember the 7th principle of context-driven testing: “Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.”
- by the iterative nature of working, there is less time to test compared to the testing most of us are used to in a more traditional (waterfall) context. It requires a different approach to testing. There is a different phasing to testing in an agile environment.
- how can I make sure that I can perform sufficient testing fast enough to keep up with the project?
- testers need to ensure that “self-managing teams” do enough testing
- cope with the changing team dynamics in which people work and where interaction is important
- integrating structured testing in an environment where change is common
- deliver added value as a tester when there is no software to test
So what is the difference? I think the testing itself is not so much different, it is the context in which you do the testing is different. If you are aiming on differences between waterfall and agile my list of most important differences would be:
- less time to prepare, execute and report (short sprints).
- iterative and incremental approach: excellent unit testing is essential.
- test automation (some rather call it automated checking or tool assisted testing) is essential for fast feedback and continuous integration.
- role change: less testing, more coaching. Testers become “test coaches” or “quality directors” to make sure the team is doing sufficient testing. Enough (not too much nor too little) and of good quality.
- cope with less certainty: change is common. Test documentation needs to deal with change by being transparent, using simple dashboards and light weight test documentation.
- team work: where many testers are used to work in TEST teams, they are now working in DEVELOPMENT teams.
- continuous critical thinking: testers need to help the team by thinking critical about the impact and risks. Where testers were used to do that upfront while writing documents like master test plans, they now have to do that continuously throughout the project: in grooming sessions, daily standups, planning sessions, etc. But also in their own work: making choices about what to cover: broad and depth.
Again: the testing itself is not so much different, it is the context in which you do the testing is different!