What Testers Can Learn From Social Sciences
This is the video of my talk at TestBash 2.0 in Brighton March 2013.
Shmuel Gershon’s tweet pointed me to an article on the scrum alliance website Agile Methodology Is Not All About Exploratory Testing by Dele Oluwole. I share Sigge Birgissons concerns: “the post clearly shows what I mean when having deep concerns about the knowledge of testing in agile community”.
I think Dele doesn’t fully understand what testing is or at least he uses a different definition than I do. And certainly he doesn’t understand exploratory testing. Rikard Edgren wrote an open letter to explain what testing is. Please read his high level summary of software testing to understand what testing means to me.
"It is imperative to state in clear terms why Agile testing cannot be all about exploratory testing"
(The text in these gray frames are quotes form the article by Dele)
I wrote a post some weeks ago about agile testing titled what makes agile testing different: agile testing isn’t that much different from “other” testing. Why do some people think agile is so different? To me there is no such thing as Agile testing. There is testing in an agile context. And every agile context is probably different. So saying that agile testing cannot be about exploratory testing makes no sense to me.
It is unequivocally the case that: you cannot estimate your time for exploratory testing, i.e., assign points realistically
Estimating testing is an interesting topic. This blogpost by Morgan Ahlström nicely emphasizes that estimates are guesses. Martin Jansson of the Test Eye writes about “utopic estimations” here. Michael Bolton wrote a lot about estimation here, here and here. He explains that testing is an open-ended task which depends on the quality of the products under test. The decision to ship the product (which includes a decision to stop testing) is a business decision, not a technical one.
Especially the fifth part of the black swan series is interesting in this discussion because Michael writes about the fallacies surrounding “development” and “testing” phases (by James Bach). He also explains why estimating the duration of a “testing project” by counting “test cases” or “test steps” is not a smart thing to do.
"You cannot plan for exploratory testing, as you do not have defined expected results."
Why are some people so obsessed by expected results? And why is there a need to have expected results to be able to plan testing? Expected results can be very helpful, but there is much more to quality then doing some tests with an expected result. A definition that I like is by Jerry Weinberg: “Quality is value to some person”. To understand this, you might want to read this blogpost to understand that there is more to quality than the absence of bugs. Also have a look at the excellent free ebook “The Little Black Book on Test Design” by Rikard Edgren. On page 2-6 he uses the “testing potato” to explain that there are more important aspects to the system than the requirements only.
"There is no defined scope for exploratory testing."
In the exploratory testing I do in my projects there is a “scope”. I do targeted and focused testing using charters, sessions and planning my testing using a dashboard that resembles a scrum board. Have a look at the slides of my presentation “Boosting the testing power with exploration“.
Using a coverage outline in a mind map or a simple spreadsheet, I keep track of what I have tested. My charters (a one to three sentence mission for a testing session) help me focus, my wrap-up and/or debriefings help me determine how good my testing was. My notes and the sessions sheets keep track of what I have done in my testing. Like in scrum, I use “standup” meetings to plan my testing. In these meetings we discuss progress, risks, priorities and charters to be executed. This helps us to make sure we do the best testing we can do continuously.
"The tester, product owner, and Scrum team are not in control."
I am not sure if I understand what you are trying to point out here. Are you saying that the team is not in control when doing exploratory testing? My model above shows that you can be in control when doing exploratory testing when done right. Exploratory testing is NOT ad hoc or unstructured when done right. If you do it right you will have control!
"There is no measure of progress, as testers cannot determine when testing is enough."
How do we determine how much testing is enough? Stopping heuristics might help here. No matter how simple the system is we are building, there are simply to many variables to test everything. So testing is about making choices what to test and what not to test. Even with a huge amount of automated checks, we can not check/test everything (to understand the difference between testing and checking read this). Testing is not about doing X test cases and when they all pass, you are done. Testing is providing information for managers to make good decisions. And when do managers have enough information to inform their decisions?
Still, not many Agile projects will require just two phases, like integration and regression. But it's definitely not only exploratory testing that's needed, as is erroneously believed in some quarters.
I am not sure what you mean by two phases. What do you mean by testing in phases? I like to use the agile testing quadrants when I try to explain how I think of testing in an agile context.
A team is developing software and the programmers do testing before checking the software in and making it available to the team. How do we call that sort of testing? Unit testing? But is that /only/ testing done by the programmer? I argue that the programmer might do all kinds of testing before checking his code in, even functional and acceptance tests. They probably will create a lot of automated checks and maybe even do some exploratory testing to see if the software meets their expectations: quickly testing some usability and performance aspects. So before the software is checked in, the programmer has covered testing in all 4 quadrants. This does not mean testing is done. More testing can be done, it depends on the context. What does the product owner wants to know? What are the risks involved? How much time do we have left? When discussing a test strategy I try not to speak about phases, I like to discuss what gets covered and why. What information is needed by the team and it’s stakeholders? When talking about coverage I do not mean code coverage but test coverage: the extent to which we have traveled over some map (model) of the system.
So I don’t think you can say “only exploratory testing is needed or not”.
Dele concludes his article with this statement:
It is the responsibility of the tester (and the Agile/Scrum team) to ensure that acceptance testing is in line with the expectation of the product owner. If we agree that there is an expectation, we therefore have to design test cases (even if minimal) that will verify the specified acceptance criteria."
Dear Dele please read about testing and exploratory testing. Some good starting points are these lists of resources on this blog or the one made by Michael Bolton: resources on Exploratory Testing, Metrics, and Other Stuff. I am happy to point you to more good sources of information if you are interested. Just let me know.
Last Friday (October 5th) a bunch of software testing nerds and one agile girl gathered in Hotel Bergse Bossen in Driebergen talk about software testing with the central theme “experience reports: implementing context-driven testing”. Ruud published almost all the photos I took on the DEWT website, so I have to write a report in text here. Thanks dude 😉
After some drinks and having dinner we gathered in the conference room called the chalet. I think they call it the chalet because of the humid smell inside because it certainly didn’t look like a chalet.
But never mind, the room was good enough to do a peer conference. Lighting talks were on the program for Friday and Jean-Paul started with a talk in which he asked the question: “Is the context-driven community elitist?”. Jean-Paul sees a lot of tweets and blogs from people in the context-driven community who seem to look down on the rest of the testing community sending the message “look how great we are!”. Is this effective behaviour given the fact that the context-driven community wants to change the way people test, he asked himself.
Joris had a short and clear answer to the first question: “yes!” (the context-driven community is elitist). We had a long discussion that went everywhere but never close to an answer to question about effectiveness and how it can be done better. Never the less it was a valuable and fun discussion. I will blog about this later this year.
We had a great evening/night with home brew beer “Witte Veer”, Belgium beer brought by Zeger and a bottle of Laphroaig quarter cask brought by Joris. Sitting at the fire place a lot of us stayed up until late…
Saturday morning the last DEWTs and guests arrived. With 23 people in the room we had a great group. Our guest were: Markus Gärtner, Ilari Henrik Aegerter, Tony Bruce, Gerard Drijfhout, Pascal Dufour, Rob van Steenbergen, Derk-Jan de Grood, Joep Schuurkes, Bryan Bakker, Lilian Nijboer and Philip-Jan Bosch. The DEWTS: Adrian Canlon, Ruud Cox, Philip Hoeben, Zeger van Hese, Jeanne Hofmans, Joris Meerts, Ray Oei, Jeroen Rosink, Peter Schrijver, Jean-Paul Varwijk and myself.
Peter was our facilitator and for the first time we used k-cards at DEWT to facilitate the discussion. I really like this method, but Lillian didn’t. She had an interesting discussion on twitter with some DEWTs.
Jean-Paul: I like the LAWST format we are using for #DEWT2
Lilian: @Arborosa i don’t. impersonal and inhibits useful discussion
Markus: @llillian @Arborosa What’s the improvement you’re suggesting? 🙂
Lilian: @mgaertne @Arborosa but i am invited and feel i at least have to try it this way
Jean-Paul: @llillian @mgaertne Invite us to an agile event to learn alternatives
Markus: @Arborosa @llillian Funny thing, I would like to introduce more focused-facilitated sessions quite often in agile discussions using k-cards.
Lilian: @mgaertne @Arborosa different ppl prefer different things 🙂
Ilari kicked of with a presentation about the introduction of context-driven testing at eBay. He had two very interesting ideas: the first is “Monday Morning Suggestion” a short email to his team with tips, tricks or interesting links. The second was that he supports his team in getting better and learning by paying for their books and conference visits. Great stuff!
Second was Markus Gärtner with a story about him coaching a colleague to become a better tester and more context-driven. He talked about his lessons learned coaching where teaching gradually turned into coaching. He also gave some insight in the transpection sessions he did using the socratic method.
The third presentation was about changing people to become more context-driven by Ray Oei. An interesting discussion developed about how to get people to change. We focus too much on the people who do not want to change instead of working with the people who do want to change. Pascal asked an interesting question which I put on twitter: “If all testers become context-driven overnight, are we happy?” James Bach replied: “Yes, we are happy if all testers become CDT” and “We are happy if all testers take seriously the world around them, and how it works, and dump authoritarianism”. To me it doesn’t really matters, I do hope that more people become CDT, but I prefer good software testing no matter how them call it.
After the discussion Joris said he was attending the wrong peer conference since he expected a lot of discussion about testing instead of what he calls “people management”. An interesting question, but isn’t testing also a lot about people? Just an hour late we went for lunch and a walk in a wet forest. After the walk the group photo was taken.
The forth talk was by Ruud Cox who talked about testing in a medical environment. He described the way he has been working. He and his colleague tester use exploratory testing to learn and explore. Scripted testing was used to do the checking. Ruud explained that exploratory testing fits very well in an R&D environment. After his talk we had an interesting discussion about auditors and their role in testing.
Around 16:00 it was my turn to do my talk about implementing context-driven testing within Rabobank. I did a short version of the talk I will do at EuroStar in November telling about the challenges we had and what we did to change our context. What did we do? What worked and what didn’t? Some interesting questions were asked. And we had a nice discussion about the personal side of becoming more context-driven where Joep explained how he became context-driven. The first stage was being interested and he went looking for a book. The second stage took him over a year where he struggled with the stuff he learned at RST and trying to adapt his way of working. The third and last stage is where he doesn’t need anything anymore because he will always find a way to make things work.
After my talk and the discussion the day ended. We did a quick round to discuss our experiences and thanked the people for attending, organizing and facilitating. Our agile girl still doesn’t know what to think about the facilitation with the k-cards, she obviously had to get used to discussing with the cards. I am curious how she thinks about them when she had the chance to think about it some more.
All together I had a great time. Pity we didn’t play the Kanban game on Friday. But I hope Derk-Jan will give me another chance at the game night at TestNet in November. Thanks all for the awesome weekend!!
DEWT3 is already being planned for April 20 and 21 next year and James Bach will attend. Let me know if you are interested in joining us in April 2013. The topic hasn’t been decided yet, but if you have great ideas please share them.
Standing left to right: Ray Oei, Jean-Paul Varwijk, Adrian Canlon, Markus Gärtner, Ruud Cox, Joris Meerts, Pascal Dufour, Philip Hoeben, Gerard Drijfhout, Bryan Bakker, Derk Jan de Grood, Joep Schuurkes, Lilian Nijboer, Philip-Jan Bosch, Jeroen Rosink, Jeanne Hofmans
Kneeling left to right: Tony Bruce, Zeger van Hese, Ilari Henrik Aegerter, Huib Schoots, Peter Simon Schrijver
Last week Rik Marselis send me an email pointing me to an article with the title “The adaptability of a perceptive tester”. He added: “Have you read this article? Should appeal to you!”. The article is written by a couple of Dutch (Sogeti) testers. And they, so the introduction tells me, get together once in a while to discuss the profession of testing and quality assurance. This time they discussed some remarkable examples of their experience that perceptive testers (who are aware of the specific situation they’re in) adapt their approach to fit the specific needs.
I replied to Rik with the following email:
Nice article, I had already seen it. But adaptive or perceptive is not context-driven. I also totally disagree with the conclusion:
TMap Next contains a number of dogmas (or rather is based on a number of dogmas) like: testing can be scheduled, the number of test cases is predictable, content of a test case is predictable, sequence of process, etc.
Rik replied with this email:
We really need to reserve some time to discuss this from different sides because some things that you say I totally disagree with. A conscious tester can handle any situation with TMap. I think whether ET is a technique or approach is really a non-discussion. TMap calls it a technique so you can approach testing in different ways in different situations. And since TMap itself is a method you cannot call ET a method too.
I think Context-driven means you choose your approach depending on the situation.
Perceptive means conscious, as you are aware of the situation, you can choose an appropriate approach. Well, it is worth discussing.
Okay, so let’s discuss!
Let’s start with the ET discussion. What does TMap say about this? ET is a test design technique. And the definition of a test design technique (on page 579 of the book “TMap Next for result driven testing”): “a test design technique is a standard method to derive, from a specific test basis, test cases that realise a specific coverage”. Test cases are described on page 583 of the same book: “a test case must therefore contain all of the ingredients to cause that system behaviour and determine wheter or not is it correct … Therefore the following elements must be recognisable in EVERY test case regardless of the test design technique is used: Initial situaion, actions, predicted result”.
Let’s connect the dots: ET is called a test design technique. A test design technique is defined as: a method used to derive test cases. But ET doesn’t use test cases, not in the way TMap defines them. It can, but most of the time it doesn’t… Mmm, an inconsistency with image, a claim, expectations, the product, statues & standards. I would say: a blinking red zero! Or in other words, there /is/ something wrong here!
What is Exploratory Testing? Paul Carvalho wrote an excellent blog post simply titled “What is Exploratory Testing?” on this topic and I suggest people to read this if they what to understand what ET is. Elisabeth Hendrickson says: “Exploratory Testing is simultaneously learning about the system while designing and executing tests, using feedback from the last test to inform the next.”
Michael Bolton and the context-driven school like to define it as: “a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to optimize the quality of his or her work by treating test design, test execution, test interpretation, and test-related learning as mutually supportive activities that continue in parallel throughout the project.”. Michael has a collection of interesting resources about ET and it can be found here.
So Rik, your argument “since TMap itself is a method you cannot call ET a method too” is total bullshit! It sounds to me as “there is only one God…”.
Don’t get me wrong, being adaptive and perceptive is great, but that doesn’t make testing context-driven. A square is a rectangle but a rectangle is not a square! Please have a look at the context-driven testing website by Cem Kaner. Also read the text of the key note “Context-Driven Testing” Michael Bolton gave last year at CAST 2011. In his story you will see that adaptive (paragraph 4.3.4) is only a part of being context-driven. I admit, it is not easy to really comprehend context-driven testing.
Do you think it was TMap Next that was the common success factor in the stories shared in the article… I doubt it!
On this blog I recently wrote about my meeting with James Bach with the provocative title: “What they teach us in TMap Class and why it is wrong“. Mid July I go to San Jose for the CAST conference. During the weekend preceding I participate in Test Coach Camp. The title of the post is the title of a proposal that I submitted to discuss at Test Coach Camp.
In the past I have been a trainer for quite a few ISTQB and TMAP courses. The groups attending the training were often a mix of inexperienced and experienced testers. The courses cover topics like: the reason for testing, what is testing, the (fundamental) processes, the products that testers create, test levels, test techniques, etc. In these three-day courses all exercises are done on paper. Throughout the whole training not once actual software is tested!? I wonder if courses for developers exist where no single line of code is written.
In San Jose at Test Coach Camp I want to discuss the approach of these courses with my peers. How can we improve them? I feel these courses are not designed to prepare testers to test well. Let alone to encourage testers to become excellent in their craft.
During my dinner with James, I asked him what he would do if he would train novices to become good testers. He replied that he would let them test some software from the start. He would certainly not start with lectures on processes, test definitions and vocabulary. During a session the student will (unknowingly) use several techniques that will be named and can be further explained when stumbled upon. A beautiful exploratory approach I would like to try myself: learning by doing! But there are many more opportunities to improve testing courses. People learn by making mistakes, by trying new things. Testing is much more about skills than about knowledge. Imagine a carpenter doing a basic training. His training will mainly consist of exercises! My neighbour is doing a course to become furniture maker. She is learning the craft by many hours of practice creating work pieces. Practice is the biggest part of her training!
One of the comments on my blog opposed to the suggestion by James Bach. Peter says: “I have been both a tester and trainer in ISTQB and TMap. Yes we can make testing fun but without a method that testing has no structure and more importantly has no measurable completion. How will those new people on “more practical” course know when they have finished? What tests did they do? What did they forget? What defect types did they target? Which ones did they not look for? What is the risk to the system? My view after 40 years as a developer and tester is that this idea might be fun but is not just WRONG but so dangerously wrong that I am sad that no one else has seen it.”
What do you think?