Yesterday I had diner with James Bach, Pascal Dufour and my testing buddy Jean-Paul. After picking James up from Schiphol airport, we took him to his hotel and after that we had diner. An enjoyable evening with great conversations and some testing challenges.
Pascal went home with this challenge: “Apply the Constructal Law to improve testing.” This challenge James gave himself since he is reading this book. Jean-Paul asked for a challenge via twitter and got this one: “Explain dendogram-based testing”. Since I start with BBST tomorrow, I didn’t ask for one.
We discussed a lot of topics, one of them was coaching. In July I go to Test Coach Camp in San Jose the weekend before CAST 2012. I told James I had an idea for a session at Test Coach Camp: “What they teach us in TMap Class and why it is wrong”. I think he likes it 😉 I asked James what he would do if he had to train newbies to be testers. He answered that he would have them test something. He certainly wouldn’t start with lectures about processes, definitions and testing vocabulary. During the testing session the aspirant-tester will do stuff that can be named. They will (ignorant) use all kinds of techniques that can be further explained and explored while they pop-up.
James also gave an interesting insight: the way testing is often trained and implemented results in dull testing jobs where most of the great people (who could be excellent testers) will run away from. I have to chew on that a little more, but I think this might be true. It definitely rings a bell.
Today I created a first draft for the proposal for Test Coach Camp:
“What they teach us in TMap Class and why it is wrong”
In my former job I have been an instructor/trainer for ISTQB and TMap classes. This was before I saw the Context-Driven light 😉 I trained ISTQB and TMap to people new to testing and also lots of experienced testers. I also was an instructor for all kinds of other training courses. These training courses are not focussing on the right things and are not preparing testers to do proper testing. Let alone that they encourage testers to become excellent their profession. For example: in these 3 day courses, all exercises are done on paper without actually testing software. This makes me wonder if there are classes where developers are trained without actually coding.
In this session I would like to point out the structure and content of a typical TMap (or ISTQB) class and the method/approach used. In the discussion later in the session the group collaborates in listing what is wrong with this approach and what can be done to improve it. This creates an insight in what is wrong with the majority of training courses in my country (and probably also around Europe and the rest of the world). This insight can help us create training courses to properly train people in testing, but also improve existing classes.
Let me know what you think and please share your experiences.
Thank you very much for this topic – where I have been struggling for moths to accept TMAP methodology where I find a sheer drawback according to my past working experience. And I am really stunt to see how this TMAP getting accepted as industry standard. More and more I try to understand TMAP – more I find its incompatibility in covering long term product development. It may somehow manage to give a little advantage over the short term small projects but over the long term projects or product development I rather see some flaws in their basic structures – risk analysis and testing strategy based on that. I am still trying to get more readings on TMAP – so before coming to a conclusive decision I have a clear view.
I am glad to see so many people take their time to explain testing to Peter Edmond. It’s obvious he doesn’t know what structure means nor has a slight clue how exploratory testing can be done. He didn’t learn these things in 40 years working on the field, yet, you guys invest your time to help him understand. I feel lucky to be surrounded by such people. Big thanks!
One thing that strikes me in your comment is that you say that the way of testing that Huib speaks for has no structure.
What think you mean is that YOU see no structure there. That does not mean that there IS no structure.
There are so many different kinds of structure. For instance visible and outer structure, that is the kind that you can touch and see. Then we have the inner and mental structure this is what going on inside of us.
If someone do have lots of outer structure but less inner structure it can at a first glance appear that this fellow has things in control and are very advanced. Likewise if one have lots of inner structure and less outer then at first glance it can appear that this fellow has lost control and are not knowing what he/she is doing.
Both of this can of course be correct but it can equally well fool you to believe something that is very wrong.
Before you claim that something has no structure you better have looked very careful and done tons of investigation of the activity. Most likely you will find structure that you did not see before.
You have the privilege to claim that ISTQB or TMap provides the structure for you to be able to do something and that Huibs way of structure is something that you do not appreciate or are not able to work in. We humans work in different ways. But do not condemn it to be unstructured.
You see, Huib recognize the structure in your preferred methods and maybe you should start be recognizing other ways of structure. Then we can discuss the differences that might lead us both forward.
Like your post and agree with its general content. But my comment is not so much a reaction to your post as it a reachtion to the previous comment by Peter Edmond. It saddens me that there are still people that believe that if you do not follow ISTQB or TMap you can only be doing unstructured and uncoordinated testing. Well Edmond the testers that Huib is aiming for do testing in a structured way and they do coordinate their efforts and when they stop with the stakeholder. Its just so that they do that in such a way that it is relevant to the stakeholders needs both with regards to the information each of them needs to take decisions and with regard to the risks they need to have covered.
The idea of seeking collaboration and coordination seems however to be so far fetched for ISTQB and TMap world that the true ISTQB and TMap fanatics just can’t imagine that any testing can be done without following their ‘best practice’ process. A process that by design and by teaching ignores the actual situation at hand, ignores the realistic behaviour of other roles and only delivers predifined process related deliverables that by their nature only deliver second order information about the product let alone about the products quality or usefulness.
So Peter your 40 years of experience doesn’t mean anything to me as you seem not have spent it thinking about what you were doing and for whom and to what end.
I have been both a tester and trainer in ISTQB and TMap. Yes we can make testing fun but without a method that testing has no structure and more importantly has no measureable completion. How will those new people on “more practical” course know when they have finished? What tests did they do? What did they forget? What defect types did they target? Which ones did they not look for? What is the risk to the system? My view after 40 years as a developer and tester is that this idea might be fun but is not just WRONG but so dangerously wrong that I am sad that no one else has seen it.
With all due respect, but you fall into the positivist illusion of complete controllability. Now, whatever you do measure, it is a surrogate measurement and may or may not give you the information you are looking for. How do you know when you are finished if your field is infinite. Come on, your use of the term “measurable completion” is kind of cute. It is your approach to testing that is dangerous, my friend, and it is the goal of us context-driven testers to stop people like you telling fairy tales about testing.
On a side note: the opposite of fun is not professionalism, but depression. So, mind your words.
Hi Peter, I have been both a tester and trainer in ISTQB and TMap too. Good that we agree testing should be more fun. And as Ilari says: the opposite of fun is not professionalism.
I understand it is hard to know when your are done. Can you explain what you mean by “measurable completion”? But do you know when you are done with a (what you call) structured method like TMap or ISTQB? You know when all the test you came up with upfront are complete. But how do you know that you have found all the important bugs? Also: please read this post by Michael Bolton or this post by Markus Gärtner about structured and exploratory testing.
Let me try to answer your questions:
Q: How will those new people on “more practical” course know when they have finished?
A: Good question. Let me compare the learning I described with the learning from the TMap course. In the TMap course you see 350 slides and do 4 practical exercises using test design techniques. But you do not test real software. So after 3 days you know a lot about definitions, processes, artifacts, but not how to test real software. In my experience a lot of people who did the TMap course had difficulties getting up to speed after the course. When you do the way of learning I described, I think after three days you tested a lot of things and you learned how to actually test something. Sure in both scenarios the student is far from done. Compare it by the way kids learn. Did you send your kids to a 3 day course in walking school? No, they learned by trying, failing, trying again (and a lot of exploring).
Q: What tests did they do?
A: Using charters, sessions, logging and debriefs as in SBTM (Session based test management) a tester is very capable of knowing/reporting what they did.
Q: What did they forget?
A: Do you know what you forget in your testing? I hope you do not claim that you don’t forget anything. Because in that case we would have bug free software…
Q: What defect types did they target?
A: It depends. But each session can target a different type. In the testing I have observed a lot of TMap and ISTQB trained people do, it struck me how un-targeted they where testing. I have seen a lot of confirmatory testing, but haven’t seen much targeting of specific defects. I don’t think a method will improve this much. Practice and thinking while testing will. My observation is that the more testers use “structured” methods like TMap, the less they think.
Q: Which ones did they not look for?
A: Same answer as before: do you know which you didn’t look for? And is this method specific or just because you think?
Q: What is the risk to the system?
A: Who says ET doesn’t do risk based testing? I would claim ET is highly risk based. ET tries to find the most important bugs first. Those are the highest risk, aren’t they? So I do not see why this would be a problem.
The questions you ask, give me the feeling that you do not know what exploratory testing is and how it is practiced well. ISTQB and TMap still consider ET as a technique. I think there is much more to it and I would describe it as a mindset and an approach. I invite you to read some of the stuff collected on the DEWT website.
Let me add a last thing to this: me and my team had the pleasure to do exploratory sessions with Michael Bolton. We explored exploratory testing. While trying stuff we were discovering how it works. Every now and then Michael would do a short lecture to explain or illustrate some things. I feel we learned more because we weren’t told, we (in a way of speaking) found it out ourselves. As Confucius said: “Tell me and I forget. Show me and I remember. Let me do and I understand.”
Thats great topic…
You know ISTQB then you can easily understand the drawbacks of it.
I would encourage you to present this topic so that it would help audience to decide what is good option for better software career 🙂
I completely understand what you mean! We are improving the courses with some more exercises, but most of it is still on paper. We are trying to get some actual testing in the courses, but achieving this is harder than it seems. The courses that actually have some practical exercises are the more technical oriented ones. This post gives me the chance to highlight Certified Agile Tester once more. This course contains a lot of theory on Agile, Scrum and testing, but also deals with practical exercises to help understanding and implementing the theory. I still agree with you that the certification part is less interesting, but the course is a great step in the direction you are suggesting in this post.
I am looking forward to our brainstorm meeting on the ‘dogma’s in Testing. Already dit some homework during the testing retreat. I find it very interesting to see what believes we ‘quote’ from the books and each other, but that are actually out-dated, not true or wrong.
And yes some bugs are found on paper, many while working with the system. Just recently got a question. My tester had a training, but now needs a really hand-on training. I suggested on site mentoring. No case can include the context.
You’re proposed session will be in line. Maybe pointing out what is wrong is the necessary first step to take. But I would focus on the what is right. It is not about T-map, it is about helping to get better testers
That is a great topic for Test Coach Camp and I am very much looking forward to hearing more about it and discuss it there in the group. You are right, software testing is not done on paper. Or more accurate: not ONLY done on paper. It resonates with so many things: The 10’000 hours practice model, the Dreyfus model of becoming an expert, tacit and explicit knowledge, etc. Cool idea, go for it! Very much fits into Test Coach Camp
As soon as someone states the obvious — it is.
It’s amazing how testing (as well as other topics) is taught like that (I know I’m exaggerating here, though not as much as you might think): One person talking, maybe presenting text laden slides, handing out binders with those slides printed (likely in a way that doesn’t leave room of comments etc.), and now & then answering questions and/or paper based exercises (could be multiple choice).
At some point something went wrong, I think. Just imagine musicians, surgeons or car drivers being trained like that: Without actually practicing… You’d hardly dare to leave the house (unless the craftsmen who built it were trained like that as well) & listen to music.
Let me finally add it’s not only about improving existing classes, but pretty much every learning experience, including but not limited to conferences, online & real life courses.