Category: Conference (Page 1 of 3)

Let’s stop talking about testing, let’s start thinking about value

This year Alex Schladebeck and I did two keynotes titled “Let’s stop talking about testing, let’s start thinking about value” at QA Expo in Spain and TestNet in the Netherlands. This blogpost has the most important points we made in our talk.

The keynote was inspired by some of our frustrations: “Testing is under appreciated” (Alex) and “Most testers are unable to explain what we do” (Huib). I wrote about my frustration back in 2016 already. This blogpost is about my frustration that most testers cannot come up with a decent definition of testing. And even worse: a big majority of the people who call themselves professional testers are not able to explain what testing is and how it works! They have trouble explaining what they are testing and why they are doing specifically the thing they are doing! How can anybody take a tester seriously who cannot explain what he is doing all day?

Alex’s frustration is that testing is not valued by others. Developers are seen as the rockstars of the project because they create the software that adds value. But why are testers often not valued?

  • Lowered expectations for testing expertise by stuff like ISO standards and ISTQB: I wrote about certification and standards before. ISTQB and standards put too much emphasis on process and documentation, rather than the real testing. By assuming there can be a standard, you say that there is one best way to organize and document your testing. But isn’t your test strategy heavily dependent on its context? When using standards we tend to focus on complying with the standard, and lose sight of the real goal. This sort of goal displacement is a familiar problem in many situations. Also, the idea that you can learn how to test is a couple of days of training is dangerous. Remember lesson 272: if you can get a black belt in only two weeks, avoid fights (Lessons Learned in Software Testing: A Context-Driven Approach by Bach, Kaner and Pettichord).
  • Avoiding controversy: nowadays more and more people advocate to be nice! I think that we confuse being nice, with being kind! An interesting article about this phenomenon is written by Marcia Sirota. Of course we need to respect other people, but to push the testing craft forward, we need to have firm discussions and disagree with others way more often. Being nice doesn’t help. Serious feedback does!
  • We devalue our own work by becoming tool jockeys: unfortunately there are too many testers (and teams) out there who focus too much on automation as much as possible. Why? Because they can! The testers in those teams are often so busy doing automation that they do not have the time to test anything…
  • We do not stand up for our craft: we do not fight back enough when other people say they do not need testers, or if they tell us how to do our jobs to name a few examples. We have to learn “testers self-defence: to stand up to people who try to dictate how do our jobs. We have to learn how to organize effective (and efficient) testing. And we need to learn how to talk about our work in a way others understand. This requires practice!
  • We do not learn or practice enough: testing is difficult! We have to deal with complexity, ambiguity, change and people. Testing is a craft, not something you do as a hobby. To become a craftsperson, you have to practice (also see my blogpost: a road to awesomeness).
  • We don’t know how to talk about testing: as said before: how can anybody take a tester seriously who cannot explain what he is doing all day? To be really valuable, testers need to learn to talk about their testing in a way others understand and find valuable.

So looking at these things, are we okay with this? I don’t think so. But what can we do about it? We are trapped in this vicious circle: we need to talk about testing! It is good for our soul to explain what I did and why, but we don’t know how to talk about our testing in a way that others understand.

Alex and I listed some traps:

  • Stories decay into Numbers: testing is about providing information to enable others to make informed decisions. The number of test cases or the number of bugs do not really matter. It is the story about the product and the risks involved. Those numbers might back up your story, but they do not tell the story!
  • A performance decays into Deliverables: testing is about finding problems, collecting information, exploring and experimenting to discover new information. Sure, documents and stuff sometimes help us, but testing is a performance. (James Bach talks about that here: a test is a performance and here: Test cases are not testing: towards a culture of test performance).
  • Test strategy decays into Test execution: when was the last time you saw a really good test strategy? In many cases I find master test plans where everything is described except the strategy. It is hard to create a test strategy and it is even harder to write it down or visualise it. Many testers I meet focus on test execution: creating test cases and scenarios and calling that the strategy.
  • Tool supported testing decays into Automation: testing using tools is a great idea. It gives us more opportunities to test and improves testability. But as said earlier: it becomes a problem when we focus too much on automation or even try to automate all our work. We cannot automate testing.
  • Many kinds of coverage decays into One kind of coverage: testing benefits from diversity! You find a certain type of bug with a certain test technique or approach. By using lots of different views, approaches and techniques, we find more problems.
  • Learning activity decays into Formalized static tasks: testing is learning about the product for our stakeholders. It’s not about verification and validation, there is much more to it. I like to replace such words with challenge the belief that (verify) and investigate (validate). Those activities provide the valuable information we need.
  • Balance risk and uncertainty decays into Certainty: people like to be comfortable and we like to give other comfort as well. But as testers we need to stay unsure, when others are sure. It is our job to keep asking critical questions. We are not here to give confidence or comfort, we are here to demolish unwarranted confidence! Also keep in mind that to find new unexpected problems, we have to go where nobody has thought of and nobody went before us. That will cause confusion which feels uncomfortable for many. I learned to be okay with confusion, since this is essential for learning new things.
  • Business Impact decays into Bugs: some testers are frustrated when bugs aren’t fixed. But that is part of the deal: some things that bug us, are just not important enough.
  • Product story decays into Testing jargon: I think this is the main problem for people not listening to testers. We talk jargon and about what we do in detail too much. We say stuff like: “We’ve executed 17 test cases in the system test, we’ve automated 50% of the test cases for area C and now have 30% code coverage. We found three major and five medium bugs”. And we are surprised that nobody will listen. We need to talk about the product! So you have found 8 bugs? Who cares? Talk about the risks involved, about the threats to the value of the product.

So maybe testers need to stop talking about testing?
Well, not exactly. We need to remember that the information from testing enables other people to do better work! So the testing itself isn’t always interesting, but the story about the results and the impact on the business is!

Just imagine a conversation between a tester and the PO.
Tester: The testing is going well!
PO: Okay, great. How is the product?
Testers: It sucks!

The role of testers

What is the role of testers? Testers see things for what they are. Testers help others make informed decisions about quality, because we think critically about software. This means creating awareness about the state of the product by staying sceptical when everybody else is sure. So we have to know what our clients want from testing. What information do they need to take these decisions? Project managers have one big question to be answered: are there problems that threaten the on-time, successful completion of the product?

Product Risk Knowledge Gap

I like to explain testing using the “Product Risk Knowledge Gap” like we teach in RST. Knowledge Gaps are the things that we need to learn in order to make good decisions. We need to learn about the product to close the knowledge gap. The more we know, the less risky our decisions will be. Testers should focus on questions like: what does the client need to know right now? What might hinder the successful completion of the product? What role do I need to take on in this situation to ensure we achieve our aims? Does this information matter? To whom?

But there is a way to avoid talking about testing. Just find enough questions and problems so that your stakeholders simply won’t have time to ask you questions back! Also, if you tell a credible story and give them the information they need, nobody cares how you got the information in the first place. In this case you need to stand your ground: tell people what they need to hear despite what they want to hear. Again: it’s your job to see things for what they are. If you give people the chance to doubt what you are doing, because you do not deliver the information they need, they will start asking questions about how you do your job. And if you have to talk about how you do your testing, then prepare to be able to tell a damned good story about your testing. Something they can understand and relate to.

The testing story

The testing story by Rapid Software Testing can help you tell that story. Tell a story about the product, what you saw, what you did to gather that information and how valuable that information is. (See “Braiding The Stories” by Michael Bolton). The testing story contains three stories that feed into each other:

  1. The product story: a qualitative report on how the product can work, how it fails, and how it might fail in ways that matter to our clients.
  2. The testing story: to make the product story credible, the testing story is about how we configured, operated, observed, and evaluated the product; what we actually did and what we actually saw.
  3. The quality of testing story: to make the testing story credible, tell a story about the quality of the testing. Describe why the testing we’ve done has been good enough. It includes details on what made testing harder or slower and what we might need or recommend in order to provide better, more accurate, more timely information.

Modern testing
As testers we do way more than only testing. We are enablers of testing by doing all kind of other things to be a service to the team and our clients. Researching this, Alex and I found the Modern testing principles by Alan Page and Brent Jensen. There is a lot of good stuff in there, and yet we feel that there is not enough focus on the actual testing in their principles. Furthermore, we think that the seventh principle “We expand testing abilities and knowhow across the team; understanding that this may reduce (or eliminate) the need for a dedicated testing specialist.” is formulated too negatively. We do not talk about dedicated test specialist as a function. But we like to talk about testing skills. And although we think there should not be a need for a dedicated testing specialist, we see too many people in teams who do not like testing. Passion (or at least motivation) for what you do, is conditional to become good at anything. So we created our own testing principles (inspired by the modern testing principles of course):

  1. Deliver insight into status of the product
  2. Practice (and enact) critical thinking
  3. Enable testing: lead, coach, teach, support
  4. Discuss testability
  5. Explore & experiment
  6. Promote waste removal / avoidance
  7. Help to accelerate the team
  8. Advocate continuous improvement
  9. Foster quality culture
  10. Keep critical distance and close social distance

Stop talking about testing?

So do we need to stop talking about testing? Not really. But we need to talk about the product, risks and value more. We can talk about actual testing only to back our story up or if they ask questions. And even then, we need to make our story understandable and relatable to others. Make sure you are a service to the team. We created our own testing principles to explain what value we add. We also have a pretty clear story on what testing is and how it adds value. We did this by practicing our stories many times. But we also figured out our own testing paradigm. That makes is easier to talk about what we do and how we add value.

Software Development is research & development: a series of experiments that ultimately lead to a suitable solution. We are dealing with customers who do not know exactly what they want. Furthermore, we are dealing with the complexity, confusion, changes, new insights and half answers. That requires research. As we team we are looking for what works and what doesn’t. Testing is of great importance for this! Testing provides insight and overview. Testing shines a light on the actual status of product and project. These insights enable others to make better decisions and eventually make better products.

The slide are here.

Note: this is an Alex Schladebeck and Huib Schoots co-production and this blogpost was co-authored by Alex. So where you read I, you could read Alex and I.

Too controversial?

On May 11 2016 TestNet (*)  held her spring conference with “Strengthen your foundation: new skills for testers” as the central theme. The call for papers that was send out made me frown.  It said:

“In the final keynote of the TestNet autumn event, speaker Rini van Solingen referred to the end of software testing as we know it. ‘What one can learn in merely four weeks, does not deserve to be called a profession’, he stated. But is that true? Most of our skills, we learn on the job. There are many tools, techniques, skills, hints and methods not typical for the testing profession but essential for enabling us to do a good job nonetheless. Furthermore the testing profession is constantly evolving as a result of ICT and business trends. Not only functional testing, but also performance, security or other test varieties. This presses us to expand our knowledge, not just the testing skills, but also of the contexts in which we do our jobs. The TestNet Spring Event 2016 is about all topics that are not addressed in our basic testing course, but enable us to do a better job: knowledge, skills, experience.”

I think that there are a lot of skills that are not addressed in our “basic testing course” where they should have been addressed. I am talking about basic testing skills! So I wrote an abstract for a keynote for the conference:

The theme for the spring event is “Strengthen your foundation: new skills for testers”. My story takes a step back: to the foundation! Because I think that the foundation of most testers is not as good as they think. The title would then be: “New skills for testers: back to basics!

Professional testers are able to tell a successful story about their work. They can cite activities and come up with a thorough overview of the skills they use. They are able to explain what they do and why. they can report progress, risk and coverage at any time. They will gladly explain what oracles and heuristics they use, know everything about the product they are testing and are deliberately trying to learn continuously.

It surprises me that testers regularly can’t give a proper definition of testing. Let alone that they are able to describe what testing is. A large majority of people who call themselves professional testers can not explain what they do when they are testing. How can anyone take a tester seriously if he/she can not explain what he/she is doing all day? Try it: go to one of your testing colleagues and ask what he or she is doing and why it contributes to the mission of the project. Nine out of ten testers I’ve asked this simple question, start to stutter.

What do you exactly do if you use a “data combination test” or a “decision table”? What skills do you use? “Common sense” in this context does not answer the question because it is not a skill, is it? I think of: modeling, critical thinking, learning, combine, observe, reasoning, drawing conclusions just to name a few. By looking in detail at what skills you are actually using, helps you recognize which skills you could/should train. A solid foundation is essential to build on it in the future!

How can you learn the right skills if you do not know what skills you are using in the first place? In this presentation I will take the audience back to the core of our business: skills! By recognizing the skills and training them, we are able to think of and talk about our profession with confidence. The ultimate goal is to tell a good story about why we test and value it adds.

We need a solid foundation to build on!

My keynote wasn’t selected. So I send it in as a normal session, since I really am bothered by the lack of insight in our community. But it didn’t make it on the conference program as a normal session either. Why?  Because it is too controversial they told me. After applying for the keynote the chairman called me to tell me that they weren’t going to ask me to do a keynote because the did want a “negative” sound on stage. I guess I can imagine that you do not want to start the day with a keynote who destroys your theme by saying that we need to strengthen our foundation first before moving on.

But why is this story too controversial for the conference at all? I guess it is (at least in the eyes of the program committee) because we don’t like to admit that we lack skills. That we don’t really know how to explain testing. I wrote about that before here.  It bothers me that we think our foundation is good enough, while it really isn’t! We need to up our game and being nice and ignoring this problem isn’t going to help us. A soft and nice approach doesn’t wake people up. That is why I wanted to shake this up a bit. To wake people up and give them some serious feedback … I wrote about serious feedback before here. But the Dutch Testing Community (represented by TestNet) finds my ideas too controversial…

 


(*) TestNet is a network of, by and for testers. TestNet offers its members the opportunity to maintain contacts with other testers outside the immediate work environment and share knowledge and experiences from the field.

Refusing to do bad work…

I talked at TestBash about context-driven testing in agile. I my talk I said that I refuse to do bad work. Adam Knight wrote a great blog post “Knuckling Down” about this: “One of the messages that came up in more than one of the talks during the day, most strongly in Huib Schoots talk on Context Driven in Agile, was the need to stick to the principle of refusing to do bad work. The consequential suggestion was that a tester should leave any position where you are asked to compromise this principle.

Adam also writes: “What was missing for me in the sentiments presented at TestBash was any suggestion that testers should attempt to tackle the challenges faced on a poor or misguided project before leaving. In the examples I noted from the day there was no suggestion of any effort to resolve the situation, or alter the approach being taken. There was no implication of leaving only ‘if all else fails’. I’d like to see an attitude more around attempting to tackle a bad situation head on rather than looking at moving on as the only option. Of course we should consider moving if a situation in untenable, but I’d like to think that this decision be made only after knuckling down and putting your best effort in to make the best of a bad lot.

Interesting because I think I said exactly that: “if anything else fails, leave!” But maybe I only thought that and forgot to speak it out loud, I am not sure. Let’s wait for the video that will give us the answer. But in the meanwhile: of course Adam is right and I am happy that he wrote his blog post. Because if I was too firm or too distinct, he gave me a chance to explain. Because looking back, I have done many projects where, if I hadn’t tried to change stuff, I would have left many of them in the first couple of days. There is a lot of bad testing around. So what did I try to say?

Ethics again.

This topic touches very closely to ethics in your work! Refusing to do bad work is an ethical statement. Ethics are very important for me and I hope more testers will recognize that only being ethical will change our craft. Ethics help us decide what is right and what is wrong. Have a look at the ethics James Bach summed up in this blog post “Thoughts Toward The Ethics of Testing“. Nathalie pointed to an article she wrote on ethics in a reply to my last post.

Ethics and integrity go hand in hand. Ethics are the external “rules and laws” and integrity is your internal system of principles that guides your behaviour. Integrity is a choice rather than an obligation and will help you do what is right even if no one is watching.

I refuse to do bad work!

Bad work is any work that is deliberately bad. I think along the lines of restrictions in a context, demands placed on them that they don’t know how to handle. Or even worse: intentionally doing stuff you know can be done better, but it is faster, easier or because others ask you to do it like that. Of course there are novices in the field and they do work that can be done better. I do not call that bad work since they are still learning. Still there is a limit to that as well. If you have been tester for several years and you still do not know how to do more than 3 test techniques without having to look them up, I will call that bad work as well. I expect continuing professional development from everybody in the field. Simply because working in IT (but in any profession) we need to develop ourselves to become better.

ethicsLying is always bad work. And I have seen many people lie in their work. Lying to managers to get off the hook, making messages sound just a little better by leaving out essential stuff. Also telling people what they what to hear to make them happy is bad work. What do you do when your project manager asks you to change your test report because it will harm his reputation? Or what do you tell the hiring manager in a job interview when he asks you if you are willing to learn? Many people tell that they are very willing to learn, but are they really?

Bad work is claiming things you can’t accomplish: like assuring quality or testing everything. It is also bad work when you do not admit your mistakes and hide them from your colleagues. Bad work is accepting an assignment when you know you do not have the right skills or the right knowledge. In secondment assignments this is an issue sometimes. I have taken on a project once where the customer wanted something I couldn’t deliver but because my boss wanted me on the position I accepted. That was wrong and the assignment didn’t work out. I felt very bad about it: not because I failed, but because I knew upfront I would fail! I won’t do that again, ever.

So how do I handle this?

I push back! Of course I do not run away from a project when I see or smell bad work. I do try to tackle the challenges I am faced with. I use three important ways trying to change the situation: my courage, asking questions and my ethics. Some examples: when a managers start telling me what I should do and explicitly tell me how I should do that, I often ask how much testing experience the manager has. When given the answer I friendly tell him that I am very willing to help him achieve his goals, but that I think I am the expert and I will decide on how I do my work. Surely there is more to it and I need to be able to explain why I want it to be done differently.

I also ask a lot of questions that start with “why”. Why do you want me to write test cases? What problem will that solve? I found out that often people ask for things like test cases or metrics because it is “common practice” or folklore not because it will serve a certain purpose. Also when I know the reasons behind the requests, it makes it easier to discuss them and to push back. A great example of this is the last blog post “Variable Testers” by James Bach.

Adam talks about changing peoples minds: “One of the most difficult skills I’ve found to learn as a tester is the ability to justify your approach and your reasons for taking it, and being able to argue your case to someone else who has a misguided perspective on what testing does or should involve. Having these discussions, and changing peoples minds, is a big part of what good testing is.

I fully agree. In my last on blog post “heuristics for recognizing professional testers” my first heuristic was: “They have a paradigm of what testing is and they can explain their approach in any given situation. Professional testers can explain what testing is, what value they add and how they would test in a specific situation.” To become better as testers and to advance our craft, we should train the skills Adam mentions: justify approach and being able to argue our case.

It will make you better and happier!

Jerry Weinberg listed his set of principles in a blog post “A Code of Work Rules for Consultants“. In this blog post he says: “Over the years, I’ve found that people who ask these questions and set those conditions don’t wind up in jobs that make them miserable. Sometimes, when they ask them honestly they leave their present position for something else that makes them happier, even at a lower fee scale. Sometimes, a client manager is outraged at one of these conditions, which is a sure indication of trouble later, if not sooner.

It will make you a happier person when you know what your limits are and you are able to clearly remind people you work with. It will prevent you from getting into situations that make you miserable. “That’s the way things are” doesn’t exist in my professional vocabulary. There is always something you can do about it. And if the situation you end up in, after you tried the best you can, isn’t satisfying to you: leave! Believe me, it will make you feel good. I have got the t-shirt! And… being clear about your values also will make you better in your work. Maybe not directly, but indirectly it will.

Daniel Pink speaks about the self-determination theory in his book “Drive“. The three keywords in his book are: Autonomy, Mastery and purpose: “human beings have an innate drive to be autonomous, self-determined and connected to one another, and that when that drive is liberated, people achieve more and live richer lives” (source: http://checkside.wordpress.com)

But but but….

Of course I know there is the mortgage and the family to support. Maybe it is easy for me to refuse bad work. Maybe I am lucky to be in the position I am. But think again… Are you really sure you can’t change anything? And if your ethics are violated every day do you resign yourself? Your ethics will act as heuristics signalling you that there is a problem and you need to do something. I didn’t say you have to leave immediately and if you are more patient than I am, maybe you do not have to leave at all… But remember: for people who are good in what they do, who are confident in what they will and will not do and speak up for themselves, there will always be a place to work.

Now you!

Have you even thought about integrity? What are your guiding principles, values or ethics? What would you call bad work? And what will you do next time when somebody asks you something that conflicts with your ethics?

 


 

While reading stuff online about (refusing) bad work I ran into this blog post by Cal Newport about being bad at work: “Knowledge Workers are Bad at Working (and Here’s What to Do About It…)“Interesting enough Cal Newport wrote a book called “So Good They Can’t Ignore You: Why Skills Trump Passion in the Quest for Work You Love” about the passion hypothesis in which he questions the validity of the hypothesis that occupational happiness has to match per-existing passion. In several recent talks and blog posts I did I talk about passion. Also in the talk discussed in this blog post I claim that passion is very important and I show a fragment of the Stanford 2005 commencement speech by Steve Jobs. Exactly the passage I showed in my talk, Cal uses in the first chapter of his book. “You’ve got to find what you love. And that is as true for your work as it is for your lovers. Your work is going to fill a large part of your life, and the only way to be truly satisfied is to do what you believe is great work. And the only way to do great work is to love what you do. If you haven’t found it yet, keep looking. Don’t settle. As with all matters of the heart, you’ll know when you find it. And, like any great relationship, it just gets better and better as the years roll on. So keep looking until you find it. Don’t settle.” Anyway. Interesting stuff to be researched. I bought the book and started reading it. To be continued…

 

 

 

Heuristics for recognizing professional testers

Via Twitter Helena Jeret-Mäe asked this question: “What are your criteria for professionalism for testers in CDT community?”. Later via Email she updated her question to: “So the updated version of my question is what are the heuristics for recognizing professional testers in your opinion? I changed “criteria” to heuristics… it’s less categorical. And I’ll leave the term “professionalism” up to you as well – I don’t know exactly what you meant by it.”

In my talk “How to become a great tester” at ContextCopenhagen last January I talked about testers and their skills. I said that most testers don’t know what they’re doing and can’t explain effectively what value they add. I have seen many testers who use the same approach over and over again. If I ask them to name test techniques, they can only name a few. If I ask them to explain techniques to me or show how they work, I get no answers. I find that shocking and I cannot understand why testers who call themselves professionals know so little about their craft and do not study their craft.

That is why I make a distinction between professional testers (of which I think there are only few) and testers by profession. Of course I know and understand that there will always be people who have a 9 to 5 mentality, do not read books or blogs and only want to do courses when the boss pays for them. I accept that reality, but that doesn’t mean I want to work with them!

Let me now answer the questions asked, have done enough ranting for now…

Professionalism is what it means to be a professional and what is expected of them. Professional testing is complex and diverse and has several dimensions: knowledge, skills, experience, attitude, ethics and values. I wrote a blog post “What makes a good tester?” in 2011 on this topic and I have a broader view on this topic now although everything I wrote then still counts. In my earlier blog posts I didn’t mention values and ethics and I now think they are extremely important. James Bach writes about them in his blog post “Thoughts Toward The Ethics of Testing”. A great practical example of ethics and values is Rapid Testing. Look at the “the premises of Rapid Testing” and “the themes of Rapid Testing” both can be found in the slides of Rapid Software Testing.

It is hard to recognize professional testers. Every tester is unique and brings different characteristics to the table. Every project is different too and to be successful in finding the right professional tester for your project different characteristics may be important. There are many characteristics to be considered so to be able to recognize professional testers heuristics can be used. Heuristics are fallible methods for solving a problem or making a decision, shortcuts to reduce complex problem or rules of thumb. They are used to determine good enough feasible solutions for difficult problems within reasonable time. On Wikipedia I found this definition: “Heuristics are experience-based techniques for problem solving, learning, and discovery that give a solution that is not guaranteed to be optimal. Where the exhaustive search is impractical, heuristic methods are used to speed up the process of finding a satisfactory solution via mental shortcuts to ease the cognitive load of making a decision.

My heuristics for recognizing professional testers:

1) They have a paradigm of what testing is and they can explain their approach in any given situation.
Professional testers can explain what testing is, what value they add and how they would test in a specific situation.

2) They really love what they do and are passionate about their craft.
Testing is difficult and to be successful, testers needs to be persistent in learning and in their work. Passion helps them to be become real professionals. Watch this video where Steve Jobs talks about passion

3) They consider context first and continuously.
To be effective testers need to choose testing objectives, techniques and by looking first to the details of the specific situation. They recognize that there are no best practices but only good practices in a given context.

4) They consider testing as a human activity to solve a complex and difficult problem that requires a lot of skill.
Testers recognize testing is not a technical profession. Testing has many aspects of social science since software is built for humans by humans.

5) They know that software development and testing is a team sport.
Collaboration is the key in becoming more effective and efficient in testing. Software development is a team sport: people, working together, are the most important part of any project’s context. A great tester knows how to work with developers and other stakeholders in any situation.

6) They know that things can be different.
Professional testers use heuristics, practice critical thinking and are empirical. They know that time to test is limited, systems are becoming increasingly complex and thinking of everything is hard. Therefore heuristics are a very helpful “tool” for testers. They also know that biases and logical fallacies can fool them. They practice critical thinking to deal confidently and thoughtfully with difficult and complex situations. Testers have to accept and deal with ambiguity, situational specific results and partial answers.

7) They ask questions before doing anything.
Testing depends on many things and what is the context of the stuff I am looking at? What is the information we need to find? What is the testing mission? Giving a tester an exercise in an interview can easily test this. If he or she starts working on it or gives you an answer without asking questions, this tells you something.

8) They use diversified approaches.
There is no approach or technique that will find all kinds of bugs or fulfil all test goals. Different bugs are found using different techniques. In order to be able to do this, testers need to know many techniques and approaches. This demands training, practice and some more practice.

9) They know that estimation is more like negotiation.
Have a look at some blog post by Michael Bolton:

10) They use test cases and test documentation wisely.
The context determines what test documentation you should make and what kind of documentation is useful. Quite recently a great (and long) article by James Bach and Aaron Hodder was published in Testing Trapeze “Test cases are not testing: Toward a culture of test performance”. Also Fiona Charles has written some interesting stuff about test documentation in her article the Breaking the Tyranny of Form.

11) They continuously study their craft seriously, practice a lot and practice “deliberate practice”.
Deliberate practice is a structured activity with the goal of improving performance. According to K. Anders Ericsson, there are four essential components of deliberate practice. It must be intentional, aimed at improving performance, designed for your current skill level, combined with immediate feedback and repetitious.

12) They refuse to do bad work, never fake and have the courage to tell their clients and the people they work about their ethics and values.
Testing is often underestimated and many believe “everybody can test”. I have experienced project managers who tell me how to do my work. Professional testers know how to push back and are able to explain why they do what they do.

13) They are curious and like to learn new things.
Testers find pleasure in finding things out. A good read is the curious behaviour by Guy Mason: “a curious mind is one that could be described as an active, engaged and inquisitive mind. Such a mind frequently seeks out new information, enjoys discovering what there is to discover and enjoys the process that comes along with this goal.“

14) They could have any of the important interpersonal skills mentioned in the list below. As stated earlier it depends on the context which skills are most important.

  • Writing skills (like reporting, note taking and concise messages)
  • Communication skills (many different like listening, story telling, presentation, saying no, verbal reporting, arguing and negotiating)
  • Social and emotional skills (like empathy, inspiring, networking, conflict management and consulting)
  • Problem solving skills
  • Decision making skills
  • Coaching and teaching skills
  • Being proactive and assertive

15) They have excellent testing skills:

  • Thinking skills (critical, lateral, creative, systems thinking)
  • Analytical skills
  • Modelling
  • Risk analysis
  • Planning and estimation
  • Applying many test techniques
  • Exploring
  • Designing experiments
  • Observation

Have a look at the Exploratory Testing Dynamics in the RST appendices where several lists of skills are listed.

16) They have sufficient technical skills.
There are many technical skills a tester needs like being able to use tooling, coding skills or willingness to learn what they need to know about the technical structure of the application they are testing. Test automation skills like scripting and SQL skills to work with databases, being able to configure and install software, knowledge and skills to work with the platform the system under test is on (Windows, Linux, Mobile, etc.).

This list is probably not complete. It is very well possible that I have made some mistakes. So please let me know if you have any contributions or improvements. Also: being a professional doesn’t mean you are an expert on all mentioned skills. Have a look at the Dreyfus model to learn more about expert level. A real professional knows what he or she can do and when to ask for help. They do not fear to learn and they are not afraid to make mistakes. I guess that would be the 17th heuristic in the list.

BTW: I googled “code of ethics software testing” and found the ISTQB Code of Ethics for test professionals. I wonder if people who pass the exam know about these AND more important practice them… What about you?


More information

Some more posts that discuss great testers:

And last but not least: have a look at the Testers syllabus by James Bach. An awesome document which lists many important skills and areas of knowledge. It inspired me read about and study different areas.

Context-driven (presentation) heuristics

James BachOctober 23 DEWT and James Bach met in Dordrecht to talk about testing. The subject was “What is context-driven testing?” and more specific “How do you recognize a context-driven presentation?”.

They came together to prepare for the TestNet Autumn Event which was fully dedicated to context-driven testing and had the theme: “Exploring context-driven testing – a new hype or here to stay?”.

DEWT decided earlier this year to write an article about the event to reflect on context-driven testing in the Netherlands. What does the TestNet community have to say about context-driven testing? What can we learn from the event? What can we do to help the community learn? How context-driven is the Dutch Testing community? While discussing this the DEWTs found it hard to find heuristics to “measure” (maybe “recognize” is a better word here) the presentations at the TestNet event.

DEWTs

What is context-driven?

Often context-driven testing is “only” seen as an approach, but it can be more. Actually there are 3 different things that are called context-driven. Testers can be part of one, without necessarily being part of the other.

  1. Paradigm (world view)
  2. Community
  3. Approach

For example: you can use an context-driven approach without being part of the context-driven community. To be in a paradigm you need to have a world view of testing.

Testing People evaluating a product by learning about it through experimentation
driven by is a matter organized and motivated by a systematic consideration of
context all the factors that significantly influence the problems and solutions that lie within the scope of their mission
  • Attitude: context-driven testing allows to change the approach. An example is “Huib’s Rapid Software Testing”. This Huib’s way of doing testing, inspired by Rapid Software Testing, but he changed his approach to his context. Changing anything he sees fit. Often factory school testers do not want to change their approach and apply their approach the same in every situation.
  • Principles: context-driven testing is NOT (only) the seven basic principles, but there is much more. Look what lies behind them, learn about the implicit principles. See slide 9 of this presentation by James Bach.
  • Mentality: context-driven testing is about skills, humanity, science, critical thinking, problem solving, non-linearity, investigation, learning, etc.
  • Metrics: you can use only those metrics if you understand them and if they solve a certain problem.
  • Critical thinking: do you know how? How do you know that you know? How do you get better?

Science is the Belief in the Ignorance of Experts” — Richard Feynman

Presentations

Good stuff to look for:

  • Heuristics over commandments
  • Learning curves
  • Compare alternative methods, trying other methods
  • To be truly context-driven you have studies and tried the practices you say you do not like
  • Discussion of causes and effects assuming open systems. Reject the believe that a project a well defined game that is predictable. Also reject the assumption that a process is unhackable
  • Acknowledgement there are people with different opinions, not speaking from authority
  • Social science
  • Humanism
  • Practitioner responsibility
  • Craftsmanship
  • Refuse to do bad work
  • Problem solving
  • Skill based work

Be aware of people:

  • talking about ‘success’ and ‘failure’ without explaining why and describing the context
  • talking about ‘structure’ and ‘chaos’ (often meaning: I am out of control)
  • showing no evidence (or using sentences like: “research shows that…”)
  • relying on folklore
  • using numbers without context
  • who are ignorant of social science
  • uncritically apply the manufacture metaphor (IT is like a factory)
  • apply premature automation
  • assuming other people read and follow what is written
  • contempt humanism
  • trying stuff once and over generalize that to the whole company/world
  • who fear variation
  • over simplify (approach the world as if it was linear)
  • misuse of statical analysis
  • who use averages without variance
  • who do narrow research
  • say or do stuff because their clients want it
  • demonize tacit knowledge and idealize explicit knowledge

This is the summary I have made of what we discussed. You could consider this as a heuristic for context-driven presentations. I used this heuristic in my presentation “What is context-driven testing?” at the TestNet event.

Heuristics Success / Failure
Learning Chaos / Structure
Compare alternative methods Lack of evidence / Narrow research
Cause and effect Folklore
Open systems Numbers without context
Social science Ignorance of social science
Humanist view Contempt of humanism
Craftsmanship IT & testing is like manufacturing
Explain context No context mentioned
Worldview of testing Generalizing after one try
Different opinions Use of averages without variance
Allows to change approach Linearity
« Older posts