Category: Agile

Test improvement in an agile/CDT environment

This post and the article have been updated on April 4 2017.

One day during a team meeting at Joep‘s previous job at a bank the Team Manager of Testing, listed a number of topics his testers could work on in the coming months. One of those topics was “testing maturity”. This topic was on the list not because this manager was such a fan of maturity models, but because the other team managers (Business Analysis and Development) had produced one for their own teams and higher management would like to have one for testing as well. And although Joep saw little value in a classic five-tiered maturity model either, he was intrigued by the question: so what can you do with respect to maturity models that is of value?

Joep asked Huib to help him think of a way to create a valuable, context-driven way to work on maturity. Since Huib had been working for the same bank, they met and discussed the possibilities. Soon they found out that the criteria should be variable since maturity depends on context. They started experimenting with stack ranking and quite soon they had the first version of their “maturity model”.

Maturity or improvement?

After discussing the first version of the article with James and Michael, we felt the need to update our article. Their comments helped us realize that we needed to explore maturity and maturity models a bit more. After doing this, we decided to rename our model into a test improvement model.

Maturity mission: better testing

What is the mission a maturity assessment? We think the assessment should be a pathway to better testing. As a part of solving problems we think the mission should be: “An investigation of strengths and weaknesses. A starting point for a discussion about potential (testing) problems and how to solve them.” Or as James Bach says: A maturity model is plan for achieving maturity. And this is exactly what we created. Our maturity model isn’t anything like the staged, fixed models available in the market. Maybe we shouldn’t call our method a maturity model, since basically it isn’t. It is a tool designed to help teams assess and improve their testing. It is a method supported by a card game that helps teams retrospect and identify strengths and weaknesses in their way of working, the stuff they create, the team, their skills and context.

Result

After a first try-out at the bank Joep worked, we let it rest for a while. After a couple of months we wrote this article. It is the first version and it needs to be refined and polished. The heuristics lists are probably to long and need to be reduced. We think of this model as a card game that can be played with teams.

Currently we are also working on an agile version of this model, a card game for agile teams to assess their “maturity” to help them to find possible areas for improvements. More about that later.

We are curious about your thoughts. What do you think? Maybe you want to try the game? Feel free to try it out. We hope you will share your experiences with us.

Article (pdf) – Card game (pdf)

The slides are of the meetup about our model are here.

Must read: A Context-Driven Approach to Automation in Testing

Test Automation is a hot item in our industry. Many people talk about it and much has been written on this topic. Sadly there is still a lot of misconception about test automation. Also, some people say context-driven testing is anti test automation. I think that is not true. Context-driven testers use different names for it and they are more careful when they speak about automation and tooling to aid their testing. Also, context-driven testers have been fighting the myths that testing can be automated for years. In 2009 Michael Bolton wrote his famous blog post “Testing vs. checking“. Later flowed up by “Testing and checking refined” and “Exploratory testing 3.0“. These tremendous important blog post learn us about how context-driven testers define testing and that testing is a sapient process. A process that relies on skilled humans. Recently Michael Bolton and James Bach have published a white paper to share their view on automation in testing. A vision of test automation that puts the tester at the center of testing. This is a must read for everyone involved in software development.

The follow text is taken from the “A Context-Driven Approach to Automation in Testing” white paper written by James Bach and Michael Bolton.

We can summarize the dominant view of test automation as “automate testing by automating the user.” We are not claiming that people literally say this, merely that they try to do it. We see at least three big problems here that trivialize testing:

  1. The word “automation” is misleading. We cannot automate users. We automate some actions they perform, but users do so much more than that.
  2. Output checking can be automated, but testers do so much more than that.
  3. Automated output checking is interesting, but tools do so much more than that.

robotAutomation comes with a tasty and digestible story: replace messy, complex humanity with reliable, fast, efficient robots! Consider the robot picture. It perfectly summarizes the impressive vision: “Automate the Boring Stuff.” Okay. What does the picture show us?

It shows us a machine that is intended to function as a human. The robot is constructed as a humanoid. It is using a tool normally operated by humans, in exactly the way that humans would operate it, rather than through an interface more suited to robots. There is no depiction of the process of programming the robot or controlling it, or correcting it when it errs. There are no broken down robots in the background. The human role in this scene is not depicted. No human appears even in the background. The message is: robots replace humans in uninteresting tasks without changing the nature of the process, and without any trace of human presence, guidance, or purpose. Is that what automation is? Is that how it works? No!

The problem is, in our travels all over the industry, we see clients thinking about real testing, real automation, and real people in just this cartoonish way. The trouble that comes from that is serious…

Read more in the fabulous white paper “A Context-Driven Approach to Automation in Testing” by James Bach and Michael Bolton.

Stop hugging, start working … on excellence!

Some context: this blogpost is my topic for a new peer conference called “Board of Agile Testers (BAT)” on Saturday December 19 2014 in Hotel Bergse Bossen in Driebergen.

I love agile and I love hugging… For me an agile way of working is a, not THE, solution to many irritating problems I suffered from in the 90’s and 00’s. Of course people are the determining factor in software development. It is all about people developing (as in research and development) software for people. So people are mighty important! We need to empower people to do awesome work. People work better if they have fun and feel empowered.

Vineet Nayar talks about people, who want to excel, need two important things: a challenge and passion. These factors resemble the ones described by Daniel Pink: autonomy makes room to excel, passion feeds mastery and a challenge gives purpose. I wrote an article about this subject for agile record called “Software development is all about people“. I see agile teams focus on this people stuff like collaboration, working together, social skills… But why do they often forget Mastery in testing?

Rapid Software Testing teaches serious testing skills by empowering testers in a martial art approach to testing. Not by being nice and hug others. By teaching testers serious skills to talk about their work, say what they mean, stand up for excellence. RST teaches that excellent testing starts with the skill set and the mindset of the individual tester. Other things might help, but excellence in testing must centre on the tester.

One of the many examples is in the new “More agile testing” book by Lisa & Janet in chapter 12 Exploratory testing there is a story by Lisa: “Lisa’s story – Spread the testing love: group hugs!” My review comment was and I quote: “I like the activity but do not like the name… I fear some people will not take it too serious… It might get considered too informal or childish. Consider a name like bug hunts.”

Really? Hugs? The whole hugging ethos in agile makes me CRAZY. Again, I love hugging and in my twitter profile it says I am a people lover. But a fluffy approach to agile in general and testing in particular makes me want to SCREAM! It makes me mad! Stop diminishing skills. If people are doing good work, sure hug them, but if they don’t: give them some serious feedback! Work with them to get better and grow. Mentor them, coach them, teach them. But what if they do not improve? Or do not want to improve? Well… maybe then it is time to say goodbye? It is time to start working on some serious skills!

Testing is serious business, already suffering from misunderstanding and underestimation by many who think they can automate all testing and everybody can test. In agile we are all developers and t-shaped people will rule the world. In 15 years there will be only developers doing everything: writing documentation, coding and testing… Yeah right! I wish I could believe that. Testing is HARD and needs a lot of study. As long as I see a vast majority of people not willing to study testing, I know I will have a job as a testing expert for the rest of my life!

This blogpost reflects some “rough ideas”. After the peer conference I will update this post with the ideas discussed in the room.

Refusing to do bad work…

I talked at TestBash about context-driven testing in agile. I my talk I said that I refuse to do bad work. Adam Knight wrote a great blog post “Knuckling Down” about this: “One of the messages that came up in more than one of the talks during the day, most strongly in Huib Schoots talk on Context Driven in Agile, was the need to stick to the principle of refusing to do bad work. The consequential suggestion was that a tester should leave any position where you are asked to compromise this principle.

Adam also writes: “What was missing for me in the sentiments presented at TestBash was any suggestion that testers should attempt to tackle the challenges faced on a poor or misguided project before leaving. In the examples I noted from the day there was no suggestion of any effort to resolve the situation, or alter the approach being taken. There was no implication of leaving only ‘if all else fails’. I’d like to see an attitude more around attempting to tackle a bad situation head on rather than looking at moving on as the only option. Of course we should consider moving if a situation in untenable, but I’d like to think that this decision be made only after knuckling down and putting your best effort in to make the best of a bad lot.

Interesting because I think I said exactly that: “if anything else fails, leave!” But maybe I only thought that and forgot to speak it out loud, I am not sure. Let’s wait for the video that will give us the answer. But in the meanwhile: of course Adam is right and I am happy that he wrote his blog post. Because if I was too firm or too distinct, he gave me a chance to explain. Because looking back, I have done many projects where, if I hadn’t tried to change stuff, I would have left many of them in the first couple of days. There is a lot of bad testing around. So what did I try to say?

Ethics again.

This topic touches very closely to ethics in your work! Refusing to do bad work is an ethical statement. Ethics are very important for me and I hope more testers will recognize that only being ethical will change our craft. Ethics help us decide what is right and what is wrong. Have a look at the ethics James Bach summed up in this blog post “Thoughts Toward The Ethics of Testing“. Nathalie pointed to an article she wrote on ethics in a reply to my last post.

Ethics and integrity go hand in hand. Ethics are the external “rules and laws” and integrity is your internal system of principles that guides your behaviour. Integrity is a choice rather than an obligation and will help you do what is right even if no one is watching.

I refuse to do bad work!

Bad work is any work that is deliberately bad. I think along the lines of restrictions in a context, demands placed on them that they don’t know how to handle. Or even worse: intentionally doing stuff you know can be done better, but it is faster, easier or because others ask you to do it like that. Of course there are novices in the field and they do work that can be done better. I do not call that bad work since they are still learning. Still there is a limit to that as well. If you have been tester for several years and you still do not know how to do more than 3 test techniques without having to look them up, I will call that bad work as well. I expect continuing professional development from everybody in the field. Simply because working in IT (but in any profession) we need to develop ourselves to become better.

ethicsLying is always bad work. And I have seen many people lie in their work. Lying to managers to get off the hook, making messages sound just a little better by leaving out essential stuff. Also telling people what they what to hear to make them happy is bad work. What do you do when your project manager asks you to change your test report because it will harm his reputation? Or what do you tell the hiring manager in a job interview when he asks you if you are willing to learn? Many people tell that they are very willing to learn, but are they really?

Bad work is claiming things you can’t accomplish: like assuring quality or testing everything. It is also bad work when you do not admit your mistakes and hide them from your colleagues. Bad work is accepting an assignment when you know you do not have the right skills or the right knowledge. In secondment assignments this is an issue sometimes. I have taken on a project once where the customer wanted something I couldn’t deliver but because my boss wanted me on the position I accepted. That was wrong and the assignment didn’t work out. I felt very bad about it: not because I failed, but because I knew upfront I would fail! I won’t do that again, ever.

So how do I handle this?

I push back! Of course I do not run away from a project when I see or smell bad work. I do try to tackle the challenges I am faced with. I use three important ways trying to change the situation: my courage, asking questions and my ethics. Some examples: when a managers start telling me what I should do and explicitly tell me how I should do that, I often ask how much testing experience the manager has. When given the answer I friendly tell him that I am very willing to help him achieve his goals, but that I think I am the expert and I will decide on how I do my work. Surely there is more to it and I need to be able to explain why I want it to be done differently.

I also ask a lot of questions that start with “why”. Why do you want me to write test cases? What problem will that solve? I found out that often people ask for things like test cases or metrics because it is “common practice” or folklore not because it will serve a certain purpose. Also when I know the reasons behind the requests, it makes it easier to discuss them and to push back. A great example of this is the last blog post “Variable Testers” by James Bach.

Adam talks about changing peoples minds: “One of the most difficult skills I’ve found to learn as a tester is the ability to justify your approach and your reasons for taking it, and being able to argue your case to someone else who has a misguided perspective on what testing does or should involve. Having these discussions, and changing peoples minds, is a big part of what good testing is.

I fully agree. In my last on blog post “heuristics for recognizing professional testers” my first heuristic was: “They have a paradigm of what testing is and they can explain their approach in any given situation. Professional testers can explain what testing is, what value they add and how they would test in a specific situation.” To become better as testers and to advance our craft, we should train the skills Adam mentions: justify approach and being able to argue our case.

It will make you better and happier!

Jerry Weinberg listed his set of principles in a blog post “A Code of Work Rules for Consultants“. In this blog post he says: “Over the years, I’ve found that people who ask these questions and set those conditions don’t wind up in jobs that make them miserable. Sometimes, when they ask them honestly they leave their present position for something else that makes them happier, even at a lower fee scale. Sometimes, a client manager is outraged at one of these conditions, which is a sure indication of trouble later, if not sooner.

It will make you a happier person when you know what your limits are and you are able to clearly remind people you work with. It will prevent you from getting into situations that make you miserable. “That’s the way things are” doesn’t exist in my professional vocabulary. There is always something you can do about it. And if the situation you end up in, after you tried the best you can, isn’t satisfying to you: leave! Believe me, it will make you feel good. I have got the t-shirt! And… being clear about your values also will make you better in your work. Maybe not directly, but indirectly it will.

Daniel Pink speaks about the self-determination theory in his book “Drive“. The three keywords in his book are: Autonomy, Mastery and purpose: “human beings have an innate drive to be autonomous, self-determined and connected to one another, and that when that drive is liberated, people achieve more and live richer lives” (source: http://checkside.wordpress.com)

But but but….

Of course I know there is the mortgage and the family to support. Maybe it is easy for me to refuse bad work. Maybe I am lucky to be in the position I am. But think again… Are you really sure you can’t change anything? And if your ethics are violated every day do you resign yourself? Your ethics will act as heuristics signalling you that there is a problem and you need to do something. I didn’t say you have to leave immediately and if you are more patient than I am, maybe you do not have to leave at all… But remember: for people who are good in what they do, who are confident in what they will and will not do and speak up for themselves, there will always be a place to work.

Now you!

Have you even thought about integrity? What are your guiding principles, values or ethics? What would you call bad work? And what will you do next time when somebody asks you something that conflicts with your ethics?

 


 

While reading stuff online about (refusing) bad work I ran into this blog post by Cal Newport about being bad at work: “Knowledge Workers are Bad at Working (and Here’s What to Do About It…)“Interesting enough Cal Newport wrote a book called “So Good They Can’t Ignore You: Why Skills Trump Passion in the Quest for Work You Love” about the passion hypothesis in which he questions the validity of the hypothesis that occupational happiness has to match per-existing passion. In several recent talks and blog posts I did I talk about passion. Also in the talk discussed in this blog post I claim that passion is very important and I show a fragment of the Stanford 2005 commencement speech by Steve Jobs. Exactly the passage I showed in my talk, Cal uses in the first chapter of his book. “You’ve got to find what you love. And that is as true for your work as it is for your lovers. Your work is going to fill a large part of your life, and the only way to be truly satisfied is to do what you believe is great work. And the only way to do great work is to love what you do. If you haven’t found it yet, keep looking. Don’t settle. As with all matters of the heart, you’ll know when you find it. And, like any great relationship, it just gets better and better as the years roll on. So keep looking until you find it. Don’t settle.” Anyway. Interesting stuff to be researched. I bought the book and started reading it. To be continued…

 

 

 

Misconceptions about testing

Shmuel Gershon’s tweet pointed me to an article on the scrum alliance website Agile Methodology Is Not All About Exploratory Testing by Dele Oluwole. I share Sigge Birgissons concerns: “the post clearly shows what I mean when having deep concerns about the knowledge of testing in agile community”.

I think Dele doesn’t fully understand what testing is or at least he uses a different definition than I do. And certainly he doesn’t understand exploratory testing. Rikard Edgren wrote an open letter to explain what testing is. Please read his high level summary of software testing to understand what testing means to me.

"It is imperative to state in clear terms why Agile testing cannot be all about exploratory testing"

(The text in these gray frames are quotes form the article by Dele)

I wrote a post some weeks ago about agile testing titled what makes agile testing different: agile testing isn’t that much different from “other” testing. Why do some people think agile is so different? To me there is no such thing as Agile testing. There is testing in an agile context. And every agile context is probably different. So saying that agile testing cannot be about exploratory testing makes no sense to me.

It is unequivocally the case that: you cannot estimate your time for exploratory testing, i.e., assign points realistically

Estimating testing is an interesting topic. This blogpost by Morgan Ahlström nicely emphasizes that estimates are guesses. Martin Jansson of the Test Eye writes about “utopic estimations” here. Michael Bolton wrote a lot about estimation here, here and here. He explains that testing is an open-ended task which depends on the quality of the products under test. The decision to ship the product (which includes a decision to stop testing) is a business decision, not a technical one.

Especially the fifth part of the black swan series is interesting in this discussion because Michael writes about the fallacies surrounding “development” and “testing” phases (by James Bach). He also explains why estimating the duration of a “testing project” by counting “test cases” or “test steps” is not a smart thing to do.

"You cannot plan for exploratory testing, as you do not have defined expected results."

Why are some people so obsessed by expected results? And why is there a need to have expected results to be able to plan testing? Expected results can be very helpful, but there is much more to quality then doing some tests with an expected result. A definition that I like is by Jerry Weinberg: “Quality is value to some person”. To understand this, you might want to read this blogpost to understand that there is more to quality than the absence of bugs. Also have a look at the excellent free ebook “The Little Black Book on Test Design” by Rikard Edgren. On page 2-6 he uses the “testing potato” to explain that there are more important aspects to the system than the requirements only.

"There is no defined scope for exploratory testing."

In the exploratory testing I do in my projects there is a “scope”. I do targeted and focused testing using charters, sessions and planning my testing using a dashboard that resembles a scrum board. Have a look at the slides of my presentation “Boosting the testing power with exploration“.

Using a coverage outline in a mind map or a simple spreadsheet, I keep track of what I have tested. My charters (a one to three sentence mission for a testing session) help me focus, my wrap-up and/or debriefings help me determine how good my testing was. My notes and the sessions sheets keep track of what I have done in my testing. Like in scrum, I use “standup” meetings to plan my testing. In these meetings we discuss progress, risks, priorities and charters to be executed. This helps us to make sure we do the best testing we can do continuously.

"The tester, product owner, and Scrum team are not in control."

I am not sure if I understand what you are trying to point out here. Are you saying that the team is not in control when doing exploratory testing? My model above shows that you can be in control when doing exploratory testing when done right. Exploratory testing is NOT ad hoc or unstructured when done right. If you do it right you will have control!

"There is no measure of progress, as testers cannot determine when testing is enough."

How do we determine how much testing is enough? Stopping heuristics might help here. No matter how simple the system is we are building, there are simply to many variables to test everything. So testing is about making choices what to test and what not to test. Even with a huge amount of automated checks, we can not check/test everything (to understand the difference between testing and checking read this). Testing is not about doing X test cases and when they all pass, you are done. Testing is providing information for managers to make good decisions. And when do managers have enough information to inform their decisions?

Still, not many Agile projects will require just two phases, like integration and regression. But it's definitely not only exploratory testing that's needed, as is erroneously believed in some quarters.

I am not sure what you mean by two phases. What do you mean by testing in phases? I like to use the agile testing quadrants when I try to explain how I think of testing in an agile context.

A team is developing software and the programmers do testing before checking the software in and making it available to the team. How do we call that sort of testing? Unit testing? But is that /only/ testing done by the programmer? I argue that the programmer might do all kinds of testing before checking his code in, even functional and acceptance tests. They probably will create a lot of automated checks and maybe even do some exploratory testing to see if the software meets their expectations: quickly testing some usability and performance aspects. So before the software is checked in, the programmer has covered testing in all 4 quadrants. This does not mean testing is done. More testing can be done, it depends on the context. What does the product owner wants to know? What are the risks involved? How much time do we have left? When discussing a test strategy I try not to speak about phases, I like to discuss what gets covered and why. What information is needed by the team and it’s stakeholders? When talking about coverage I do not mean code coverage but test coverage: the extent to which we have traveled over some map (model) of the system.

So I don’t think you can say “only exploratory testing is needed or not”.

Dele concludes his article with this statement:

It is the responsibility of the tester (and the Agile/Scrum team) to ensure that acceptance testing is in line with the expectation of the product owner. If we agree that there is an expectation, we therefore have to design test cases (even if minimal) that will verify the specified acceptance criteria."

Dear Dele please read about testing and exploratory testing. Some good starting points are these lists of resources on this blog or the one made by Michael Bolton: resources on Exploratory Testing, Metrics, and Other Stuff. I am happy to point you to more good sources of information if you are interested. Just let me know.

What makes agile testing different?

Last week Pete Walen asked me the following question via twitter: The Question (read it carefully!): What is it that makes Agile Testing different from “other” testing?

Agile vs agile?

What is agile testing? And what is agile? Let me first say that I do not distinguish agile and Agile. For me it is all the same. Agile is a mindset, a way of looking at the world. For me it is not a process or method. It is more a container than a way of working. This blogpost discusses agile vs Agile and I like it. It describes agile as a mindset and Agile with a capital A as something commercial: “The problem here is that all of these sensible suggestions got formalized into “Agile”, with a Capital A. This set of suggestions for “things that seem to work” got boxed up into a package with a bow on top, to be sold to companies and managers.”

Agile testing?

And agile testing? What is agile testing? I rather say testing in an agile context instead of agile testing. Good testing in an agile context is done by looking first to the details of the specific situation. Remember the 7th principle of context-driven testing: “Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.”

Testing is an essential part of software realization. Implementing testing in an agile context is a challenge. It comes with some interesting challenges for testers:

  • by the iterative nature of working, there is less time to test compared to the testing most of us are used to in a more traditional (waterfall) context. It requires a different approach to testing. There is a different phasing to testing in an agile environment.
  • how can I make sure that I can perform sufficient testing fast enough to keep up with the project?
  • testers need to ensure that “self-managing teams” do enough testing
  • cope with the changing team dynamics in which people work and where interaction is important
  • integrating structured testing in an environment where change is common
  • deliver added value as a tester when there is no software to test

Testers need to deliver immediate value. In an agile environment rapid feedback allows the team continuously forward. Testing should instantly provide useful and understandable information about the status of the products in development. It allows the team to deliver insightful value to the business continuously and to make maximum progress.

Different?

So what is the difference? I think the testing itself is not so much different, it is the context in which you do the testing is different. If you are aiming on differences between waterfall and agile my list of most important differences would be:

  • less time to prepare, execute and report (short sprints).
  • iterative and incremental approach: excellent unit testing is essential.
  • test automation (some rather call it automated checking or tool assisted testing) is essential for fast feedback and continuous integration.
  • role change: less testing, more coaching. Testers become “test coaches” or “quality directors” to make sure the team is doing sufficient testing. Enough (not too much nor too little) and of good quality.
  • cope with less certainty: change is common. Test documentation needs to deal with change by being transparent, using simple dashboards and light weight test documentation.
  • team work: where many testers are used to work in TEST teams, they are now working in DEVELOPMENT teams.
  • continuous critical thinking: testers need to help the team by thinking critical about the impact and risks. Where testers were used to do that upfront while writing documents like master test plans, they now have to do that continuously throughout the project: in grooming sessions, daily standups, planning sessions, etc. But also in their own work: making choices about what to cover: broad and depth.

Again: the testing itself is not so much different, it is the context in which you do the testing is different!

Agile Testing Days: a gathering called PATS

November 14-17 I attended Agile Testing Days in Potsdam. It was a great experience! In a series of experience reports on my blog I will reflect this conference. I want to start with the gathering of testers Jean-Paul Varwijk and I organized called PaTS (Potsdam agile Testers Session).

Jean-Paul and I love to discuss testing with other passionate testers. That is why we
both are in DEWT, participate at TestNet and some other initiatives. Zeger
van Hese
, fellow DEWT (and Programme Chair EuroSTAR 2012. GRATS again
Zeger!) told us about the Rebel Alliance at EuroStar 2010 and Jean-Paul and I decided to try to organize something similar.

So we arranged a room, made a shortlist of special and interesting people from the
(agile) software testing scene and send out an email announcing PaTS: a get
together on Wednesday evening 16th of November 2011 in Potsdam. We defined our
event as an informal off-conference evening, with drinks and pizza to talk and
debate about testing, have fun and debate some more. It turned out to be a fun and interesting evening!

Rob Lambert, Rob van Steenbergen, Daniel Levy, Janet Gregory, Simon Morley, Brett L. Schuchert, James Lyndsay, Stevan Zivanovic, Jim Holmes, Bart Knaack, Lisa Crispin, Olaf Lewitz, Mike Scott, Jurgen Appelo, Thomas Ponnet, Cecile Davis, Michael Bolton, Jean-Paul Varwijk and me gathered in the test lab and we first ordered some jumbo pizza and beer. Thanks to Telerik for sponsoring the beer!

We made a list of interesting topics and conducted a “dot vote”. This is the list
and the dot ranking:

Great Testers – 10
Management of testers – 8
Acceptable level of risk – 8
DEWT / Peer groups – 6
Tool topic – 3
Coin game – 2
Retrospect ATD talks – 2
Bathtub – 1
Continuous delivery – 1
Cloud – 1
Testlab – 0

We agreed to give every topic 10 minutes and “Caesar vote” after the time box to see if the group wants to give the topic extra time. During the evening the 4 top topics were discussed. Rob Lambert made some great mind maps of each topic. He let me take pictures so a massive thanks to him! You can also find them here.

Great Testers

Lisa Crispin started the discussion: “Most important is attitude, we will teach them the
skills”. And I fully agree. The group also mentioned passion as very important. When somebody is passionate about his work, he wants to improve and learning will become an essential part of his daily routine. Curiosity and quick learning are skills which were mentioned here as well. Another characteristic of a great tester is to be able to see things different or as Michael Bolton said: a great tester is capable of seeing complexity in apparent simple things and simplicity in apparent complex things.

Of course communication was mentioned as an important skill to be able to exchange knowledge. Michael Bolton ask what was meant with communication and he summed up 27 different forms of communication. I have to study the list, but I think that will end up as a separate blogpost here.

There was also a discussiuon about structured vs. unstructured testing (scripted vs. exploratory testing). What the conclusion was of this skill I can’t remember. But for me I value testers who can do both and are not affraid of thinking of their own. Exploratory testing gives a tester freedom but also forces him to keep thinking.

Other characteristics/skills mentioned during the discussion: cultural fit, humility, empathy, honesty, domain knowledge and sense of humour. Good to see a lot of human factors discussed.

I will further work on the other 3 topics discussed. But the mind maps by Rob Lambert give you a nice overview what was discussed.

Manage / lead testers to become great

 

 

 

 

 

DEWT / Peer groups

 

 

 

 

 

Acceptable level of risk

 

 

 

 

 

You can find some other experience reports of PaTS by Rob van Steenbergen, Jean-Paul and Olaf Lewitz.

The final exit of the test manager?

Last week I attended the autumn event by TestNet. This was an awesome event (as always!) where Leon Bosma did a interesting talk on the role of the test manager in the future with the title “Test Manager: the final exit”. His slides can be found here (in dutch).

In his presentation, he focused on the question what makes a manager? He used the model of Quinn as the basis for all tasks/roles of a manager. Then he compares these tasks to the testing activities according to TMap. In this part of his talk he covers the role description, tasks in testing and its importance to the tester. Then he talked about the developments in the testing profession covering developments in approach, the testing craft itself and the systems under test. The latter coincided with the theme of this event “The Cloud”. In the last part of his talk Leon gives his view on the impact of these developments on the role of a test manager. I have summarized them in the table below.

Role From To
Director He determines what I should do I’ll decide what to do in consultation
Producer He makes sure I can do my job The team make sure we can our job
Innovator He comes up with new solutions We come up with new solutions
Broker He makes the stakeholders happy The actors make each other happy
Facilitator He inspires the team We inspire and build the team
Mentor He teaches me the craft I learn from other specialists
Coordinator He decides what to do when I decide in coorperation the team what to do when
Monitor He sees to it that I do the right things The team ensures that I do what is needed

So, is the test manager a dying breed or not? I think they are! With developments like agile it is obvious that the team will be more responsible to do planning, coordination and determine the test strategy. For that we no longer need test managers. I never really understood the role of a test manager in many projects. Why should we divide the role of the tester in test executer, test analyst and test manager? We don’t do this in the other disciplines in IT, do we?

I think it has to do two major reasons:
1. The immaturity of the testers and therefore the (wrong?) interpretation of the role tester
2. The career path in many organizations

Immaturity
A good tester IMHO “deals” with all aspects of his job, maybe in consultation with the project manager. The tester controls the entire process from writing test plans containing an appropriate strategy to the reporting to the Steering Committee. I know many testers who find it scary to facilitate a workshop (for example, product risk mapping) or to give a presentation (e.g. reporting to the steering committee). The unfamiliarity of the testing craft by other IT professionals makes it worse. Testing is a difficult discipline but most people in IT do not recognize that. A project manager is often unable to make a good estimate of the testing activites is his project. Let alone that he is capable of creating a proper test strategy. Acceptance criteria are very difficult to determine and test managers are often asked to take care of this.

Good testing is quite difficult. Traditionally, business analysts and developers have a relatively easy job. The analyst has only a few dependencies: he talks with stakeholders and writes that down. The developer only needs a development environment and he can do his work. Testing is often much harder: always at the end, started too late with fixed deadlines, organizing various test environments, the many procedures your organization has to prepare acceptance environments, different user groups who should be involved in testin, etc. A lot of arranging and organizing to do! Hiring a test manager is an easy thing to do. Especially if there are people you think the testing itself is not as interesting. Or even worse: if you are not so good at your craft… It is easy to grow (read: flee) towards coordination.

The interpretation of the test role
In a situation where dedicated testers are present, test managers are often needless. Some time ago I worked as a test manager at a customer. In that organization a group test managers were the only “testers”. Most of the actual testing was done by the functional application managers and the users. The test managers did the coordination, wrote test plans, made sure that the testing went as smoothly as possible and were supposted to coach the users. In my opinion this organization of testing is terrible. Sure, users are involved, but testing is a profession and that cannot be delegated to untrained users. Writing requirements or programming is also done by professionals! The users in this situation had a short training. Enough to do proper testing, right?

Test carreers
But who did invent the test managers role? And why? I do not know where the test manager role came from, but I have a suspicion. Testing in many places is still a role that “used” as a stepping stone to another role in IT (WRONG!). Because everybody can test and testers do not need a lot of training (WRONG AGAIN!) so he can focus on the next steps in his carreer. The knowledge and experience of the the systems he gains along the way may well be used in the rest of his career…

Of course a tester grows and his role will be different over the years. I see many junior testers who are doing far less than a senior tester. I often wonder why. The junior needs to learn the craft and he can learn it by just doing it! So let the junior tester do everything himself. Maybe under the guidance of a senior tester as coach. Learning is making mistakes and evaluating them with your (senior) colleagues.

There is only one role: Tester! Test coordinators are in my opinion just senior testers (foremen) responsible for a certain part of the project. One tester is happy to coordinate, the other likes the content. Ultimately, a good tester has mastered his craft in every aspect!

Imagine that after ten years you still want to grow as a tester. What do you do? This is very difficult in many organizations. The salary scales of a tester will not let you. Test manager is a common role in which testers grow. This is a curse for our profession: it makes people want to make money and grow quickly by leaving the role of “ordinary” tester and become test manager. Without focusing to become a great tester they are busy becoming managers. Too bad!

So, will the role of test manager disappear soon? I think not! The test manager is still needed in many places to fill the gaps created by bad testers not in control of their own job!

A new cash CAT?

Two weeks ago I had a discussion with an account manager of a testing consulting company about CAT: Certified Agile Tester. His company sells this training and he told me he sees a certification training as a driving license: once you obtain it, you still need to learn how to drive like an expert. I do not believe that ISTQB or TMap certification makes anybody a better tester. And this agile tester certification is even worse… How can you give somebody a certificate for a mindset?

I do not think you’re a good tester if you have an (advanced) certificate in ISTQB or whatever. It tells me nothing about your motivation. Especially not when the certificate could (or even worse, should) be obtained during working time paid by your employer… I think that certification is not necessarily bad, but it is certainly no blessing to our profession. James Bach wrote a great blog post about it in 2005 (!). IMHO certification gives a kind of false security about testing skills. And unfortunately this is maintained by the whole market. The big companies (let’s just for convenience call them “demand side”) have difficulty selecting the right external hires. So they look for certificates which makes selecting easier. The “supply side” is more than happy to participate in the certification hype. Because they earn a lot of money by providing training. But also from a commercial interest because the “demand side” keep asking for these certificates. And if they want to sell their people, they must have certificates. Sadly a mechanism that keeps the certification alive!

Certification says very little about a tester. I know plenty of testers with lots of certificates that I would never hire. Why not? Because they are no good in testing! And the certificates do not make them any better.

The CAT website explains: to become a Certified Agile Tester you have to succeed in three different ways:

  • A soft skills assessment on capacity for teamwork
  • An exam, which requires free answering – no multiple choice questions
  • A practical section where the examinee’s testing skills are put to the test.

This soft skill assessment makes me wonder. As a manager I find it difficult to assess my testers after some months or even a whole year, but the trainers of this course can in only 5 days?

The exam doesn’t prove the examinee is a good agile tester! How many exams have we taken during our lives? And do we still know everything we have learned? Like I said in my last blog post, you become a better tester by continous learning, not by taking exams!

The practical section sounds good. Although I cannot imagine that you can fail your certification on this part.  

The course itself looks okay: the subjects trained are interesting and on the reading list are books I have read and would recommend to others. But the whole idea of certification is bad! Agile is a mindset and how can a mindset be certified?

The testing/training industry has found another cashcowCAT!