Testing to Make Product Better vs. Perfect

Reading Seth Godin’s post Perfect vs. important I realized that his idea is very relevant to testers. To rephrase, the main thought of his post is:

Spend more time on making something better (more useful) than polishing it to perfection

When it comes to testing, frequently testers jump into a habit of reporting every minor issue found which leads to quantity vs quality sometimes. Have you ever reported an ugly progress indicator or not the prettiest alignment of UI elements? I have. And I even fought for these to be fixed.

Obviously, UI is important. Distortion bug on IE9 can make you lose customers who use IE9, for example. Ugly UI is not inviting to be used. However, let’s stop for a minute – what is the actual importance of these issues for your product? Are they more important than a security bug where user can access different user’s account by changing their user id in the URL?

Sometimes we are wasting our energy, effort and even nerves with bugs which are for “polishing to perfection” rather than making the product better.

Think for a moment: what is the main purpose of the product?

The art of being a good tester is the ability to ask good questions, so let’s ask ourselves some questions when we test:

  • Does the product work as expected?
  • Are there any areas which may cause trouble and were not thoroughly tested?
  • Does my testing concentrate on making product better or perfect?
  • Do we (testing + other departments) have time to polish the product to perfection? (If yes – yay, there is time to fix minor issues as well!, if no – then concentrate on the important functionalities)

Sometimes you have to let go of the minor bugs – there are more important features to test/improve. Be smart with your priorities: work on making the product better, not perfect.

Advertisements

Dear tester! Others care about quality, too.

Dear tester,

I know that sometimes it feels like people you work with just want to mark the cards in JIRA as Done without proper testing. Sometimes they tell others “…once it passes the QA” or create tasks for you subjected “QA X” like it is being done just because “they have to”. The feeling of annoyance caused by urgency to complete the task immediately without reporting any bugs is inevitable because of the way they tell just to “pass the QA”. I did write of that before on why “pass the QA” makes me cringe, so I can feel your pain really well. Especially, that even after me trying to explain QA vs Testing vs Checking many times in my company and clarifying, the very same wording is still being used. I would like to share with you a story that happened to me which made me think that sometimes we exaggerate a little bit assuming that others do not care of quality as much as we do.

Today one of developers in my company came back with initial implementation and results to “QA”. The task was fairly simple – there is a lot of data generated by an algorithm and we should check it (I’m using here check consciously as it’s not really testing at this point): does it make sense, what patterns of fault we notice, does algo actually work? All of this should evaluate the quality of results produced by this new algorithm.

The wording of this task’s formulation and documentation with the data had QA mentioned around 5 times in various forms and I’m sure you are familiar with most of them: “data to be QAed”, “for the QAing” or my least favorite “pass the QA”. These terms do not feel too good as they are not correctly used and it may feel slightly insulting sometimes that your colleagues may not bother to even understand what you’re doing. However, you cannot teach all people to use the terms and it’s important to let it go sometimes. Remind yourself that we all have biases (and I do have a story on Managing your biases which made me slow down a little bit before judging). I decided not to exaggerate and think from that developer’s point of view: we both know what he wants as a result – the quality should be evaluated even if he is using the wrong terms.

Some colleagues may use the wrong terms and confuse testing/checking/QA, but don’t go and nit-pick on that. Words matter, but not everyone cares either how to name the rose: all you can do as an empathic quality specialist is to show people that you are open to explain to them, but only if they want to. 

This is not why I’m writing to you, though – this colleague of mine may have used the wrong wording, but letting go of that wasn’t the main takeaway I got.

When the colleague created the task description, it lacked one thing: any description of implementation details of algorithm. No documentation was yet created, no code mentioned, only thing provided was the generated data and vague explanation what should be done (compare columns and say if it’s okay or not using some human sense and research on each of options).

I really wanted to see implementation details: how else can I assess actual risks? Maybe there are areas and patterns that are design flaws and can be seen before even looking at the data generated. This developer tends to work alone as well, so there isn’t much of code review going on.

When I asked if there is any documentation on this algorithm, this was the response I got from the developer:
“Not yet, this is not ready for production yet. When it passes QA there will be a documentation page with all the changes that have come out of the QA process.”

This wasn’t something that I expected to be honest – I replied that to do the QA process we need to know the implementation details and this shouldn’t be made visible only when the algo goes to production. We shouldn’t check in the dark.

My reply has made this developer write to me personally and the words that were used by them again were a little bit rough I could say. The arguments on why the documentation wasn’t created were that “it is too big overhead” and then eventually “it seems that we disagree on the QA process here: for this task, there is no need for implementation details”. How would you react to this, my dear tester? Developer is claiming that as someone who is hired to test and give quality evaluations you shouldn’t look at implementation details at all.

As someone who recently encountered several design flaws in built products which caused issues and could have been spotted years ago, I felt ridiculed. Of course testers or QA (whatever way people want to call these specialists) should see implementation details. Is this developer really thinking that their design and implementation is perfect that we should look just at the results produced?

Issues can be spotted when getting to know algorithms and implementation: you may spot a logical error which causes certain bugs before you even look at the data obtained from running the algorithm

I stood my ground then, though. I tried to explain that I would love to see the implementation because it will help me to do the “QA processes” faster, more efficient and may display me some of issues before I actually look at the data. I want to be familiar with what it is actually doing.

And, to my surprise, it worked. This very same developer who was fighting that QA does not need any details on implementation shared with me the code they wrote to produce the results. It turns out that they thought I needed detailed documentation, but even code was enough which could easily be provided.

In the end, I realized that I could have given up. I could have closed myself up and exaggerated thinking that it’s only me who cares about proper quality judgement and people just assign tasks blindly without even considering that there may be issues in their logic of implementation. I could have felt hurt by the words used and impressions I got from this person, but in the end, even if we spoke in different terms, we both aim to finalize quality assesment (not to pass the QA, just understand if this implementation is good enough). I stood up for myself trying just to do my job better and I got help even if it took an extra step.

So, dear tester, believe that your colleagues are there to help you – you all want your products to be successful and of great quality. It is not only you, just sometimes others don’t know what you exactly need to do your tasks – open yourself up and ask for it. Only by sharing your needs and communicating you can make others understand your tasks better. 

 

 

 

My first pair testing session – Day 21 of 30 Days of Testing

PAIR TEST WITH SOMEONE

Some time ago I’ve read a great article by Katrina the Tester on Pair Testing. This topic since then would often pop up, but I never had a chance to actually try it out.

I am very grateful and lucky because I have super supportive colleagues when it comes to 30 days of testing challenge. Not only that they show a sincere interest and ask about the progress, but they also agree to get involved into some of the challenges. At the very start of July, I was talking to one of our great front-end developers and mentioned pair testing as one of the challenges. He immediately said “I can do it with you!”.

Ironically, today I did not have any specific task which we could pair test, so we thought of testing an existing feature.

In the very start it was a lot about seeing how he does testing. It is fairly different than my testing, so it was very beneficial to me. He showed how he configured dev environment locally for testing so he could fix any coming up issues as well immediately.

Then, we tried to specify on how this feature should be tested. This involved some UI tests verifying requirements, but not only that – we looked at monitoring, requests being sent. Usually, when I test, I concentrate on UI when the feature is mainly seen on the UI, but it was great to see how differently the very same feature is viewed by a developer who knows what events are being recorded and which requests should be sent for the feature to work. He as well showed me some of his unit tests for that feature.

To sum up, we asked when we can say that we have tested the feature completely. How much is enough? What should we do to be sure that it’s well tested?

This question is a challenging one. We decided that in addition to what we discussed, we should as well just try to be users for a while and try to use the product. Funny enough, trying a few of scenarios – we found a bug. In a tested released feature in production.

Here came in a very important point in testing: while testing one of the features you can have blind spots on some of inconsistencies on other existing features, so it’s best to stay alert and check for regressions which concentrate not only on the new feature.

Pair testing definitely exceeded all my expectations: I learned a lot new ways how to test, and, seeing the product with the help of different set of eyes even helped us to find a bug!

And, it was all about improvisation and sharing knowledge rather than dictating the rules.

128istqb

Illustration above taken from brilliant Cartoon Tester.

 

 

 

 

What I’m doing at work: Day 2 of 30 Days of Testing

TAKE A PHOTO OF SOMETHING YOU ARE DOING AT WORK

I had to cheat a bit on this one: it’s Saturday today, so I am not at work. I had to take a picture yesterday, however, I’ll share it now.

13570268_10209801873230058_84405356_o

This looks like a wonderful job for a woman – doesn’t it? All day looking at clothing. My boss sometimes remarks “You’re again shopping!”.

I am testing fit solutions and visual similarity algorithm. It is a pretty exciting task to do with many layers and challenges. This picture just reflects a common screen view you can see on my computer when you pass by.

Rebirth as an Omega Tester

Last time I wrote an entry in this blog was 2 years ago. Definitely, a lot has changed and it’s time for a rebirth.

Reasons why I got silent were simple: I got tired at my work, it became monotonous and lacking of challenges (later reasons became simpler: I forgot about the blog, it was not a habit anymore and I was pretty busy).

More than a year ago, I decided to look for a new job. With a bit of luck and accidents, I stumbled upon an ad for a first full-time software tester in a startup. As I have met all of the requirements and job sounded challenging enough, I just had to apply even if it was based in other country (where I have never been before).

Response was quick and inspiring: I got homework (I always think it’s a great sign of a company – you can prove yourself with actual task). After that, I had multiple Skype interviews and… quit my job (which is definitely a tough thing to do when it’s your first job), packed my bags and moved to a new land of opportunities.

I must tell you that this change was the happiest change in my life. Now it’s more than a year that I work as an only full-time tester in a promising and exciting startup! I have many stories to share, so, it’s about time to make a rebirth.

Changing from manual tester executing test scripts in a big international company to being a sole tester in a (sometimes chaotic) startup is a huge difference, but, oh boy, how worth it is! Advice for you:

If you ever wake up at least a couple of days in a week thinking “I don’t like my work and don’t want to go there” – it’s about time to change your work. New opportunities are around the corner waiting for you.

And, if you find yourself wondering, why did I call myself an Omega tester, there is a great article by James Bach: Omega Tester: Testing with a Team One. Definitely worth reading it!

Fail is for the Winners Part Two: Accidents

Best advice for a software tester: make mistakes.

When it comes to black box testing (testing software from user’s perspective), let all your imperfections loose! You don’t need to be a professional user all the time, you can be a clumsy user misclicking something. This kind-of-out-of-the-box-testing-strategy hits security parts. And, trust me, those are the most interesting bugs which can even break the system.

Most of the best bugs I found were found by accident. I did something that I was not intended to do, and, bang – beautiful bug appears. Of course, usually bugs like that are really difficult to reproduce because being such a free spirit and doing something unplanned, you cannot remember exactly what you did.

Trying to reproduce that bug is a long journey sometimes. However, it is wonderful. Your adrenaline is rushing to catch the leg of that bug again. You are a little bit bitter that you slipped it and cannot remember how to reproduce it. And, you just cannot give up because you had it in your hands, and, it’s one of the best bugs you found. The ‘destination’ of finally finding that accidental bug is more than words can describe. It fills you up with harmony and self pride.

Accidental bugs are the best in my opinion. They are the ones that define the purpose of manual testing:

Humans do mistakes and that’s why automated testing will never substitute manual testing done by imperfect humans.

Set yourself free! Get crazy with your software. Misclick, choose another option, experiment, and… break it.

Fail is for the Winners Part One: Neglected Bugs

“Ever tried. Ever failed. No matter. Try again. Fail again. Fail better” – Samuel Beckett

Sometimes software testing is like a roller-coaster ride: fails (bugs) make you happy, but at the same time sad, and… to make matters worse: sometimes you may fail by reporting something that is not actually a bug.

At the very start of my work, I was scared to file a bug in order to avoid a… failure. How would I feel like if it actually figures out to be not a bug, but a feature? Maybe I’m just not aware of how the software must work.

Later on, I got braver and braver…

I have filed more than 100 bugs in past 3 months. In this new project we just started, I’m the tester who has filed the biggest amount of bugs. Does it make me the best tester? No.

The best tester is the one who gets the most bugs fixed.

Of course, filing a lot of important bugs gives you a bigger chance of actually getting a lot fixed, but… another side of this is that there will be a lot of Closed bugs which will be neglected and it makes you feel a little bit stupid for filing them after developer’s responses like “it works as designed”, or, “cannot reproduce”.

It takes time and practice to learn how to deal with them, but…

Don’t get down because your bugs were neglected. If there is a chance – file it as a task or enhancement. Even if bugs sometimes get closed for various reasons, it is always very useful to put attention on them. What you are doing is great and maybe first 2 times that bug will roughly be marked as Works as Designed, but in future releases someone will take it as an amazing suggestion to improve the software.

And, remember, don’t let failures bring you down. Don’t be scared of being laughed at. It would be way worse not to do anything and sit silent.

There is only one way to become a professional at what you do: Fail.
Fail a lot and never stop moving forward. Only leaving your comfort zone, you will reach some of the most amazing things in the world…

How to Attract Interesting Bugs: Use your Talents and Be a Software Terrorist

The release was supposed to be tomorrow, and, today I found a really interesting bug. I think it is the bug that I’m the most proud of so far… or at least today. (:

I will tell what that bug was like in this post, but before that I must say how my colleagues reacted after getting an e-mail with developer’s commit and bug’s description: “What? Why did you do that? How did you manage to find that bug?”. And the answer is pretty simple: I try to make my testing nice and not monotonous…

Collect the best pictures: art, photography, travel, whatever you like and use it when you need to test something with Insert Picture, or, with image files in general. 

This seems so unimportant, right? Who cares what kind of picture you insert while testing. But, hey! You get tired of those default Windows pictures with tulips or penguins. And, let me tell you a secret – a picture which you like looking at can do wonders. Your imagination strikes with all this creativity/nice view in the image file. You suddenly feel happier and testing is more fun. To make it even more interesting, you could save quite many images you love with random names, so each time inserting an image you wouldn’t know what you are inserting exactly and get a bit surprised.

Practice your foreign language while testing. 

Modifying a file (usually it’s a text modification) may seem such a boring task – just type in ‘Aaaaa’ or any other ‘asafsfjdsf’. So dull, repetitive and what a mess it creates when you try to save a lot of files with random file names like that and you are never sure whether this random name is unique. Then again you need to change it, oh, boy, so annoying.
One way could be to think of interesting words, people to name your files, but…
Why not to practice your foreign language skills? Try to use testing as a chance to repeat newly learned words and just have fun while doing so. 🙂 It will definitely be more fun to write a funny word in a language you are studying instead of ‘Aaaa’. 🙂

Take inspiration from music.

If you lack of words in a foreign language, why not to listen to a radio of that language you are learning while testing? Then when cannot think of any word – just write down the one you hear in the song. In general, music can help to create a good vibe and mood for testing. However, this is not for everyone (for example, I cannot listen to it all the time, I need to get silence, but sometimes it really helps to refresh my mind).

Be a Software Terrorist. 

A what? It’s time to tell about my lovely bug which made our release shift even more and a developer send me a message saying “You’re a terrorist” (I am not a terrorist, just a software terrorist…). 🙂
Try things out, have fun and you will find bugs like these…

***

I turned on the software, inserted a lovely travel photo as a background picture, and, thought of a song I just heard on a Swedish radio (I am just a beginner, and, they say it’s really good to hear the language constantly, so I try to listen even if I understand only a few words). It had lyrics like: “wherever i turn in the world i’m standing here with empty hands”, so, I thought that this time my modification will be “empty hands”, but in Swedish. I wrote: “tomma hander” (without Swedish characters). Then I wanted to check some shortcuts and pressed Ctrl and got surprised. We test integration of products like that with a file system product of ours. And, there showed up the login to our file system. This is weird, I thought. Ctrl itself should not make any dialogs open. After log in there came integrated Open dialog… So, basically, it meant that if a user wants to save the new document with “tomma hander” and presses Ctrl+S he won’t be suggested to Save the file, but Open it from that file system product. This may be very annoying for a user working with large files and using shortcuts.
I would have never found it if I typed in ‘asdajdaskdj’… The bug itself was as follows:
every time typing the letter o in the document (either it alone, or, in the text) and then pressing Ctrl key would work as Ctrl+O and give log in to our software and integrated Open dialog. 

A secret of a happy and productive tester: be not only passionate about testing, but fuel it with other passions you have in life. 

 

Psychology of a Software Tester: Expectations and Moral Lessons

Even if at the very start my colleagues laughed at me when I felt bad that our software is ‘buggy’ telling me that ‘it’s our job, silly, be glad’, but… I still have that good versus bad fight inside of me. It is good that I find something bad which means I work pretty well, however, I work for a company, for that specific project, for the specific software. Is it bad if the software that I work on (our product) is bad? 

I am not the only one with questions like that. Understanding how common this ‘crisis’ is between testers, Cem Kaner, Jack Falk, Hung Q. Nguyen in their book ‘Testing Computer Software’ gave a lovely introduction which even had a few psychology notes there. I am going to share some of my findings. 

You get what you expect. 

Imagine that you’re a scientist and you just discovered a new wonderful theory. You have a lot of methods which you could apply to prove that it’s correct. However, take a minute, think about it… Would you choose methods which could make your theory fail? 
There actually has been a research about how intelligent experimenters unconsciously avoid test experiments which show that their ideas are wrong (Rosenthal, 1966). 
Similar example showing how expectations rule our mind is proof reading. We expect words to be written with no typos and we manage to raed wdors the way we expect them to be.
Exactly the same thing happens with a software tester…. If you have an attitude that your software is good – you won’t find any bugs, you will miss the ones that actually are there and take them as details or accidents.

You have to want software to fail. 

This sounds rough. I know. It is your software, you work on its quality and you want it to be bug-free. But your attitude must be very strict and judging – in order to make your software really good, you must put high standards on it.
It just reminded of a quote which I liked way before I was a software tester (maybe it’s a sign!) by Baltasar Gracian: 

Attempt easy tasks as if they were difficult, and difficult as if they were easy; in the one case that confidence may not fall asleep, in the other that it may not be dismayed.

You can do your job well (even if it will never be perfect). 

There are always bugs left. This is a one more ‘sore’ point for testers. You will never catch them all. Therefore, you will never be a perfect software tester. You will never do your job completely, but… you can do it really well. 

Sometimes it’s hard, but hey… Being a software tester is way more than catching bugs, it teaches you… life. It may be difficult at times, but noticing problematic areas in software is actually a path to a better software, and, even if testing does not win you any popularity contests, but secretly, everyone is grateful that you point out the defects which could have caused a lot of trouble. 

Love Story between Software Testers and Developers

It has been a pretty hectic period of time in my life: loads of bugs, and, experience. I’ve been thinking what I should write this post about as at the same time I have so little, but so much to tell that I don’t know where to start. After work, I decided that this day and my feelings about it would be best described in a post about a relationship… Between testers and developers…

The Love and Hate Relationship Between Testers and Developers

From my first day, developers for me were an object of investigation. I would try to get what kind of relationship I should expect us to have (sounds serious, doesn’t it?). I am a pretty funny and silly person at times and my contact with developers has been somehow… awkward. I would talk about my beloved bugs with lots of inner passion, love, and, laugh… I guess this may be called a bit unprofessional, but it’s me, and, I love my job with all its problems which I take as simple challenges, and, allow myself to joke around. However, this sometimes creates misunderstandings and developers are not used to see someone fight for their bug so much. 

1. Developers may hate testers only in two cases: 
    a) bug was found
    b) bug was not found
To sum this up: no outcome is good enough. 

Of course, we are a team. We try to create bug-free software and seek for the best product possible.

However, in the case a) usual scenario is like this: a developer writes to the tester asking whether it really is our problem. Maybe we could blame something else? Maybe it was there before, not only in this release. Check this, and, that, and, this also more.
While dev leader also always asks us how many bugs we are going to find in the future, how many they should expect – my answer was very simple: as many as you have done in your code (he is always very confident, but that time he blushed and got silent). 
Image
In general, developers are lazy. They try to avoid work as much as possible. Sometimes testers have to explain 5 times and share screen to make them fix bugs. Also, it happens that they don’t bother to read notes, details, comments, on bugs we give them. Just reads the start and assumes. Afterwards, marks it as cannot reproduce without even trying. Or, another possibility is, that they blindly follow steps and don’t even waste time to try to get the same result in a different way which is quite obvious, but it’s not written down. 
I must admit that in a case, I hate developers a little bit, too (I love them more, though!). 

In the case b) as I mentioned before… Testers are going to be blamed for leftover bugs. Developers may just put all responsibility on our tiny shoulders by saying: your fault that you did not see that obvious bug and now users suffer, bad tester.

2. Developers and testers actually silently love each other: for finding bugs, fixing them, and, making a good product together.

Today I got so amazed by our developers as they fixed so many new defects quickly. Of course, my happiness did not last too long as some of their fixes had to be rejected and did not work properly. However, I really feel that it’s magical that they manage to fix those bugs that we just write down. Some bugs are very tricky, and, developers get to know them, understand them, find their reasons, and, fix them. I feel happy when my bugs get attention, are taken seriously, and, are taken care of!

And, even if they will never admit that because of 1 a), but… I believe that they secretly love testers back. A little bit. For helping them improve and giving a lot of interesting riddles (aka bugs) to solve. 

I am not sure whether any film company will buy a script for a movie about a relationship like that, but, I find it fascinating.