Automate all the RIGHT things

“We don’t need this E2E test if all teams have their pipelines green” – hearing this made me uneasy and slightly annoyed. I went on a tiny rant about automation, checks, tests, integrations, and how pipeline being green may not mean that the product is perfect. What do I mean with that? I believe that we need to make sure that our test automation is correct, extensive, and meaningful to give us a good foundation for product quality.

With the arrival of DevOps, many companies started adopting Continuous Integration, Continuous Delivery, Continuous Deployment principles: there are green/red pipelines, quicker releases, faster feedback… To make sure we build in the quality, more and more teams are learning about the advantage of creating checks for their code (intentionally, I am avoiding the word tests here, even though many of definitions for specifics are including tests like unit tests, integration tests, contract tests, etc.). If there are enough of automated checks, we would have a better safety net preventing any major issues and allowing us to release faster to production.

With this comes a lot of trust in those checks, though. A lot of people have a tendency to believe the correctness of checks if they are there, and that’s a danger zone.

What if the check for a certain things: a) does not even exist – all the deployment will be green; b) exists, but does not check what it should exactly (for example, the check verifies the wrong assumption and just confirms what was wrongly understood).

Automation checks should be meaningful

Checks should be created correctly, not just for the sake of having them. So, as a result, a healthy test pyramid, has various levels of checks – not only unit tests are included, but E2E tests as well. Their count may be way smaller, but verifying a user journey can be extremely beneficial  – the approach goes from user perspective and may reveal some issues which were not covered in lower levels.

Question the validity of checks

You could very easily write wrong unit tests. Imagine that for some reason you have a strong belief that 2 + 2 should return 5, so you implement an addition function which yields exactly what you think is correct, and then you write a unit test to verify it which passes. Tests are green, pipeline screams yay, but is it correct? Not at all. Only human judgement writing the checks can make sense if they are correct. A nice article on the correctness can be found here with this example and more.

Validity is a very common problem I notice, sometimes it could be that the product does not work as expected, but checks created during implementation pass. The passing of them is not as trivial as 2 + 2 equaling 5. Sometimes the mocks used in automation can be silently misleading.

Observe the right level for checks

If you can write a unit test, is that big Selenium suite really necessary checking exactly the same functionality? There may be cases where it is when the product is very UI heavy, but in most of the cases it is very useful to question if test automation we are doing is being done smart, rather than done in order to have something. Questioning levels of checks can be a good start.

Aim for the healthy amount of checks

It is easy to make the pipeline green if there are missing checks – if you never write a test, how can it fail? This reminds me of this meme we once printed for the team I was in:


On another side, we also may over-automate, so we have to balance our checks. How much should we automate? I really like Alan Page’s stock automation phrase (his article which introduced it): You should automate 100% of the tests that should be automated”.

So, if I had to summarize my thoughts, I’d say:

Instead of looking if pipeline is green or not, implemented test automation should be observed, too: its meaningfulness, correctness, and balance on certain test levels & amount. 

Assumptions are a breaking force – if we assume that every team has a green deployment, it does not tell anything about their quality apart from the fact that their written automated checks passed.  This does not assure the correctness of the written checks or that in general they make sense and there is a good coverage.


How “Clean Code” by Robert C. Martin helped me to get my automation tests going

Even if I am a pretty decent bookworm, it took me months to finish “Clean Code”! It definitely was not an easy evening read for me (even though I do know some people who read it overnight) and it was the very first book I have read on programming.

In the first years of my testing career my tasks were limited to setting up my own testing environment and manual testing (very often with manual test scripts). No wonder it started making me feel like a monkey and bored to death!

When I changed my job to being the lone tester in a startup, my time resources became very limited. Manual routine tasks started to hurt a lot, so automated UI tests became a long term plan. Luckily I have wonderful colleagues who gave me a lot of needed support to get going.

In the very start as a very beginner programmer (never done it before except in university basic classes) and a practical person, I just created some automated tests based on examples I found. My main purpose there was to make it work. If it works – then I move on to the next thing I want to automate, google, find a solution, modify and move on.

Then, the fun began. We did want my automated tests to be actually used and scheduled. For this purpose GitHub was chosen to store it and for that I had to start creating pull-requests and get them reviewed by a senior Java developer. It was definitely terrifying. First pull request I made got 30 comments in one review session.

The comments were super useful, but it made me realize so well that to make it work is not enough – we need to make it maintainable, readable not only to me, but others, and scalable for the future. There were so many things I did not know about programming. And there came an advice from the developer for me to read “Clean Code” by Robert C. Martin.

I did read it and I realized how important it is to actually write clean code. It definitely should be a long term goal of everyone.

Some very important points mentioned in the book which I loved:

  • Meaningful names: Code has to be readable and understandable to others as well, so use intention-revealing names without disinformation.
  • The single responsibility principle (SRP): there should be only one reason (responsibility) for class or module to change.
  • Tests should be clean: readable! One assert per time if possible and single concept per test.

“Clean Code” is full of useful advice on how to make your code cleaner. It includes some easy tips like the ones I listed above, but it also goes in depth and looks into code examples and tries to make the code cleaner hands-on.

Getting software to work and making software clean are two very different activities.

I think the quote above sums up the main idea of the book. Very often whoever is doing programming forget this very important aspect: clean code can make your life way easier in the long run and even help you to avoid making mistakes (or bugs!).

Months later, I finally finished this book (reading bit by bit) and my automated tests are now scheduled for nightly runs. I am far from being a great programmer, but, due to a lot of useful code reviews and information obtained from “Clean Code”, I have improved my code quality a lot and can make changes way easier than before.

Even if you are a not a full time programmer, but have to do some programming, do not underestimate the power of clean code. It is not only about making it work, it’s about making it work in the long run, too. It will scale, get used and be easy to maintain if you work on writing cleaner code. 

Why you should improve your test automation code – Day 5 of 30 Days of Testing


Did you know that it’s quite easy to find great testing blogs following just one blog? 5 Blogs is the one! Everyday there is a post with 5 best blog posts read. This blog has been a huge help for my 5th day’s challenge. I got a big list of great articles and chose Learn Development Practices To Improve Your Test Automation Code by Jeroen Mengerink.

This article pretty much expressed what I’ve learned myself. I started coding at my current job wanting to automate some Web UI tests with Selenium. In first month I was googling a lot, reading practical examples and playing around with code. My main focus was to make it work: if I want it to click on a button, it should do that.

I was glad to have something to show: I could run my code and a lot of operations I used to do manually would quickly be executed in the browser. However, even if my code worked, but it was not scalable at all.

After seeing my interest in coding, company’s developers decided to help me and move my project of automated tests forward. This is how my code started to be reviewed. I must admit that I was terrified. My first pull request got around 30 comments about structure, naming, all kinds of conventions. The main advice from the developer was to read “Clean Code: A Handbook of Agile Software Craftsmanship” by Robert C. Martin.

With time passing, I did read bits of the book and started noticing patterns of what mistakes I usually do and a lot of comments under my pull requests would be known conventions which were mentioned in the book. I improved my tests according to feedback I got and… it felt great. I really value all kinds of comments as usually they are helping me and my code to be better.

Thus, I definitely can say from my own experience, that if you want to do test automation, then you have to learn some software development practises as well. This is exactly what article I’ve mentioned in the start is saying: software development conventions are very beneficial to you. I cannot agree more.

Development practises ease your automation work: it is easier to add new tests, you don’t get lost in your own code (with time it’s going to expand), it is easier to refactor your code and change it when the need comes.

Code improvements even become like an addiction in the end – you want to refactor more and more to make your code cleaner and nicer to work with. Do not forget to invest some time to that because if you don’t – you will lose way more time trying to maintain messy code.