Why testing and how?
Testing is a practice with several objectives, the obvious one is to verify that your job is done correctly but also is a trigger to…
Testing is a practice with several objectives, the obvious one is to verify that your work is done correctly but also it’s a trigger to change your designs.
Software testing is an investigation conducted to provide stakeholders with information about the quality of the software product or service under test. Software testing can also provide an objective, independent view of the software to allow the business to appreciate and understand the risks of software implementation. Test techniques include the process of executing a program or application with the intent of finding software bugs (errors or other defects), and verifying that the software product is fit for use.
https://en.wikipedia.org/wiki/Software_testing
In the above definition there is nothing said about how to test software, can be tested in different ways:
Manual testing
Automatic testing
Manual testing is using the time of someone to go into an application and try to find bugs.
Automatic testing is making developers to write code to test the production code they are doing.
How can we select how many tests must be done manually and how many automatically, which are the KPI’s to use?.
Feedback loop
The tests have as the first mission to tell you that you have an error, so the sooner I have that information the sooner I will fix it. This means less money lost creating the incorrect solution. But how can we compare Automatic Tests vs Manual Test to select between them:
How much time do you need to cover your tests?. If the time is long those tests will be executed less frequently.
How much can I trust in the result of my tests. Are those tests failing because of the test itself, or because of the code tested.
Where is the error if one test fails?. A test is better if it can point me directly to the part of the code that make it failed.
What is the cost of executing those tests?.
What is the time of creating those tests?.
When and where can I execute those tests?. Can I execute them locally, do I need any kind of infrastructure?.
How easy is to refactor the code with your tests in place?.
For one test you want to do, think about these values. You will achieve the concussion that automatic tests are in general a better approach. It’s very difficult to avoid manual testing, but we can minimize the number.
Automatic tests
There are several types of automatic tests, I’m not going to talk about all of them, just a subset:
Unit tests, they test units of our code. We will discuss what is a unit.
Integration tests. They cover several units of your code and the integration between them. Sometimes they require some infrastructure to be executed.
E2E tests. They are trying to test from the interface of our whole system the final output (this can be the user interface, database, etc).
Based on the above explained characteristics of testing, unit tests are:
The fastest tests to write and to execute. The cost of executing unit tests usually is super low, a hundred of unit tests can be executed in a few seconds. Write unit test can be low or a lot depending on your strategy. Low if you do TDD, you write your production code through your tests. This is giving you also feedback about the design, and a lot of benefits more. If you write tests after you wrote your code then the cost of writing tests can be high.
If one unit test fails you know the place where is failing and why, depending on how well is written the test and your design this is accurate or not. But in general unit tests are the most accurate tests.
After Unit tests, we could consider Integration tests as the second best option and the third one should be E2E tests. This is the test pyramid, isn’t it?.
Inverted Test Pyramid
Why is bad having much more E2E tests than Integration than Unit tests?. The problem is the time to execute all those tests. When you are in this situation you realize that a developer cannot execute all the tests locally at any moment. So we usually decide to split the test execution in different phases. Unit tests are executed while developing, integrations in the Continuous Integration server and E2E once every X days.
When you have an inverted test pyramid the Integration and E2E tests are the ones that are better to find your bugs, but those tests are going to be executed less frequently than your unit tests.
The cost of fixing a bug depends on when did you find that bug. The sooner you find it the cheaper will be. This is one of the reasons for limiting the build time to a threshold and using that limit to realize that your test approach can be improved. Normally in an inverted pyramid the limit is not taken into account, this is a symptom of the inverted test pyramid.
On the other hand E2E and Integration tests usually requires debugging to understand what was the error introduced. Debugging is one of the most time-consuming tasks a developer can do. I started learning TDD because of the pain of debugging my code.
Fast tests and executed at any moment is great because they open us a door of possibilities. If all my code is documented with my tests and the build is fast then I can start thinking in Continuous Deployment. Explain the benefits of CI/CD is too much here, but you can read from them in this article.
Summarizing, time to execute your tests is crucial for the quality of your code. A fast test suite (seconds) with a good coverage will help you to refactor your code easily.
Unit tests
Wait, but what is a unit?.
The problem with unit tests is that developers have matched the unit word with classes in OOP languages, functions in functional programming, etc. So they are trying to cover each class with tests. It is not easy to define what is a unit test explaining what is a unit but is easier to define unit tests based on their characteristics:
Fast to execute.
Isolated from your infrastructure.
Each test fails for only one reason.
To go deep let’s imagine we have some code to store a user in our system with this design:
A controller to receive a call from another system to store the user.
One adapter to translate data to our domain.
One service that needs to calculate some stuff for our domain, like the taxes our client has to pay.
The repository in charge of talking with our database to store the user.
We could write tests for each part of the system and mock the rest but what happen if I want to refactor this design to another approach, how many tests do I have to change?. The number of tests changed once we need to refactor is a good value to know how difficult is to change your system without risk to introduce a bug.
Let’s imagine that I want to test my controller and my controller is using the adapter like a collaborator. We will have tests for my Controller and in those tests the adapter will be replaced by a test double. My test double will take some assumptions of the real adapter, this is an indirection, I’m not testing the real adapter (this is the cost of isolation). Now imagine I want to refactor my adapter to pass a new parameter then I have to change all my test doubles to accept this new parameter. I’m changing my tests for a refactor, that increases the risk of introducing bugs.
In someway setting the unit as a class and have unit tests for each public method in a class is limiting our capacity to refactor so is limiting the evolution of our design. This practice pushes us to maintain the first design done by the first developer working on that task.
I don’t want misunderstandings here, to refactor anything you need tests covering that code. I’m just saying that a refactor over an internal part of your system should not produce always to fail 10 different tests (I’m also including here compilation problems).
So where to put your tests doubles is the key to reduce the number of tests rewritten because of your refactors. Basically our tests doubles are defining the borders of our units. The unit in this example could be everything until achieve the repository and the repo could be replaced by a test double.
With this new approach we are giving new devs the possibility to refactor all between the controller and the repository without this kind of problems. So the unit is not defined by default, it depends on each case, it depends on our design.
Even we could say that the tests are telling us how difficult is for our system the experience of our client.
But the opposite is also true, sometimes it’s better to introduce a collaborator and test it independently just to reduce the explosion of tests.
I suggest to set the unit with the borders of your design as the first option. That means testing the entry points of your domain and mocking your interfaces (repositories). If you have a test explosion because of one collaborator then just create tests only for this component and mock it in the rest of your tests.
This is why TDD is great because while you are writing code you are receiving feedback of your design. Just listen to that feedback and you will improve your design and your tests.