What is a test automation pyramid?

Test automation is there to ensure software application works as expected. But in order to do so, you need directions to run the automation processes. The test automation pyramid helps in this respect by organizing and giving structure to the whole test cycle. Thanks to this hierarchy, you can streamline the testing processes, manage time more efficiently, reduce testing effort and the testers get a time-tested blueprint to shape their projects accordingly.

The concept of a test automation pyramid originated in the book by Mike Cohn "Succeeding with Agile"”. The author explains the difference between traditional testing frameworks and testing for agile development.

But what is the goal of this test automation pyramid (sometimes called also Agile Testing Pyramid)? It has been designed to find the equilibrium in testing. How? By representing the best feasible case for software development.

It is extremely helpful to know the definition of the test automation pyramid and how to execute it step by step for any organization who implement their testing projects. A test automation pyramid, in an easy and transparent way, represent visually a testing methodology that can be applied to a QA strategy. Such a visualization enables teams to determine the amount of time necessary to focus on different test type, testing categories and groups.

Typically, a test automation pyramid works on three levels:

Unit testing is the widest foundation layer in the test automation pyramid. It is because most testing is done at the unit level. Here, both developers and testers break down larger functions into smaller pieces for testing and validation.

Here, testers validate functions which work together and API and services that enable end-user functionality. For tests exceeding the scope of unit tests, it is best to use tests that communicate with the software application at the service or API level. Most of the today’s applications offer some sort of API connection. It can be either via the actual programming interface or a web service. The tester can use API to test the application. These tests are definitely quicker than UI tests, much less brittle (APIs and service interface change less often than UIs) and give fewer false negatives.

At the top of the testing pyramid, we have the UI-level automated tests. The general approach is to run as few of UI tests as possible. They take long, both in terms of test case development and test execution. It is recommended to use this form of test automation only when the UI is actually being tested or if there is no other option. The final layer comprises of components that have been individually tested. Less end-to-end testing is required and that is why testers can now shift their focus to the broader user experience rather that feature function and other, more granular, components.

We’ve mentioned that companies who want to succeed with their testing project should follow a test automation pyramid. But why is it so important? One of the reasons is the complexity of software application. They consist of multiple features, API usage, etc. This means that, inevitably, testing these applications have also become more convoluted. The other reasoning behind using the testing pyramid is that you don’t want to delay deployment. Following an organized and structured approach facilitates but also accelerates testing process itself.

There is no way or need to automate everything. And manual testing takes more time and most importantly, is far less accurate. In order to deal with these bottlenecks, you need to make the process efficient and smart and this is what testing pyramid is all about. Test automation is not all. You also need a plan to follow it through. With test automation pyramid you get specific guidelines that provide you with better testing results and allow you to minimize the effort involved in testing planning and executing testing activities.

Testing pyramid bring order to chaos. Its main benefit is to make the testing process as efficient as possible. However, using the testing pyramid is one thing. What else can you do to improve the testing and reach the most efficient stage (which ultimately also means minimizing the operational costs)?

1. Choose the right automation tool: for your software but also for your business needs
2. Plan and decide what to automate: there is no sense to automate everything. Plan which test cases will be involved in the rest automation pyramid and what the scope of the tests will be.
3. Prioritize the tests: obviously, with the testing pyramid you have it all sketched out. However, bear in mind that sometimes you might need to use the inverted pyramid (more about this below) or allow some additional time for certain test types.
4. Write clean test code. There is not really much to add here. If you engage more developers, make sure they stay on the same page and cooperate.
5. Prepare good quality test data. There is little or no room for flaky tests or rerunning the tests due to irritating bugs. Make sure that you test cases and test scenarios are based on solid and good quality test data.
6. Avoid test duplication. This is related to the automation testing tool. Make sure that the tool allows full repeatability so that you don’t have to do any manual makeovers.
7. Do exploratory testing. If you want to double and triple check that you have not missed out on any important product risk areas, consider exploratory tests. They will give you all the information you need regarding the achieved coverage and the detected problems.
8. Put tests into your deployment pipeline.

For certain type of software – such as ERP system – an inverted testing pyramid is not an anti-pattern but a model to follow. There are also some circumstances where inverted test automation pyramid is necessary, e.g. in order to rebuild an API in a different language.

In some extreme cases of the inverted test automation pyramid, the middle layer does not even exist.

However, before implementing one test automation approach you need to think very carefully as a lot of money can be wasted on development and maintenance of the automated test cases.

Some other situations where you might want to invert the testing pyramid are:

1. Integration to a third party API: you need the lowest level tests to ensure that the API has stayed the same. Most of your tests will be end-to-end tests to ensure that the application correctly handles responses from the API (and formats the data for the API)
2. Integration to a specialized physical device: here, most of the end-to-end tests are likely to be manual but the inverted model still applies. Specialized devices are not necessarily good candidates for automation and simulators might not give accurate responses.
3. Testing communications between separate applications/ systems/ APIs. The testing pyramid will be inverted if the communication between different systems is the primary focus.
4. Legacy code: if you are working with legacy code, unit testing might be challenging if not impossible, therefore there might simply be no choice in how you automate.

Different shapes of testing pyramid have their own privileges. Here are some of the ones that you might want to consider for your test automation project:

• Cost to run: A full set of unit tests runs quickly. More quickly than end-to-end testing. That is because end-to-end tests often involve complex setups and tera downs and lots of waiting. In the end, a preferred option to be used is a test that runs fast rather than the one that is slow.
• Maintenance costs: end-to-end tests tend to be more fragile and require more control than unit test. If you are planning to invest with a limited test budget, you might want to consider if end-to-end testing justifies the higher upkeep costs.
• Diagnostic vs user-relevance: As end-to-end testing models actual user interactions it feels closer to how the entire system works, contrary to unit tests. However, in case end-to-end tests fails, you have little to go on as to what caused the fire. That is why end-to-end testing is considered to bring almost no diagnostic value. However, unit test provide a lot of information in this respect. You can easily deduce the root cause and do not have to make assumptions. Again, if you have a lot of interdependencies between different component, you cannot rely only on unit tests to verify how the system performs. You might need more tests then.