Although more and more companies switch to test automation, in most organizations manual testing prevails. It is understandable as automated testing comes with many challenges. However, so does manual testing – it comes with very high risk of human error, slow testing speed, lower test coverage and insufficient knowledge of what to test… That is why automation testing is very tempting as it guarantees software quality that cannot be gained with manual testing alone.
What are the most usual challenges in automation testing? We have tried to compile a comprehensive list of those. Knowing the obstacles on the way helps to make a more informed decision.
1. Test automation means high initial investment.
Test automation requires a high initial investment in terms of tools but also infrastructure and resources (mainly professional developers). Sometimes you also need to consider licensing costs on top of that, plus costs incurred in training and onboarding. Even if you decide on an open-source automation testing tool to reduce the licensing costs, you need to invest significantly in learning, training and tool maintenance (not to mention the hidden costs like collaboration, meeting, etc.).
This is the challenge #1 in our list of challenges in automation testing as it seems to be a major obstacle for a lot of Ce-level management when they need to invest in test automation. Roughly speaking, for the majority of organizations, between 30-49% of their budget is spent on test automation.
In order to convince the management that long-term benefits of test automation outweigh an initial investment by far, you need to calculate costs involved in setting up automation framework and production defects. It is also extremely helpful to have a ROI model at the very start. Although there is always a huge payoff when it comes to using test automation, convincing the stakeholders to invest is still problematic. Often, due to budget constraints, test automation is given up (even with clear executable goals and high demonstrated ROI).
2. The test automation strategy is not (clearly) defined.
No proper test automation strategy in place (not to mention the complete lack of it) is doomed to bring the test project to failure. The strategy for test automation is instrumental in the success not only of the project itself but it is also key in showing its business value accurately.
A complete and precise test automation strategy should encompass all of the below elements.
- The risks, functionalities and capabilities and way of communicating these to the stakeholders
- Goals and plans
- Overall thought process and proof of the concept.
- Auditing tool
- Definition and description of high business value tests.
- Definition of automated testing environment
- Test data management
- Technology, tools and resources
If any of those points are not addressed or covered, they might impact the success and the cost of the automation testing project.
3. Test automation frameworks don’t match with the pace of development.
Let’s say you are working agile. To build a test automation, particularly with rigid test automation framework, often takes more time than doing the required tests manually. It takes a very long time to implement code-based tests together with changes in the framework. Moreover, there’s often no time to implement automation and, at the same time, execute testing. The frameworks simply do not keep up with the changes in the application. Testers come up with short-term fixes which, however, lead the frameworks to start crashing eventually. It is impossible to program a framework flexible and adaptive enough for all unexpected factors. There are also a lot of question to be answered (and actioned): who can run those frameworks, who will make changes, who will determine the testing scope, etc.. That is the reason a lot of clients look out for script-based tools. And here comes challenge number 2.
4. Script-based automated testing frameworks are costly and take away time from testing itself.
Script-based frameworks consume a lot of resources. Creating testing scripts takes a lot of time. Although script-based testing is considered by many “the king” of test automation frameworks, still to drive this framework and to keep the test scripts functioning is challenging. The other thing is that you need to have skilled developers in place to build, develop and maintain those scripts. For a lot of IT managers, it is, however, not a value-added task. Developers should rather be focusing on creating new functionalities and business operations that bring tangible value for the organization. Not to mention the fact that skilled automation specialists with professional-level programming are also difficult to find. Usually, this requires a compromise between either leaving an expert person to do their job and upskill them in automation or letting a developer or a testes learn how to automate. At the end of the day, it is very difficult not to compromise on the quality of the automation testing.
5. Custom automation testing frameworks are difficult to scale and are limited in scope.
Unfortunately, custom automation frameworks are very often project specific. This means they are simply not suitable for handling and working across various types of applications and technologies. Most test automation framework are built for one specific application type, be it web, desktop or virtual. This does not allow the testes to use the framework outside of this application to test different apps within the organization. This requires building new code, resources, time so this is simply not doable. Also, if custom frameworks are built for a specific purpose, they are not future-proofed. They have limited usability and if they also make it difficult to scale automation across applications, they are not really needed for the business.
6. Frameworks are often person-dependent and have a short lifespan.
Testing frameworks are often introduced or developed by a single person in the organization. This means that their code might be incomplete or difficult to understand for other developers who need to work with a given framework. And if the inventor leaves or changes job, the responsibility for the code changes hands. This means that the framework will most probably die – either due to the code intricacy (use of substandard coding practices) and difficulties in deciphering the processes or due to the lack of proper handover. Any developer-dependent test automation framework, in the long run, means heaps of lost time and effort.
7. Automation testing infrastructure is often inadequate.
When talking about challenges in test automation, it is important to mention that companies are often missing the adequate infrastructure for automation testing. What this inevitably means is that proper test coverage and execution speed are in danger. If, let’s say, an application needs testing on a number of operating systems, you need to be able to execute test scripts in parallel. And in order to do so, the testing infrastructure needs to be robust so that is supports running tests against many configurations in a shortest possible time.
As a lot of companies who are just getting familiar with test automation are not prepared to develop a new internal testing infrastructure, sometimes a good choice is to go for a cloud-based testing infrastructure.
8. Choosing the right test automation tool is difficult and complicated.
With a lot of test automation tools available on the market, it is increasingly difficult to make an informed choice. Each tool supports particular situations. A common mistake is to go for a tool (often free or open source one) and not choose a one dedicated for a software or product. Also, many of us do not do enough research before deciding on automation tool. As a result, companies tend to go for the most popular, commercial tools based on inadequate evaluation. In the long run, as the tools seems incompatible with the business (and system) requirements, a lot of budget goes to waste when a new (suitable) automation testing tool has to be implemented. This means that test automation will be failed and sometimes not applied in the testing activities.
A sufficient assessment of a test automation tool defines a set of tool requirements criteria that are based on AUT (application under tests) and the experience of the experts who have used the tool a lot. However, a lot of companies do not have resources to run such an assessment. Also, each test engineer should look carefully into the tool’s functionality, particularly its ease of use, scalability, maintainability and integration with third-party tools (e.g. Jira or GitHub).
9. There are issues with data reliance.
Another – often initially overlooked – on the list of challenges in automation testing is data management. In order to execute test scripts properly, data has to be reliable, otherwise, it might become a source of serious issues in the automated testing scenarios. Let’s consider the need to use the same data in multiple instances of the script. Will the test script fail then as any change to data in one test execution may cause another instance to fail? And what if the data in one of the automated testing scripts is built up through the execution of yet another test script?
Developing independent and self-contained automation testing scrips is one way to deal with the data management and reliance problems. However, this is a very time-consuming task that requires a lot of skill. Test scripts should ideally be written in such as way so that create and clean all data that is required for a successful execution.
10. There is a lack of effective collaboration and communication across the teams.
Mentioned as the last in the thorough list of challenges in test automation as this one might as well relate to manual testing teams. However, effective communication and collaboration is just as challenging as the rest of the above mentioned factors which – if missing – can cause the test project to fail.
Effective cooperation on every level is even more problematic in test automation than in manual testing. As test automation is a significant investment, the whole team should be involved and relied upon when it comes to identifying test automation objectives and setting the right targets. This requires a lot of effort put into communication: providing evidence, digging historical data and doing a proof of concept. All the people involved need to be on the same page so that the purposes and goals are clear and transparent to everyone. Contrary to manual testing, automation testes not only discuss the plan, scope and timeframe of test automation with the business and the technical teams but also talk about what should and should not be automated with developers and technical architects (and manual testers, too). The management team support is also vital in establishing the test automation strategy and accepting the ROI and cost and benefit analyses. Communication is therefore key but also challenging – it can decide either on the success or the failure of the whole test automation project.