Articles and documents
Test Automation: ROI, Cost Estimation and Effort Analysis
Moving from manual to automated testing is not “optional” and it plays a crucial part in delivering high quality services. Even so, it’s not mandatory to automate all tests! Automation is unquestionably efficient for testing regression scenarios. They require plenty or repetitive actions taken one by one, frequently with large data sets included. It is not only time consuming for testers, but also gives a place for human mistakes that could be avoided while using automation.
Keeping in mind the above, the vast majority of businesses agree that test automation is a good idea. But uncertainty around appropriate resources that should be committed, as well as the cost of building and sustaining an automated testing framework, sometimes prevents moving it forward. Also, it usually leads to questions about the return on the investment: what part of the budget should be dedicated, and how much value will this venture deliver in the future?
Measuring the fiscal benefits of test automation can dispel the above doubts. A quantitative justification may be also helpful to consider investing in an automated testing tool, especially for those outside of the technology processes in the organization.
The organization is facing system updates every quarter. Estimating 5 minutes spent on each scenario with (positively assumed) 1 case per scenario that needs to be tested, leads us to 25 days per year dedicated by manual testers (Fig. 1).
Getting back to the investments. To start generating savings, the first months have to be dedicated to the preparation of automated test scripts.
Depending on the test case complexity and validations type, due to complex process of moving files to LCS and Azure DevOps and further execution, an average Test Script creation in RSAT usually takes between 30-180 minutes (approx. optimistic 60 mins). Due to the superior flexibility and script reusability features that EA provides, tests do not need to be continually re-created from scratch so the average time for Test Script creation is 10-30 minutes. For our example, we’ll say pessimistic 30 minutes.
It gives accordingly 30 (RSAT) or 18 (EA) days invested in the automation while number of automated test cases with EA (288) is significantly higher than with RSAT (240) (Fig. 2).
But initial script creation is just one part of the process. The second part that is critical, considering the number of system updates needed, is test script maintenance. Companies face changes made into Business Process Scenarios.
The third factor to consider is test execution time, which also demands testing team supervision. These 3 factors considered together is how we receive 36 and 20 days spent on preparation, maintenance, and execution of test scripts accordingly for RSAT (6 additional days) and EA (2 additional days) (Fig. 3 and 4).
Once again the difference between both tools is explained by the EA’s flexibility for recreation. Meanwhile, manual testing takes just the execution time of each and every scenario. We already calculated that it is 25 days per year.
It is noticeable that the first year is invested in the preparation of an test automation framework. On the other hand, each subsequent year brings less time spent on manual testing. That saves the tester’s precious time and lets them focus on more challenging tasks.
Eg. year 1 with RSAT gives 15 days spent on manual testing and 36 days on automatic scripts preparation (15 + 36 = 52). 52 - 25 (just manual testing scenario) = -26
Calculating Return On Investment
ROI = Savings(Benefits) / Investment (Costs)
Here, to present the ROI already compared with manual testing, the formula takes total accumulated savings (time) against manual testing (Fig. 6) and divides it by the total time invested in automation (Fig. 5).
For EA the first year would be: (-5 / 20 )*100% = -24%
At first glance, it may seem expensive to pay for extra weeks of work just for constructing the test framework. But after running the test automation script more times the deficit is made up. For instance, the ROI for EA starts to be positive during the first year. The difference is noticeable at first sight in the second year: (-8 / 22 )*100% = 39%
ROI for RSAT is also continuously getting higher.
As a good business practice, it is important to ensure that ROI is positive during the first three years of the automation project. Any investments in automation that will not start returning in this amount of time should be verified and probably dropped.
To prepare your own case scenario, use the editable spreadsheet, available HERE. Play with your data and forecast if the investment in automation for your business is profitable.
ROI is not the only factor. What else should be considered in terms of investment in test automation?
Time and security are the most valuable assets in business. Automations are constructed to fight against every additional minute that employees waste performing repetitive tasks. That investment pays itself off with use, and the real profits can be made by providing appropriate test automation tool along with a holistic automation strategy.
Companies relying just on manual regression testing, not only fail to achieve these profits but put the product and possibly the organization at risk, especially as a product and business grows. The ROI model can, at a glance, quickly and easily help business decision makers understand the value that test automation delivers over time and drive the right decision making with regards to automation.