Decision makers in ERP-managed companies realize how important it is to keep their internal system operational and up-to-date. To get and sustain a good-quality ERP product it is not enough just to buy one. After the implementation of the system and “go-live”, it is almost obligatory to perform testing and in many cases it is mandatory. Sooner or later business owners realize how time-consuming manual testing is, and they face the decision about investing in automation.
Moving from manual to automated testing is not “optional” and it plays a crucial part in delivering high quality services. Even so, it’s not mandatory to automate all tests! Automation is unquestionably efficient for testing regression scenarios. They require plenty or repetitive actions taken one by one, frequently with large data sets included. It is not only time consuming for testers, but also gives a place for human mistakes that could be avoided while using automation.
Keeping in mind the above, the vast majority of businesses agree that test automation is a good idea. But uncertainty around appropriate resources that should be committed, as well as the cost of building and sustaining an automated testing framework, sometimes prevents moving it forward. Also, it usually leads to questions about the return on the investment: what part of the budget should be dedicated, and how much value will this venture deliver in the future?
Measuring the fiscal benefits of test automation can dispel the above doubts. A quantitative justification may be also helpful to consider investing in an automated testing tool, especially for those outside of the technology processes in the organization.
On average 35%-40% of test cases are able to be covered by RSAT. However, the Microsoft tool does not support negative tests, system warnings, „try-catch” functions, or workflows that are also important to test and may be automated. With Executive Automats, which can handle 100% of what RSAT handles and more, businesses are able to cover 70%-80% of their test cases. For the purposes of our calculation, let’s assume a little bit worse scenario in which EA covers “just” 60% of the testing scope.
Getting back to the investments. To start generating savings, the first months have to be dedicated to the preparation of automated test scripts.
Depending on the test case complexity and validations type, due to complex process of moving files to LCS and Azure DevOps and further execution, an average Test Script creation in RSAT usually takes between 30-180 minutes (approx. optimistic 60 mins). Due to the superior flexibility and script reusability features that EA provides, tests do not need to be continually re-created from scratch so the average time for Test Script creation is 10-30 minutes. For our example, we’ll say pessimistic 30 minutes.
It gives accordingly 30 (RSAT) or 18 (EA) days invested in the automation while number of automated test cases with EA (288) is significantly higher than with RSAT (240) (Fig. 2).
But initial script creation is just one part of the process. The second part that is critical, considering the number of system updates needed, is test script maintenance. Companies face changes made into Business Process Scenarios.
The third factor to consider is test execution time, which also demands testing team supervision. These 3 factors considered together is how we receive 36 and 20 days spent on preparation, maintenance, and execution of test scripts accordingly for RSAT (6 additional days) and EA (2 additional days) (Fig. 3 and 4).
Once again the difference between both tools is explained by the EA’s flexibility for recreation. Meanwhile, manual testing takes just the execution time of each and every scenario. We already calculated that it is 25 days per year.
It is noticeable that the first year is invested in the preparation of an test automation framework. On the other hand, each subsequent year brings less time spent on manual testing. That saves the tester’s precious time and lets them focus on more challenging tasks.
Adding up total time spent for automations and manual testing, and comparing the number with the scenario of having just manual tests, we receive the total accumulated savings.
Eg. year 1 with RSAT gives 15 days spent on manual testing and 36 days on automatic scripts preparation (15 + 36 = 52). 52 - 25 (just manual testing scenario) = -26
With all the costs and savings prepared, we finally get to the ROI. Return on investment is calculated as savings (gained by replacing manual regression tests with automated tests) divided by the cost of investment into test automation:
ROI = Savings(Benefits) / Investment (Costs)
Here, to present the ROI already compared with manual testing, the formula takes total accumulated savings (time) against manual testing (Fig. 6) and divides it by the total time invested in automation (Fig. 5).
For EA the first year would be: (-5 / 20 )*100% = -24%
At first glance, it may seem expensive to pay for extra weeks of work just for constructing the test framework. But after running the test automation script more times the deficit is made up. For instance, the ROI for EA starts to be positive during the first year. The difference is noticeable at first sight in the second year: (-8 / 22 )*100% = 39%
As a good business practice, it is important to ensure that ROI is positive during the first three years of the automation project. Any investments in automation that will not start returning in this amount of time should be verified and probably dropped.
To prepare your own case scenario, use the editable spreadsheet, available HERE. Play with your data and forecast if the investment in automation for your business is profitable.
The ROI model presented above is intentionally simple and keeps the number of inputs low to make it generally applicable to most of the companies. In reality, many other components can impact ROI but are not incorporated in the calculation. These include eg. qualitative measures as security of data or an additional capacity of testers’ workforces.
Time and security are the most valuable assets in business. Automations are constructed to fight against every additional minute that employees waste performing repetitive tasks. That investment pays itself off with use, and the real profits can be made by providing appropriate test automation tool along with a holistic automation strategy.
Companies relying just on manual regression testing, not only fail to achieve these profits but put the product and possibly the organization at risk, especially as a product and business grows. The ROI model can, at a glance, quickly and easily help business decision makers understand the value that test automation delivers over time and drive the right decision making with regards to automation.