Every software product requires a personalised approach. However, in order to ensure its impeccable functioning, regression testing is indispensable. This last step before every release of the ready-to-use product is a key part of QA strategy that guarantees that it will function as expected. Regression tests – as will be shown later on – are considered to be a crucial factor determining the success of any new implementation, especially when we discuss ERP systems.
However, a lot of companies do not execute regression tests. For some of them it seems to be a redundant activity, others are concerned with investment and resources necessary to run them. Regardless of the reason, such approach – neglecting the importance of such tests – should be discouraged. This publication shows that for any software – and especially ERP systems – regression testing is an obligatory requirement. Testing revision for any system vulnerabilities, bugs and any underperformance cannot be overlooked. With Microsoft Dynamics 365 regularly issued update rollouts that may affect the overall flow of processes, it is particularly important to conduct systematic tests that check system regression.
1.1. What is regression testing?
Regression testing is a maintenance activity that involves checking already tested features after any system modifications or upgrades of the given software/ system. The testing team is responsible for investigating whether the existing features and functionalities have not been adversely impacted during those changes. Whenever software goes through regression testing, the testing team checks if the changes in one part of the system have induced changes in other parts, too. This not only helps to see how certain features are correlated but more importantly, it helps in developing new features and modifying the existing system.
Regression testing generally has a very wide scope and requires several different cycles. This allows for more comprehensive and effective testing strategies. Knowledge gained from the results of each test cycle may contribute to future testing cycles in order to highlight the areas where testing is necessary for any new changes.
1.2. When do we run regression tests?
In theory, regression testing can be conducted at any stage of the software development lifecycle to make sure that the introduced modifications or deployment have not impacted anything that should not be changed. Regression tests are thus required whenever there occurs a change in system code/ configuration/ business process/ business data/ environment specifications. It can be either a case of:
adding a new functionality: when developers add a new feature or functionality to software, it may adversely impact current system/ business flows and/or pre-existing functionalities. Regression testing can therefore identify any possible issues.
system changes: sometimes it is necessary to change the existing system by adding or changing/ enhancing certain features. Regression testing in this case helps to determine if implemented changes have impacted overall functionality and if given software can perform with new enhancements. An example can be modifications in user interface introducing a different language.
software integration with other products: when integrating software with some external business data/ applications, there is a risk that changes introduced in the code could compromise existing integrations and functionalities
software upgrade: any software updates require performing regression tests. Retesting all new features and patches against the whole system prevents occurrence of any new bugs, issues or even misconfigurations that might have been caused by system improvements.
performance issue: even when no system changes have been introduced, it is still recommended to run regression testing session as soon as a performance issue has been detected. Retesting the original test suite allows to save significant amount of time for developers, too.
testing overall performance of any system: so called “Sunday regression tests” should be executed regularly to prevent any unexpected flaws and issues and minimise the number of emergencies that testers have to deal with in the product’s lifetime.
However, it is important to note that regression testing should also be executed regardless of the above-mentioned changes. Ideally, regression tests should be run as often as possible. That is because regressions can occur also in connection with certain changes that might be outside of your knowledge. A recommended practice is to run regression tests regularly and in order to do so, use an automatic tool for scheduling them.
1.3. Why do we perform regression tests?
Regression testing ensures a more stable product. To a large extent, it mitigates risk associated with collateral damage as it helps to determine if the system has not „regressed” due to some new bugs or re-appearance of pre-existing flaws. Regression tests are vital for ensuring high quality of the software for the longest time possible. As long as they are performed, any potential problems that might be induced by updates or upgrades are under control. Hence, they have a direct impact on user experience. Regardless of the changes in the system, its reliability and functionality stay the same. Regression testing also plays a crucial role in building confidence in modified system, therefore it should not be overlooked or only partially performed. Without adequate regression testing, the quality of software/ data/ system configuration will decrease with time.
What is also important is the efficiency of the process itself. It has to be noted that fault detection during regression testing because of its specificity provides feedback on the system much faster than might be possible otherwise. This in turn means that testing team can start their work earlier and avoid release delays. It is also a cost-effective strategy as it saves the implementation projects from going over budget. Also, the costs of fixing bugs in production environments are much higher than fixing them at development stage.
1.4. Regression testing vs functional testing – scope, purpose and execution
Software testing is performed to ensure that the product meets the requirements. Testing phase has two objectives: one is verification, i.e. checking if we are building the right software and the other is validation which is understanding if the software is built in the right way. Broadly speaking, tests can be divided into two categories: functional and non-functional (which among others, such as performance, integration or usability tests, include regression tests). Both functional and regression testing ensure a stable and high-quality software. To deliver and later on maintain this software/ system, it is vital to perform both kind of tests as they complement each other. Developers and testers need to be aware that regression testing allows to check all interrelations within the system while functional tests look at the acceptance criteria for specific features. Let us take a quick look at both functional and regression testing. We will try to show how functional tests differ from regression testing terms of their scope, purpose and method of execution. Due to the fact that regression tests are the topic of this e-book, other types of non-functional tests will not be discussed.
Functional testing primarily validates if the system/ software is functional and if it works as it is supposed to. The purpose of the tests is to verify the functionality of the application, program, action module or feature of a given environment. In other words, only a fragment of the system is investigated independently to see if it works according to requirements. Functional tests, broadly speaking, can check:
any missing functionalities or gaps
persisting interface errors
The focus of these tests is on functionality which means that user experience and design are not taken into consideration. A well-derived functional test helps to deliver an error-free product or software and check if all requirements are developed and tested as expected. Test suites for functional testing are based on fresh set of functionalities and cases which have not been tested before. Also, as functional tests have different purpose (to check a certain feature/ functionality) than regression tests, they are fewer in number and testing scripts are easier to create and maintain.
Regression testing can cover the same functionalities and features that are checked during functional testing. However, it is not feature-specific. Its focus is on the whole system – tests are performed in order to check if introduced modifications/ upgrades have not caused any regression, i.e. breaks or flaws in code or in already tested functionality. Regression testing basically means testing the whole set of functionalities and not just a single feature.
Test suites consist of cases that were previously tested and on top of that they might also include newly added features and modules depending on the requirements. Regression test cases are very often re-executed multiple times to see if the new modifications have not impacted previously tested environment. Once the defect is raised, test cases are re-executed, then it gets fixed and re-assigned for next re-tests.
In contrary to functional testing, regression test scripts require more maintenance than functional ones as the test suite aims to test overall functionality. Due to its scope, regression testing suite should be updated after each release. The high number of regression tests and their recurrence are the main factors that speak for their automation. However, it has to be noted that automated tests – similarly to tests ran manually – are also very time-consuming because testing scripts have to be constantly amended and updated. New modifications may also change the existing functionalities on purpose which regression tests may incorrectly interpret as regression defects.