Articles

Regression testing in D365 – purpose, scope and automation

5min read

Test Case Tutoral Video

Go back to
Knowlage Base

Introduction

This article is intended to introduce the topic of regression testing with focus on tests created and executed for ERP (Enterprise Resource Planning) platform.

Our aim is to show how automated regression testing is an indispensable part of digital transformation and in the long run helps mitigate business continuity risks.

Regression testing – purpose, scope and automation
CHAPTER #1.

Regression testing – an overview

Testing is crucial for a successful implementation, especially in companies using an ERP software system for conducting business. It is therefore extremely important for the customer as well as for any manager and developing team to understand the importance of this process. Next to UAT (user acceptance testing), functional and integration testing, regression testing is a primary tool for validating functionality of a given software or application.

Every software product requires a personalised approach. However, in order to ensure its impeccable functioning, regression testing is indispensable. This last step before every release of the ready-to-use product is a key part of QA strategy that guarantees that it will function as expected. Regression tests – as will be shown later on – are considered to be a crucial factor determining the success of any new implementation, especially when we discuss ERP systems.

However, a lot of companies do not execute regression tests. For some of them it seems to be a redundant activity, others are concerned with investment and resources necessary to run them. Regardless of the reason, such approach – neglecting the importance of such tests – should be discouraged. This publication shows that for any software – and especially ERP systems – regression testing is an obligatory requirement. Testing revision for any system vulnerabilities, bugs and any underperformance cannot be overlooked. With Microsoft Dynamics 365 regularly issued update rollouts that may affect the overall flow of processes, it is particularly important to conduct systematic tests that check system regression.

1.1. What is regression testing?

Regression testing is a maintenance activity that involves checking already tested features after any system modifications or upgrades of the given software/ system. The testing team is responsible for investigating whether the existing features and functionalities have not been adversely impacted during those changes. Whenever software goes through regression testing, the testing team checks if the changes in one part of the system have induced changes in other parts, too. This not only helps to see how certain features are correlated but more importantly, it helps in developing new features and modifying the existing system.

Regression testing generally has a very wide scope and requires several different cycles. This allows for more comprehensive and effective testing strategies. Knowledge gained from the results of each test cycle may contribute to future testing cycles in order to highlight the areas where testing is necessary for any new changes.

1.2. When do we run regression tests?

In theory, regression testing can be conducted at any stage of the software development lifecycle to make sure that the introduced modifications or deployment have not impacted anything that should not be changed. Regression tests are thus required whenever there occurs a change in system code/ configuration/ business process/ business data/ environment specifications. It can be either a case of:

adding a new functionality: when developers add a new feature or functionality to software, it may adversely impact current system/ business flows and/or pre-existing functionalities. Regression testing can therefore identify any possible issues.

system changes: sometimes it is necessary to change the existing system by adding or changing/ enhancing certain features. Regression testing in this case helps to determine if implemented changes have impacted overall functionality and if given software can perform with new enhancements. An example can be modifications in user interface introducing a different language.

software integration with other products: when integrating software with some external business data/ applications, there is a risk that changes introduced in the code could compromise existing integrations and functionalities

software upgrade: any software updates require performing regression tests. Retesting all new features and patches against the whole system prevents occurrence of any new bugs, issues or even misconfigurations that might have been caused by system improvements.

performance issue: even when no system changes have been introduced, it is still recommended to run regression testing session as soon as a performance issue has been detected. Retesting the original test suite allows to save significant amount of time for developers, too.

testing overall performance of any system: so called “Sunday regression tests” should be executed regularly to prevent any unexpected flaws and issues and minimise the number of emergencies that testers have to deal with in the product’s lifetime.

However, it is important to note that regression testing should also be executed regardless of the above-mentioned changes. Ideally, regression tests should be run as often as possible. That is because regressions can occur also in connection with certain changes that might be outside of your knowledge. A recommended practice is to run regression tests regularly and in order to do so, use an automatic tool for scheduling them.

1.3. Why do we perform regression tests?

Regression testing ensures a more stable product. To a large extent, it mitigates risk associated with collateral damage as it helps to determine if the system has not „regressed” due to some new bugs or re-appearance of pre-existing flaws. Regression tests are vital for ensuring high quality of the software for the longest time possible. As long as they are performed, any potential problems that might be induced by updates or upgrades are under control. Hence, they have a direct impact on user experience. Regardless of the changes in the system, its reliability and functionality stay the same. Regression testing also plays a crucial role in building confidence in modified system, therefore it should not be overlooked or only partially performed. Without adequate regression testing, the quality of software/ data/ system configuration will decrease with time.

What is also important is the efficiency of the process itself. It has to be noted that fault detection during regression testing because of its specificity provides feedback on the system much faster than might be possible otherwise. This in turn means that testing team can start their work earlier and avoid release delays. It is also a cost-effective strategy as it saves the implementation projects from going over budget. Also, the costs of fixing bugs in production environments are much higher than fixing them at development stage.

1.4. Regression testing vs functional testing – scope, purpose and execution

Software testing is performed to ensure that the product meets the requirements. Testing phase has two objectives: one is verification, i.e. checking if we are building the right software and the other is validation which is understanding if the software is built in the right way. Broadly speaking, tests can be divided into two categories: functional and non-functional (which among others, such as performance, integration or usability tests, include regression tests). Both functional and regression testing ensure a stable and high-quality software. To deliver and later on maintain this software/ system, it is vital to perform both kind of tests as they complement each other. Developers and testers need to be aware that regression testing allows to check all interrelations within the system while functional tests look at the acceptance criteria for specific features. Let us take a quick look at both functional and regression testing. We will try to show how functional tests differ from regression testing terms of their scope, purpose and method of execution. Due to the fact that regression tests are the topic of this e-book, other types of non-functional tests will not be discussed.

Functional tests

Functional testing primarily validates if the system/ software is functional and if it works as it is supposed to. The purpose of the tests is to verify the functionality of the application, program, action module or feature of a given environment. In other words, only a fragment of the system is investigated independently to see if it works according to requirements. Functional tests, broadly speaking, can check:

any missing functionalities or gaps

incorrect specifications

persisting interface errors

The focus of these tests is on functionality which means that user experience and design are not taken into consideration. A well-derived functional test helps to deliver an error-free product or software and check if all requirements are developed and tested as expected. Test suites for functional testing are based on fresh set of functionalities and cases which have not been tested before. Also, as functional tests have different purpose (to check a certain feature/ functionality) than regression tests, they are fewer in number and testing scripts are easier to create and maintain.

Regression tests

Regression testing can cover the same functionalities and features that are checked during functional testing. However, it is not feature-specific. Its focus is on the whole system – tests are performed in order to check if introduced modifications/ upgrades have not caused any regression, i.e. breaks or flaws in code or in already tested functionality. Regression testing basically means testing the whole set of functionalities and not just a single feature.

Test suites consist of cases that were previously tested and on top of that they might also include newly added features and modules depending on the requirements. Regression test cases are very often re-executed multiple times to see if the new modifications have not impacted previously tested environment. Once the defect is raised, test cases are re-executed, then it gets fixed and re-assigned for next re-tests.

In contrary to functional testing, regression test scripts require more maintenance than functional ones as the test suite aims to test overall functionality. Due to its scope, regression testing suite should be updated after each release. The high number of regression tests and their recurrence are the main factors that speak for their automation. However, it has to be noted that automated tests – similarly to tests ran manually – are also very time-consuming because testing scripts have to be constantly amended and updated. New modifications may also change the existing functionalities on purpose which regression tests may incorrectly interpret as regression defects.

Regression testing – purpose, scope and automation
CHAPTER #2.

How to perform
regression
testing?

Regardless of employed strategy and preferred methodology, there is a couple of essential steps.

testing process defines as follows:

step 1: preparing scripts for functional tests: on the basis of requirements description, regression testing scripts are built and later on, they are used as a basis for functional tests, too.

step 2: re-using scripts: prepared scripts are used to build regression test suites. They collect test scenarios prepared beforehand (see point 1). Regression test suites can be based e.g. on functional or integration tests.

step 3: systematic regression tests execution: regression tests need to be run on a regular basis, according to a pre-defined schedule (e.g. before every new release/ build of the software)

2.2. Methods and approaches

Regression testing is a complex and intricate procedure that requires specific planning and diligence. Depending on the case scenario, you might want to use a different methodology, for example test a complete functionality or select tests for checking only business critical features. Test suite design and method of regression testing can affect the effectiveness of the process. The strategies of performing regression testing are further discussed below:

Retest All
Complete approach is frequently used when the testing team aim to define the number of and their impact on system functionality as it undergoes changes. It allows to see the full snapshot of system functionality. It is also a required step in the deployment stage of the release. This approach is usually adopted during deployment phase in the final stages before the release of a build. This method implies that every test case within a given environment should be re-executed to ensure that a change in code has not caused any bugs to emerge. Retesting all technique is an all-around method, however it requires more resources than other methods and is extremely time-consuming. That is why in cases where it is required sessions should be automated and test modules need to be re-usable.

Test Selection
Contrary to Retest All method this approach allows to reduce time by selectively retesting the modified system by means of re-running tests. The entire system is not tested but depending on the approach, testers select cases from test suite for regression testing sessions to see whether modifications in the code affect the software application in any way. There are several practices that might be used for selective regression testing.

2.3. Regression testing for ERP systems

An implementation or an upgrade of an ERP system usually entails the digital transformation of bringing the business to the cloud. That is why those two require specific scale and method of regression testing. Once a company decides to move to the cloud or purchase an ERP application, they have to be aware that efficient and cohesive management imposes the need of numerous updates. An up and-running ERP system should be flexible, adaptable to business requirements and by being so also risk-mitigating. It requires frequent customization, adding further functionalities so that it guarantees full integration with different add-ons/ applications, and ultimately testing and integrating those features.

With new Microsoft ERP – Dynamics 365 F&O – there is a scheduled update plan and a very demanding testing scope. Companies who decide to move forward with Dynamics 365 need to calculate risks and costs related to resource planning in the context of application maintenance. It is not a recommended solution to opt for less frequent updates or stop them entirely. The issue of update management is an obligatory consideration for ERP system. Continuous development of the application requires a great deal of man-day amount for regression testing purposes. This calls for an automated approach to regression testing.

Automated regression testing tools

Automated regression testing tools have been used and developed in many business areas, however for ERP environment they are quite a novelty. Especially for Dynamics 365 F&O customers, Microsoft has introduced a free package called Regression Suite Automation Tool (RSAT) that largely helps streamline the update process and management. However, in case of complex ERP systems, large amount of data, numerous testing teams with many testers, in order to ensure better efficiency and testing coverage (and ultimately, to mitigate any business continuity risks), it seems a reasonable option to use an independent testing tool. In XPLUS we developed a tool designated for automatic testing regression particularly for ERP environment of Dynamics 365 (which is discussed in detail in chapter 3.2)

Regression testing – when to do it in ERP environment

Generally, we can distinguish two different scenarios when there is a need for automated regression testing:

Implementation phase (close to deployment)
Running an automated regression testing tool before moving on to UAT will both minimise the risk associated with lack of user acceptance and help to perform any necessary adjustments once the discovered bugs and errors have been fixed. This is a critical phase in an ERP implementation process whereby the number and scale of run tests is the highest. Using automated regression testing tool will also provide faster ROI (Return on Investment). Companies deciding to run tests at this stage focus on enhancing their operations rather than technology.

Post go-live phase
Alternatively, you might want to postpone the adoption of the automated tool and postpone it till the post-launch phase. You can then benefit from the knowledge of the whole cycle of internal updates as well as e.g., the vendor. Once the entire system is operational, all the processes within a given ERP solution are clearly visible. This, in turn, allows for easier risk-assessment, evaluation of most critical processes and systematic forecast of changes and updates required throughout the product life. Businesses who are focused on making correct calculations and prefer to spend more time on adjusting their ERP system to their operational and financial capabilities are more likely to move forward this automatic testing post the go-live phase.

Regression testing – purpose, scope and automation
CHAPTER #3.

Regression testing –
impact on business
performance

3.1. Benefits of using automated testing

A manual approach to regression testing is both inefficient and very costly. Manual execution of such tests requires significant involvement of multiple users in preparation of test suites and then running tests, not to mention the gigantic amount of time required. That is why automated regression testing is a recommended solution. Most businesses who have invested in regression test automation were able to cover minimum 30% processes (in most cases with the appropriate tool and methodology test coverage, however, can be as high as 70%). Even considering the dynamic nature of today’s operational reality that requires frequent software updates and flexibility to changing business requirements, keeping automated scripts that can be regularly updated in the long run is still less costly and effort-consuming method than manual regression testing.

An ERP implementation phase should start with automation, too. As the business processes are not stable yet, regression testing that can be automated brings significant value when creating and maintaining test scripts as changes grow incrementally. For clients who are already live in D365 automation, automated regression testing is a natural part of the journey. Every update or software modification entails regression tests and their automation enhances quality and speeds up the release process. In some areas they also help while planning rollouts in various locations. When updating the system requirements for MS D365, regression test automation should also be considered as it helps to determine precisely which existing modules or features are not working. It makes testing process both more efficient and less problematic in further perspective.

Regression tests make ideal candidates for automation in MS Dynamics AX and 365 due to their:

repeatability: due to consistent and thus predictable performance, automated tests are highly repeatable. With modern ERP systems such as MS Dynamics 365, creating new regression test suites anew with every new release and periodic update would not only be extremely time-consuming but also challenging in terms of resources and workforce. Automated repeatable tests eliminate the need for involving the whole testing team in running continuous iterations of each test.

stability: conducting each test manually requires a high level of expertise of each tester involved. As software functionality grows and platform structure becomes increasingly complex, regression testing also gets more error prone. Introducing automated testing guarantees that all components across the system are checked with the same level of accuracy, expertise and focus.

minimum supervision: automated regression tests can be recorded by any system user with the knowledge of the business process, even without programming skills. They can be run 24/7 which cuts down the time needed to test the system as well as human resources needed to execute such tests.

increased coverage: automated regression testing can cover every process and aspect executed in ERP system. More tests can be run (also simultaneously) which examine the system more thoroughly.

3.2. Automatic regression testing – Executive Automats

Our proprietary tool – Executive Automats (EA) – is an automated regression testing framework to perform automated testing processes implemented locally even from the earliest stages of the rollout procedures. It was created and developed as a response to challenges related to deployment speed, testing accuracy and quality for MS Dynamics AX/365/CRM system. Automation coverage for tests with EA ensures end-to-end business process continuity across each and every organisation with an ERP system in place. With Executive Automats tool, you can cover up to 70-80% of business test cases , which translates into 65%-time savings when it comes to actual testing.

EA tool serves as a central platform for automated tests that can be utilised to cover all functionalities of Microsoft Dynamics AX/365FO/CRM during upgrades and/ or modifications. In this way, EA allow for regression testing in complex systems without compromising on quality. Business processes are prevented from failing due to continuous software development and application updates offered by Executive Automats tool. What is more, Executed Automats, although designed for Microsoft ERP systems, is a safe environment that also works independently of D365 in any web-based platform or application.

Main advantages of Executive Automats:

zero coding – automated test scenarios can be built without the help of developers. There is no need to engage development teams. The main idea of EA test is that each customer should be able to record and play their processes as simply as possible. Seamless collaboration allows to empower people with different skills to prepare and run automated tests within each business area.

reusability – 100% reusability and flexibility guarantee greater stability of regression tests as well as improve time to value. In the context of EA tool, we can differentiate between three aspects/ levels of reusability.

there is no need to record a test script multiple times – once a test script is created, the test suite can then be used in exactly the same setup by every user for different purposes, e.g. as a part of UAT testing

every test created within EA can be safely re-used multiple times. Usually, by re-using the same test, we can cause data integrity issues whereas with EA tool we have mechanisms in place that allow safe and effective reusability, without a need to build a new database

EA reusability feature allows for data and process verification during rollouts. We are able to check precisely if data is correct and system configuration and environment specifications matches exactly different business requirements.

scalability: EA test design supports web-based platforms and allows a well-structured testing. The tool provides all-in-one features for managing the whole automation project from one place, regardless of its scale.

When we talk about Executive Automats and regression tests, main functionalities that make the testing swifter, more flexible and ultimately cheaper are:

practical test recording: in order to improve efficiency of the test recording and its modifications, you can use Advanced Recorder. Contrary to typical circumstances when you have to record the test case, then switch pages and play back the recording for verification and potential changes, modifications can be made on the fly and playback is not necessary. Every click is registered and recorded, and so individual steps can be adapted and enriched with logic and context. Out of the recordings, each user can prepare business process documentation.

quick installation: the customer receives a ready-to-go deployment package whereby no installation is necessary and whole setup is completed in 15 minutes only. Test suites can be used in day 1 already. The setup of security permissions is automated, too.

easy and flexible test execution: the testing platform itself is complete and easy to configure and navigate. Regression tests can be easily and quickly launched via a build-in feature (Scheduler) or performed directly via Microsoft Azure DevOps (with the Test Plans module). This integration allows you to conveniently manage the Executive Automats test scripts, as well as a team of testers and developers. Both features make it easy to handle automated tests created in EA by means of test plans or pipelines. A single test case based on a business process takes 2-10 minutes to complete. Average time to create and run a simple automated test (a so called “happy path)” is just 2-3 minutes.

parallel testing: the interface allows for individual processes to be batched and executed in parallel to simulate load testing and real-life system stress test situation. EA shows how system behaves under stress and allows system adjustments before peak times. This ensures 100% system stability during actual peak loads.

easy maintenance: EA framework allows for creating, updating and modifying hundreds of various test scripts. It is easy to change in order to use them for different business/ testing purposes. Scripts can be merged, combined or cloned. New steps can be added to an existing script (not only at the very end but also in the middle). Additionally, within each script certain steps can be added, deleted, moved or copied over if need be. You can also perform jumps between test steps, both forward and backward, as well as switch users when running one script. There is also a test project feature that allows grouping scripts into a more complex project).

versatility: EA offers many opportunities (apart from typical regression testing) for investigating specific cases, introducing new conditions for double-checking various processes, generating reports, planning tests ahead in time, etc.

Example of a test case recorded with Executive Automats

3.3. Value of automated regression testing for clients

Robust manual regression testing consumes a great deal of time and resources. In the case of fast-paced ERP environment with frequent updates and releases of Microsoft One Version, Executive Automats is a preferred solution as each upgrade requires regression testing of the application. As soon as business processes are deployed, automated testing should be triggered, too. During an ERP implementation phase, tests are either not stable for a very long time or non-existent. However, EA automated testing to check process regression is valuable, especially with very fast and easy test creation and maintenance. Automation for companies planning an upgrade to Microsoft D365 from AX2012 allows to find out obsolete processes or modules that can be excluded during upgrade. A complete framework offered by Executive Automats apart from complete process documentation brings value during stabilization months while maintaining live D365 application. It also greatly helps to keep up with Microsoft One Version approach.

Regression testing is inevitable after deployment, too. Test automation with continuous ERP system development is a crucial component of the after-go-live phase. EA tool allows for automated process detection. That means that entire operations are investigated thoroughly and on this basis regression tests are run and can be modified later in case of any defects. By being able to see what works and works not in the system, XPLUS team can better tailor the tests for each client and deliver standardised versions with each new implementation. Our approach is to involve users, even those without in-house knowledge, with a ready-made solution for recording real business processes and further for creating test cases based on those recordings. In this way, use and test cases requirements as well as testing scenarios are much better defined. Additionally, whatever is recorded by each process owner may be re-used and re-applied as a template for business process documentation to enhance onboarding experience and create comprehensive training as well as audit trail materials.

Conclusion

To sum it up, regression testing significantly improves software quality, drives higher productivity, and reduces risks that might affect business continuity. Although regression testing does involve financial investment, undetected defects in the application can be much more costly. No regression testing suite in place, using tests as ad hoc solutions to occurring problems or relying merely on functional and UAT testing represents a reactive approach that might be called „put-out-fires”. In the long run, it is bound to have negative consequences for the ERP system performance, health of the software and hence also company’s operational side.

Using an automated solution for performing and maintaining regression tests is a strategy of mature and conscious companies. Although there is a number of tools available for automatic regression testing, they need to be flexible enough to allow updating the test suite, reusability of the test cases, thus guaranteeing their stability. The decision when to use automated regression testing tool depends on the employed testing strategy, application maturity and company profile. For ERP systems, regression testing is a must and with a user-friendly, advanced and comprehensive tool which is Executive Automats this process is easier, faster and more cost-effective. EA reduces the complexity of each test projects, thus shortening the implementation as well as maintenance of testing phases.

“We originally started our cooperation with XPLUS because of their unique testing software for Dynamics AX – Executive Automats. The effectiveness of this software let XPLUS being awarded the Dynamics 2012 support service and maintenance contract as well as several related development contracts.

Here is an excerpt of some of the work that was done:
• Detailed Security AX add-in
• Sharing and reassigning user related parameters AX Add-in
• Grid filters improvements
• Advanced queries preparation for third party reporting tool

In addition of traditional development work, Brakes is using XPLUS Executive Automats for performance, trainings, security management and application testing. Also, XPLUS provides go-live support services for AX environments and SQL server. Because of the work done, we consider XPLUS to be a trusted and reliable Microsoft Dynamics partner.”

Stewart Mackey,
IT Director of Brakes

We use Executive Automats to further improve the quality in the development process through automated tests optimized to identify possible sources of errors in advance and to rule them out

Michael Roedeske,
‍Member Of Management Board at webtelligence IT consulting GmbH

Thanks to Executive Automats, Suez Groupe was able to reduce dramatically the number of man-days required to complete a major Dynamics AX upgrade project (from AX 2012 R2 CU1 to AX 2012 R3 CU11)

Jean Francois Neuville,
Head of Dynamics AX Competence Center at SUEZ

Thanks to their [XPLUS] innovative approach to fast and codeless solution, we have an automated testing solution that spans across multiple applications for our current and future needs

Pratap Krishnamurthy,
Software Quality Assurance Manager at BlueCrest

Regression testing in D365 – purpose, scope and automation

XPLUS S.A.

Contact

Support

Get in touch