In this article we will share and elaborate on the Top 5 Automated Testing concerns in Microsoft Dynamics 365. These are all raised by those who have approached us regarding testing for their Microsoft Dynamics application. Since 2013, we have been advising and running test automation projects in Microsoft Dynamics AX2009, AX2012 and now on Dynamics 365 FO. We have conducted hundreds of small and several enormous test automation projects and this is to summarize different questions, statements, and our position and advice on these.
Together with elaboration on these concerns, we have listed our advice on what to consider during test strategy definition process and what should be challenged to find the best way to proceed. This may be used by anyone interested in the area, but it is no secret anymore that robust manual testing (even if combined with RSAT) consumes too much time and resources to keep up with the rapid pace of updates and releases of the Microsoft One Version. All that considered, we have focused this article on the only viable option on the market, Executive Automats, which has been designed and developed from the very beginning to address these specific problems.
In summary, by the end of the document, we will have explained how Executive Automats may support your Dynamics 365 automated testing project to finish it quicker (in a few days or even hours), better (reaching up to 80% automated test coverage), and cheaper (to create real savings and ROI)
It is sometimes a consequence of other concerns, but phrased this way, is illustrated with the following statements:
Following “cloud revolution”, more and more applications were moved to cloud to improve its delivery, time to value and further maintenance. Software vendors were given a fantastic opportunity to provide value faster and better with more frequent minor and major updates and releases. Together with its unquestioned value it has a very specific impact on all applications that are tailored to a customer needs. Customizations and modifications are based on standard solutions and when standard is being improved or updated it may affect existing modifications. Critical business processes may fail in the application. Regression testing of the application after any upgrade becomes essential.
Aside of application updates and releases provided by software vendors, customized applications are evolving all the time. Not only because of internal business needs, but also, to follow new regulatory affecting business requirements. Testing (all types from unit tests, integration to a User Acceptance Tests and regression) is a natural part of the software development lifecycle and regression testing takes a huge role in this process. Automated regression testing gives a chance to adopt application changes (business value) faster with a lower number of bugs. What may be interesting, following different researches, shortening application lifecycle is usually accessed more important than savings that are brought by automated testing (which is usually a case as well).
Advice: questioning importance of automated testing is usually a risk evaluation statement and probably a consequence of cost (efforts) vs effects (ROI) analysis performed by the customer. This is often the position of those who are maintaining a non-cloud, mature application, that are not expecting significant changes and perceiving automated testing scope very exhaustive thus of a limited value.
We suggest re-analyzing a following aspects (all supported with Executive Automats):
More about Microsoft Dynamics 365 One Version https://docs.microsoft.com/en-us/dynamics365/fin-ops-core/fin-ops/get-started/one-version
Summary : a root cause for limited or no automated testing scope is usually a consequence of resource and budget estimations (see point 2 below) necessary for the project setup and/or general risk assessment. In most cases, a more detailed analysis reveals that the budget is overestimated (due to inappropriate tools and methods) and risk is underestimated (ie. Life after go-live is often underestimated considering the time and effort needed to maintain a constantly updated application.)
Traditional test automation, with its history starting in early 2000’s, still has a “bad reputation” because of its complexity (see point 3), limited value (point 4) and inefficiency in ERP platforms under continuous development. The need to involve technical people and dedicated resources made test automation expensive and useful only where absolutely necessary. Those who tried test automation in the past have suffered on automation project costs when it was tried on the implementation phases and were suffering even more during the maintenance phase.
Together with the cloud revolution, new testing and process automation tools appeared. Low or now code UI based test automation tools, have been introduced, but the main questions concerning test automation are still valid:
On one side, low or no-code tools (mainly based on Selenium frameworks) have been introduced to make it possible to design automated test cases based on “trigger and action” automation approach (Robotic Process Automation, RPA, type of automation). It allows to automate things, but the way a test script is created makes it very time consuming and not really useful for test automation project needs. Testing projects requires hundreds or thousands of test cases (average for an ERP) and thus, RPA type of solutions are difficult to accept due to budget challenges to start and maintain. These tools, supported with developers, allow you to create robust test cases and do integration testing but are heavily limited in scale use, missing necessary DevOps (TFS/VSTS) integrations and the possibility to document test plan executions (ie. test reports in regulated environments).
Recognizing this limitation, in the beginning of 2019, Microsoft introduced its Regression Suite Automation Tool (RSAT). With a different approach for faster test cases creation (recording “happy path” processes and not focusing on single actions) it gives a way faster solution to automate scripts and, more importantly, it gives native Azure DevOps integration that is usually essential for project needs (Pipelines and Test Plans). Addressing the speed aspect of things, it is still woefully lacking when it comes to different needs of a testing team. Without specific functions to support no-code validations, negative tests, try-catches, optional steps, loops, conditional steps and many others, it is not providing real life support for the test team to automate much from a current manual testing scope and therefore, the expected test automation coverage is low and test automation project remains expensive.
Another common challenge for existing automated testing solutions is the maintenance and reusability aspect of things. Dynamics 365 testing projects are specific, the application is live, under continuous maintenance, and with Microsoft updates, business processes are constantly affected with changes. Following these changes, test plans are modified and thus, part of each test cases and scenarios needs to be updated. Neither a “trigger and action” approach (RPA type) tools, nor RSAT are prepared for flexible script modifications and testing teams should be prepared to recreate test cases from scratch EACH and EVERY time a script modification is needed. We have found that this aspect is rarely considered during normal analysis and is one of the main causes for exceeding the planned test automation budget.
Advice: For limited budgets, focus on the main testing problems first. If there are a few heavy and critical integrations to automate, consider an RPA type of tool, but be prepared for difficult maintenance and limitations in scaling. If you need to focus on automated testing in Dynamics 365 FO (without any integrations) consider RSAT, but do not plan much for testing coverage as you will find out it has severe limitations and difficulties regarding script maintenance. Once you are able to accurately calculate the return on this investment (ROI), it will be easier to extend your scope and scale of test automation.
We suggest (suggested approach with Executive Automats):
Summary:
There are a few requirements to meet to make Microsoft Dynamics 365 automated testing project viable and affordable. For a limited testing scope, it is crucial to be able to start quickly in order to create ROI in short term. For extensive test plans, it is important to have an all-in-one tool that may support all testing needs and limit the extra effort needed to manage multiple tools. Must have include:
There are a few common expectations from functional automated testing in Dynamics 365.
Besides the automation challenges listed in the previous points, current automation tools are also inadequate with regards to low reusability and flexibility of automation scripts. While no-code tools (non-enterprise solutions) provide too few specific testing features, large RPA type tools are complex solutions dedicated only to complex testing needs and are limited in test case volumes.
While RPA tools (“trigger and action”, flow type of design) are good to support integration tests and it makes a business case to spend long hours creating and stabilizing single critical interface testing, it is not really a solution when it comes to easier test cases that may be supported with click-by-click type solutions.
Being aware of that, the following concerns are being expressed:
As mentioned in point 2 above, the need to use different solutions is making the project unviable from the cost perspective (potential savings from manual tests), but it is also making things more difficult from the management perspective.
Testing teams are usually dedicated to perform a wide extent of tests. Any need to use different tools or methods is complicating their tasks. Different tools requires specialization, creates bottlenecks, and information flows. In Dynamics 365 projects, teams are interested in managing all ERP testing (including integrations) from a single place (80% Azure DevOps), and are expecting a tool to be integrated into this common workspace. Different test requirements concerning Dynamics 365 are covered with manual or automated test cases. Managing the entire project from a single place allows to keep test completeness, quality, and reports tracking. Limiting the number of supporting tools is essential to simplify the area and improve project effectiveness.
What is more, when working with Dynamics 365, you are probably utilizing part or more of Microsoft ecosystem and solutions. Why not to use tools that are already there? Power Automate is not dedicated for automated testing in Dynamics 365 but may become a perfect match with your testing tools where it comes to a specific interfaces testing. You may support desktop applications, trigger actions and invoke test cases within more complex cross-platform testing.
Advice: Regardless you are planning to switch to an automated framework or yet planning a test project from scratch, it is important to decide on the planned integration testing scope. Select solutions and methods that will allow you to maximize coverage while minimizing tools. If there are very specific needs of very high significance, we suggest to solve them individually (including hard coding methods), for any others, we suggest to focus on a single solution that will cover maximum requirements.
We suggest (all supported with Executive Automats):
Summary: Different testing needs require different approaches. There are already dedicated tools for different applications and methods available to support integration testing, but to properly manage test plan completeness, quality, and reporting we suggest to minimize the number of tools. We recommend using a “satellite” supporting solution only for very specific and individual test cases of a very high significance.
Performance testing is another type of test automation and depends who we are talking with, this is of higher or lesser importance in MS Dynamics 365. Of course, together with Cloud and scalable infrastructure, performance issues have become less common. It is easier now to react to system overloads, but it is still extremally difficult to create necessary benchmarks for cloud infrastructure to support your business as usual.
When talking about UI based test automation projects and mentioning robots mimicking user actions (virtual users) at the application interface, it comes very quickly to the mind why not to use these robots to run in parallel. Running in parallel and mass executing the same functional test scripts may provoke a “close to real” controlled system overload for its measurement (ie. when preparing for system stress at “Black Friday” traffic).
What is disappointing, most existing test automation tools (used for regression testing) are not supporting this idea (to support performance testing). There are many reasons either technical or financial. Tools are not prepared to multiply test executions, schedule scenarios, or just handle specific platform limitations to accept dozens of the same virtual user’s runs. On the other side, to make a real life performance test, we need to simulate different users with different application permissions and thus operating with different virtual users’ accounts. Any tools licensed by robot instance quickly become unaffordable.
Together with the above, a performance test project is getting more and more complicated.
Importantly, this is all to create a single performance test scenario.
Advice: Look for a tool that allows you to create scripts easily and run multiple sessions in parallel without multiplying virtual uses licensing costs. It would be great if you could reuse functional tests prepared in the past (reflecting real use cases) to schedule them into possible performance test scenarios.
We suggest selecting a tool that supports (all supported with Executive Automats):
Summary: It should be a nice side effect of having a UI based functional (regression) test automation framework in place to reuse it for performance testing purposes (to mimic 100’s of real users actions) . The supporting tool should allow you to run multiple, simultaneous session runs with the same virtual account or should allow you to use multiple virtual user accounts for a reasonable price. Having that it is easier and faster to plan different performance tests scenarios in a limited time. Be aware that many tools on the market charge per robot (Virtual User) which is often not openly presented and can quickly balloon project costs.
This point, listed here as the last one, may be just as important as the preceding points. It is common that companies are convinced and determined to proceed with test automation, but are not really convinced about when and how to start. On the other hand, testing strategy and your application maturity may determine whether test automation is already necessary or not.
There are 4 main questions considering “when to start”:
On the other hand, there are questions concerning “how to start” automated testing in Dynamics 365:
Answering the above is definitely not trivial and there is no single answer, but there are few factors worth considering to estimate possible best fits:
Advice: Our advice depends on the combination of the answers to the above, but there are few common rules:
When analyzing “pros and cons” of the automated testing project in Dynamics 365, defining strategy and tools to use, there are several factors to consider and evaluate.
Most of concerns relates to efforts vs effects evaluation, testing risk assessment, tools usability or test project setup. We have learned in time, that the above concerns, not being properly addressed, are stopping companies to change their manual testing “status quo”. What is important here, at the end, all of these concerns are about a tool and methods considered in automated testing. Tool selection determines efforts, time to value, usability and possible strategy and framework to adopt.
Executive Automats solution and testing methodology have been created, by design, to address these concerns and simplify the decision “if” and “how” execute automated testing in Dynamics 365.
With its design, technology and usability, it has become a real solution and best fit for those looking for Dynamics 365 test automation tool :
Following the above article, we have created a Dynamics 365 Testing Tool Checklist to let you evaluate different tools and their possible adoption into you D365 project. Article available here D365 Test Automation Tool Checklist
Executive Automats recording panel in Dynamics 365 Finance & Operations
