Articles

Top 5 Automated Testing Concerns in Dynamics 365

Testing Project Management
D365 Testing
To ensure that your Microsoft Dynamics 365 business applications run smoothly and meet rigorous business standards and objectives, you need to test them. And testing does not come easy. However, with the right (and automated) testing tool for Microsoft Dynamics 365), you can overcome the challenges and make your Dynamics 365 testing less of a burden for all the testers and business users.

In the below article, we will discuss the most common automated testing concerns when it comes to Microsoft Dynamics 365. They have been discovered during our test automation projects. As a result of discussions with our Clients, we have summarized the most important questions and presented our approach and advice in detail on Microsoft Dynamics test automation.
TOP 5 AUTOMATED TESTING CONCERNS IN MS DYNAMICS 365
Chapter #1

Is automated testing in Dynamics 365 really necessary…?

It is sometimes a consequence of other concerns, but phrased this way, is illustrated with the following statements:

 

… it is not really that much to test recurrently; we can handle it with manual testing


…application updates from Microsoft are not really a big risk – we are not afraid
 
…we may involve business people for UAT and manual testing – enough for our needs
 
…testing is critical only during the upgrade or implementation, there will be no big need for testing
BG-Chapter

WHAT THE CLIENTS FACE

When questioning the need and importance of Microsoft Dynamics 365 test automation, you probably take into account risk evaluation and ROI analysis. We see that it is often the position of clients maintaining a non-cloud, mature Dynamics 365

application. They are not expecting or predicting any changes and so Dynamics 365 testing scope (and automation) seem too burdensome and of little business value.

WHAT WE ADVISE

With the Microsoft One Version approach and more and more companies moving to the cloud to improve delivery, time to value and application maintenance, testing is a must. More frequent updates and releases, on the one hand, make it possible to tailor the system to specific business and industry needs. On the other hand, when standard solutions are improved with new customizations and modifications, this is very likely to affect existing modifications. If you want to prevent critical business processes to fail, regression testing after each update and upgrade is essential. What you also need to remember is that customized apps also evolve and are continuously developed. It might be due to business needs but also the need to comply with new business requirements. Consider all types of testing (including regression and performance testing for Dynamics 365, UATs, etc.), a natural consequence of your software development lifecycle.

Therefore, Microsoft Dynamics 365 automated testing allows you to adopt any application changes faster and with fewer bugs on the way. As research and our testing experience shows, shortening the application lifecycle should be considered more important than the savings resulting from automated testing.

SUMMARY

A root cause for limited or no automated testing for Dynamics 365 in place is usually a consequence of two things: either too general risk assessment or inadequate resource and budget estimations.

A closer look and more detailed analysis reveal that in most cases the budget is overestimated (due to inappropriate tools and methodology in use) and risk is underestimated. You need to take

into consideration also the time and effort invested in the maintenance of the continuously updated application if there is no Dynamics 365 testing tool or methodology in place.
TOP 5 AUTOMATED TESTING CONCERNS IN MS DYNAMICS 365
Chapter #2

Dynamics 365 automated testing project is too expensive

Traditional test automation, with its history starting in early 2000’s, still has a “bad reputation” because of its complexity (see point 3), limited value (point 4) and inefficiency in ERP platforms under continuous development. The need to involve technical people and dedicated resources made test automation expensive and useful only where absolutely necessary. Those who tried test automation in the past have suffered on automation project costs when it was tried on the implementation phases and were suffering even more during the maintenance phase.

 

Together with the cloud revolution, new testing and process automation tools appeared. Low or now code UI based test automation tools, have been introduced, but the main questions concerning test automation are still valid:

 

… it is a huge investment to start

 
… it costs even more when it goes to maintenance

… I can’t automate much of manual test, so manual tests are still essential
BG-Chapter

WHAT THE CLIENTS FACE

Test automation tools for Dynamics 365 that are low-code or codeless allow to automate the testing projects. However, the test script creation in most cases is till very time-consuming and hence, does not fit the test automation projects needs. For instance, RPA testing solutions, apart from budget challenges around their start and maintenance, are heavily limited in scalability, they are missing DevOps (TFS/ VSTS) integrations and do not allow test plan execution documentation. And while RSAT addresses the specific needs in terms of test scrip automation, it still lacks a lot of functionalities for

testers such as negative tests, try-catches, loops, optional steps, conditional steps, functions for no-code validations, etc. The challenge lies mainly in finding the tool that allows adequate test script maintenance so that test automation coverage is high. The inability to achieve impacts negatively the test automation project costs.

 

WHAT WE ADVISE

We have found out that neither a “trigger and action” approach offered by RPA tools nor RSAT are adequate for fully flexible test script modification and maintenance. One of the main aspects that is often overlooked during cost analysis – at the same time one of the main causes for going over planned test automation budget – is test reusability. It should take away the need to create test cases from scratch each and every time any script modifications are needed.

With limited budget, focus on the main testing challenges first. You also need to be able to accurately calculate ROI Consider the difficulty in maintenance and scarce scalability if you want to go for an RPA testing tool for Dynamics 365. For Microsoft Dynamics test automation, choose a tool that enables full reusability and flexible script maintenance and modification and possibly allows integrations. Check out how RSAT and Executive Automats compare in this respect (kotwica do artykułu, który powstaje)

SUMMARY

You need to meet certain requirements if you want to make your Microsoft Dynamics 365 test automation project viable and affordable. A limited testing scope calls for a quick testing project start in order to create ROI in the short term. With extensive testing plans, you need an all-round (and dedicated) Dynamics 365 testing

tool that supports the needs of the testing teams, limiting the extra effort for managing multiple tools.
TOP 5 AUTOMATED TESTING CONCERNS IN MS DYNAMICS 365
Chapter #3

Testing different integrations requires different tools…

There are a few common expectations from functional automated testing in Dynamics 365.
 
Besides the automation challenges listed in the previous points, current automation tools are also inadequate with regards to low reusability and flexibility of automation scripts. While no-code tools (non-enterprise solutions) provide too few specific testing features, large RPA type tools are complex solutions dedicated only to complex testing needs and are limited in test case volumes

 

While RPA tools (“trigger and action”, flow type of design) are good to support integration tests and it makes a business case to spend long hours creating and stabilizing single critical interface testing, it is not really a solution when it comes to easier test cases that may be supported with click-by-click type solutions.
 
Being aware of that, the following concerns are being expressed:
…we need to use different tools to automate different needs
 
…integration tests are not supported easily, heavy coding required
 
… validations and specific testing needs are not supported with most solutions
BG-Chapter

WHAT CLIENTS FACE

Testing teams are required to perform a wide variety of tests. What is more, different test requirements regarding Microsoft Dynamics 365 testing project call for either manual or automated test cases.

As mentioned in Concern #2 (interlink?), the need to use different solutions for different testing needs often makes the Dynamics 365 testing project unviable from the cost and management perspective.

Different tools require not only testing team members specialization but also create unnecessary bottlenecks and hinders information flows.

Dynamics 365 testing projects teams aim to manage all ERP testing (including integrations) from a single place (80% Azure DevOps). They also expect the tool to be integrated into their common workspace. That is why it makes the most sense to utilize the Microsoft ecosystem, its solutions and tools.

WHAT WE ADVISE

Regardless of what your Dynamics 365 testing plans are – whether you are switching to an automated framework or planning a big test project and need to start from scratch, first define and decide on the planned integration testing scope. You should select solutions, methods and matching tools that will allow for a maximum testing coverage while minimizing the number of tools. For example, although Power Automate is not a tool dedicated to Dynamics 365 testing, it becomes a perfect match for automated testing with other testing tools when it comes to specific interface testing.

For specific testing needs, we suggest to solve them individually (including the hard coding), for others, it might bring more value to find a single Dynamics 365 test automation solution that will cover the maximum number of the testing requirements.
TOP 5 AUTOMATED TESTING CONCERNS IN MS DYNAMICS 365
Chapter #4

Why can’t I use automated functional test cases for performance tests…?

Performance testing is another type of test automation and depends who we are talking with, this is of higher or lesser importance in MS Dynamics 365. Of course, together with Cloud and scalable infrastructure, performance issues have become less common. It is easier now to react to system overloads, but it is still extremally difficult to create necessary benchmarks for cloud infrastructure to support your business as usual.

When talking about UI based test automation projects and mentioning robots mimicking user actions (virtual users) at the application interface, it comes very quickly to the mind why not to use these robots to run in parallel. Running in parallel and mass executing the same functional test scripts may provoke a “close to real” controlled system overload for its measurement (ie. when preparing for system stress at “Black Friday” traffic).
BG-Chapter

WHAT CLIENTS FACE

Most existing test automation tools (that are used for regression testing) unfortunately do not support the idea of performance testing. Reasons being either technical or financial. Those tools cannot handle multiple test executions, test scheduling or have platform limitations when it comes to dozens of runs of the same virtual user. However, a real life performance test requires simulating different users with different permissions and roles, thus such test requires different virtual users’ accounts. And so, tools that are licensed per robot instance become an unaffordable option. Apart from that, creating scripts for executing performance test scenarios is very time consuming. Executing testing scripts in order and scheduling mass executions during Dynamics 365 testing might also not be possible in some cases. The multiple virtual users’ accounts need not only to be created but also assigned to different scripts which involves a lot of manual work. And ultimately you expect a tool that will support a number of virtual users – ideally with a single performance test scenario – at a reasonable price.

WHAT WE ADVISE

Look out for a tool that allows an easy and fast testing script creation and with which you can run multiple sessions in parallel without multiplying virtual users licensing costs. You might also want a Dynamics 365 test automation tool that can reuse functional (regression) tests that reflect real use cases (prepared earlier on) and schedule them into performance test scenarios that you need.

SUMMARY

The option that is most cost- and time-effective is to have a UI based functional Microsoft 365 test automation framework in place that you can reuse for your performance testing for Dynamics 365. The tool should allow to mimic 100’s of real users actions, run multiple, simultaneous session runs with the same virtual account or should allow you to use multiple virtual user accounts for a reasonable price. This makes it much easier (and faster!) to plan performance tests scenarios within a limited time. Mind you, many tools on the market charge per robot (virtual user) which is often the hidden price that can make the Dynamics 365 testing projects costs in turn quickly balloon out of control.
TOP 5 AUTOMATED TESTING CONCERNS IN MS DYNAMICS 365
Chapter #5

Dynamics 365 automated testing scope or strategy is not defined – I’m not sure when and how to start…

This point, listed here as the last one, may be just as important as the preceding points. It is common that companies are convinced and determined to proceed with test automation, but are not really convinced about when and how to start. On the other hand, testing strategy and your application maturity may determine whether test automation is already necessary or not.
There are 4 main questions considering “when to start”:
Should I start automation during the D365 implementation phase or wait closer to the go-live phase?
I’m already after go-live, is it still reasonable to start test automations?
…Should we automate in AX2012 if planning upgrade to D365?
…When to start if planning a series of rollouts?
On the other hand, there are questions concerning “how to start” automated testing in Dynamics 365:
What is the best project setup, who to involve?
Can I fully outsource that and get Test Automation as a Service?”
 
Should we “start small” from critical processes only and extend in time or is it better to plan long term already from the beginning?”
 
If having manual test plans in place should I recreate 1:1 into automation or search for modification?
BG-Chapter

WHAT CLIENTS FACE:

Although there is no single answer, you are probably thinking about the below issues when looking for the best possible solution:
  1. Can we dedicate a QA team to create automated test cases?
  2. Should I dedicate developers to support test automation script creation?
  3. Do we have updated process documentation in place to understand what is needed for functional tests?
  4. Can I involve business people (Subject Matter Experts, Functional Analysts , Process or Module Owners, etc.) to help us design better test cases?
  5. Are there any existing manual test plans that may be used?
  6. Can we prioritize where the application is most prone to errors through updates or which processes are critical for our business continuity?
  7. Do we know how many business processes are supported with our application?
  8. Can I estimate how much I want to cover with automated testing in Dynamics 365?
  9. Is there any deadline when we need to have it done?
  10. Do you need any specific test execution reporting in place?

WHAT WE ADVISE

The question when and how to start Microsoft Dynamics 365 test automation is definitely not trivial. Our answer to the above depends on the combination of those issues and your needs, however here are a few common rules to follow when planning Microsoft Dynamics test automation:

  1. PLAN LONG TERM BUT START SMALL

    It is important to know what the long term goal is (applications tested, test coverage, timeline), but it is even more important to start where it is most critical.

    Our suggested approach is to start from a high level review of application failure risks, determine points of weakness or high significance processes and start from there. With an iteration approach, go back to the plan and extend test plans for other areas, processes and additional test cases to cover more specific risks.

  2. TEST REAL BUSINESS PROCESSES

    It is common to create tens of test cases that will not touch properly on real business needs. Without proper business process documentation or without business people involvement, testers are usually neither able to define what is to be tested, nor where the real risks of the application failure lie.

    Our suggested approach is to involve those who know business processes and empower them with a solution that they can use to record these processes (for its further recreation into test cases). This approach provides two huge values:

    · test cases are based on a real business processes and use cases and tests cases requirements and defined better,

    · what has been recorded by a process owner may become a template business process documentation for further training and onboarding purposes.
  3. DO NOT INVOLVE THE TECHNICAL TEAM IF NOT REALLY A MUST

    They are always in shortage. Whatever done, they will not stay forever in automated testing. Test automation requires maintenance and today’s chance to have it coded may not be the case in the future.

    UI based functional tests with the Record & Play approach are a few times faster than coded test scripts. Using a tool that is easy to use by non-technical people will allow you to keep scripts maintained in the long run.
  4. DO NOT THINK OF AUTOMATED TESTS THE SAME AS MANUAL TESTS

    Of course it should be doable to re-create manual test cases into automated test cases if only test results validation may be automated. When creating manual test plans, it is quite common that these are tailored to testing team capacity. In test automation frameworks, you may think “wide” on the possible test cases as there will be no real limitation on how many cases may be executed in a specified period of time. If there is a need to test 10 000 cases in 1 day – it is doable. Think of the test plan as something that you might need to execute often.
  5. FOCUS ON BUSINESS PROCESSES FUNCTIONAL TESTING, NOT SINGLE BUTTONS OR CONTROLS

    TEST AUTOMATICALLY, MIRRORING HOW A PERSON WOULD PERFORM THE WHOLE PROCESS.

    ERP platforms are enormous and “old-style” test plans to test single buttons, one by one, creating tens of thousands of tests are just not effective. If you know how business is operating you may focus on the processes that test lot of buttons, controls, functions and integrations at once.

    UATs are usually the latest test phase but also the most important. If you involve those performing UATs to re-create manual tests into functional automated test cases, you will be able to avoid manual UATs in the future. Again, this is necessary with continuous periodic updates required by Microsoft’s One Version approach..
  6. IF YOU NEED TO START FAST AND SMALL IT TAKES ONLY A FEW HOURS TO GET THERE…

    Together with the cloud, test automation delivery takes just a few minutes (SaaS is available). If you chose the right solution, you will be able to create your first test suites and have them running by the end of day 1. If you decide to start small, you can create test cases to test your critical business processes

    and start test execution in only a few hours. A few hours to cover a continuous risk or the critical business process failure in the application.

    Together with the cloud it is not even needed to do it by yourself. Test Automation as a Service (TAaaS) is already here and you may simply involve those who create tests cases based on business processes to run these cases against your application whenever needed – on demand.

SUMMARY

WHEN ANALYZING THE “PROS AND CONS” OF THE AUTOMATED TESTING IN DYNAMICS 365, and DEFINING STRATEGY AND TOOLS TO USE, THERE ARE SEVERAL FACTORS TO CONSIDER AND EVALUATE.

Most of the concerns relate to an efforts vs. effects evaluation, testing risk assessment, tool usability and/ or test project setup. We have learned with time that the above concerns – when not properly addressed – prevent a lot of companies from changing their manual testing “status quo”. What is important in the end is that all of these concerns are about a tool and methods considered in automated testing for Dynamics 365. Tool selection determines efforts, time to value, usability and possible strategy and framework to adopt.

IT HELPS TO CREATE TEST AUTOMATION VALUE VERY FAST TO MINIMIZE ENTRY COSTS

  • It is 100% codeless that dramatically improves usability, time to value and adoption
  • Allows to start in few hours and get first test suites running in Day 1.
  • It requires only 2-10 minutes to create a single test case (business process test)
  • Provides testing features required by a real test automation team needs
  • It may be provided flexibly (when needed) as SaaS or TAaaS (Test Automation as a Service) from Microsoft Partners

IT ALLOWS TO SCALE UP AND SUPPORT INCREASING TESTING NEEDS

  • Supports integration testing (web-based and more)
  • Provides all-in-one features to manage a whole test automation project from one place
  • Is integrated with Microsoft Azure DevOps

IT IS PROVEN BY THE MARKET

  • Already a 100+ customers using it worldwide
  • Is certified by Microsoft
  • Incorporated by number of local and global Microsoft Partners.

IT GIVES MORE THAN AUTOMATED TESTING

  • Allows simultaneous virtual user runs to create performance testing scenarios
  • Provides robust test documentation
  • May be used to create business process documentation

MAY BE INCORPORATED AT ANY TIME OF THE APPLICATION LIFECYCLE

  • AX2009/2012 to D365 upgrades
  • Implementation
  • After go-live maintenance and keeping up with Microsoft One Version approach