articles and documents
Top 5 automated testing concerns in MS Dynamics 365
Is automated testing in Dynamics 365 really necessary…?
It is sometimes a consequence of other concerns, but phrased this way, is illustrated with the following statements:
… it is not really that much to test recurrently; we can handle it with manual testing…application updates from Microsoft are not really a big risk – we are not afraid…we may involve business people for UAT and manual testing – enough for our needs…testing is critical only during the upgrade or implementation, there will be no big need for testing
What the Clients face
When questioning the need and importance of Microsoft Dynamics 365 test automation, you probably take into account risk evaluation and ROI analysis. We see that it is often the position of clients maintaining a non-cloud, mature Dynamics 365
application. They are not expecting or predicting any changes and so Dynamics 365 testing scope (and automation) seem too burdensome and of little business value.
What we advise
With the Microsoft One Version approach and more and more companies moving to the cloud to improve delivery, time to value and application maintenance, testing is a must. More frequent updates and releases, on the one hand, make it possible to tailor the system to specific business and industry needs. On the other hand, when standard solutions are improved with new customizations and modifications, this is very likely to affect existing modifications. If you want to prevent critical business processes to fail, regression testing after each update and upgrade is essential. What you also need to remember is that customized apps also evolve and are continuously developed. It might be due to business needs but also the need to comply with new business requirements. Consider all types of testing (including regression and performance testing for Dynamics 365, UATs, etc.), a natural consequence of your software development lifecycle.
Therefore, Microsoft Dynamics 365 automated testing allows you to adopt any application changes faster and with fewer bugs on the way. As research and our testing experience shows, shortening the application lifecycle should be considered more important than the savings resulting from automated testing.
A root cause for limited or no automated testing for Dynamics 365 in place is usually a consequence of two things: either too general risk assessment or inadequate resource and budget estimations.
A closer look and more detailed analysis reveal that in most cases the budget is overestimated (due to inappropriate tools and methodology in use) and risk is underestimated. You need to take
into consideration also the time and effort invested in the maintenance of the continuously updated application if there is no Dynamics 365 testing tool or methodology in place.
Dynamics 365 automated testing project is too expensive
Traditional test automation, with its history starting in early 2000’s, still has a “bad reputation” because of its complexity (see point 3), limited value (point 4) and inefficiency in ERP platforms under continuous development. The need to involve technical people and dedicated resources made test automation expensive and useful only where absolutely necessary. Those who tried test automation in the past have suffered on automation project costs when it was tried on the implementation phases and were suffering even more during the maintenance phase.
Together with the cloud revolution, new testing and process automation tools appeared. Low or now code UI based test automation tools, have been introduced, but the main questions concerning test automation are still valid:
… it is a huge investment to start… it costs even more when it goes to maintenance... I can’t automate much of manual test, so manual tests are still essential
What the Clients face
Test automation tools for Dynamics 365 that are low-code or codeless allow to automate the testing projects. However, the test script creation in most cases is till very time-consuming and hence, does not fit the test automation projects needs. For instance, RPA testing solutions, apart from budget challenges around their start and maintenance, are heavily limited in scalability, they are missing DevOps (TFS/ VSTS) integrations and do not allow test plan execution documentation. And while RSAT addresses the specific needs in terms of test scrip automation, it still lacks a lot of functionalities for
testers such as negative tests, try-catches, loops, optional steps, conditional steps, functions for no-code validations, etc. The challenge lies mainly in finding the tool that allows adequate test script maintenance so that test automation coverage is high. The inability to achieve impacts negatively the test automation project costs.
What we advise
We have found out that neither a “trigger and action” approach offered by RPA tools nor RSAT are adequate for fully flexible test script modification and maintenance. One of the main aspects that is often overlooked during cost analysis – at the same time one of the main causes for going over planned test automation budget – is test reusability. It should take away the need to create test cases from scratch each and every time any script modifications are needed.
With limited budget, focus on the main testing challenges first. You also need to be able to accurately calculate ROI Consider the difficulty in maintenance and scarce scalability if you want to go for an RPA testing tool for Dynamics 365. For Microsoft Dynamics test automation, choose a tool that enables full reusability and flexible script maintenance and modification and possibly allows integrations. Check out how RSAT and Executive Automats compare in this respect (kotwica do artykułu, który powstaje)
You need to meet certain requirements if you want to make your Microsoft Dynamics 365 test automation project viable and affordable. A limited testing scope calls for a quick testing project start in order to create ROI in the short term. With extensive testing plans, you need an all-round (and dedicated) Dynamics 365 testing
tool that supports the needs of the testing teams, limiting the extra effort for managing multiple tools.
Testing different integrations requires different tools…
…we need to use different tools to automate different needs…integration tests are not supported easily, heavy coding required… validations and specific testing needs are not supported with most solutions
What Clients face
Testing teams are required to perform a wide variety of tests. What is more, different test requirements regarding Microsoft Dynamics 365 testing project call for either manual or automated test cases.
As mentioned in Concern #2 (interlink?), the need to use different solutions for different testing needs often makes the Dynamics 365 testing project unviable from the cost and management perspective.
Different tools require not only testing team members specialization but also create unnecessary bottlenecks and hinders information flows.
Dynamics 365 testing projects teams aim to manage all ERP testing (including integrations) from a single place (80% Azure DevOps). They also expect the tool to be integrated into their common workspace. That is why it makes the most sense to utilize the Microsoft ecosystem, its solutions and tools.
What we advise
Regardless of what your Dynamics 365 testing plans are – whether you are switching to an automated framework or planning a big test project and need to start from scratch, first define and decide on the planned integration testing scope. You should select solutions, methods and matching tools that will allow for a maximum testing coverage while minimizing the number of tools. For example, although Power Automate is not a tool dedicated to Dynamics 365 testing, it becomes a perfect match for automated testing with other testing tools when it comes to specific interface testing.
For specific testing needs, we suggest to solve them individually (including the hard coding), for others, it might bring more value to find a single Dynamics 365 test automation solution that will cover the maximum number of the testing requirements.
Why can’t I use automated functional test cases for performance tests…?
Performance testing is another type of test automation and depends who we are talking with, this is of higher or lesser importance in MS Dynamics 365. Of course, together with Cloud and scalable infrastructure, performance issues have become less common. It is easier now to react to system overloads, but it is still extremally difficult to create necessary benchmarks for cloud infrastructure to support your business as usual.
When talking about UI based test automation projects and mentioning robots mimicking user actions (virtual users) at the application interface, it comes very quickly to the mind why not to use these robots to run in parallel. Running in parallel and mass executing the same functional test scripts may provoke a “close to real” controlled system overload for its measurement (ie. when preparing for system stress at “Black Friday” traffic).
What Clients face
Most existing test automation tools (that are used for regression testing) unfortunately do not support the idea of performance testing. Reasons being either technical or financial. Those tools cannot handle multiple test executions, test scheduling or have platform limitations when it comes to dozens of runs of the same virtual user. However, a real life performance test requires simulating different users with different permissions and roles, thus such test requires different virtual users’ accounts. And so, tools that are licensed per robot instance become an unaffordable option. Apart from that, creating scripts for executing performance test scenarios is very time consuming. Executing testing scripts in order and scheduling mass executions during Dynamics 365 testing might also not be possible in some cases. The multiple virtual users’ accounts need not only to be created but also assigned to different scripts which involves a lot of manual work. And ultimately you expect a tool that will support a number of virtual users – ideally with a single performance test scenario – at a reasonable price.
What we advise
Look out for a tool that allows an easy and fast testing script creation and with which you can run multiple sessions in parallel without multiplying virtual users licensing costs. You might also want a Dynamics 365 test automation tool that can reuse functional (regression) tests that reflect real use cases (prepared earlier on) and schedule them into performance test scenarios that you need.
The option that is most cost- and time-effective is to have a UI based functional Microsoft 365 test automation framework in place that you can reuse for your performance testing for Dynamics 365. The tool should allow to mimic 100’s of real users actions, run multiple, simultaneous session runs with the same virtual account or should allow you to use multiple virtual user accounts for a reasonable price. This makes it much easier (and faster!) to plan performance tests scenarios within a limited time. Mind you, many tools on the market charge per robot (virtual user) which is often the hidden price that can make the Dynamics 365 testing projects costs in turn quickly balloon out of control.
Dynamics 365 automated testing scope or strategy is not defined – I’m not sure when and how to start…
What is the best project setup, who to involve?Can I fully outsource that and get Test Automation as a Service?”Should we “start small” from critical processes only and extend in time or is it better to plan long term already from the beginning?”If having manual test plans in place should I recreate 1:1 into automation or search for modification?
What Clients face:
Although there is no single answer, you are probably thinking about the below issues when looking for the best possible solution:
Can we dedicate a QA team to create automated test cases?
Should I dedicate developers to support test automation script creation?
Do we have updated process documentation in place to understand what is needed for functional tests?
Can I involve business people (Subject Matter Experts, Functional Analysts , Process or Module Owners, etc.) to help us design better test cases?
Are there any existing manual test plans that may be used?
Can we prioritize where the application is most prone to errors through updates or which processes are critical for our business continuity?
Do we know how many business processes are supported with our application?
Can I estimate how much I want to cover with automated testing in Dynamics 365?
Is there any deadline when we need to have it done?
Do you need any specific test execution reporting in place?
What we advise
The question when and how to start Microsoft Dynamics 365 test automation is definitely not trivial. Our answer to the above depends on the combination of those issues and your needs, however here are a few common rules to follow when planning Microsoft Dynamics test automation:
PLAN LONG TERM BUT START SMALL
It is important to know what the long term goal is (applications tested, test coverage, timeline), but it is even more important to start where it is most critical.
Our suggested approach is to start from a high level review of application failure risks, determine points of weakness or high significance processes and start from there. With an iteration approach, go back to the plan and extend test plans for other areas, processes and additional test cases to cover more specific risks.
TEST REAL BUSINESS PROCESSES
It is common to create tens of test cases that will not touch properly on real business needs. Without proper business process documentation or without business people involvement, testers are usually neither able to define what is to be tested, nor where the real risks of the application failure lie.
Our suggested approach is to involve those who know business processes and empower them with a solution that they can use to record these processes (for its further recreation into test cases). This approach provides two huge values:
· test cases are based on a real business processes and use cases and tests cases requirements and defined better,
· what has been recorded by a process owner may become a template business process documentation for further training and onboarding purposes.
DO NOT INVOLVE THE TECHNICAL TEAM IF NOT REALLY A MUST
They are always in shortage. Whatever done, they will not stay forever in automated testing. Test automation requires maintenance and today’s chance to have it coded may not be the case in the future.
UI based functional tests with the Record & Play approach are a few times faster than coded test scripts. Using a tool that is easy to use by non-technical people will allow you to keep scripts maintained in the long run.
DO NOT THINK OF AUTOMATED TESTS THE SAME AS MANUAL TESTS
Of course it should be doable to re-create manual test cases into automated test cases if only test results validation may be automated. When creating manual test plans, it is quite common that these are tailored to testing team capacity. In test automation frameworks, you may think “wide” on the possible test cases as there will be no real limitation on how many cases may be executed in a specified period of time. If there is a need to test 10 000 cases in 1 day – it is doable. Think of the test plan as something that you might need to execute often.
FOCUS ON BUSINESS PROCESSES FUNCTIONAL TESTING, NOT SINGLE BUTTONS OR CONTROLS
TEST AUTOMATICALLY, MIRRORING HOW A PERSON WOULD PERFORM THE WHOLE PROCESS.
ERP platforms are enormous and “old-style” test plans to test single buttons, one by one, creating tens of thousands of tests are just not effective. If you know how business is operating you may focus on the processes that test lot of buttons, controls, functions and integrations at once.
UATs are usually the latest test phase but also the most important. If you involve those performing UATs to re-create manual tests into functional automated test cases, you will be able to avoid manual UATs in the future. Again, this is necessary with continuous periodic updates required by Microsoft’s One Version approach..
IF YOU NEED TO START FAST AND SMALL IT TAKES ONLY A FEW HOURS TO GET THERE…
Together with the cloud, test automation delivery takes just a few minutes (SaaS is available). If you chose the right solution, you will be able to create your first test suites and have them running by the end of day 1. If you decide to start small, you can create test cases to test your critical business processes
and start test execution in only a few hours. A few hours to cover a continuous risk or the critical business process failure in the application.
Together with the cloud it is not even needed to do it by yourself. Test Automation as a Service (TAaaS) is already here and you may simply involve those who create tests cases based on business processes to run these cases against your application whenever needed – on demand.
WHEN ANALYZING THE “PROS AND CONS” OF THE AUTOMATED TESTING IN DYNAMICS 365, and DEFINING STRATEGY AND TOOLS TO USE, THERE ARE SEVERAL FACTORS TO CONSIDER AND EVALUATE.
Most of the concerns relate to an efforts vs. effects evaluation, testing risk assessment, tool usability and/ or test project setup. We have learned with time that the above concerns – when not properly addressed – prevent a lot of companies from changing their manual testing “status quo”. What is important in the end is that all of these concerns are about a tool and methods considered in automated testing for Dynamics 365. Tool selection determines efforts, time to value, usability and possible strategy and framework to adopt.
It helps to create test automation value very fast to minimize entry costs
It is 100% codeless that dramatically improves usability, time to value and adoption
It may be provided flexibly (when needed) as SaaS or TAaaS (Test Automation as a Service) from Microsoft Partners
It allows to scale up and support increasing testing needs
Is integrated with Microsoft Azure DevOps
It is proven by the market
Incorporated by number of local and global Microsoft Partners.
It gives more than automated testing
May be used to create business process documentation
May be incorporated at any time of the application lifecycle
After go-live maintenance and keeping up with Microsoft One Version approach