Microsoft Dynamics 365 Performance Testing
with Executive Automats
45 min read
Microsoft Dynamics 365 Performance Testing with Executive Automats
CHAPTER #1.
How to understand application
performance in ERP?
performance in ERP?
The performance of an application or software, in general, is most often defined as the number of activities that can be performed in a given time. The more there are, the greater the efficiency. In practice, however, it is a much more complex issue. Especially in the case of the Microsoft Dynamics 365 system. It integrates the functionality of ERP, CRM and many other solutions in the cloud.
In the context of ERP performance, specialists take into account various issues, including application fault tolerance, data processing performance, performance concerning shared resources, workload management, and many others.
About performance tests, as well as daily use of the system, performance is most often considered in terms of:
USER
user experience, sensory performance
application load by many users at the same time
session handling efficiency
A good example would be an employee of a production company. One of his tasks is to prepare a specific document. It must contain several products from the company’s assortment. For this purpose, the employee uses a form with a drop-down list of available products containing tens of thousands of elements. To easily select the ones he needs, the user activates the available filter. The entire operation should take from a few to several seconds. But in practice, it turns out that it takes a whole minute.
Why?
Because the search algorithm is Why? not efficient enough.
In this case, application performance issues may be due to its architecture. So we are talking about inadequate preparation of the software in terms of technology. A perfect example will be the aforementioned product list filter, in which an ineffective algorithm was used, or an element of the form that does not work well with others, etc. This is mainly because ERP software vendors like
Microsoft are constantly developing their products. That is why not all elements of a given solution will immediately work with optimal performance, e.g. a search algorithm. It should be pointed out that ERP systems are usually operated by many users simultaneously.
This implies the need to test session handling performance. It determines the system’s ability to scale in terms of the number of simultaneous user sessions. In other words, to scale under a load generated by a certain number of users at the same time.
In this case, apart from the problems resulting from the application architecture, the reason for the lower performance of the software operating in the cloud, e.g. on the PaaS (Platform as a Service) platform, maybe the service provider’s allocation of too limited resources of hardware infrastructure
(including network infrastructure): insufficient computing power, amount of memory, disk space, etc.
Limiting the speed of execution of business processes is critical for handling more transactions and increasing productivity. Performance testing is therefore one of the basic factors supporting a company’s business liquidity.
Microsoft Dynamics 365 Performance Testing with Executive Automats
CHAPTER #2.
When can I take Performance tests?
In theory, the best practice is to test the system’s performance every time something new comes out. This is due to the simple fact that the vast majority of business processes are interdependent, which means that a change made in one process may translate into the effectiveness of tasks in another process.
In practice, a newly implemented functionality is tested if there is a risk of its operation with lower than expected performance. Also, if the implementation is likely to impact critical business processes, the entire system is tested.
In the context of ERP performance, specialists take into account various issues, including application fault tolerance, data processing performance, performance concerning shared resources, workload management, and many others.
Be fore go - l I v e
For this reason, when a company implements a Dynamics 365 solution, it performs an upgrade to Dynamics 365 - as part of the Microsoft One Version
program - it is recommended to conduct erformance tests of the system. Especially that before initiating ERP in the cloud, its load is limited to a dozen or several dozen people, including process owners and IT experts. While the system is perfect for the load generated by 5, 10 or 30 people under test conditions, it is advisable to check its performance after running several hundred user sessions simultaneously, often in different locations and roles. This also applies to
situations when enterprises plan to rapidly increase their business scale, e.g. by employing a larger number of employees. In practice, this is synonymous with the go-live stage and testing.
Another practical question arises here. What to do with a free
filter or the aforementioned form?
application load by many users at the same time
A similar argument for performing performance tests also appears after initiating
go-live, in the case of a system update. In this context, the details contained in the release notes preceding the Planned Update are relevant. If there is a risk that the update may affect the performance of critical processes, then performance tests are recommended. Another example of post-go-live system testing is a situation when a company expects an increased system load, e.g. in connection with planned trade events, such as Black Friday, Black Weekend or Cyber Monday. This allows verifying how the system will perform larger amounts of tasks and prevent many unpleasant situations. For example, in a situation when the enterprise is unable to respond to the needs of its customers because the system does not handle the processes with the expected performance.
Microsoft Dynamics 365 Performance Testing with Executive Automats
CHAPTER #3.
Performance issues within the standard
The implementation of Dynamics 365 means the adoption of the Microsoft standard and modifications required by the customer. In the first case, the company must take into account that the standard may not be properly optimized yet. Microsoft and its partners are aware of this state of affairs. That gives the opportunity to take actions that counteract reduced application performance.
As a rule, external companies cannot change the standard but can extend it. This means that if, for example, a specific form is too slow, it can be extended with new elements (and hide the standard ones).
Another solution might be to change the way you use the app. A good example is the aforementioned too slow process of selecting items from an extensive assortment list. Especially when searching by incomplete names. In this case, the solution would be to instruct users to search using the full product name.
Yet another solution may be the modification of the form itself. In Dynamics AX /
365 systems, it can be built from scratch, including a more efficient algorithm adapted to support the database of tens of thousands of entities. However, this is an expensive solution, and at the same time, it means tampering with the standard (or bypassing it). And that entails the need to re-test the performance of the Dynamics AX/365 system.
Microsoft Dynamics 365 Performance Testing with Executive Automats
CHAPTER #4.
Best practices and performance testing
Also, creating modifications in the Microsoft standard involves the need to follow Dynamics 365 best programming practices. Their use is to ensure the best possible application performance. This means that architects and software developers create code with a view to the fast and efficient operation of all applications and all functionalities within the MS Dynamics AX/365 system.
However, the use of best practices does not guarantee that each programmed solution will operate with optimal performance. For example, as part of best practices, the use of raw SQL commands is discouraged, but developers are aware that the use of pure SQL commands provides the best performance. Therefore, when a given problem is critical, from the application performance point of view, programmers write commands in pure SQL to reduce the risk of slower operation of a part of the entire application.
This shows that not all the recommendations contained in the best practices are optimal from the point of view of application performance, and they also evolve. Hence, it may happen that the developer will not be able to circumvent some problems, such as in the case of a filter in the form, and the prepared application will cause performance problems.
Microsoft Dynamics 365 Performance Testing with Executive Automats
CHAPTER #5.
Cloud scalability - ERP practice
In this case, the company may turn to the operator of the cloud service, i.e. Microsoft, and report a demand for the use of more efficient cloud infrastructure. As a result, Microsoft must update the data in the so-called subscription estimator, i.e. in a tool that allows you to estimate the parameters of the cloud service that will be made available to the client. Additionally, the latter has to estimate his peak load. On this basis, it is possible to determine what configuration will meet the customer’s business requirements.
Subscription
estimate
activated
estimate
activated
Performance testing telemetry data available
(if applicable)
Production environment sized and deployed
Final round
of validation
of validation
But that’s not all. In the case of modification of the standard, Microsoft may request confirmation that the changes made to the application code are prepared under best practices and the problem to which they relate results from the standard proposed by the company from Redmond.
Moreover, the manufacturer can expect the customer to demonstrate that, despite the best practices, the application still operates at lower than expected performance.
In practice, this means – In the case of MS Dynamics 365 Finance or Dynamics 365 Supply Chain Management – that Microsoft provides the so-called T5 (Tier 5) performance environment, the parameters of which are similar to the production environment. After performance tests as part of this solution and proving that it is insufficient, the manufacturer will improve the parameters of the infrastructure (for the production environment) in the cloud, according to the customer’s needs.
Unfortunately, this process takes time, even several days. This is one of the reasons why it is worth conducting Dynamics 365 performance tests before go-live. This prevents application performance issues from being detected too late, thereby causing business downtime that businesses are typically unwilling and unable to afford.
This prevents application performance issues from being detected too late, thereby causing business downtime that businesses are typically unwilling and unable to afford.
Microsoft Dynamics 365 Performance Testing with Executive Automats
CHAPTER #6.
Who and when identifies application problems?
Application performance issues in the Dynamics 365 service package are discovered at various stages of implementation. They are usually identified by architects and developers. They know the specifics and limitations of the standard, the current problems, and the imperfections of the best practices recommended by the manufacturer. Besides, architects and developers are also responsible for designing and testing the performance of the projects. These should operate on a specific scale and fully meet the needs of customers.
Problems with performance are observed mostly during tests even before go-live. However, there are cases when a company notices difficulties resulting from limited application performance after implementation. This usually happens when the smooth implementation of business processes is disrupted, e.g. long waiting times for invoices, delays related to documenting the receipt of goods, etc. Especially during large trade events, such as Black Friday or sales promotions.
Most often, it results from negligence at the stage of implementing or updating the system. Usually on the side of the company or the implementers who conducted the tests incorrectly or using the wrong tool. In such situations, it is usually too late and resolving the problems requires further and substantial investment to mitigate or overcome the negative effects of the slow operation of the program.
Microsoft Dynamics 365 Performance Testing with Executive Automats
CHAPTER #7.
How to perform Performance tests?
Environemt
When deciding to implement the Dynamics 365 system, as part of the purchased license, the company receives 3 different environments:
The development environment is built with the work of the development team in mind, therefore its performance is adapted to such requirements. The same applies to the test environment (UAT), intended for acceptance tests performed by up to several dozen people.
In turn, the production environment is provided for the load declared by the company. Its parameters are prepared according to the subscription estimator. Many companies very often use the practice of limiting production licenses at the stage of application implementation. It is due to costs and the fact that a relatively small number of people work on it in the course of creating and implementing a production application.
Generally, a production application is very efficient, but not for testing purposes. For this reason, Microsoft provides appropriate testing tools on various layers (tiers: Tier-1, Tier-2… Tier-5). The higher the tier, the more efficient (faster) it is, which means that the environment offers more computing power and a database. For example, the UAT application is Tier-2 (multi-box).
Tier-4 and Tier-5 production environments are most often used for performance tests. It is their parameters that constitute the starting point for determining the performance of the cloud, provided by Microsoft for Dynamics 365. Also, at Tier-5, application performance is tested according to best practices. The maximum system load is determined by simulating traffic for a fixed number of user sessions, the number of specific operations, transactions, etc. The goal is to gather all elements into a coherent scenario that will allow for accurate performance testing on the customer side.
Single user
Within the framework of the environments described above, it is possible to carry out various types of performance tests. For example, in a UAT or a development environment, a company can successfully test performance related to a single user: working filter, a group of buttons, UI form.
And this is the case of the Tier-5 environment, which usually contains a copy of the customer’s production environment, i.e. it has access to all product data. In practice, this means that if in the case of the customer’s production, something works with lower efficiency, then the dynamics with a discount will be within Tier-5. Having to test, the developer or architect can limit himself to the tested application and check the quality of individual elements by entering the role of the user. This kind of testing is relatively simple and not very difficult. Customer consultation usually costs additionally with the use of Tier-5.
Several hundred user sessions
It is different when it comes to the traffic of several hundred users. This can be done using programming methods, but it is an extremely time-consuming and costly process. Ultimately, it does not pay off for any company. Therefore, the project requires an appropriate tool that will record all user activities. For example, when issuing an invoice and then duplicating the entire process for 200 simultaneous user sessions (issuing 200 invoices).
At the same time, it is also worth initiating other processes in which users run, for example, 200 production orders and 200 sales orders for a similar number of user sessions. As a result, the environmental load of 600 employees (performing their tasks in the tested system) will be simulated.
The test scripts must do the same things as the real user. This way each robot will launch the indicated interface, click on the appropriate buttons, activate the selected functions, expand the product list and wait for the searching algorithm to find the element or group of products defined in the filter.
Tier 4 or 5 environment
For performance testing
Robots simulating real users performing real business processes in the system
Performance testing tool
Meanwhile, most performance test tools on the market allow to run scripts that perform specific tasks bypassing the user interface. And it is the UI that is one of the key elements influencing the performance of a business application in the cloud and more. This is because activities in the area of the interface usually generate much more dependencies and processes that involve computing power and memory resources than it would result from the load on the database alone, including the handling of relationships between individual tables in the database.
In other words, by focusing on performance tests based on queries and database relationships, it is impossible to recreate the actual system load generated by the user using the UI. In performance tests for Dynamics 365, it is important to map real projects in specific business processes. This is the only
approach that enables the customer and/or testing team to verify and evaluate the system performance.
Considering the above-mentioned requirements and functionality, one can ask: which of the ERP system testing solutions available on the market – Microsoft Dynamics AX / 365 – is the most effective and profitable to use?
Microsoft Dynamics 365 Performance Testing with Executive Automats
CHAPTER #8.
Executive Automats (EA)
Executive Automats is a tool that allows you to simulate specific user actions within the application, including repetitive and controllable traffic on the interface. In this way, a load is created that is subject to detailed analysis.
It can be implemented from two perspectives:
Load on the processor,
RAM and hard drives
responsiveness
for example, measured by the response time of the user interface. It is about the average system response time between individual clicks made by robots in parallel sessions.
Therefore, Executive Automats allows measuring important elements related to the user experience. For example, initiating 200 concurrent user sessions and measuring 20,000 steps taken, each taking 0.3 seconds to complete, gives an insight into the real capabilities of an application running under a given load. You can run an identical scenario on different infrastructure and get an average time of 0.25 seconds, which is a relatively significant and, importantly, measurable change. Without robots (test scripts) and the possibility of their multiple uses (repeatability), performing similar tasks is impossible.
Currently, there are many tools from the Robotic Process Automation (RPA) category available on the market, with a recorder that allows you to record user actions and then play this session in a test script. Nevertheless, their licensing makes these solutions expensive (per user license). If the test procedure simulates the operation of dozens or hundreds of users, the costs of such a tool are astronomical. In addition, creating test scripts often requires appropriate skills and deeper technical knowledge, and thus is more time-consuming.
Executive Automats is a tool whose primary mission is to allow customers to record and replay their processes as simple as possible. That includes the possibility of building automated performance test scenarios without the help of developers (100% codeless). As it was mentioned before, this tool allows you to run many parallel users sessions, simulating real-life situations, in which the user is using the Dynamics 365 application’s UI.
The XPLUS automated testing suite ensures the stability of the system during peak load. It will be necessary, for example, during the Black Friday or Cyber Monday performance testing – a real-world stress test simulation. In other words, Executive Automats can helpExecutive Automats can help to solve real testing problems in highly customized and integrated environments.
EA makes it possible to implement the Software as a Service (SaaS) or Test Automation as a Service (TAaaS) approach in the best and most practical way: for the customer, no installation is necessary and the setup is completed in 15 minutes, and the tool is ready to perform the first test.
Entities using ERP software, such as Microsoft Dynamics AX/365, gain a considerable number of benefits from the implementation of EA, among others:
intuitive and easy to use automated tests based on recorded scripts
functionalities allowing you to re-use Scripts and combine them with other scripts – minimizing the work required to maintain automated tests that are compatible with business processes
manage automated tests of all environments in one place
planning and autonomous running of test suites
migrate complex data between applications without having to develop integrations
measuring the length of processes (time and step).
Putting it all together Executive Automats offers an UI-based functional testing framework in place to reuse it for performance testing purposes and mimic hundreds of real user’s actions. It allows to run multiple, simultaneous sessions within the same virtual account or multiple virtual user accounts for a
reasonable price. This makes it easier and faster to plan different performance test scenarios in a limited time. Many testing tools available on the market demands payment per robot (a Virtual User). Most of the practices like that are not openly presented and may cause ballooning project costs.
Executive Automats deweloped by XPLUS
Makes automated testing simple
Helps to create test automation value very fast to minimize entry costs
it’s 100% codeless that dramatically improves usability, time to value and adoption
allows to start in few hours and get first test suites running in Day 1
it requires only 2-10 minutes to create a single test case (business process test)
provides testing features required by a real test automation team needs
it may be provided flexibly (when needed) as SaaS or TAaaS from Microsoft Partners
Allows to scale up and support increasing testing needs
supports integration testing (web-based and more)
provides all-in-one features to manage a whole test automation project from one place
is integrated with Microsoft Azure DevOps
Is proven by the market
already a 100+ customers using it worldwide
is certified by Microsoft
incorporated by number of local and global Microsoft Partners
Gives more than automated testing
allows simultaneous virtual user runs to create performance testing scenarios
provides robust test documentation
may be used to create business process documentation
May be incorporated at any time of the application lifecycle
Dynamics AX 2009/2012 to Dynamics 365 upgrades
implementation
after go-live maintenance and keeping up with Microsoft One Version approach
Microsoft Dynamics 365 Performance Testing with Executive Automats
CHAPTER #9.
About XPLUS
XPLUS was established in 2002. Today, the company is co-created by over 100 experts specializing in the implementation and development of ERP systems – Microsoft Dynamics 365. The XPLUS portfolio contains more than 300 successful projects in this area. All of them supported large organizations in the digital transformation process.
XPLUS activities focus primarily on complex projects, including big international companies, usually with multi-departmental, dispersed structure.
Many years of experience, business and technical competencies, and expert knowledge guarantee that customers – from the construction, production and commercial and other industries – will receive the measurable quality of implementation, as well as the implementation of projects within the agreed budget and schedule.
The company uses the Conference Room Pilot (CRP) methodology recommended by Microsoft to implement large, integrated ERP systems. CRP significantly shortens the time of delivering business value in comparison with classic methods (waterfall).
