Articles

Step-by-Step Guide to Automated Testing in Microsoft Dynamics

Getting Started

Test fundamentals: Planning and Control

A solid house requires a strong foundation. Using the same logic, a software-based company needs a trusted ERP system. The reliability of which is provided, in particular, by proper testing. The relevant test is not only well thought out and designed, but also constantly monitored, maintained, and reviewed.
TEST FUNDAMENTALS: PLANNING AND CONTROL
Chapter #1

Test Plan components

Test planning, monitoring, constant supervision, analysis and then design. Finally, implementation - which is not even the last step. Then the execution and ultimately - closure, completion of tests. These are the main components of a testing process. Adding that each of them affects the others, it becomes clear that the accurate design of all parts defines the agile character and success of the performed tests.
BG-Chapter
A well-constructed test plan is the base for the next phases. The first element of a solid ERP company foundation.

The test plan is designed to answer the questions of WHAT, HOW, WHO, WHERE, and WHEN to conduct the testing process. This crucial document is developed as a result of the test planning process. It defines, among others, features to be tested, test environments, test cases, and testing tools.

According to the ISTQB glossary, a “test plan” is a document describing software testing scope and activities. It is the basis for formally testing any software/product in the organization.
A test plan simply means a detailed document consisting of the
 
  • test strategy,
  • goals,
  • timeframe or schedule,
  • analysis and estimation,
  • planned outcomes (deliverables),
  • and resources needed to perform testing.
The software test plan helps to specify the effort required to validate the quality of the tested application, product update, or any new component. It is an outline to prepare and run software testing activities as a specified process - well determined, analysed and controlled. Led and supervised preferably by the Test Manager.

Test plans describe not only the scope but also the risks. It indicates which areas should be tested, but also limitations - excluding the range that is not necessary to be involved. It is a desired practice to know what not to test as such knowledge, indeed, saves time and money.

Planning of ERP software tests, like Microsoft Dynamics 365, is no different in terms of its compositions. What makes ERP testing exceptional is the priority and importance of testing for business. Each company has the option to customize the ERP software and adjust the platform to its needs. It requires extensive integration testing to make sure that the ERP system is set up in line. Also when the software is being updated all customized parts should be verified to be ready for new features’ implementation.

Test plans vary across teams, industries, and projects. There is no single set of parameters that makes a test plan the only one “correct”. However, certain elements can make test plans more useful and valuable. For ERP companies, for instance, one typical part is security testing.

To conclude, a proper test plan document indicates what to test, areas to skip, testing pass and fail criteria, deliverables, testing environment needs, responsibilities, requirements of the staff, the schedule, risks, and the scale for feedback. Whoa! Quite a lot, right? Such a comprehensive project demands small steps. This is how we slightly move to the pre-phase of planning.
 
Choosing the methodology and testing standard to create a test plan.
TEST FUNDAMENTALS: PLANNING AND CONTROL
Chapter #2

Identify the business needs and scope of the testing

There is no one-size-fits-all software testing process, but there are typical testing activities, without which, testing goals will not be achieved. The accuracy depends on many factors. Each organization defines in its testing strategy which ventures are part of the process, what test activities are required to be implemented at specific points in the software life cycle.
BG-Chapter
A variety of components influencing the test process in an organization includes among others, software used, the test type, products of tests, project risk factors, business fields involved, and last but not least - operational limitations.

What can restrain the scope of testing? Budgets and resources, as well as the complexity of the ERP process and design. If the company works with sensitive data, contractual and regulatory requirements must be involved. The organization’s test policy and practices, internal or external norms impose special testing standards.

In ERPs, company departments work as an interlinking system. Integrations between sectors protect the smooth running of the business and at the same time create a complicated structure of dependencies. In fact, a business working this way has more modules than the company is even able to test! The ISTQB glossary constitutes a great beginning for understanding all components of the test plan. The more essential part is, however, understanding the company’s environment.

In an enterprise with a complex nature of business processes and information systems, sometimes it is difficult to follow what is actually going on within every single piece of the company. However, the lack of knowledge can lead to a waste of money and an ineffective selection of business processes and its testing needs.
Introducing the APQC process classification structure makes it possible. This framework helps to align business processes with industry best practices and perform benchmarking. APQC developed the classification framework for companies as an open standard to facilitate process management and benchmarking, regardless of industry, size or location. The Process Classification Framework is a categorization of cross-functional business processes, created to objectively compare organization modules and performance.

The APQC model allows to conduct a complex business analysis of the company, as well as to organize the process outlook and, consequently, the methodical approach to the complex project, which is the implementation of an ERP class system and its testing criteria. This is why APQC is frequently used by ERP providers. We do not need to look far - it is the standard for Microsoft Dynamics. Due to that, after the software implementation, APQC matrix can be used as a checklist on what and how deep the company should test its areas.

APQC may serve as an elimination methodology. It is easier to pick from a big jar of processes (these ones that are performed by the company) than trying to write them down from scratch. In a simple world, it is a deductive way to evade the risk of wasting time.

The scope of testing based on business analysis should be set by business representatives as well. Who can tell more about processes conducted in the company, if not people that perform them on a daily basis? Surely it is not easy to specify all the details developed every day in the company. Even the best employee can forget about some parts or consider them as not important to mention. The APQC matrix can help to avoid this danger.
TEST FUNDAMENTALS: PLANNING AND CONTROL
Chapter #3

Tricky EPR testing: the complexity of the system

Organizations adopt and implement ERP software to manage business processes in a more efficient way and better conduct business operations. Testing an ERP means making sure that it works properly in every aspect.
BG-Chapter
Testing validates the business processes and reduces the probability of operational risks. Provides the proper functioning and configuration of each module according to the organization’s needs.

What complicates testing in ERPs is complex dependency. Systems are usually very cross-functional. A simple exemplary process of online orders includes also sending the delivery and collecting the payment which engages at least a few departments during different phases.

While preparing the tests, the manager should be able to predict the whole end-to end process to choose the best-fitted testing technique and prepare different test scenarios.

 

TEST FUNDAMENTALS: PLANNING AND CONTROL
Chapter #4

Test Types

In ERP testing, the focus is made on the functional and performance aspect of the system in specified types of scenarios. Aside from functional and performance points, it should also stress the integration quality.
 
BG-Chapter
To talk about different test types, we need to make a basic distinction between manual and automated testing.
 

Manual tests are performed personally by testers literally clicking through the application elements or using appropriate tools to interact with software and APIs. These types of tests are usually expensive because they require the physical presence of testers who set up the environment and conduct tests. Manually performed parts may also be prone to human error due to the fact that testers may, for example, skip some steps in the test script.

 

Automated tests, on the other hand, are performed by software that implements the assumptions of a previously prepared test script. Tests of this type can differ significantly in their complexity, including both very simple tests consisting of verifying the operation of a single method in a given class, as well as tests checking whether the execution of a sequence of complex actions in the user interface will lead to the same results each time. Automatic software testing is much more efficient and more reliable compared to manual testing. However, the quality of automated testing directly depends on the quality of the developed test scripts.

ERP testing activities may be performed, either manually or by the automation tools using different testing techniques.

One of them, functional testing, focuses on the business outcomes that the process is about to affect. These types of tests center solely on checking the results of the operation, completely ignoring the intermediate states of the system during their execution.

The functional test analyzes the actions of the particular system in terms of available functionalities. The tester plays the role of a potential application user- examining the availability and ergonomics of functionality and the actual operation of the product in relation to its specifications. These types of tests detect errors and malfunctions but do not provide information about the causes of errors. Why? Because they do not see the particular process as a part of the greater system.
A distributed model of data flowing from different places requires verification of all dependencies. It demands integration testing. That type checks the accuracy of operation in functionalities available in the software and the interaction of integrated parts. Verifies if the various modules or services used by the software work well together. This level of testing is more expensive to run because it requires multiple elements of the application to be involved in the procedure.

The integration test may simply check that the application is able to query the database, while the purpose of the functional test is to get a specific value from the database according to the product requirements.
Talking about testing techniques, it is impossible not to mention performance tests. Each day users work with many applications connected to servers via the Internet, which are used by a certain number of recipients logging in, whether in offices, in companies and enterprises, or on ordinary websites. It happens that such an application or website works very well in a limited environment for a small number of recipients, but under the influence of increased activity, various unpleasant incidents occur. Higher website activity - when a number of users log in at the same time, can cause bugs, resulting in a decrease in resource loading. In the end, it may cause a complete server overload. Performance tests should be carried out to simulate the load, which will allow testing the strength of the system. They are intended to make companies aware of the possibilities and criteria of the software operation in conditions of very high use by a group of users, as well as to prevent unforeseen and unauthorized activities.
The techniques listed above are the main testing types most common in ERP companies. Obviously, the full list is more than that. There are plenty of other testing techniques including data handling, adaptability, usability, and implementation. Or - often in data-driven companies - security testing.
TEST FUNDAMENTALS: PLANNING AND CONTROL
Chapter #5

Risk Assessment

Forecasting the risk of failure simply supports the success of a project. Testing means controlling the risk. Performed tests provide feedback by measuring the effectiveness of critical error removal and prevention plans.
BG-Chapter
The risk-based test method includes the chances to counteract errors before they occur and thus reduce the level of failure. It identifies product risks and incorporates information about them in test planning, specifications, and test execution preparation.
 
To ensure that the chances of product failure are minimal, risk management activities:
 
  • provide the assessment of what could go wrong,
  • determine what risk and of what importance the company may be dealing with
  • introduce actions to counteract the risk.
Testing can support the identification of new risks, it helps detect the risks that can be reduced, and it can lower the uncertainty of their occurrence. As part of the planning analysis, the leader of a testing process may indicate risk areas to stress more testing activities there.
TEST FUNDAMENTALS: PLANNING AND CONTROL
Chapter #6

Monitoring and Control

ERP software reduces human errors in business. But as the company’s departments affect each other and ERP modules are connected, a slight change in one can significantly influence the other.
Test monitoring and control is basically a manager activity. The test leading person whose role is to help with planning and controlling the testing process, should know all process connections and include them in the testing plan to properly supervise the works. Test monitoring is the process of evaluating and feedback on the „ongoing” testing phase. Test control is about guiding and developing corrective action based on certain metrics or information to improve performance and quality.

Test monitoring includes gathering feedback on test progress, communicating test results, tracking test metrics, and planning future action based on the calculated indicators. All rests on the shoulders of one man? Not exactly.

Larger ventures frequently require participating business specialists or consultants - people with in-depth knowledge of a business area that enhances a team’s understanding of a given process. Their engagement is crucial, as they play a significant role in each automated testing team - defining business needs, test requirements, and software functionality. Officially they are called Subject Matter Experts, standing on the business side of the testing team. Talking about which…
TEST FUNDAMENTALS: PLANNING AND CONTROL
Chapter #7

The importance of a Test Team

Automated or manual, functional, security, or any kind. Tests require people to perform them. From the process of creating the testing plans, through the realization, monitoring and evaluation.
BG-Chapter
A well structured team that responds to business needs is a key part of a successfully driven testing process. Although automation testing is usually considered to be the script created by developers that test another piece of code, nowadays the script can be made even by non-technical people by using modern automated testing tools (e.g. Executive Automats).

It does not mean that with the automation there is no human component needed. Nevertheless, if the test is performed manually or with the automation tool, a dedicated testing team helps with the performance and delivery of all the benefits. Business representatives that record test-related activity in the system, developers that review it and adjust missing parts, and a Test Manager that controls and verifies if everything goes correctly with the testing plan.
Certainly, testing teams need two points of view on board:
 

Technical frontmen - testers, developers, technical-skilled people responsible for ensuring the code and - in case of automating the process - requirements to test automation solutions that match technical needs. This part of the team should also involve the leader that drives the project - with technical knowledge and understanding of business processes, as the above mentioned Test Manager.

Business insight - when the tech experts do not have the inside knowledge, or if the scope for testing is complicated, the team needs someone to set business priority and ensure that the business value is delivered. In specific situations, testing teams call for more specialized roles such as an expert on a specific technology or Subject Matter Experts - already described. The test team does not require to be expanded, especially when the company uses automated testing tools. For instance, Executive Automats can cover a huge dose of work performed manually by testers - even up to 70-80% of test cases.

TEST FUNDAMENTALS: PLANNING AND CONTROL
Chapter #8

Testing system and data security

To start testing, testers receive system user training to be familiar with the interface and have the knowledge about the main functions of the testing process. They then follow a user acceptance testing (UAT) script to go through all possible business scenarios related to a functional system area and, in the end, record the test results.
BG-Chapter
The UAT script is a formal checklist that guides a tester through test scenarios and allows them to report the outcomes. Evaluated scenarios are categorized as “Pass” or “Fail”. How does it look in practice? Considering Microsoft Dynamics, UAT is typically required before implementing the application update or applying custom code and configurations to the production environment.

If the ERP user cares about the security of data and wants to be sure that no failure of updates occurs, then applying more advanced testing techniques is desired. To accelerate the results, and reduce the long-term costs, business owners decide to use automation testing tools. They let functional power users record business tasks by the recorder and convert the recordings into a suite of automated tests, without having to write source code. It may significantly reduce the time and cost of user acceptance testing (UAT).

Developing software or adding new functions to the existing environment requires running the entire testing process in order to verify how the system is handling specific conditions, like the weight of a large database. In the perfect situation, the entire process is performed under conditions as close as possible to those that will occur when the system is operating in a real production environment. Therefore, it is best for applications if their extensions and updates are recreated on the basis of databases that faithfully reflect the facts. That could be manually impossible, yet available when it is automated.

Setting up a testing system for ERPs raises compliance and sensitive data questions. If the organization implemented data security compliance strategies e.g. to protect the individuals as well as the organization from violating any laws like HIPPA or GDPR, should also include the security aspect in testing. Most companies pay a lot of attention to securing production environments against data leakage, but they treat test environments less restrictively. And this is a large area for abuse.
TEST FUNDAMENTALS: PLANNING AND CONTROL
Chapter #9

Test planning recurrency - monthly modifications to the standard

Tests should be prepared with attention to whether it is worth testing the modules with each release. The test plan should also assume the frequency of the tests.
BG-Chapter
Vendors, like Microsoft Dynamics 365, continuously enhance the system with new functionality to provide businesses with a well-adjusted tool. As a first-part player in the ERP landscape, it is important for Microsoft to do so to stay ahead of the competition. Frequent updates allow the contestant to be on top. However, for clients to benefit from enhancements they must keep adjusting the system to the latest version. Microsoft used to release two major updates per year. In November 2018 they introduced a monthly update cadence that includes application and platform updates. From April 2019 forward, all Dynamics 365 customers need to be on the same, current version - ‘One Version’ which will always be up-to-date. This is a significant change compared to the earlier two per year releases. Clients are advised to respect that in terms of avoiding deadlocks.

Even a small component of a customized system that is not ready for an update may cause a domino-effect. Thus, a small update change in the sales module can require significant modification in the inventory and supply parts. To predict that, the test plan must be comprehensive.
The main goal of the testing outcome is to confirm that all functions and integrations work properly. Software providers usually issue a standard testing kit for users. Microsoft is signified by its Azure DevOps - a toolkit to plan, execute, and track scripted manual testing. DevOPs supports the APQC approach using the logic of the aggregation. That knowledge enables preparing the testing plan and its execution in a more structured way. Microsoft customers receive the software standard that is based on the main processes of the APQC matrix. They can be sure then, that also these components are pre-tested before the official launch. Due to that, the company already knows what parts are going to be supported and tested before the ERP update announcement. The conscious Test Manager is aware then of what components should be tested on a company’s side in a more detailed and meticulous way. It is also a priceless insight for choosing the proper automation testing tool that responds to all needs and speeds up the testing process.
Various automation test providers support more or less comprehensive solutions. Usually offered by ERP suppliers are free of charge standard testing tools. These kinds of instruments do not support all customized environments that the organization introduced to adjust the system to its needs. Companies with such modifications are more likely to be exposed to the update risks. It is recommended for them to conduct more comprehensive and structured tests, provided by commercial alternatives (as the above-mentioned Executive Automats). Effective, tailored, and large-scale test programs are perfect for demanding and sensitive businesses. It is also confirmed that test automation helps in improving the quality of the product while reducing cost and time.
WITH EXECUTIVE AUTOMATS
Build Automated Test Suites For Your MS Dynamics AX/365 without Developers
Ask for a Free Trail

Reviewing the test basis: Analysis and Design

ACCORDING TO THE ISTQB DICTIONARY, THE TESTING PROCESS CONTAINS FIVE STAGES:
  • Test Plan
  • Analysis & Design
  • Implementation & Execution
  • Evaluation & Report
  • And finally, test Closing Activities
In a well created ERP test life cycle, after sketching a comprehensive Planning, the next phases should be a piece of cake. A proper scheme with company-required specifications is a basis for the later step - test Analysis and Design, where specifications are deeply studied and test cases are created. Forethought in Planning can simplify analysis and design by avoiding re-work due to incomplete testing parameters or undefined company requirements.
Analysis & Design test activities include key tasks such as:
 
  • Evaluation of test fundamentals (i.e. requirements, risk reports, interface specifications, software integrity, construction, and design part)
  • Estimation of the testability of an application and its parts
  • Recognition and prioritization of the test conditions based on the test elements, specifications, software structure, and behavior
  • Design of test cases and assignation of properties
  • Identification of the test data needed to handle the test
  • Configuration of the test conditions and recognition of the necessary infrastructure
IN SIMPLE WORDS:
Analysis and Design are about putting a Test Plan into practice. A bitter clash with reality. During Planning many scenarios are defined. When Analyzing, the expectations and the possibilities are compared.
THIS PHASE ANSWERS THE TEST PROCESS QUESTION OF:
WHAT to test by determining the software or its parts.
HOW to do that by determining if the test of a given element should be carried out manually or automatically.
WHEN When is the best time to test by indicating how often the test should be run (365 days a year, only once, or in a special period of time).
REVIEWING THE TEST BASIS: ANALYSIS AND DESIGN
Chapter #10

Analysis & Design Basics

Analysis and Design is the way to estimate and prepare testing activities in the most beneficial way for the company, taking into account the resources at its disposal.
BG-Chapter
In the beginning, the team analyses test specifications to understand the application - the proper flow of data, error messages, and outcomes. All information is diagnosed to verify how clear and complete the specifications are. The reviews and clarifications made at the beginning of this step help to better understand the product and, thus, decrease spending, improve the quality, and reduce time in further phases.
 
ISTQB recommends a 3-phase test Design process which includes creating:
  • test conditions (specifications)
  • test cases
  • test procedures (properties).
The test condition checks a smaller part of the functional specification (it could be signing in or, submitting a form). Test cases indicate an input data set, results that are expected to occur, and test verification conditions. While test procedures constitute an order of steps that must be taken to execute the test. Thus, a major functional specification or an application is fragmented or divided into small, reasonable pieces using test conditions. The test cases center on testing these conditions in negative and positive scenarios. Such fragmentation makes the big software easier to manage and allows to focus on minor details without becoming disoriented by the large documentation.
The positive consequence of spending time on analyzing and designing all the test cases are shown as a good return on the reduced amount of re-work, and gives more comprehensive risk coverage. It has the benefit of less time spent on reviewing and correcting the outcomes. Understanding this dependence can significantly improve the efficiency of the testing process.
REVIEWING THE TEST BASIS: ANALYSIS AND DESIGN
Chapter #11

Company’s “Golden template” and rollouts

Accurately prepared Analysis is essential for large companies, where many local businesses are exposed to a series of rollouts.
BG-Chapter
Global ERP process standards called “golden template” or “core model” are the basis spread out to the other locations. The system is rolled out this way to decrease costs, standardize the product, and enable high-level management. In such cases, ERP is never implemented in each location separately, instead it is copied. Obviously, most regions have to add some changes to the “core” process, for instance due to the financial requirements unique in each country or the special method of storage that may not be available.

The test Planning phase should take the “golden template” into account with special attention. Even though the “core” should mean “the same”, these parts have to be tested in all locations, because of its variables - local modifications.
The Test Manager should bear that in mind while creating the Test Plan and to reduce the amount of work in the future. That is correct! Along with the ERP “core” rollouts, the “gold template” testing can be designed just once and spread into other locations. The company can benefit from the Design created once and share the results.
REVIEWING THE TEST BASIS: ANALYSIS AND DESIGN
Chapter #12

WHAT to test? - Analysis essentials

This is the phase of reviewing and understanding in detail all documents from which component or system requirements can be inferred. They are e.g. documentation of requirements, functional specifications, or design documents.
BG-Chapter
After that the test objectives are defined. This includes answering the question of WHAT actually should be tested in the form of test conditions.

It can be run at the same time, together with or separately from the test Design. The test Analysis apprises and revises the test objectives and risks, defining detailed indicators and goals for success.
 
The decision on the level of detail in the testing should take into account:
 
  • Test basis meticulousness
  • The complexity of the software
  • Product and testing project risks
  • The connection between the test database (what should be tested and how it should be tested?)
  • Testing tools used to manage the project
  • The advancement of the testing processes in the company and the skills of the testing manager
  • Accessibility of the business part (for instance is there a dedicated Subject Matter Expert for consultation purposes?)
Once you have a Test Plan (the general assumptions), now it is the time to analyze the risks. If, for example, the company considers the marketplace as the most essential part of the system - crucial for the business, then in the analysis phase a larger budget should be allocated to this area.

The Analysis limits the scope for testing verifying the balance between benefits and efforts (as the time and capital). As part of the Analysis, using the benefits method, the company defines also what should be automated.

The more often a given area requires testing, the more profitable automation is.
The team analyzes the criteria for risk assessment. The Test Plan considers specific critical modules. During the Analysis step, the project crew is answering the major questions:
 
  • Whether the application part is an element of the software “golden template”? If so, then it must be tested, because of the future rollouts and inevitable scaling effect.
  • Is it an essential and critical area from the business perspective? The team needs to prioritize and estimate the necessity of testing. For example, if the element is essential but not critical, the test cases can be prepared more selectively.
  • Was the part modified from the standard during the implementation phase? Each modification needs more testing.
As the next step in the Analysis phase, the team gives priorities, creates a test suite (catalog of parts and functions divided by the testing method e.g. manual, functional, cross-functional, or smoke test suites), and builds the collection of functions.
The company decides which part of the software should include more testing cases, what should be prioritized, and which functions should and should not be tested. After that assessment, the cross-functional test scenarios are built.

In the analysis, the team considers WHAT to test (what to omit) and WHEN the test should be performed. Imagine that in the test plan there are 500 test scenarios. The Analysis is the moment to choose the valuable ones.
REVIEWING THE TEST BASIS: ANALYSIS AND DESIGN
Chapter #13

HOW to test smart? - Design in practice

The Design phase prepares the element or event of a component or system that can be investigated with one or more test cases. It includes HOW the selected part should be tested step by step by identifying test cases along with developed test conditions, based on the previously carried out test Analysis.
BG-Chapter
In ERP testing, the success of the test depends on the test cases used. To reduce the response time and duration of the project, it is important to design an effective set of test cases that allow the detection of the maximum number of errors.

This phase may begin as soon as the test conditions have been identified and sufficient information is available to allow the creation of test cases.

The test case is a set of input values, execution preconditions, expected results, and post-execution conditions created for a specific purpose or test condition. A test case can include practicing a specific program path or verifying compliance with a specific requirement.

While it can be combined with test Analysis, Design is still a separate activity. Some tasks that normally occur during test implementation, or are included in the “golden template” designing are likely to be incorporated here.

Planning could be done in an Excel sheet or on a piece of paper. Design and Analyze parts are more demanding and require a tool that will simplify the test managing. One of them is DevOps which can transfer testing plans into the tree structure of test suites and test cases.
REVIEWING THE TEST BASIS: ANALYSIS AND DESIGN
Chapter #14

Dedicated testing tools. Are they necessary?

The Test Plan is a document (and literally can be a material document scratched in a notebook). The assumptions included inside are able to come into force, only after the proper analysis of its terms, and with the designated execution path. A paper sheet is no longer enough for that.
BG-Chapter
The ERP consists of different modules that usually reflect the company structure. In the Test Plan, the areas are scanned for their testing needs. Simultaneously the company analyses what kind of test should be delivered in particular departments. Smoke tests, functional, regression, or others are divided into manual and automatic parts, depending on the effective criteria.

For instance, the planning phase assumes to perform functional testing in a production part of the ERP. During the design phase, a hierarchical tree of activities is created. The area has its functions that should be covered by the application (e.g. creating orders in the production department). Based on defined functions test cases are created. The coverage of test conditions by prioritizing test cases can be optimized by creating properties attached to each test case. Properties determine whether the test case belongs to the “golden plate” or if it has high or low priority.

DevOps covers all of the activities in the „Define” tab where all test cases, properties, steps to be performed, and definitions of positive and negative results are determined. Based on them, appropriate tasks to be made by testers are automatically created in the following, „Execute” tab. All chosen tests can be performed in predefined configuration. Repeating them in a different configuration is just a matter of changing the parameters, without the need of creating an additional Design. The Design prepared once allows testers to perform the testing activities with modified conditions. What is more, execution history shows all tests made within the particular test case in various configurations.

As you have probably noticed, we dedicated more space for the test Planning chapter. This is not a coincidence. When the test Planning phase is given much thought and done properly, the Analysis and Design and the following take much less time. Using the proper managing tool, the number of activities that need to be done within this step of the test process shrinks significantly!

Testing shifts into high gear: Implementation & Execution phase

ACCORDING TO THE ISTQB DICTIONARY, THE TESTING PROCESS CONTAINS FIVE STAGES:
ERP testing is not a one-time activity. It is a process that should be planned and well thoughtout. Testing success and efficacy depends primarily on the quality of all the steps and tools used along the way.

There is a time when all the testing points are identified, risk assessment is finished, and the preliminary definition of “what?”, “when?”, and “who should be involved?” are done. The analysis and design steps are finished, and test cases - involving a list of steps, input data, variable data, and the expected result - are created. This is the moment to put all the preparation work into effect: Implementation & Execution.

A comprehensive test plan along with the well-thought-out control results in an accurate analysis and precise design. These are the foundations of the testing iceberg, whose visible tip belongs to the Implementation and Execution phase followed by its “subphases”, Evaluation and Report preparation. Finally, the tip is capped with the Test Closure activities
Before digging into the details, the importance of the selection of suitable testing software should be highlighted. An incomplete tool may significantly decrease the effectiveness of a well developed test. Even if the current testing tool does not fulfill the company’s requirements it still can be adjusted! Taking the example of a testing process developed in the widely-known DevOps. It is a comprehensive solution, yet for the best coverage, that tool should be extended with:
  • An automated testing component, that will handle the repetitive and time-consuming test cases (such as free of charge, but limited RSAT, or more advanced - Executive Automats).
  • A reporting element (DevOps has narrow reporting availability, that can be expanded with an external tool like Executive Automats or others).
TESTING SHIFTS INTO HIGH GEAR: IMPLEMENTATION & EXECUTION PHASE
Chapter #15

Implementation

Test implementation is an action in which test procedures or scripts are specified by combining test cases in a specific order. This includes any other information necessary to execute the test, the test environment set up, and tests ready to be run.
BG-Chapter
Test implementation is an action in which test procedures or scripts are specified by combining test cases in a specific order. This includes any other information necessary to execute the test, the test environment set up, and tests ready to be run.

During the Implementation phase, detailed test cases are created based on previously prepared test scenarios. Then the test procedures are prioritized. Here, often omitted and marginalized, documentation of business processes plays a crucial role. It allows you to accurately describe the test cases, structure the nomenclature, and if there is a need, it enables recently involved people to smoothly join the project. The documentation has to be up-to-date, which may be challenging considering the number of updates an average application has. In this case, a key part is played by Subject Matter Experts, who are always supposed to answer questions on how the process should go if the documentation is invalid.

The implementation of testing does not only refer to the manual testing of the software. When the documentation is prepared and a decent tool is chosen, it is the time for constructing and recording a swath of automated test cases. Most importantly, such cases must be designed to be repeatable and reusable in other test cases. Automated test scripts should be implemented in order to avoid needlessly wasting time and money, and to ensure a good return on investment. Automation only applies to the most profitable elements that are manually ineffective meaning that test plans are not and shouldn’t be entirely automated. To automate the repetitive testing, scripts cannot have any leaks in the data delivery. The input supply process should be as smooth as possible. In the best-case scenario, the closing scripts have the input data introduced from another automated test script.

During the recording of the automated script, user access should be unlimited. It is advised to create scripts using an admin user with full permissions. It ensures no failures caused by the lack of access at some point. Only after the proper fluidity of the script is provided, the user can be changed to a targeted one, with limited permissions. Also, workflows, in which the preparation, review, and approval stage demand different platform access, can be easily automated. Yet the rule stays the same - all scripts must be prepared on an admin account, and only after they are verified as complete, the user can be changed. A good practice is to introduce the „two person” rule and catch all the leaks in data exchange and input. The verification can also be covered by advanced test automation tools like it is done within Executive Automats.
To sum up, in the Implementation phase, Test Managers should ensure that:
  • Manual and automated test cases are checked and scheduled and the test environment is verified.
  • Test data is provided.
  • All risks and priorities are located, set, and verified.
  • Test tools are prepared and checked.
  • Input criteria are established.
Testing can support the identification of new risks, it helps detect the risks that can be reduced, and it can lower the uncertainty of their occurrence. As part of the planning analysis, the leader of a testing process may indicate risk areas to stress more testing activities there.
TESTING SHIFTS INTO HIGH GEAR: IMPLEMENTATION & EXECUTION PHASE
Chapter #16

Execution

Once the automation and the manual solutions are prioritized, and the execution order is established – it is time for carrying out the project. The system or its components are being tested - test cases are run, and their results are collected.
BG-Chapter
Before launching this phase, it is important to ensure that all entry criteria are met. All of the criteria must be defined in the test planning stage and should be mentioned in the approved test plan documentation.
 
A “To-do list” before firing the test execution should check:
  • if all tests are defined and designed
  • if the tools for test management and test automation are prepared
  • if all documentation, including standards for test recording and reporting, is evaluated
Test Implementation and Execution determine test procedures or scripts by linking the test cases in a specific order and taking into account any other information necessary to prepare the tests, set up the environment, and run the tests. But how should the testing be scheduled? There are various tests like automated, regression, or functional tests. Their execution should depend on the actual circumstances and the shape in which the company and the ERP are currently in. During the software implementation, automatic tests should be performed quite often. When new functionality is added, but before merging it with the rest of the platform, a functional test to check whether this build will work fully needs to be arranged. The further along in the ERP deployment, the more variety of test types are needed.

What triggers tests? Execution should occur i.a. with every attempt to include a new update, after finished sprints (regression tests), to check the overall system during a non-working day (automated “Sunday regression” tests), at the construction stage (performance tests verifying if the application can run at full speed and is secure), or suited to the current circumstances (like performance tests of an e-commerce platform before Black Friday).

The testing Execution runs when all test objects are delivered and input criteria are met. Then, Test Managers monitor if the progress is going according to the plan, launch, and control testing, and make sure the test logs gather the record of important details for test cases.
To provide a valuable Implementation and Execution, the best practices are:
  • To automate only manually unprofitable test cases.
  • To make automated scripts reusable, especially in end-to-end tests. The practicality of automated tests means combining scripts that make them functional for each other (the script for input data is used for closing scripts).
  • To avoid test case duplication. Creating one group of scripts that are identical in multiple locations (like allowing for the use of different currencies) prevents wasting money on repeating similar case scenarios.
TESTING SHIFTS INTO HIGH GEAR: IMPLEMENTATION & EXECUTION PHASE
Chapter #17

Evaluation and Reporting

Test implementation includes developing and prioritizing test cases and creating test suites (which are basically collections of test cases used to test the application). It is also the moment of rerunning tests that previously failed. Detection of the badly performed tests is provided by Evaluation and Reporting, where all the test execution results are reported and logged as “successful” or “failed”. Then actual outcomes are compared with expected ones and evaluated.
BG-Chapter
The performance criteria of a test are assessed with previously established objectives. Each test level should be classified accordingly. The assessment of exit criteria means checking test logs against exit criteria, evaluating the demand for additional tests or the urgency to change specified exit criteria, and writing a final test report. Reports are prepared depending on the need - some can be used just for internal teams, some may be a base for compliance or audit purposes. In any case, it is a good approach to have them all collected within one tool - which does not overwrite test results!
TESTING SHIFTS INTO HIGH GEAR: IMPLEMENTATION & EXECUTION PHASE
Chapter #18

Evaluation and Reporting

After the software or its components are tested, it is time for the closure activities. The test completion activities collect data about completed test operations to summarize the experience, test tools, facts, and figures. This subtle end of a testing lifecycle puts the major focus on completion criteria, reporting, assessment, as well as the lessons learned. It is a moment for retrospective meetings that can help the entire project team understand “what went wrong?” and “what was done well?”.
BG-Chapter
Planned results are compared to check if they have actually been achieved and to ensure that all reported incidents have been resolved. The test is also evaluated to gather all the good practices created, which may be useful for future projects, releases, and to improve test maturity.
Among the major tasks of the Implementation & Execution phase are prioritization of test cases and test procedures, creation of test data, creating automated test scripts, and certainly - deployments. Efficient test execution cannot be completed without appropriate test suites and the verification of the configuration of the test environment. Finally, when everything is verified, test procedures are performed. Execution is done manually or within the test tools, following the prioritization sequence. Then the results of the test run are recorded, and during the reporting phase, they are compared with the expected outcomes. No improvement in the software can be made if any of the above phases fail. It is important to keep all of the testing process elements (including the Implementation & Execution part) working smoothly, since at the end of the day, they can significantly affect the quality of each product or project.