Think of the test automation framework as the scaffolding. You have your test project and testing tool ready so probably wondering what is the next step? Well, you also need outlines and execution environment for your automation test scripts. A test automation framework is basically a combination of guidelines, concepts, processes, best practices, coding standards. Also, it covers project hierarchies, modularity, reporting mechanism and all that is necessary for efficient development and execution of the test scripts.
Having said that, it is clearly visible that using a test automation framework is a must, no matter what the size of the project is. There are numerous benefits to using such a frameworks. The most obvious ones are ease of scripting, scalability, modularity, process definition and reusability. Developers might also use more than one automation testing framework to maximize their efforts. However, it is important not to overdo it. Using a mixture of approaches might also introduce confusion and increase the likelihood of errors. That is why a recommended practice is to choose one ideal framework for your specific testing and business requirements and apply it to each tested module.
Other benefits of using test automation frameworks include:
- Maximum test coverage
- Available recovery scenario
- Minimal manual intervention
- Intuitive and ready-to-use reporting
The main advantage of an automation framework for the software testing team is to provide support for automated testing. This means low cost for maintenance. Also, it is worth noting that cloud-based frameworks allow you to run automated tests anywhere there is a connection to the cloud. You do not have to own the actual device you are testing on. This is more cost-effective than other test automation frameworks which require the device to be on-premise, in a test lab or within a closed environment.
Before you decide on one test automation framework, you need to evaluate each of them and determine which one is the most fitting for your organization.
What criteria do you need to take into consideration?
- Maximizing reusability (minimized effort e.g. for repeated regression test runs)
- Communication with additional automation tools
- Ability to automate scenarios based on multiple logics
- Data retrieval from multiple input data stores
- CI readiness
- Tool extensibility
- Logs (error analysis)
- Error handlers
- Custom messages for report
- Parallel execution
- Re-runs for failures
- Result presentation
In order to establish which test automation framework is appropriate for your organization, also ask the following questions:
- What are your project requirements? The test automation framework that you will choose should focus on a specific problem. There are no good or bad approaches to testing – all depends on the particular solution, project scope and software development methodology.
- What types of testing do you plan on executing? Whether you will need to run integration testing, performance testing, load testing or other, you might need to consider them in order to define the project scope and the expected test coverage.
- How was the application built? Consider the application and the technology in question. What is the user experience like?
- Does your application have a highly complex workflow? This will dictate the testing requirements.
- What is the testing budget?
- What costs are associated with each framework? Consider here the license cost of the tool.
- What skillset does your team already have? Think if the team are ready to dive into one of the test automation frameworks or if you need to develop their skills or look outside for specialists.
Creating a good test automation framework is not a one-off task. It involves many steps which require building a detailed set of instructions for test suites automations for a number of applications/ modules. Depending on what needs to be tested, the set of instructions will differ.
Let’s take a closer look at the most common test automation frameworks.
Linear Scripting Framework
This framework is also known as record and playback framework. It is used for creating test scripts for individual test cases (or small code snippets). It is mainly used for testing smaller applications – for a sequential manner of scripting the test scripts and then executing these on simple apps.
Pros: Test scripts are created very quickly. This framework is easily compatible with the existing test automation tools (as they support the record and playback feature).
Cons: Scrips aren’t reusable. Also, they are difficult to maintain.
Behavior Driven Framework (or Behavior Driven Development (BDD) framework)
It is an agile test automation framework that creates test suites to validate application specifications which are provided to the user. It ensures that the application testing results confirm the expected outcome.
Pros: you can create easily readable test scripts. It allows automation of functional validations in a clear, understandable format to business analysts, developers, testers, etc. As this framework does not require programming language, it also allows non-technical staff to change the framework as and when required.
Cons: It requires a lot of communication between the end user and the software testers. Also, BDD framework is not practical for short term projects.
Data Driven Framework
It helps maintain separate sources for data and test scripts. This is useful for multiple application test suites that need to use the same data input for their tests. Multiple data source can be accessed and used for different test scenarios.
Pros: The Data-Driven Testing framework allows you to store the test data in an external system and access it at any time you need to add new scripts to the test case. This considerably reduced the total number of scripts and a lesser amount of code is required to test a complete set of scenarios. Data-Driven Testing also means increased flexibility and maintainability.
Cons: The testing process using DD framework is quite complex and requires proficiency in programming language for developing tests scripts.
Keyword Driven Framework
This framework extends the concept behind the Data-Driven Testing framework. It only allows the tests to extract the test data to an external source but also organize the data by specific codes (which are also referred to as “keywords”). The codes can be used to alter the test scripts and conclude different possible outcomes from the test framework. The keywords ultimately dictate the functions performed by every application.
Pros: The big advantage of this framework is that each keyword can be used multiple times across different test scripts.
Cons: As the framework gradually grows as new keywords are introduced, it becomes more and more complicated.
(Test) Library Architecture Framework
This test automation framework divides the application intro functions by grouping similar tasks within the test scripts instead of breaking up the test cases. Then, the functions are stored in a reusable library. As the functions include tasks with common goals, whenever functionality is needed, they can be called upon by the test script.
Pros: Library Architecture Testing allows greater reusability and offers flexibility in testing.
Cons: However, writing the test script is more time consuming. Test data is lodged in the test scripts so whenever changes are needed, test scripts need to be altered, too.
Hybrid Framework
This approach is a combination of some of the others outlined above, mainly data-drive and keyword-driven frameworks. It is sometimes called the ideal automation framework as it leverages the benefits of all associated frameworks. It is usually based on the combination of many types of end-to-end testing approaches (two or more). It incorporates their better parts and mitigates their weaknesses. Such a framework can be created according to the specific project needs.
Pros: Hybrid Framework can be used for test automation of any application. It also reduces the man-hours spent on scripting the automation code. Once created, it can then be re-used for different applications. Hybrid Testing allows for more scope of testing, increased productivity and higher efficiency.
Cons: Despite the high code reusability, Hybrid Framework requires knowledge of automation testing.
