What is a Proof of Concept (POC) in Automation Testing?

Failures happen but can be prevented. Imagine that you need to run a test automation project and need to choose an appropriate framework. You do the evaluation process, implement the tool and start testing. Everything seems fine at kickoff. However, as more and more time passes, you see that the software is not tested in the way you would like it to be and you can’t go beyond some other tool limitations, too.

Proof of Concept (POC) prevents such scenarios from happening. A Proof of Concept that is prepared and used for automation testing tool implementation projects can provide far more information than any trial evaluation or even an extended Request for Proposal (RFP). Provided that you involve the right people and communicate the testing strategy clearly.

As the implementation of any automated testing solution requires significant investment and resources, instead of jumping straight into implementation, it is best to conduct a Proof of Concept that get the project plan audited in a very short time and with much less effort.

Let’s analyze automation testing POC in more detail to see what it is exactly, why it is a useful and critical part of the implementation of automated testing and learn how to implement it successfully.

Proof of Concept in itself is a very general concept that reaches beyond testing and software. It is crucial in the decision-making process – the results of POC can be enough guidance to highlight any potential issues and risks before these happen.

For automation testing POC, it might be the case that a consultant is brought in to do it. It is considered the fastest way to bring the experience of a new testing tool framework or approach. Once the POC is successful, the implementation then is highly likely to happen in-house. Still, it might be useful to engage consultants or other experienced professionals to review the ongoing work on a regular basis so that you make sure that the implementation is not bringing any risk or future technical debt.

Proof of Concept shows stakeholders a small-scale showcase so that they see your action plan. It is often used when companies want to adopt a new automation testing tool. It might happen that POC implementation might be the first step of the automation transformation – once the automation tool has been evaluated and selected.

A Proof of Concept idea is to provide most of the necessary information in a very short time. So make sure that you work with experienced professionals and set out the correct questions.

Let’s see how the automation testing POC is helpful for the implementation project.

First of all, it is the final step to test and challenge all the prior assumptions after QA teams have done the tool evaluation. POC aims to verify certain aspects of test automation, such as:

• Defining what can and cannot be tested automatically.
• Distinguishing between manual and automation testing when it comes to their expected outcomes and test quality
• Defining how much automation testing saves, compared to manual testing
• Defining the capability to mee the testing requirements of a given automation tool.
• Identifying any hidden issues of the project that might require changes
• Determining the predicted ROI of test automation

Apart from testing and QA teams, another group that needs a help of a clear and well-defined Proof of Concept are the stakeholders.

The automation testing POC shows the decision-makers if the planned automation solution and its implementation is viable financially. In this way, business owners and investors do not venture irresponsibly. Also, Proof of Concept is the perfect way to show the healthy RO and thoroughly evaluate the projects that stakeholders are about to invest a lot in. This shows all the teams and the business what are the advantages of automation testing and what exactly they will gain from the implementation.

DEFINING EXPECTATIONS

In automation testing, it is necessary to clarify the expectations prior to starting any work. This will definitely prevent any misunderstandings along the way as at such an early stage, the client might take certain things for granted, although the team(s) implementing the project know that those things are outside the estimate scope or simply not feasible.

The main expectations to be defines are:

• Priority of scope
• Reporting frequency
• Maximum time available
• Output from POC

Generally, it is recommended that you aim not for what you need to actually achieve but rather for what has been looked into and is feasible. As POC is to discover issues that might impact the work, any stated aim is a best effort.

Also, it is important to remember not to set the expectation in isolation. They need to be thoroughly discussed and shared between the client and the implementor. By doing so, they will share the common understanding of the venture even before the planning and the project kicks off. Also, any mandatory outputs from the client might be identified and turned into time and cost factors. Discussing the POC expectations also makes it possible to identify the success pre-requisites and agree to them, such as access to environments, etc.

IMPLEMENTATION

When implementing POC, it is recommended to follow the below steps:

• Define the scope of work within Proof of Concept

Here, you don’t necessarily need to cover all the test cases for the upcoming project, contrary to the popular belief. It is enough if the team picks up the most critical ones. Those should relate to the core software functionalities or features that will be mostly used by the end users.

• Know the benchmark

The team should be able to benchmark the POC results against the current testing state. Once the milestones have been established, it will be easier to measure the progress later on during the project.

• Demonstrate both manual and automated testing

The automation testing POC should clearly show that automation will not decrease the test quality.

• Have at least one failed test case ready

To be complete, the Proof of Concept for automated testing project should demonstrate at least one test case that fails. This shows that the automation testing tool is able to find a software bug or defect.

• Define areas where automation can and cannot be applied

The general idea of the automation testing POC is to prove that automation tools is indispensable. However, what POC also needs to demonstrate is the areas where testing workload cannot be automated and the team needs to resort to manual testing instead.

• Determine the expected outcomes

Last but not least, a properly defined Proof of Concept should highlight certain key factors that will help to analyze the results upon completion. These are:

1. Is the automation tool capable of automating all intended features of a given application or software?
2. Can the automation tool run on all the browsers required by the project?
3. Will automation require a change in-app implementation?

Once the Proof of Concept is completed, the team needs to define its outcome. There are three possible scenarios here that need to be considered:

• The automation tool fully satisfies the project requirements

This scenario states that automation tool is feasible. This means that it can be implemented and is approved by the management team. However, if during the execution of automation testing POC are issues might crop up, they should be assessed accordingly and any changes should be introduced, too. The next step is working on a pilot project.

• The automation tool does not satisfy the project requirements

This scenario means that the automation transformation project either need to be cancelled altogether or postponed and evaluated further.

• The automation tool partially satisfies the project requirements

This outcome of POC means that the team need to gather more data to be able to make the final decision. In order for this decision to be more informed, the team might need to, e.g., revise the project requirements.

There are also some best practices and tips that might come in handy when doing the automation testing POC:

• Keep the Proof of Concept as simple as possible. There is no need to include a lot of priority questions, a few will do.
• Make sure that you involve experienced people in POC and its scoping
• Communication is key so make sure that all the parties are always on the same page.
• Reduce the time of the Proof of Concept to a minimum. It is always better to extend a successful POC that is short than invest time and find out only later on that it was failing.
• You might need to create a “legacy” description of the POC work which also covers the recommendations.
• Find time for handover within POC. Obviously, the final report might make the handover not necessary, however it is best to keep a time window open if it is required anyway.