Abstract

There is this myth in most IT organizations and teams that Test Automation is some sort of a magical pill that can take care of improving the quality of the software. It is often considered as a gateway in improving the SDLC processes to achieve quality products. However, one faces the real challenges at the time of implementation of Test Automation.

It feels overwhelming at first place to decide the scope and timing of automation as well as the selection of tools for the same. Even if one clears these initial hurdles, there are always some things that would have been ignored in the first place, which could eventually lead to an inefficient test automation setup. Despite following Automation Best Practices, investing time and effort results does not yield the expected outcome.

The KEY problem is one is NOT adopting the right process at the right time under the right circumstances which results in poor ROI and high cost.

This whitepaper looks at potential issues and proposes ways forward to overcome the same as well as suggests some best practices that when followed could help in achieving maximum returns from a test automation strategy. The main objective of this white paper is to assess, how can one make test automation more cost-effective and improve success percentage practically.

Essence of Test Automation

With the ever-growing demand for reduction in time to market, there is significant pressure on the IT industry to adopt innovative approaches and technologies. To remain competitive in the market, most of these IT organizations face challenges in bringing out new products/features that is stable, reliable, and of very high quality.

Initially, manual testing was considered as the easiest, most cost-effective, and less risky approach to ensuring high-quality deliverables. However, manual testing takes a longer time to complete thereby delaying the product release. Hence, it becomes inevitable for organizations to migrate from manual to automated testing. This has the potential for timely release of the product with desired quality and in a cost-effective way.

Test automation has thus become more and more popular in recent years and has been a constantly increasing activity in the existing software industry. Test automation is often seen with many benefits e.g. to reduce the costs of testing, increase test coverage and effectiveness, shorten testing cycles etc. Since test automation is initially costlier than manual testing, it should be thoroughly studied before being adopted. However, success in test automation can only be achieved if certain best practices are followed diligently before and during the implementation phase of test automation suite. 

Pre-requisites for Test Automation

Before jumping into implementing a test automation plan, one should understand that it is a long process and the desired results are not immediate. One should be pragmatic in the approach in planning an automation strategy based on the business needs.

Before migrating from manual to automation, one should pass through the following checkpoints.

Adhering to these checkpoints is the one of the most practical and easy approaches in strategizing test automation and will help in making a rational decision, thereby saving quite a lot of effort, time and cost.

All these checkpoints are sequential. Hence, one cannot skip one checkpoint and jump to another.

Why to Automate?

It is very much essential to determine the objectives, benefits, and ROI before deciding on Test Automation. Before starting any activity in this area, one should ask, “Why am I automating?”

One should look at:

1. Objective and Benefits to move to Automation (purpose and benefits of migrating from manual to automation must be very clear)
2. Calculate ROI for the current setup considering the following factors:

         a. Validity of application

         b. Frequency of testing

         c. No. of planned releases in the near future (at least the next two years)

         d. Time for implementation of automation project

         e. Automation tool cost

         f. Automation resource cost

         g. Cost of current manual testing

The overall objective of this exercise should be to determine how effective the test automation will be for the project.

Research and experience indicate that it takes almost 10-12 releases to recover ROI on test automation. Also, test scripts keep changing all the time based on the change in the requirement or design. Thereby, a lifetime of a test script is limited.

The main goal of this checkpoint is to ascertain;

• Reduction in cost of testing
• Reduction in release time
• Quality of deliverable
• Effectiveness of test automation as value addition

If all the above goals are achievable and measurable, then it makes sense to move to the next checkpoint i.e. When?

When to Automate?

Once it is decided to go for test automation, the timing of implementing the same becomes very important. One should not attempt to do test automation if the application is not stable and most of the basic regression tests are failing when executed manually. A reasonable amount of stability in the application is expected before implementing test automation. Hence the minimal criteria for the right timing for test automation should be as follows:

1. All the regression tests must be consistently passed with 100% results.
2. All the test cases should be clearly defined with preconditions, test data, and expected results. All these test cases must have been reviewed and finalized.
3. The entire application’s manual testing should be completed for at least two releases with at least 70% pass percentage as the test result of the overall application.

If all the above things are in place, then it makes sense to move to the next checkpoint i.e. Which?

Which test case to Automate?

It is important to know that all the test cases cannot be automated. Some of them will still have to be tested manually. Hence it becomes critical to identify the test cases for automation based on priority and ease of implementation.

Typically following test cases are ideal candidates for automation:

1. Regression Test Cases – Those test cases that need to be run with every new build.
2. Data-Driven Test Cases – Those test cases that use multiple values for the same actions.
3. Test cases that have passed manual validation.
4. Test cases that have more data verification points.

Overall, those test cases that are more repetitive are the best for test automation.

The following test cases are not fit for automation:

1. Usability Test – How easy is the application to use?
2. Test cases that have failed in manual testing
3. Test cases without expected results
4. Test cases that are executed only once
5. Adhoc or Exploratory test cases

Once the test cases for automation are identified, then it makes sense to move to the next checkpoint i.e. How?

How to Automate?

The selection of right tool/framework for test automation is very critical for a successful test automation implementation. It is very essential to select the right tool that best suits overall test automation.

The following points should be taken into consideration while selecting the right tool/framework for test automation:

1. Technology Compatibility – Language used for application development (C#, Java, JavaScript etc.)

2. All the automation functionality/features available – GUI, Event Handling, Database, Value verification, etc.

3. Ease of learning, adapting, and training in the team

4. Ease of usability

5. Scalability, Maintainability and Support 

6. Availability of competent resource

7. Licensing Cost

If based on the above-mentioned factors, a tool/framework is selected for test automation, then it makes sense to move to the next checkpoint i.e. Who?

Who should Automate?

Test automation is not very trivial in terms of selecting a tool and starting with it. Test automation involves writing test scripts for the identified test cases and involves a significant amount of coding. Hence it is imperative to have skilled programming resources who are passionate about testing and test automation. Typically, one would need to identify the right resource based on the type of testing being done in test automation such as Unit Testing, GUI testing, Web Services/API testing or database testing.

The test automation team should include:

1. Automation Testers who are good programmers. They are at par with developers.

2. Resources who have knowledge and experience of manual testing.

3. Resources who are passionate about automation. 

It is very much essential for automation testers to design and develop reusable test programs, and review and test the test scripts to enable a quality test automation process.

If the test automation team is now set in place, then it makes sense to move to the final checkpoint i.e. Where?

Where to Automate?

The test Automation environment is very similar to the overall QA environment as mentioned below:

1. Test automation should be performed on the QA environment and not on the dev environment.

2. Test automation should be done on the same build/release version on which the manual test is going on.

3. Test automation can be on a local setup or VM or cloud. But the same environment should be replicated for other testing.

Benefits of Successful Test Automation Strategy

Test Automation is a long-term strategy that requires an initial investment in terms of effort, time and cost. However, a successful implementation of test automation strategy can bring the following benefits across the table:

1. Reduced time and cost for testing.

2. Increased testing quality level.

3. Shortened release cycle.

4. Reliable results.

5. Increased test coverage.

6. Avoiding mundane and cumbersome tasks.

7. Repeatability (running the same test cases again and again).

8. Ensure the process is in place for future success.

Causes for Test Automation Failure

Even though the intent is good, many times implementation of a test automation strategy fails. Following are some of the primary causes for failure:

1. Lack of time, competent resources, and budget.

2. Lack of process.

3. Wrong selection of automation tool.

4. Application under test being unstable.

5. Lack of programming resources – Purely dependent on GUI automation using record and playback.

6. Automating those test cases that failed during manual testing.

7. Poor test coverage.

8. Tight schedule.

9. Lack of resources with the testing background.

10. Automation test script not reviewed and maintained.

11. Too many changes in the existing features in each release.

12. Improper test environment

13. Inconsistent and unreliable behavior of the application and framework

14. Too much of automation – Not delivering on time

15. Regression tests failing

Conclusion

Based on our knowledge and experience, one can achieve success in implementing an effective test automation strategy by following the above-mentioned checkpoints diligently.

Test automation is a strategic investment and should be done by adopting the right approach at the right time. The main objective is to develop and follow best practices that highlight the various factors to be considered in order to make a wise and rational decision for test automation.