Essential Tips for Developing a Test Automation Strategy

If we want to briefly explain what a test automation strategy is, it is a general plan on how to introduce test automation into software testing. Test automation strategy usually includes such things as goals and results, risks and their mitigation, what should be the first thing to automate, which technologies should be used, test result reports and test statuses, the environment in which to run the test automation solution, management of test data, and plans for further expansion of the test automation. As you might already know, test automation takes a great deal of effort to do, the same as the development of the software that is going to be tested. There is very little room for error when it comes to bigger choices, like which technologies to use and proper mitigation of potential risks of failure. This is why having a plan and strategy is very important prior to creating a test automation solution.

What are the main items of a successful test automation strategy?

Goals and results of test automation

When considering getting into test automation, one of the first things you should think about is the goals that you want to achieve with test automation. Here is a short list of common test automation goals:

• Increased test execution speed
• Broader code coverage
• Broader web browser coverage
• Broader device coverage
• Broader operating system coverage
• Faster identification of critical bugs and failures
• Reduction of manual testing effort

If the software product is already being regression tested, then it is very likely that the top goal for automation would be increased test execution speed. On the other hand, if the software product is an application for smartphones, then the top goal could be a broader device coverage. It can also be a combination of various goals, depending on the software that is going to be tested with the test automation solution.

Risks of test automation and their mitigation

Another important aspect of a test automation strategy is to know the potential risks that the test automation could introduce and how to deal with them quickly so that no precious testing time is lost. Usually, every test automation solution will be different, the same as with the software that is being tested, so the risks and their mitigation will differ. However, we can take a look at a few examples of potential risks and their mitigation that you might come across:

• Test execution can't be completed in a few hours (or overnight) - Multiple tests could be run in parallel, cutting down the overall test execution time/
• Instability of the test environment leads to test automation solution failures, false positives, and false negatives - Test execution can be done on off-peak hours when the test environment is not being actively used.
• Very high maintenance of the test automation solution leading to false positives and false negatives - Have up-to-date information about upcoming software changes that might affect the test automation so that the test automation solution can be updated accordingly prior to changes being deployed to the test environment.

Which part of the software should be automated first?

As we get to choose which platform/application and its features to automate, we have to consider which one will have the greatest impact. Most times, the priority of each application/platform and its features will be set by the criticality of each of the features and how difficult it is to automate, meaning that the first thing to automate should be easy to do but not too trivial so that the automation has a noticeable impact on the quality of the software rather quickly. The next things to consider should be how much time it takes to test the features manually and how difficult they are to test - how error-prone they are to human error.

Which technologies should be used for automation?

It is always difficult to decide which technologies to use for automation; however, there are a few main points we can consider when choosing technologies:

1. What kind of software is going to be automated? Is it a native mobile application, or is it a web application? Or perhaps it is a hybrid mobile application.
   1. If it is a web application without any complex objects, then you might consider using such popular automation frameworks as Cypress or Playwright.
   2. If it is a web application that contains a lot of rather complex objects, then perhaps a good option would be Selenium in combination with other frameworks that are usually used together with Selenium. This will work great, especially in case you want to automate mobile browsers as well.
   3. If it is a native mobile application or even a hybrid mobile application, or all three combined, then you might want to consider using something like Appium together with Selenium and other frameworks.
2. Another determining factor for tool and technology selection is the alignment with currently used technologies, of course, if there are any.
3. Important to consider is how well the technologies align with the skills of your team. In case it is not applicable and you are not so familiar with the technology or programming language that you are considering using, you can at least check with the development team and find out what they are familiar with so that, when you are in need of assistance, they can assist you.

Reports of Test Results and Test Statuses

There are a few options for reports of test results and test execution statuses as well. For reports of test execution results, we can consider using software that is already available and in use, like test management tools. You might ask why that is even an option for automation. Well, you might be able to integrate the test automation results into your current test management tool. Two very popular options for that are Jira and TestRail. This way, you are reusing the existing tools you have and also getting the reports of test execution in the format you need, which makes it even easier to make them transparent and accessible for every involved party. We also have a blog post that explains how to integrate a test automation framework with test management tools. If, for some reason, this is not a suitable solution, you can also use such tools as ExtentReports or Allure framework, or perhaps even a test reporting tool that is built-in for the test automation framework you are using. Another thing to note is that you could attach the test automation solution and the system under test logs/log files to the test results in test reports, be it only for test cases that are failing (in a sense, that the logs are recorded anyway, but easy to access in case a test has failed for some reason) or for all test cases. This way, it is easier to debug everything and see if any failures have occurred in the system under test.

Another smaller but rather important and useful thing is chat notifications for test results that essentially give real-time updates on testing progress for the whole team. Such notifications can be implemented in communication platforms like Slack or Discord.

To complement the test execution reports and make them more accurate, you should consider adding additional test execution statuses, like skipped, error, or other, where each indicates that a test was skipped for some reason, there was an error within the automation solution or other failure or some other reason why the test wasn't run.

Where will the test automation solution be stored and run?

You should also ask yourself where you want to run your automation. If it is a solution to automate web tests, then CI/CD service on GitLab or Jenkins might be your best choice. If that is the case, you can also check our blog post about 10 Guiding Principles for Effective E2E Test Automation in CI/CD. In the case of mobile or hybrid mobile app automation, you might want to have a local test automation solution with real devices. You have to consider the best option for your situation, as well as consider if the possible choice will be viable in the future. As an example, you currently might have a web application that only supports desktop browsers but, in the future, will support mobile browsers. In such a case, you need to consider whether you will need to switch where you run your test automation solution or perhaps introduce a separate one to run on real devices in parallel with the existing one. Another thing to consider is the storage of the test automation solution itself (or rather its code), and, of course, you should consider a solution that includes version control, too. A few popular options that you might use are Github, GitLab, and Bitbucket. The choice of where to store the code for the test automation solution could very likely be the same as for the code of the software that is being tested.

Test data management

Another aspect of the test automation solution is test data management. This is where you have to figure out what kind of test data you will need for the automated tests. There are a few main things to consider for test data management:

• Is there going to be a lot of test data? If yes, then you will need to use a relatively fast test data storage solution. For this, you should avoid using Excel spreadsheets as they tend to get considerably slow to work with as the amount of data grows. In a worst-case scenario, you should at least switch to using .csv files for test data storage.
• Is the test data confidential, such as that containing personal information? In this case, you need to provide secure storage of test data and perhaps encryption as well.
• Where will the test data be stored? As mentioned earlier, excel spreadsheets and .csv files are an option for this, or, if there is very little test data, you might even store it in the .feature files in case you are going to use Cucumber to write your test scenario. Perhaps even a remote database will be a fitting option. Another way to approach this might be a hybrid test data storage strategy, with small parts in feature files and large sets of test data in databases.

Further plan for expansion of test automation

At some point, you will want to expand in one way or another with your test automation solution. Once the initial features/parts of the software are fully automated, you will definitely want to move forward with automating some other parts of the software. This is why we should plan ahead on where to move forward with the test automation solution once the initial software features have been automated and the automation solution deemed successful.

Same as with initial software features to automate, you should consider which other features/modules/parts of the software have the highest priority and greatest return on investment when automated. This way, you can set a small roadmap of what to automate next and in what order, getting the best benefits of automation.

What will you gain from developing a test automation strategy before starting the development of a test automation solution?

Once you have written your test automation strategy, you can look forward to gaining the following benefits while creating your test automation solution and later running it efficiently:

• You will have clearly defined test automation goals that you want to achieve, resulting in a clearer vision of the test automation solution.
• You will have all your test automation risks known and have ways of mitigating them when necessary.
• You will also know which parts/features of the software to automate first and in what order to automate the rest of the eligible features based on business requirements and risk criticality.
• You will have defined the best-fitting technologies and programming languages to use for the test automation solution at the start and for the expansion later on.
• The test execution reports and test execution statuses will show the information exactly how you need it for the best analysis of the results.
• The test automation solution will be run in the best possible location for your needs, and the test automation solution will be stored in the most efficient way.
• A clear test data management strategy will improve and enhance the acquisition of test data for test execution.
• There will be a clear plan on how to move forward with the expansion of the test automation solution once the initial test automation solution is created.

At first glance, this might seem like a lot, but don't worry. If you need help creating a test automation strategy or consulting on one of the related subjects, feel free to contact us.