What Is Test Semi-Automation And When You Should Use It

When the topic of user acceptance tests is brought up in projects, most often the discussion comes to the decision between test automation or manual execution. But why not semi-automation?

In this article, I will explain what exactly is meant by semi-automation and why you should keep it in mind when discussing the plans of test execution in the project.

What is semi-automation

As the name suggests, semi-automation is a mix between manual and automatic approach, therefore partially done completely independently by the machine and partially includes careful human attention. It does not matter what is the proportion of automated parts compared to manual as long as at least one crucial aspect of test execution is covered by each and not just a click of a button to execute everything or the investigation of the generated report.

For example, instead of going through the test scenarios completely manually, a script is executed before each test case to open the application, register as a completely new user and approve the user as valid through REST API. This way the work becomes quicker, an annoying and long part of the scenario is skipped and the specialist is not using a corrupted, old user that has been messed with for 5 months in a row and has not represented the real-life user since the first few tests.

Or another example would be to execute manually a part of the test case that seems to be impossible to sufficiently validate automatically. Like evaluating how comfortable the text background is for the reader’s eye or grading whether the generated fantasy character face looks cool and realistic enough for the game standards.

Of course, anything is possible, but why willingly make your life a nightmare?

Why use semi-automation

Manual and automatic test execution both have pros and cons, thus the reason we don’t completely drop one or the other. But not enough thought and time is spent on mixing them and combining the best of two worlds. If not looking only at the extremes, it is possible to create a solution that is more suited to the specific needs of the project.

With that kind of mindset, new questions can flourish in one’s mind. Instead of just “How can I automate that?”, a new question of “Should I automate that?” appears.

But to argue which parts should or should not be automated we have to take a look at the general structure of a single test and what are the pros or cons of executing it manually or automatically.

Test execution structure

Well-designed testing should include the following parts described below. Each of them will be evaluated for manual or automated approaches, although I will not repeatedly mention that if the functionality is quickly and frequently changing then it most likely does not make sense to invest time on automation. Same for the situations when it is physically not possible to execute something either manually or automatically.

1. Preconditions

Before executing the main test steps the system needs to be prepared and set to the specific state, which is needed to start performing the main steps of target functionality. Generally, this is a very repetitive part of the work and is meant only to prepare the system, not to catch a specific bug, so automation should be seen as a clear winner. Most often the same preconditions repeat for several test scenarios, therefore the same code could be reused multiple times.

2. Main test steps

After the system is in the desired state, the main steps can follow. During these steps, the machine or a human is interacting with the target functionality with the objective of going through the previously specified user flow that is expected to be working. The benefits of a manual approach would be the possibility to note down undesirable user experience or a visual defect. On the other hand, an automated approach gives the benefit of speed, parallelisation, precision, and repeated execution. Unless the user experience of the visual resources is of the highest priority, automation should again be the goal.

3. Result validation

This is the most tricky, but also the most important step of test execution that can make or break the whole process. After finishing the main test steps, it is required to test whether the result is as expected, the system works correctly and the end-user will be satisfied. Automation still brings things like parallelisation and repeated execution to the table, so if the results are easily verifiable by a machine, this approach still is the king. But if not, issues begin. Some results are often either extremely difficult and time-consuming to check by a machine, for example, image or audio quality, visual defects or the available specialists lack the required experience for the task. Consequently for the validation part most often the human eye does a better job of evaluating whether the user will be satisfied with the product and whether the result is sufficient.

Let’s take a look at an example of a test case and how to split the flow into these three parts. Imagine we have an online shopping website like eBay or Amazon. Now imagine how rich we would be… okay, now imagine we want to test the item removal from the cart. First, let’s define all the necessary steps and actions that have to be taken to accomplish this imaginary task of an imaginary website.

1. Open the homepage of the website
2. Search for an item
3. Choose the item
4. Put the item into the cart
5. Go to the cart
6. Select the added item
7. Remove the selected items from the cart
8. The cart should be empty

Now we can try and split these into three parts. Let’s start from the end as that part is much more comprehensive to a human mind. Result validation steps answer the question “what should’ve happened?” or “what should the user see?”. Look for a step (or steps) that checks whether the previous steps finished successfully or not. Result validation also most often doesn’t actually interact with the system as the results are received by the previous steps. In this case, step 8 is the only step that validates the result.

Next, the main steps. This part answers “How do I do it?” Any action that is directly using the logic behind the target functionality is considered to be one of the main test steps. In this case, those are steps 6 and 7. In this example, step 5 is not one of the main steps because there might be different ways of getting to the cart and navigation most likely is not directly connected to the removal of items. On the other hand, selecting one or multiple items in the cart (step 6) most likely is directly connected to the actions that can be performed afterwards (step 7).

And the last one, or if we follow the order of steps then the first one—preconditions. This part answers “What do I need to get there?” or “What am I missing when starting fresh?” You probably already guessed which ones are preconditions, those are steps 1 to 5. These are steps that use the main, positive flows to get the system prepared for the actual desired test actions. The big difference between these and the main steps is that if preconditions fail, the defect is considered to be in another part of the system and not connected to the target functionality, in this case, removal of items from the cart.

A simple analogy would be that if you want to play a game but you have to install a Windows update first. This update will be an annoying precondition and actually opening the game and playing will be the main desired part.

Also, if you have used the Cucumber framework, you might be really good at this as Cucumber is designed to split the Gherkin scenarios into “Given”, “When” and “Then” steps AKA preconditions, main steps and result validation. A feature file in Gherkin language would look like this (steps are slightly improved by changing the format and adding specific test data):

Feature: Cart item management

  Scenario: As a user I can remove an item from the cart
    Given I open the homepage of the website
    And I search for an item "playstation 1"
    And I open the searched item
    And I put the item into the cart
    And I go to the cart
    When I select the added item 
    And I remove the selected items from the cart
    Then there are no items currently in the cart
    And the total price of cart items is $0.00

How to use semi-automation

By keeping these conclusions and general test structure in mind, as well as taking into account the limitations of the given system, it is possible to create a better solution for quality assurance. When designing a solution, the basic principles apply – consider the available resources, knowledge, possible risks, and advantages of each approach. But by combining the best of the two worlds you are now ready to create something more suited for the project needs!

It is true that the tools are advancing incredibly fast, we are able to test things that just a few years ago were incomprehensible. For example check out these blog posts about how we test video quality in VoIP applications and how we test mobile applications with our GPS spoofing solution. So it might feel like manual testing should be thrown out of the window as everything can or will be possible to automate. But as these tools grow, so does the complexity. Time and knowledge of the specialist is also an incredibly important resource to consider when deciding on the testing solution. Therefore, use the semi-automation mindset to your advantage when considering these limitations.

Summary

Don’t think of manual or automated tests as a whole, think of them as a combination of separable parts and approaches. If the first objective is semi-automation, then all options and the possibility of any mix are still on the table; each part of test execution can be analysed and a solution can be offered. If it turns out it’s better to automate every part of the process, congratulations, you have chosen a solution that you can be more confident about and that has a higher chance of succeeding. But if the solution is a mix, then you have created a path to something that not everyone could have thought about!