Test Automation Framework Integration With Test Management Tools

Many things in the world are connected or integrated either naturally, or someone did that for a reason. Test automation frameworks and test management tools are two different things, and each of them has its purpose, but what if both were joined together and used as one complete solution? Let’s find an answer to this question and see if it is worth considering such an implementation.

Imagine that you could select all the tests you would like to execute, click a single button and get execution results in the same tool. Sounds good, right? It is possible, and depending on the test management tool in use, it can be easier or a bit more challenging to integrate it with your test automation framework, but in the end, there can be several benefits that make this time investment well worth it. Considering the fact that quite a few Test Management tools have a license fee, you might want to use their full potential and capabilities to make the most out of your or your company spendings.

Integration benefits

Automated test scenario management: Test Management tools can be used as storage for your automated test scenarios. If you are using BDD or some keyword-based test automation framework, test scenarios could be stored in your test management tool and fetched directly from there for each execution. The advantage, in this case, would be that any team member could easily do any changes on the scenario step level even without a need to access code in the repository.

Test execution management: Easy preparation of test executions and the possibility to combine required tests. Usually, test management tools contain sections for test executions where users can create their executions, combine multiple tests and test sets together to perform manual test runs. During these runs test execution statuses and additional comments are added manually. The same actions can be done automatically by your test automation framework – gathering required tests for execution from previously prepared test execution tickets and sending back results after test execution is completed. It is also possible to return more details such as failure reasons, failure screenshots, and other valuable information.

Test execution results: Ensures test automation execution result analysis using a test management tool. Test automation frameworks should generate reports after execution is completed to visualize execution results, describe failure reasons, include screenshots in attachments, and even show execution time of individual steps and total execution time. There are enough open source solutions and libraries to choose from for report preparation but also test management tools can be an excellent option to display the mentioned data. In addition, after execution is completed, results can be sent to the test management tool.

Test execution history: Another advantage of integrating your test automation with the test management tool is a handy way to store and access all your previous test execution results. Depending on the tool, each scenario could have its execution history indicating stability, previously occurred problems, and regularity of execution. This ensures less chance of losing valuable data if backup or result storage duty is someone’s responsibility, so there could be a chance that something can be deleted, not copied, etc. No matter how reliable your colleagues are, something like that can happen. Of course, test management tool data must be backed up as well.

Popular test management tools that support integration

Xray plugin for Jira: Jira with Xray plugin ensures all mentioned features – test scenario storage, test scenario export for execution, execution result import, execution history. Xray plugin is offered at an additional cost, so depending on the number of team members and Jira plan, it might become pricey since it adds up to the already existing Jira subscription fee. Read more about the Xray plugin in this blog post – Using Xray for Test Management in Jira.

TestRail: TestRail test management tool has its API to perform all necessary actions for successful test automation framework integration, which means that there are no additional costs involved if you decide to use its full capabilities. In the case of the TestRail Professional Cloud plan solution, you are limited to 50 GB of storage which by time might become an issue as execution history would grow, so clean up at some point might be required, especially if failure screenshots are attached to results. If this becomes a potential issue and money is not a problem, then the TestRail Enterprise Cloud plan with 500 GB could be considered.

TestMonitor: Although TestMonitor’s API is still in beta, it seems pretty advanced and should ensure smooth integration with your test automation framework. Of course, as it is subject to change, there still might be some adjustments that could impact your implementation but probably nothing major that could not be resolved with minor fixes.

Besides these test management tools, there are several more providing API, and if you are already using a specific tool, start with checking its documentation as it might be good enough for such integration, and a change of tool would not be needed.

Implementation

Prerequisites – To better understand this section, it would be beneficial to be familiar with API – Application programming interface, CI – Continuous integration platforms such as Jenkins, and solutions for local server exposure to the public internet, for example, Ngrok (if planning to use TestRail’s cloud solution).

To read more about Jira and Xray plugin usage, you can see this blog article Using Xray for Test Management in Jira, but this time, we will focus on TestRail integration with the test automation framework.

To acknowledge before actual implementation

TestRail API Bindings: TestRail provides API bindings for the following programming languages:

• Java;
• PHP;
• Python;
• Ruby;
• .NET.

For Javascript, it is possible to find open source TestRail API wrappers.

TestRail API Rate limits: For TestRail Cloud, there are the following rate limits (per instance per minute):

• TestRail Cloud Professional subscription – 180 requests;
• TestRail Cloud Enterprise subscription – 300 requests.

For TestRail Server installations, there are no API rate limits.

To avoid exceeding these limits, it is recommended to use bulk API endpoints when adding results – use endpoints for updating all test cases instead of updating each test case individually.

Implementation example

The implementation can be divided into two main tasks – getting scenarios from TestRail and sending back execution results.

Fetching scenarios from TestRail and preparing feature files for execution

In the TestRail, a test run with required scenarios to execute should be prepared. Each scenario also has its own section based on the grouping you use – usually, functionality-based. It makes sense to keep the same grouping during execution to have scenarios correctly grouped in execution results as well in case you use some additional reporting options. In order to keep the same grouping when preparing feature files for test execution, several sequential API requests should be used.

1. Getting project’s Id by run Id:

GET index.php?/api/v2/get_run/:run_id

2. Getting all project’s sections by project’s Id:

Suite Id is optional if a project is operating in single suite mode.

GET index.php?/api/v2/get_sections/:project_id&suite_id=:suite_id

3. Getting tests by run Id:

Additional filters can be applied to select, for example, only failed tests in case of a fail rerun, a limited number of tests (default is 250), or skip a certain amount of tests by using offset, which defines starting. For limit and offset filters, TestRail 6.7 or later is required.

GET index.php?/api/v2/get_tests/:run_id&status_id=:status_id&limit=:limit&offset=:offset

With these requests, you got test scenarios to execute with their steps and all project sections. This information is almost enough to proceed with building feature files. Unfortunately, the missing part is the scenario section which requires additional API requests. Then it is a matter of looping through gathered tests and getting their section. The logic for each iteration is also quite simple – if a section was previously used, then add a scenario to the previously created feature file but if not, then make a new feature file.

For feature files, content writing and formatting is very important so that execution can recognize them – for feature titles section names can be used. Then a proper title for each scenario must be defined, which is also available from previous API responses, and of course, it should be followed by all scenario steps. In the case of an existing feature file, a new scenario entry using the same approach is appended.

Sending execution results to TestRail

Compared to scenario fetching, result sending requires only one API request, but a specific request format must be prepared.

The following endpoint must be used, which requires run Id, which we already used for fetching scenarios, and for content, JSON Array must be provided with each scenario’s execution status and error details in case of failure. This information could be found by looping through your results output file and gathering the required information.

POST index.php?/api/v2/add_results_for_cases/:run_id

Request example:

{
  "results": [
    {
      "case_id": 1,
      "status_id": 5,
      "comment": "Cart submit button not found!"

    },
    {
      "case_id": 2,
      "status_id": 1,
      "comment": "",
      "elapsed": "5m",
    }
  ]
}

Test execution trigger button

As a more advanced but nice to have feature, TestRail offers to add custom buttons to different screens. Using this functionality, you can add a custom button that would trigger your test execution in your CI platform by calling a predefined URL which could also include the actual run Id as a parameter. Run Id is available in TestRail’s test run screen context.

In this example, the trigger button launches the ‘TEST_AUTOMATION’ Jenkins job. It must send a token and run Id to Jenkins to ensure access and fetch required tests from TestRail. Ngrok is used to make Jenkins accessible as in this case it is hosted locally and could not be reached from the TestRail cloud solution.

name: Trigger tests for run
description: Triggers automated tests for a test run
author: Gurock Software
version: 1.0
includes: ^runs/view
excludes: 
js:
$(document).ready(
  function() {
    /* Create the button */
    var button = $('<div class="toolbar content-header-toolbar"><a class="toolbar-button toolbar-button-last toolbar-button-first content-header-button button-start" href="javascript:void(0)">Start tests</a></div>');
    /* Add it to the toolbar */
    $("#content-header .content-header-inner").prepend(button);    
    /* Bind the click event to trigger the automated tests */
    $("a", button).click(
      function()
      {
        $.ajax(
        {
          url: "https://98f406369.ngrok.io/buildByToken/buildWithParameters?job=TEST_AUTOMATION&token=TestRailTriggerToken&runId=" + uiscripts.context.run.id,
          dataType: "json",
          type: "GET",
          success: function()
          {
            location.reload();
          },
          error: function()
          {
            App.Dialogs.error(
              'An error occurred while trying to trigger the automated tests.'
            );
          }
        });
        App.Dialogs.message(
          'The tests are being processed in the background and the results are automatically posted back to TestRail. This page refreshes once the tests are finished.',
          'Confirmation'
        );
        return false;
      }
    );
  }
);

For more information, check the TestRail API documentation, which is very detailed and can help you achieve even more.

Conclusion

Test automation framework integration with Test Management tools can be an excellent way to involve other team members in the test automation process and keep track of product quality changes by using execution history, which can be very important for team and management. Depending on the chosen solution, it might have some additional costs, but the implementation itself shouldn’t cost too many resources, making it a potentially good investment.