Using Xray for Test Management in Jira

First things first – this article will not be about the black & white broken bone images that you can get from your doctor’s visit after doing something stupid. But if you are familiar with various Jira add-ons then, yes, you might have already guessed it. We will take a look at one of the most popular Jira add-ons for test management purposes – Xray. As the number of new tools in this niche keeps on growing and the old tools keep on living, we figured that a comprehensive article of Xray’s main advantages and shortages could be useful – especially if you are currently still looking around the market for the right test management setup for your project.

So, Xray’s first release for Jira was back in 2013. We can say it is fairly established in the testing community and I would say that it has proven to be worth standing next to the most popular commercial test management tools like TestRail, Zephyr, and Hiptest. It has always been maintained by a fellow Portuguese company called ‘Xpand IT’, but no worries – this review, article, summary or however you want to call it is not sponsored by them. Actually, quite the opposite – it will be an open, honest summary from a QA engineer who has been working with this tool for about a year and wants to share his experience.

This article will mention all the necessary topics you need to know, but before we dig in, I want to mention specifically what we will take a look at and which points were chosen for this review to evaluate this tool. These points are:

• Organizing test cases;
• Integrating with test automation;
• Managing release test plans;
• Customizing reports.

Organizing test cases

While Jira wasn’t initially designed to serve its purpose as a test case storage space, with Xray it can be configured to support this functionality in a couple of different ways.

What we basically get with this add-on are a few new pages in the sidebar – Xray Reports, Xray Test Repository, Xray Test Plan Board and new adjustments for ticket types like ‘Bug’, ‘Pre-condition’, ‘Test’, ‘Test Set’, ‘Test Execution’ and ‘Test Plan’ to help us better organize our daily testing work.

Here’s a quick example of how a manual test case creation process will look like. When you start creating a new scenario, in this case, a new manual test, a separate ticket with type ‘Manual’ needs to be created.

After that, with this ticket type, you are able to document your test steps, add test data, write down what are the expected results for each test step, and, finally, add necessary attachments. Also, a useful thing to note is that you can avoid re-writing repetitive pre-condition steps, like logins, navigation actions to pages and so on by simply adding previously created ‘Pre-condition’ type tickets at the start of each test scenario.

But is there automated, behaviour driven test scenarios written in Gherkin you may ask? Yes! There’s a specific ‘Cucumber’ ticket type to support that, but I will get to that later when we will look specifically at Xray’s test automation & CI integration.

Let’s say you have successfully completed writing some of your test scenarios. What’s next? One of the key advantages of keeping your tests as separate Jira tickets is that they can be easily linked to already existing development User stories. This gives the opportunity for not only us, but also developers, stakeholders, and everyone else, who is checking the Jira stories, to quickly see the testing status and coverage for each feature.

This is done via Jira’s handy ‘Link’ functionality, which allows us to connect all of our test scenarios to each corresponding story that needs to be tested. Sounds great, right? Yes, in theory… Personally, this ‘Test Coverage’ section was where I noticed most malfunctions with this integration. Maybe it was the Jira version, maybe it was mismanagement in the settings, but there have been more than a few times where I’ve noticed that the test status (passed, failed) does not update properly or feature is clearly linked with tests, but the test coverage status is displayed as “Uncovered”. I’m not saying this will happen to you, but you have been warned!

In conclusion, if you’re familiar with Jira, maintaining your tests with Xray will be a piece of cake. There’s also a separate “Xray Test Repository” page available to quickly access all of your tests in one place where you can categorize them by however you wish – either by creating new folders and sub-folders or by using countless filters. For example, you can filter those test cases which are not yet linked to stories and so on to make your life easier.

Integrating with test automation

Essentially with Xray you can manage both manual and automated tests in a similar way. To enable Xray’s support for test steps written in Gherkin syntax, first, you will want to open a separate ‘Test’ ticket and change its type to ‘Cucumber’. Secondly, you will obviously want to implement your test automation code for this specific scenario. And, lastly, you will want to find a solution on how to publish the results after each test run using Jira’s REST API.

Is your test automation setup able to push the results in Jira/Xray? The answer is simple – yes, if it can achieve these minimum requirements:

• Results get parsed in a specific Xray JSON format after each run;
• Test cases have the corresponding Jira ticket links as tags (f.ex, @TDL-123);
• Statuses for each test case & step (pass or fail).

Xray has in-depth REST API documentation that explains the whole process of importing your results once you have them in this final, valid payload. Here’s an example of two tests run:

{
  "info": {
    "summary": "Automated test run - 27.06.2019 14:17",
    "description": "This execution is automatically created when importing execution results from an external source",
    "startDate": "2019-06-27T11:03:02+02:00",
    "finishDate": "2019-06-27T11:05:33+02:00",
    "testPlanKey": "Q4E-42",
    "testEnvironments": [
      "UAT",
      "chrome"
    ]
  },
  "tests": [
    {
      "testKey": "Q4E-32",
      "start": "2019-06-27T11:03:02+02:00",
      "finish": "2019-06-27T11:05:33+02:00",
      "comment": "Successful execution",
      "status": "PASS"
    },
    {
      "testKey": "Q4E-34",
      "start": "2019-06-27T11:03:02+02:00",
      "finish": "2019-06-27T11:05:33+02:00",
      "comment": "Execution failed. Example Step #2 FAIL.",
      "status": "FAIL",
      "steps": [
        {
          "status": "PASS",
          "comment": "Comment for Test Step 1"
        },
        {
          "status": "FAIL",
          "comment": "Comment for Test Step 2"
        }
      ]
    }
  ]
}

The kicker here is the following – your framework most likely won’t generate this format on its own right away. How to get to that particular report in JSON will be a journey, and additional scripting is usually necessary to parse your test result information in a valid format. There is good news, though. Github community already offers a list of JSON reporter extensions for almost every popular test automation framework there is. Personally, I am using Webdriver’s JSON reporter powered by JavaScript and some work still had to be put in to get the desired format, but this combo dealt with this task pretty well – mainly thanks to JS built-in methods like JSON.parse() and JSON.stringify().

In the end, we should get a separate ‘Test Execution’ type ticket that displays a nice looking summary of our automated test results:

An important thing to note here is that a new ticket gets generated each time you run your automated tests. A project’s backlog can pile up pretty fast with these new tickets, especially in the starting phase when you are trying things out.

Managing release test plans

Together with Xray’s version 2.0 came a specific ‘Test Plan’ issue type which gives us the ability to add one or more separate tests and also multiple test executions.

How is this useful? Usually, you have several test sets and not just one that you are planning to use for testing the release candidate. For a simple, practical example let’s take only these: the previously discussed automated test execution and some specifically selected manual tests. What you can do is, again, link this previously run test execution ticket to your new test plan and Xray will automatically add all of its contents with its latest statuses.

Personally, I feel like this is one of Xray’s most valuable features and it has saved me tons of time, where I would otherwise add each test case one by one. Also, if the contents of your test plans don’t change radically from version to version then what you can use is the ‘clone ticket’ functionality that Jira offers.

Customizing reports

Yes, Xray does offer some built-in reporting features and test coverage metrics, however, they seemed too basic and with a lack of flexibility for my specific project requirements.

Luckily, I was able to find another custom solution on how to make the quality reports much more presentable and customizable. Instead of default Xray reports, I would advise building personalized reports using integration with Confluence (another Jira’s by-product for managing documentation) with enhanced JQL (Jira Query Language) filters and charts. Main benefit for creating reports in Confluence is that data gets taken from Jira and it stays dynamic, so you are able to adjust your own layout – this is exactly what we need.

So, let’s pick an example – if we would want to include information regarding the total number of bugs found within a specific version, then we can simply use the following JQL filter: project = yourProject AND issuetype = Bug AND fixVersion = 1.7.0

Considering that in this particular case bug tickets can also be sub-categorized by priority (major, minor, trivial, critical & blocker), I would choose to display this information as a chart, but it’s totally up to you how the final report will look like. There’s also an option to just display a number of tickets that the JQL filter was able to find within your project.

Xray offers in-depth readme with useful JQL examples how you can crunch out the numbers you might need with these filters, but, again, you will have to use your own imagination and style to craft a good looking report.

Here’s a list of some JQL filter suggestions that you might also find useful and can include in your reports.

• Total number of bugs found within project:
  project = yourProject AND issuetype = Bug
• Total number of bugs fixed within project:
  project = yourProject AND issuetype = Bug AND status in (Resolved, Closed) AND resolution in (Fixed, Done)
• Critical/ blocking/major issues found within version:
  project = yourProject AND issuetype = Bug AND priority in (Blocker, Critical, Major) AND fixVersion = 1.7.0
• Total number of User stories delivered:
  issuetype = Story and project = "yourProject" and (labels not in ("NO_QA") or labels is EMPTY) and resolution = Done
• Stories delivered and covered by at least 1 test case:
  issueType in ("Story") and project = "yourProject" and status in ('Closed','Resolved') and resolution = "Done" and status not in ("New") and (labels not in ("NO_QA") or labels is EMPTY) and issue in requirements("OK|NOK|NOTRUN","yourProject","","","true")
• Number of test cases finished:
  project = yourProject AND issuetype = Test AND status in (Resolved, Closed)
• Number of automated test cases:
  project = yourProject AND issuetype = Test AND "Cucumber Test Type" = Scenario
• Number of passed tests within a test plan:
  issue in testPlanTests('Q4E-1588','PASS')
• Number of failed tests within a test plan:
  issue in testPlanTests('Q4E-1588','FAIL')

So is Xray the answer? Is it the best and the only solution in terms of the whole testing process management? Definitely not for the masses, but it could be if you’re specifically looking at Jira add-ons. There are tons of really great alternatives out there like Zephyr, TestRail, qTest, etc. It really depends on the organization and functionality requirements.

Truth is – every product team is different, but there’s always the right tool for each one of them in the market. We have been helping companies to pick the right tools and make the right choices in terms of the test management process for more than a decade. What’s the best way for your organization? Contact us for more information and see how we can help in your specific case!