Adding Browser Logs to Your Capybara/Cucumber UI Test Report

UI automation is a big part of the testing industry, because people appreciate reliable advantage of finding regression errors. Even the most advanced QA can make mistakes during monotonous manual testing, but automated tests do not. If you are a test automation engineer or don’t know what kind of test framework to use this is my positive experience with UI test framework called Capybara.

Capybara provides good test starter pack to automate web UI tests as well as many helper methods such as click_in, fill_in, click_button etc. Capybara includes a feature file with already defined test examples and a well-written documentation.

Capybara framework provides an interface for interacting with websites from Ruby. It is a very handy tool, because it uses HTML structure to simulate real user actions. To be more precise Capybara does not interact with website, it is like layer or a bridge between actual web driver.

Capybara supports 4 drivers – RackTest, Selenium, Capybara – Webkit and Poltergeist:

• RackTest is written in Ruby and doesn’t have JavaScript support. It’s used just for Rack applications. Besides that you cannot use the RackTest driver to test remote applications.
• Capybara – Webkit is used for headless testing. It uses QtWebKit to start a rendering engine process. It is significantly faster than drivers like Selenium since it does not load an entire browser.
• Poltergeist is another headless driver which integrates Capybara with PhantomJS. It is truly headless, because it doesn’t require Xvfb to run on your CI server. It will also detect and report any Javascript errors that happen within the page.
• Selenium webdriver – which is mostly used in web-based automation frameworks. It supports JavaScript, can access HTTP resources outside of application, and can also be set up for testing in headless mode which is especially useful for CI scenarios.

For step definition Cucumber uses Gherkin language. The first part of this syntax starts with the keyword Feature, which in this case is “Login page”. Feature can also include optional description, which is meant to help understand the context. Scenario is the case name, which also will appears in test reports. Scenario name can be appended with case number if you are using some kind of QA management tools, that compares manual and automated case coverage. Given, Then, And, But are just Cucumber acceptance test format keywords. Order of these words can be changed, the main idea is to keep case simple.

Feature: Login page
Scenario: Successful login
 Given I am on the Main page
 Then I enter user credentials
 And I press login
 Then I check if user is located on main page
 And I log out

Our main goal is to improve Capybara/Cucumber reports with debug data and keep it working separately for each case. The following are tips on how to generate Cucumber JSON test result file, by using it locally and even in Jenkins CI.

Basic Capybara setup

• Install Ruby: https://www.ruby-lang.org/en/documentation/installation/
• Install Bundler: http://bundler.io/
• Clone or download Capybara Git project: https://github.com/teamcapybara/capybara.git
• Move to your Capybara project and run terminal command: bundle install
• To run tests use command: cucumber or bundle exec cucumber, if you want to use just your Gem file included gems

After your Capybara/Cucumber framework has been successfully installed, your project structure should look like this:

Don’t forget to select Selenium driver in your env.rb file:

Capybara.default_driver = :selenium

Generate HTML report from your Cucumber JSON result file

Cucumber already can generate JSON file with result values that can be converted in report. To generate formatted JSON test result file run the following command:

cucumber  --format pretty --expand --format json -o "file_path.json"

If you want to build an HTML report right away, use Ruby gem called “ReportBuilder”. This Gem converts Cucumber generated JSON file in responsive HTML report. This HTML report will provide these sections:

• Test overview;
• Features;
• Summary;
• Errors.

HTML Report builder installation

gem install report_builder

After “Reportbuilder” gem has been installed, create a new ruby file “report_builder.rb”. It will contain logic for building HTML file.

Report_builder.rb code:

require 'report_builder'

time = Time.now.getutc

ReportBuilder.configure do |config|
 config.json_path = 'report.json'
 config.report_path = 'cucumber_web_report'
 config.report_types = [:html]
 config.report_tabs = %w[Overview Features Scenarios Errors]
 config.report_title = 'Cucumber web automation test results'
 config.compress_images = false
 config.additional_info = { 'Project name' => 'Test', 'Platform' => 'Integration', 'Report generated' => time }
end

ReportBuilder.build_report

You can add other config options, but I listed the most necessary ones for the overall QA report. Here is the full list of options: https://github.com/rajatthareja/ReportBuilder.

Generate HTML report

ruby report_builder.rb

Cucumber JSON Report in your freestyle Jenkins project

After you have generated a JSON report file you can also start to using it in your Jenkins CI. First, you have to install Jenkins CI plugin – “Cucumber Report”. To use the Cucumber report plugin you have to configure your Jenkins job to save report.json file build artifact. Now you can add the final post build action – Cucumber reports.

Next time when you will build a new project, Cucumber section will appear in your project page. Report will be available under the Cucumber icon.

Adding debug info to a failed case

Standart Cucumber reports are very useful for gathering your test results. But in a case of reproducing bug, you have to do that manually. To avoid that add debug info, such as screenshot, browser console logs and even location link.

To implement such mechanism, you have to be familiar with Cucumber hooks. Hooks allow to run code blocks before and after each scenario and more. Hooks can be used as preconditions in order to prepare the test environment. You can place hook logic anywhere under Capybaras support folder. This time let’s keep it in support/env.rb file.

Basic “Before” and “After” hook syntax

Before do
# Do something before each scenario
end

After do |scenario|
  # Do something after each scenario. The scenario argument is optional, but if you use it, you can inspect status with the #failed?, #passed? and #exception methods.
end

Adding screenshots to your Cucumber report

In this case Selenium driver supports screenshot saving function and all you have to do is point out the file saving path. Adding screenshot to the Cucumber report is more complicated, because you have to include it in JSON report file. To do that you have to encode the image by using Base64 module. Base64 module translates image binary data into purely printable characters and that can be included as a string in your report.json file. After that don’t forget to add your add_screenhot method to failed scenario afterhook.

support / env.rb

require 'capybara/cucumber'
require 'capybara-screenshot/cucumber'
require 'base64'

Capybara.register_driver :selenium do |app|
 Capybara::Selenium::Driver.new(app, browser: :chrome)
end

Capybara::Screenshot.autosave_on_failure = false
Capybara::Screenshot.prune_strategy = :keep_last_run

After do |scenario|
 if scenario.failed?
   add_screenshot
 end
end

def add_screenshot
 file_path = 'screenshot.png'
 page.driver.browser.save_screenshot(file_path)
 image = open(file_path, 'rb', &:read)
 encoded_image = Base64.encode64(image)
 embed(encoded_image, 'image/png;base64', 'SCREENSHOT')
end

Adding browser logs and current link to Cucumber report

Selenium driver provides access to browser console output and it’s quite desirable to add that in your Cucumber report. In case the test fails you will be able to check the console error message and code without spending time reproducing the issue. Also I suggest to add details such as current link and time stamp.

We will create new method called add_browser_logs that also involves fail case occurrence timestamp. This browser log gathering method is also provided from Selenium driver. Challenges begin when you have to iterate through the log array and remove unnecessary information, such as warnings and info messages.

support / env.rb

require 'capybara/cucumber'
require 'capybara-screenshot/cucumber'
require 'base64'

Capybara.register_driver :selenium do |app|
 Capybara::Selenium::Driver.new(app, browser: :chrome)
end

Capybara::Screenshot.autosave_on_failure = false
Capybara::Screenshot.prune_strategy = :keep_last_run

After do |scenario|
 if scenario.failed?
   add_screenshot
   add_browser_logs
 end
end

def add_screenshot
 file_path = 'screenshot.png'
 page.driver.browser.save_screenshot(file_path)
 image = open(file_path, 'rb', &:read)
 encoded_image = Base64.encode64(image)
 embed(encoded_image, 'image/png;base64', 'SCREENSHOT')
End

def add_browser_logs
 time_now = Time.now
 # Getting current URL
 current_url = Capybara.current_url.to_s
 # Gather browser logs
 logs = page.driver.browser.manage.logs.get(:browser).map {|line| [line.level, line.message]}
# Remove warnings and info messages
 logs.reject! { |line| ['WARNING', 'INFO'].include?(line.first) }
 logs.any? == true
 embed(time_now.strftime('%Y-%m-%d-%H-%M-%S' + "\n") + ( "Current URL: " + current_url + "\n") + logs.join("\n"), 'text/plain', 'BROWSER ERROR')
end

Conclusion

Capybara/Cucumber framework includes useful features for web-based UI tests, such as methods to manage browser actions, check elements and even headless characteristics.

In my opinion it’s good practise to use Capybara together with Cucumber, that adds hook options and easy test step definition. Cucumber Hooks allow to run code blocks at various points at the Cucumber test cycle, that lets you automate features like browser console error gathering, current link saving, screenshot capture, save fail occurrence time stamp and more. You can easily use all these features in your CI by using JSON report file.