Executions and Reports

In order to see the report for your cloud executions, click on the name of the execution set of interest. This will show you all of the Executions of that Execution Set.

Open

Execution Status

To see the overall status of each Test Case within this Execution Set, click on the specific Execution Set of interest. 

At this level, you can click on the video, image, and log icons to see more information about the test case. Video and logs get created for both passed and failed test cases, however screen shot is only created for failed test cases. 

Open

When clicking on the links under the ID column, you will see the individual test run belonging to that specific test execution.

While your test case is being executed, you can view gradual reports by clicking Refresh to constantly update and see these reports. You can also click Enable Auto Refresh icon as highlighted below and it will automatically update the results for you using the rate provided in your User Preferences. Refer to Set Preferences to a Test Case for more information about how to enable Auto Refresh by default and set a rate.

Failure messages will automatically appear in the details panel and in the Test Run Summary bar.

Execution Logs and Details

1. Browser Log tab will show you a log of browser (Console tab) generated during execution

2. Detail Log icon will show you a log of the execution session generated during execution. Sample log:

3. Metadata tab shows you additional information pertaining to your test execution as well as the Test Run Datasets which shows your dataset selections made prior to executing in cloud.

4. Variables will show you all variables declared during execution

Create related ticket link allows you to connect to JIRA to create a ticket in case you need to. Videos, screenshots and step-level reports are attached to the generated ticket. The number of the JIRA ticket created will be available in the execution set report so that you can navigate the ticket from Subject7. For more information, refer to Ticketing System and its sub articles.

Charts and Reports

You can access the reporting from the execution set main window by checking the box next to the execution set and selecting either: History Chart, Aggregated Summary Report or Aggregated Test Case Report under Reports.

History Chart: To see a pie chart for execution history of an Execution Set, click on the checkbox to the left of an Execution Set name and click History Chart icon under Reports.

Aggregated Summary Report: Aggregates failures of test cases with the same reason as well as successes to help with stabilization of cloud executions. Lets the user query by: Execution Set Name, a from/to date window, or time frames: Yesterday, Last week, Last 30 days.

Below is the result of running a query for execution set and it includes the date window that the execution occurred. The result includes each test case that is a member of the execution set as well as Status, Browser(s), output message (Pass or message as to why the test case failed), and Artifacts: video, image, and detail log.

Aggregated Test Case Report: Shows the result for the last 20 runs of test cases to show consistency of cloud executions. Lets the user query by: Execution Set Name, a from/to date window, or time frames: Yesterday, Last week, Last 30 days with more details of each step

You can click on check marks/X for each execution ID to view the details of each step. You can also click on the timestamp under Latest to view the latest execution details of each step.