Executions and Reports

In order to see the report for your cloud executions, click on the name of the execution set of interest. This will show you all of the Executions of that Execution Set.

Filtering Executions

1- To filter the executions listing on the Executions page, click on the Add Filter dropdown list shown below:

2- After clicking on the Add Filter dropdown list, the following options will appear

3- Select the type of filter you would like to apply on the execution listing and then provide the value for that type of filter if needed. The execution listing will get filtered down based on the filter selection. An example of an applied filter is shown below:

4- To reset the filter, click Reset Filters icon highlighted above.

Tip

Overall status of an execution counts FAIL_SKIP as a passed test case.

Tip

Overall status of green check-mark with fail_skip steps will appear as an orange check-mark

Column Selection

You can control how many columns appear by clicking on the Gear icon highlighted below:

Execution Results Columns

NOTE: Requester field is an optional column which you can enable as below. This column will show you who initiated the cloud run request.

Test Case Table Columns

Steps Table Columns

Execution Status

To see the overall status of each Test Suite within this Execution Set, click on the specific Execution Set of interest.

At this level, you can click on the video, image, log, variables and details icons to see more information about the test case. Video and logs get created for both passed and failed test cases, however screen shot is only created for failed test cases.

Tip

Video and log in URLs are named like this: TestCaseName_ExecutionID_DateToMilliSeconds.mp4

While your test case is being executed, you can view gradual reports by clicking Refresh to constantly update and see these reports. You can also click Enable Auto Refresh icon as highlighted below and it will automatically update the results for you using the rate provided in your User Preferences. Refer to Set Preferences to a Test Case for more information about how to enable Auto Refresh by default and set a rate.

In case the Status of the Test Case is Fail, you can hover over the Status highlighted below to show the failure message.

You can also click the Status fail icon to copy the result of the failure to your clipboard.

Execution Logs and Details

1- The left log icon will show you a log of the execution session and variables will show you the variables generated during execution. Details will show you the Data Template and Data Set used in the execution as well as Pool, Executor and Re-run:

Sample log:

2020-06-09 06:09:34 [pool-6-thread-1] INFO  c.s.p.e.cloud.ExecutionListenerImpl - Test execution id=1166 start, number of steps 9
2020-06-09 06:09:39 [pool-6-thread-1] INFO  c.s.p.e.cloud.ExecutionListenerImpl - Step index=0 PASS 4240(ms) [Goto_Url HALT https://reference.subject-7.com] message='https://reference.subject-7.com'
2020-06-09 06:09:41 [pool-6-thread-1] INFO  c.s.p.e.cloud.ExecutionListenerImpl - Step index=1 PASS 1080(ms) [Click HALT subject7_reference_login.link.login] message='XPath: "//a[contains(text(),'LOGIN')]" was used to find element 'subject7_reference_login.link.login''
2020-06-09 06:09:43 [pool-6-thread-1] INFO  c.s.p.e.cloud.ExecutionListenerImpl - Step index=2 PASS 1128(ms) [Fill HALT clear='false' subject7_reference_login.text.email_address "${reference_multiple.username}" ] message='Resolved: p.thorne@subject-7.com
XPath: "//input[contains(@placeholder, 'Email')]" was used to find element 'subject7_reference_login.text.email_address''
2020-06-09 06:09:45 [pool-6-thread-1] INFO  c.s.p.e.cloud.ExecutionListenerImpl - Step index=3 PASS 902(ms) [Fill HALT clear='false' subject7_reference_login.text.password "${reference_multiple.password}" ] message='Resolved: *****
XPath: "//input[contains(@placeholder, 'Password')]" was used to find element 'subject7_reference_login.text.password''
2020-06-09 06:09:47 [pool-6-thread-1] INFO  c.s.p.e.cloud.ExecutionListenerImpl - Step index=4 PASS 862(ms) [Click HALT subject7_reference_general.button.login_button] message='XPath: "//text()[contains(.,'Login')]/ancestor::button" was used to find element 'subject7_reference_general.button.login_button''
2020-06-09 06:09:51 [pool-6-thread-1] INFO  c.s.p.e.cloud.ExecutionListenerImpl - Step index=5 PASS 3008(ms) [Wait HALT 3] message='Waited for:3 secs.'
2020-06-09 06:09:52 [pool-6-thread-1] INFO  c.s.p.e.cloud.ExecutionListenerImpl - Step index=6 PASS 0(ms) [Call subject7_reference_logout] message='null'
2020-06-09 06:09:54 [pool-6-thread-1] INFO  c.s.p.e.cloud.ExecutionListenerImpl - Step index=7 PASS 832(ms) [Click HALT subject7_reference_general.link.profile] message='XPath: "(//a[contains(text(),'Profile')]/ancestor::li/descendant::a)[1]" was used to find element 'subject7_reference_general.link.profile''
2020-06-09 06:09:55 [pool-6-thread-1] INFO  c.s.p.e.cloud.ExecutionListenerImpl - Step index=8 PASS 842(ms) [Click HALT subject7_reference_general.link.logout] message='XPath: "//a[contains(text(),'Logout')]" was used to find element 'subject7_reference_general.link.logout''
2020-06-09 06:09:56 [pool-6-thread-1] INFO  c.s.p.e.cloud.ExecutionListenerImpl - Test execution id=1166 finished. Status='PASS', Message='Test passed successfully, duration = 12894'

2- The right log icon will show you a log of the browser session captured during execution. The icon under Log will show you the browser (Console) logs for the entire execution while the icon next to each step will only show the logs generated in that particular step.

Variables:

Details:

Execution Test Case Steps

To see the steps of test cases, click on the name of the execution set which will take you to the Test Case Results page.

Afterwards, click on the arrow next to the test case name. You can also click Expand All icon to view the details of all the test cases. Collapse all icon will revert the view to the previous stage.

Execution Reports

You can access the reporting from the execution set list by checking the box next to the execution set and selecting either: Aggregated Summary Report or Aggregated Test Case Report under Reports.

Aggregated Summary Report: Aggregates failures of test cases with the same reason as well as successes to help with stabilization of cloud executions. Lets the user query by: Execution Set Name, a from/to date window, or time frames: Yesterday, Last week, Last 30 days.

Below is the result of running a query for execution set and it includes the date window that the execution occurred. The result includes each test case that is a member of the execution set as well as Status, Browser(s), output message (Pass or message as to why the test case failed), and Artifacts: video, image, and detail log.

Aggregated Test Case Report: Shows the result for the last 20 runs of test cases to show consistency of cloud executions. Lets the user query by: Execution Set Name, a from/to date window, or time frames: Yesterday, Last week, Last 30 days with more details of each step

You can click on check marks/X for each execution ID to view the details of each step. You can also click on the timestamp under Latest to view the latest execution details of each step.