Execution Set Summary and Results

Owned by Joseph Hamdan
Last updated: Dec 30, 2024
5 min read

After your execution set is created and execution starts, you can click on the execution set link displayed below to get to the summary and results page. Alternatively, you can click See all results from the detail pane.

image-20241224-152832.png

After opening, the most recent run which has failures will be automatically selected for you so you can easily begin checking the failures. You will also notice that the header title, left panel icon and breadcrumb now reference the ID and Name of the execution set for easier identification.

image-20241224-153816.png

Details Pane

On the detail panel on the right, the following information will be displayed in the top section along with the main status:

More actions (ellipsis) icon allows you to Refresh, Delete execution or export the result to XLSX or PDF.

 

In the bottom section, you will see general information related to your executions. Here’s a brief description on all of them:

  1. ID: This is dynamically generated and contains the ID of the execution set (shown in the first image above) and the ID of the selected execution separated by a dot.

  2. Started By: This shows the username of the person who started the execution. It may or may not be the same user who created the execution set

  3. Started On: This shows the date/time in which the run was triggered (in other words, sent to cloud run, but may not necessarily have started execution yet)

  4. Completed On: This shows the date/time in which the execution finished

  5. Duration: This shows the duration of the execution

  6. Runs: This shows the number of execution runs

  7. Reruns: This shows the number of reruns that were either started automatically or manually by the user

  8. Type: This shows the type of cloud run request. The value here could be one of 3:

    1. Portal: Started directly by the user (clicking Cloud Run icon)

    2. Scheduled: Started by Subject7 automatically using a schedule defined by the user

    3. Web Service: Started by API (Either directly via the API or by using Jenkins CI/CD plugin)

  9. Pool: This shows you the pool configured in the execution set where execution took place

  10. Browsers: This shows you which browser(s) you configured in the execution set. The greyed out ones have not been select in the below image.

Navigation Between Layers

In order to avoid opening multiple tabs, all layers of execution will be open on a single tab and you can use the arrows on the top left (next to Add to Bookmarks) to navigate between the layers.

Schedules

If you would like to review your schedule, you can do in one of the following ways:

  1. Click Schedule tab. It will show you a fixed schedule for you to review (cannot be edited from here)

  2. Click Edit Settings which will show you the execution set edit form where you can adjust the schedule

Filtering on All Execution Results

On this window, you can click Add Filter to display all available filters to select a filter. The filters will assist you in locating different results at this level.

Example of State filter locating the value ‘Pass’. You can reset filters by clicking the x on the displayed filter or by clicking Reset Filters.

Overall status of an execution counts FAIL_SKIP as a passed test case.
Overall status of green check-mark with fail_skip steps will appear as an orange check-mark

Execution Details

In the last section of the detail pane, you will see your Test Runs separated into 2 categories: All and Failed. Since the user will mostly be concerned with checking the failures, the Failed view will be activated by default, but you can click All to see all of the executions.

The link under Test Case column will navigate to a different screen that will show you a more detailed view of your executions. This high-level overview shows you the widgets as follows:

  1. Execution Result Status: which shows the status of the entire execution

  2. Run State: which shows the execution completion state

  3. Pass Rate: which shows 3 levels:

    1. Initial Run

    2. Reruns

    3. Final Run

  4. Pool and Number of Agents

  5. Browsers

In the table below the widgets, we will see the executions in each run. The summary column will show a summary bar to graphically highlight where the test failed. While the details pane on the right will automatically show the image highlighting where the failure took place in the video. The bottom section of the details pane also shows you the ID, Status, Failure Message as well as additional information that may be of help to you.

 

Execution Metadata

If you would like to view additional details about your execution, you can click Metadata tab.

This will show you all the information you need related to your execution as follows:

Comparison and Export

The table also shows you the status of each execution as well as the datasets on the right side of the status. More Actions (ellipsis) icon in both the last column and the details pane allows you to compare executions, download videos or view any additional data as shown below. You can also compare using the icon highlighted on the left side.

If you would like to export these results, you can click the export icon shown below:

Â