Create an Execution Set

Owned by Former user (Deleted)
Last updated: Nov 13, 2024 by Joseph Hamdan
7 min read

In order to create a new Execution Set, follow the steps below:

  1. Click Add while on Execution Sets tab. You can select Folder to create a folder to organize your execution sets. To create a new set, select Execution Set.

  2. General Setup:

Main: Provide the Name, Pool (one or more virtual machines assigned to your account), Browser(s), Rotations and click Video Captions checkbox if you need to add captions to the videos. Multiple browsers can be selected for an individual execution set under Browsers label.

Scheduling options: The Schedule Maker (similar to cronmaker.com) assists in creating the run schedule for the execution set. Cloud execution uses EST (Eastern Standard Time) to execute your test cases.

Notifications: You can also set Notifications to receive an e-mail when the execution set meets a specific condition (Any, Any Failure, Success, Failure, Timeout). The e-mails sent will also include a link to the log, video and images (in case of failures). You can choose to receive an e-mail immediately, hourly or daily based on what you prefer. [Immediately] sends an e-mail depending on the selected Level. There are two levels: 1- Test Case: an email for each test case upon completion 2- Execution: One email upon completion of the entire execution set. [Hourly] aggregates the results of all the test cases that have been executed in an hour and sends you an e-mail notification with the results of all those test cases. [Daily] aggregates the results of all the test cases in an execution set and will send a single e-mail with the results of all those test cases at a configurable time in a day. In case of failure, your e-mail will include selected data sets, failed step number, failed step and the reason for failure.

Tip

Notification e-mails will include the following additional fields:

  1. Browser Type

  2. Pool Name

  3. Retry Number

  4. Speed

  5. IP of the processing VM

Callback: You can also set a REST callback to the execution set with conditions (Any, Any Failure, Success, Failure, Timeout). Please refer to Execution Callbacks for more information on this.

Timings: you can set the number of milliseconds for waiting between steps (Speed), page timeout and locator timeout in seconds. You can also set a Rerun to determine how many times to rerun in case of failures.

Rerun Timeout (s) in conjunction with No. of Reruns, will calculate an additional period of time which will be automatically used for reruns to Page Timeout, Locator Timeout and Speed.

For example, if your test case fails due to timeout error, your No. of Reruns is set to 1, your Rerun Timeout is set to 5, and the default parameters are:

  1. Page Timeout: 30s

  2. Locator Timeout: 5s

  3. Speed: 200ms

Your rerun will be triggered automatically with these new values:

  1. Page Timeout: 35s (Additional 5 seconds)

  2. Locator Timeout: 10s (Additional 5 seconds)

  3. Speed: 700ms (Additional 500 milliseconds)

The numbers displayed on the execution set form under Timings section are controllable by Admin users under Administration > Account Preferences > Timeouts. This menu allows to set the maximum allowed limit which applies to all executions. Please refer to Account Preferences for more information.

3. Test Case and Data Set Selection: Drag and drop test suites from the Test Library on the left-hand side to the Selected Test Cases in the middle and then click on each test case in the Selected Test Cases area to set data sets for on the right hand side and then save at the end. Note that this list will only display test cases that were created as part of the selected project. The selected test cases are automatically sorted alphabetically. You can sort ascending or descending for easier viewing experience.

Tip

Pre-execution and post-execution test case options are available on (Test Data Selection) screen. When selecting to include a test case in an execution set, you will have the option of choosing pre-execution or post execution with Execution being the default. If pre-execution test cases fail, the rest of the execution set will be skipped.

 

Data Set Selections and Combinations

  1. To select which datasets will be used in your executions, you can click on each test case individually or click on All icon highlighted below. This will show data template names and dropdown lists with datasets on the right side. It will also allow you to chain and unchain test cases into groups (Scroll below to Chaining Test Cases) and select Phases (Execution and Pre execution).

2. To show the available datasets, click on the Search icon highlighted above next to each data template. This will show you all datasets available where you can select specific ones to use or click on the icons displayed below if you wish to select all, deselect all or apply selected datasets to the same templates in the execution set. Note that based on your selections, the number of executions will be triggered. For examples, if we choose the users "abaldwin" and "ballan", we will get two executions, one for each selection.

3- If you would like to view and pick specific combinations for several datasets, select all datasets as shown earlier and click Save on your execution set form. You will see an icon appear next to Data Set Selection as shown below:

4. Click on the arrow next to the test case name and you will see all available combinations based on your selection. You can choose whether to keep a selection or skip it by using the available checkboxes next to each template/set. Once finished, click Save under Actions for each test case.

Execution Phases

Execution Phases allows users to select which test cases start execution first, so they would be marked as "Pre Execution" and which ones will start afterwards "Execution". Note that this execution here is still in parallel (for sequential execution, scroll below to Chaining Test Cases).

1- To set up Pre Execution test cases, click on the checkboxes next to them and select Pre Execution from the dropdown list next to Phase.

2- To set up execution test cases, deselect the previously selected test cases and select the execution test cases. Now you can select Execution in the dropdown list next to Phase.

Chaining Test Cases

Chaining is used to execute test cases run one after another (sequentially) on the same executor machine (Virtual Machine), which is almost against every automation best practice (independent test cases, highly paralyzed, maintainability, easy reporting, etc.). It's only introduced as a feature for projects where for example, concurrent login with the same credentials was not possible. The QA team had no access to the Dev team (government projects) so their tests could not run in parallel and they have to chain it to make sure they will run one at a time.

If you would like to set up chained groups, you will need to select 2 or more test cases and click Chain icon as below:

Once your test cases have been chained into groups, both of these groups will execute simultaneously, but everything inside them will run sequentially based on the order shown below:

If you would like to change the default name of the group, you can click on the checkbox next to that group and click Rename icon as shown below:

4. Browsers Settings: Set the browser settings for all of the different browsers to be used in this execution set. 

5. Emulators Settings: Set the type of device to be used for web-based emulation next to Chrome/Safari Device Emulator, or Android and iOS emulators next to Real Emulator Platform.

6. When finished, click Save (Pool will need to be selected before the execution set can be saved).