In order to create a new Execution Set, Click Add while on the Execution Sets tab. You can select Folder to create a folder to organize your execution sets. To create a new set, select Execution Set.

Open

Basics

General Setup

Main: Provide the Name, Pool (one or more virtual machines assigned to your account), Rotations and click the Video Captions checkbox if you need to add captions to the videos. Then select one or more browsers you want to use with your test cases.

Open

Schedule Maker

The Schedule Maker (similar to cronmaker.com) assists in creating the run schedule for the execution set, and it uses EST (Eastern Standard Time) to execute your test cases. After you have saved your execution set, you will then start the schedule for your execution set (see below).

Notifications

You can receive an e-mail when the execution set has completed, based upon the condition (Any, Any Failure, Success, Failure, Timeout), the level of detail (Test Case or Execution Set), and the frequency (Immediately, Daily, or Hourly):

• Immediately sends the email as soon as either the Test Case or Execution Set has completed (depending on the level of detail

• Hourly aggregates the results of all the test cases that have been executed in an hour

• Daily aggregates the results of all the test cases in an execution set, at a configurable time in a day, that have been executed since the prior notification time.

The e-mails sent will include a link to the log and video. In the case of failures, it will include a link to the images, selected data sets, failed step number, failed step and the reason for failure.

Test Case and Data Set Selection (Test Data Selection)

The Test Library on the left-hand side displays all the test cases that are part of the current project. To add test cases to the Execution Set, drag and drop test suites from the Test Library to the Selected Test Cases in the middle. The selected test cases are automatically sorted alphabetically, though you can sort ascending or descending for an easier viewing experience. To specify data sets for each test case, click on a test case in the middle, and in the right-hand side it will display the Data Templates specified in the Test Case. For additional information about specifying data sets, see the Advanced section below.

Starting your Schedule

After you have completed the above items, click Save & Close, and you will then be ready to start your schedule. On the Execution Set Main screen, there are two ways to start the schedule:

1. Select your Execution Set, and in the Detail Pane on the right click the toggle under the Recurring Portal Runs heading.

2. Select one or more Execution Sets, and in the row of actions above the table, click the Start Schedule action.

Once you have started the schedule, the test cases in your Execution Set will be queued at the exact time you specified and will begin running. The Schedule column in the table and the fields in the Detail Pane will reflect the current status of the schedule, and the next time that it will run. If at any time you need to stop the schedule, you can click the toggle to stop it, or click the Stop Schedule action.

Advanced

Timings: you can set the number of milliseconds for waiting between steps (Speed), page timeout and locator timeout in seconds. You can also set a Rerun to determine how many times to rerun in case of failures.

Rerun Timeout (s) in conjunction with No. of Reruns, will calculate an additional period of time which will be automatically used for reruns to Page Timeout, Locator Timeout and Speed.

For example, if your test case fails due to timeout error, your No. of Reruns is set to 1, your Rerun Timeout is set to 5, and the default parameters are:

1. Page Timeout: 30s

2. Locator Timeout: 5s

3. Speed: 200ms

Your rerun will be triggered automatically with these new values:

1. Page Timeout: 35s (Additional 5 seconds)

2. Locator Timeout: 10s (Additional 5 seconds)

3. Speed: 700ms (Additional 500 milliseconds)

Callbacks: You can set a REST callback to the execution set with conditions (Any, Any Failure, Success, Failure, Timeout). Please refer to Execution Callbacks for more information on this.

Data Set Selections and Combinations

1. To select which datasets will be used in your executions, you can click on each test case individually or click on All icon highlighted below. This will show data template names and dropdown lists with datasets on the right side. It will also allow you to chain and unchain test cases into groups (Scroll below to Chaining Test Cases) and select Phases (Execution and Pre execution).

3- If you would like to view and pick specific combinations for several datasets, select all datasets as shown earlier and click Save on your execution set form. You will see an icon appear next to Data Set Selection as shown below:

4. Click on the arrow next to the test case name and you will see all available combinations based on your selection. You can choose whether to keep a selection or skip it by using the available checkboxes next to each template/set. Once finished, click Save under Actions for each test case.

Execution Phases

Execution Phases allows users to select which test cases start execution first, so they would be marked as "Pre Execution" and which ones will start afterwards "Execution". Note that this execution here is still in parallel (for sequential execution, scroll below to Chaining Test Cases).

1- To set up Pre Execution test cases, click on the checkboxes next to them and select Pre Execution from the dropdown list next to Phase.

2- To set up execution test cases, deselect the previously selected test cases and select the execution test cases. Now you can select Execution in the dropdown list next to Phase.

Chaining Test Cases

Chaining is used to execute test cases run one after another (sequentially) on the same executor machine (Virtual Machine), which is almost against every automation best practice (independent test cases, highly paralyzed, maintainability, easy reporting, etc.). It's only introduced as a feature for projects where for example, concurrent login with the same credentials was not possible. The QA team had no access to the Dev team (government projects) so their tests could not run in parallel and they have to chain it to make sure they will run one at a time.

If you would like to set up chained groups, you will need to select 2 or more test cases and click Chain icon as below:

Once your test cases have been chained into groups, both of these groups will execute simultaneously, but everything inside them will run sequentially based on the order shown below:

If you would like to change the default name of the group, you can click on the checkbox next to that group and click Rename icon as shown below:

Browsers

Set the browser settings for all of the different browsers to be used in this execution set. 

Mobile Emulators

Set the type of device to be used for web-based emulation next to Chrome/Safari Device Emulator, or Android and iOS emulators next to Real Emulator Platform.