Executing the Test Plan

Executing a node or branch

You can run a part of the test plan by selecting one or nodes in the test plan and right-click and selecting Execute from the context menu. The test plan item and all sub-items will execute. Executing a Test Plan in this fashion also works in the Repository.

📘

If your password policy prevents saving of credentials in the database, you will be prompted to provide the credentials required to complete execution.

Creating an execution plan

The checkboxes to the left of each node in the test plan are used as an execution plan. To manually define an execution plan:

• Select the part you want to execute using the checkboxes.
• Click the Execute Test Plan button in the ribbon menu.

The checkboxes define precisely what is executed. If e.g. a data processing task uses the output of a report task as its source, and that report task is not selected, the output from the previous execution of that report task is used.

This is useful for example in a scenario where you have run through a test case and added approvals to the reconciliation task, you can re-run only reconciliation without having to re-run the underlying reports.

If the dependency report task has never been executed, and no last output is available, the report task is included in the execution automatically, regardless of the checkbox state.

Advanced execution plan

Instead of toggling many checkboxes manually you can use the Add to Execution Plan function on the Test Plan context menu.

The Add to Execution Plan dialog allows you to include specific task types and specific systems from any of the selected branches. The function operates additively (it never un-checks any tasks), so you can run it repeatedly. If you for example realize you need to include the import tasks for the reference system as well, you can simply add them through the user interface:

👍

This is useful if you want to refresh the target data and re-run the reconciliations without refreshing the reference data. You can select the task types you want to include and the systems used for the target system.

To clear the execution plan, use the ribbon menu Clear button.

Result of the execution

The Logs view shows a series of all executions of the item currently selected in Test Plan.

The key information is the Success Rate figure, indicating a percentage of success for the currently selected item. Data processing and reconciliation task can be successful to a degree like 98%, whereas report tasks are fully successful or failed.

The success rate of a group item, such as a test case, is calculated as the average success rate of all immediate children. For a test case with only tasks as children/sub-items the success rate is the average, unweighted success rate of the tasks of the test case. For a group with multiple test cases, the success rate is the average, unweighted success rate of all test cases within the group.

Result of partial executions

The Logs view shows one row per execution, meaning one row per click to the Execute button. If e.g. a reconciliation task of a test case has been re-run using the previous output of the source reports, the log for the whole test case will show another log row representing the partial execution.

The column Number of logs shows the number of task logs (on any sub level) that the row aggregates. When only a part of the test case has been executed the logs from previous executions of all other tasks are included in the calculation to provide a picture of the most recent state of the test case.

In the above screenshot, the task has been executed three times partially.

Using the drill-down function it is possible to compare two executions, and also see what task logs are included to form the aggregate success rate.

Result reporting

OmniFi supports result reporting using result reporting extensions. Extensions are custom and built in Python or .NET allowing extremely flexible and tailored behaviors, including synchronizing execution result with a test management system, producing execution audit logs etc.

Result reporting extension is configured both on repository and test plan level, in the Settings view.

When a result reporting extension is configured and enabled, the column Export Result is visible in the Test Plan view. The checkbox determines for what nodes in the test plan to execute the result reporting extension.

The level of result reporting you want to use depends on what extension you are using, but notably this switch allows you to turn off result reporting for a branch in the design phase.

Running result reporting manually

To re-execute result reporting on an ad-hoc basis it is possible to right-click on the item in the test plan you want to run result reporting for, and selecting Export Result from the context menu.

Result reporting is executed for the last execution of all reportable items in the selected branch. This means that results from more than one Execution Id can be reported in the same batch.

To report an older execution, you can change to logs time window by adjusting the From and To -parameters in the Task Logs ribbon menu.