# Part 1 : Basic techniques for improving the reliability and execution time of test cases

Background

We have to run thousands of test cases that are expected to perform series of operations such as creating the customer accounts, inserting and loading the entries to/from databases, invoking REST APIs etc in a testing environment.

This looks simple but there are challenges such as:

This leads to huge impact when thousands of test cases are required to be executed.

Scenario

Let’s consider that there is a typical test case takes overall 30 seconds for execution and has 20% failure rate because of the reliability issue in testing environment and there can be maximum 10 parallel threads as one of the component has usage limit in testing environment.

Iteration #1 : High level approach for execution

In order to execute these test cases, runtime engine adds the test cases to the Task Queue and creates pool of threads based on max allowed capacity for shared component (e.g. 10 threads) and assigns the task queue to threads for execution.

The following diagram illustrate the high level flow:

Iteration #1 Approach

The following diagram demonstrates the run time where pool of 10 threads are executing group of 10 test cases in parallel.

Runtime View

If this approach is followed then in order to run 500 test cases, the runtime engine would require 50 cycles. And, if the failure rate is 20% then 100 out of 500 test cases would fail. The following diagram illustrate the formulas and sample results.

The above mentioned approach has one major problem — Total Failed Test Cases are huge and re-running them manually will lead to extra effort and time.

Iteration #2 : Add Retry Count for each test case for minimizing the failures

Iteration #2 Approach

This approach will help in solving a major problem and will reduce the overall failures so that manual runs is no longer required but this approach is still not fully optimized as

Iteration #3 Divide and Conquer

The idea of this approach is to split the test cases in multiple logical steps such that each step logically performs one type of operation.

Divide

Once steps are logically divided they can managed more efficiently and each test step can have its own retry count.

Conquer
Step Level Retries

This approach helps in solving two critical issues that was there with previous approach.

This revised approach help in significantly optimizing the execution time and retries but can this be further improved and the answer is “Yes”.

I will share the further details in next article.