# Part 1 : Basic techniques for improving the reliability and execution time of test cases
We have to run thousands of test cases that are expected to perform series of operations such as creating the customer accounts, inserting and loading the entries to/from databases, invoking REST APIs etc in a testing environment.
This looks simple but there are challenges such as:
- Testing environment is not 100% reliable and there are failures time to time.
- Testing environment has few shared components that has usage limits.
This leads to huge impact when thousands of test cases are required to be executed.
Let’s consider that there is a typical test case takes overall 30 seconds for execution and has 20% failure rate because of the reliability issue in testing environment and there can be maximum 10 parallel threads as one of the component has usage limit in testing environment.
Iteration #1 : High level approach for execution
In order to execute these test cases, runtime engine adds the test cases to the Task Queue and creates pool of threads based on max allowed capacity for shared component (e.g. 10 threads) and assigns the task queue to threads for execution.
The following diagram illustrate the high level flow:
The following diagram demonstrates the run time where pool of 10 threads are executing group of 10 test cases in parallel.
If this approach is followed then in order to run 500 test cases, the runtime engine would require 50 cycles. And, if the failure rate is 20% then 100 out of 500 test cases would fail. The following diagram illustrate the formulas and sample results.
The above mentioned approach has one major problem — Total Failed Test Cases are huge and re-running them manually will lead to extra effort and time.
Iteration #2 : Add Retry Count for each test case for minimizing the failures
This approach will help in solving a major problem and will reduce the overall failures so that manual runs is no longer required but this approach is still not fully optimized as
- The entire test cases is being retried which means that Account creation, Database setup, REST API invocation etc would have to be repeated again.
- The retry of overall test cases adds significant time to overall execution. E.g. For 5000 test cases with 20% failure rate, 1000 test cases will be re-tried and if test case takes 30 second then it will add (1000 / 10 )* 30 Sec (~ 50 minutes) and this will go on and on till all test cases are successful.
Iteration #3 Divide and Conquer
The idea of this approach is to split the test cases in multiple logical steps such that each step logically performs one type of operation.
Once steps are logically divided they can managed more efficiently and each test step can have its own retry count.
This approach helps in solving two critical issues that was there with previous approach.
- This doesn’t require re-run of all the steps in a test case.
- This also reduces overall execution time. E.g. if the failure was at step 4 while invoking an API then it gets retried immediately and other previous steps are not required to be re-run again.
This revised approach help in significantly optimizing the execution time and retries but can this be further improved and the answer is “Yes”.
I will share the further details in next article.