This blog is by Puttajunjaiah Krishnamurthy who will be presenting along with Arun Srinivasan on their talk – Pairwise Testing in Agile World – at XP Conference India 2016. This post first appeared in Puttajunjaiah’s blog site
The fundamental challenge in testing any system is that the exhaustive testing is often impossible! This is owing to a large number of combinations of inputs to the system under test. The number grows exponentially high as the number of input parameters and the number of values each of those parameters can take increases. Take for example, the aircraft cockpit control system which has at least 100 switches and controls.
Even if each of these controls take 2 values (ON or OFF), it takes 2 ^ 100 = 1.2676506e+30 combinations to thoroughly test the system. That takes more than a lifetime for someone to test this system!! Not to forget the regression testing when things change. World is moving towards agile methodologies which means shorter release cycles. So quick test feedback on every build is absolutely necessary. Even with 100% automation of these test cases, it takes several days/months to give this feedback. So, what is the alternative?
The combinatorial techniques like Pairwise testing (also known as All Pairs) hold some promise. The intuition behind Pairwise technique is that some problems only occur as the result of an interaction between pairs of parameters/components. For e.g, a failure occurs if pressure < 10 & volume > 300 (2-way interaction).
The researchers Wallace and Kuhn at NIST (National Institute of Standards and Technology) have determined that 98% of the reported software defects in recalled medical devices could have been detected by testing all pairs of parameter settings.
Consider a system that takes 3 input parameters. Each parameter takes 2 values.
Total no of combinations = 2^3 = 8
Pairwise Reductions by hand
We can get rid of test case T2, since all pairs of this combination are covered in at least one other combination other than itself. Hence it is redundant.
- AC is covered in T1
- CF is covered in T6
- AF is covered in T4
With little bit of paperwork, you’ll be able to get rid of T3, T5 & T8.
So the reduced subset consists of just 4 test cases. Not a great reduction but the quantum of reduction is significant when there are more no of input parameters and their values.
Fortunately we don’t have to do this by hand. There are plenty of tools available in the market that can do this for us. Like PICT from Microsoft, TestCover etc.
1. Pairwise testing will not be effective if you choose the wrong input test data values
2. Pairwise testing will not be effective if you don’t have a good enough oracle. We often assume defects can be immediately observed. In fact, there may be slow internal corruption occurring
3. High-use (including defaults) or high-risk combinations probably don’t get enough attention
4. We rarely understand the n-wise connectivity within programs we are testing, so it’s not clear that pairwise testing is an appropriate choice 39
Note: These dangers are not exclusive to pairwise testing. They affect all testing efforts