Pairwise Testing in Agile World

This blog is by Puttajunjaiah Krishnamurthy who will be presenting along with Arun Srinivasan on their talk – Pairwise Testing in Agile World – at XP Conference India 2016. This post first appeared in Puttajunjaiah’s blog site


The fundamental challenge in testing any system is that the exhaustive testing is often impossible! This is owing to a large number of combinations of inputs to the system under test. The number grows exponentially high as the number of input parameters and the number of values each of those parameters can take increases. Take for example, the aircraft cockpit control system which has at least 100 switches and controls.

cockpit-100624

Even if each of these controls take 2 values (ON or OFF), it takes 2 ^ 100 = 1.2676506e+30 combinations to thoroughly test the system. That takes more than a lifetime for someone to test this system!! Not to forget the regression testing when things change. World is moving towards agile methodologies which means shorter release cycles. So quick test feedback on every build is absolutely necessary. Even with 100% automation of these test cases, it takes several days/months to give this feedback. So, what is the alternative?

The combinatorial techniques like Pairwise testing (also known as All Pairs) hold some promise. The intuition behind Pairwise technique is that some problems only occur as the result of an interaction between pairs of parameters/components. For e.g, a failure occurs if pressure < 10 & volume > 300 (2-way interaction).

The researchers Wallace and Kuhn at NIST (National Institute of Standards and Technology) have determined that 98% of the reported software defects in recalled medical devices could have been detected by testing all pairs of parameter settings.

An Example:

Consider a system that takes 3 input parameters. Each parameter takes 2 values.

example

Total no of combinations = 2^3 = 8

all_comb

Pairwise Reductions by hand

We can get rid of test case T2, since all pairs of this combination are covered in at least one other combination other than itself. Hence it is redundant.

  • AC is covered in T1
  • CF is covered in T6
  • AF is covered in T4

With little bit of paperwork, you’ll be able to get rid of T3, T5 & T8.

reduced_subset

So the reduced subset consists of just 4 test cases. Not a great reduction but the quantum of reduction is significant when there are more no of input parameters and their values.

final_subset

Fortunately we don’t have to do this by hand. There are plenty of tools available in the market that can do this for us. Like PICT from Microsoft, TestCover etc.

Caveat

1. Pairwise testing will not be effective if you choose the wrong input test data values 

2. Pairwise testing will not be effective if you don’t have a good enough oracle. We often assume defects can be immediately observed. In fact, there may be slow internal corruption occurring

3. High-use (including defaults) or high-risk combinations probably don’t get enough attention

4. We rarely understand the n-wise connectivity within programs we are testing, so it’s not clear that pairwise testing is an appropriate choice 39

Note: These dangers are not exclusive to pairwise testing. They affect all testing efforts

Share: Pinterest

    Your email address will not be published. Required fields are marked *

  • Name *
  • Website