EXAMPLE 7.20 The Effects of Altering a Software Parameter

Example 7.7 (pages 368–370) describes an experiment to compare the measurements obtained from two software algorithms. In that example we used the matched pairs test on these data, despite some skewness, which make the -value only roughly correct. The sign test is based on the following simple observation: of the 76 parts measured, 43 had a larger measurement with the option off and 33 had a larger measurement with the option on.

To perform a significance test based on these counts, let be the probability that a randomly chosen part would have a larger measurement with the option turned on. The null hypothesis of “no effect” says that these two measurements are just repeat measurements, so the measurement with the option on is equally likely to be larger or smaller than the measurement with the option off. Therefore, we want to test

The 76 parts are independent trials, so the number that had larger measurements with the option off has the binomial distribution if is true. The -value for the observed count 43 is, therefore, , where has the distribution. You can compute this probability with software or the Normal approximation to the binomial:

As in Example 7.7, there is not strong evidence that the two measurements are different.