EXAMPLE 7.27

The effect of altering a software parameter. Example 7.7 (page 419) describes an experiment to compare the measurements obtained from two software algorithms. In that example, we used the matched pairs t test on these data, despite some skewness, which makes the P-value only roughly correct. The sign test is based on the following simple observation: of the 51 parts measured, 29 had a larger measurement with the option off and 22 had a larger measurement with the option on.

To perform a significance test based on these counts, let p be the probability that a randomly chosen part would have a larger measurement with the option turned on. The null hypothesis of “no effect’’ says that these two measurements are just repeat measurements, so the measurement with the option off is equally likely to be larger or smaller than the measurement with the option on. Therefore, we want to test

H₀: p = 1/2

H_a: p ≠ 1/2

The 51 parts are independent trials, so the number that had larger measurements with the option off has the binomial distribution B(51, 1/2) if H₀ is true. The P-value for the observed count 29 is, therefore, 2P(X ≥ 29), where X has the B(51, 1/2) distribution. You can compute this probability with software or the Normal approximation to the binomial:

= 2P (Z ≥ 0.98)

= 2(0.1635)

= 0.3270

As in Example 7.7, there is not strong evidence that the two measurements are different.