EXAMPLE 13 Is the Experiment Realistic?

Clinical trials give medical treatments to actual patients with the condition that the treatments are supposed to help. But some experiments are less realistic in terms of how well experimental conditions align with the usual circumstances of what is of greatest interest. For example, a researcher studying stages and cycles of sleep observes patients overnight in a special “sleep lab” that can monitor their electroencephalography (EEG) waves and other data. However, individuals may sleep quite differently in a lab setting than in the natural sleeping environment of their bedroom at home.

Another type of example is that some studies on animals may be limited in how reliably their conclusions might apply to humans. Penicillin, for instance, is highly toxic to guinea pigs, but it has been a very helpful medicine for humans.

These are not statistical questions. Researchers must use their understanding of their academic domain to judge how far their results apply. Good statistical design enables us to trust results for the participants in the study at hand, but additional knowledge and judgment are needed to decide the extent to which conclusions might be generalized to other settings.