7.5 7.4 Cautions About Sample Surveys

Random sampling eliminates bias in the choice of a sample from a list of the population. Sample surveys of large human populations, however, require more than a good sampling design to eliminate bias.

To begin with, we need an accurate and complete list of the population. Because such a list is rarely available, most samples suffer from some degree of undercoverage. A sample survey of households, for example, will miss not only homeless people but also prison inmates and students in dormitories. An opinion poll conducted by random-digit-dialing of landline phones will miss the more than 40% of U.S. households without landline phones. Hispanics, African Americans, younger adults, and the poor are more likely to rely solely on cell phones; hence, their views won’t be represented in the opinion poll.

303

Pew Research Center Spotlight 7.2

On January 15, 2014, the Pew Research Center announced that it was changing how it surveyed Americans by telephone. Starting in 2014, 60% of interviews in its national polls would be conducted via cellphones and 40% on landline phones. This represents a change from the 50-50 split used in 2013. Although cellphone surveys are more expensive to conduct than landline surveys, Pew believes that this change will improve the quality of its surveys. For example, with the 50-50 split used in 2013, only 19% of the respondents to their surveys were under 35. By making the shift to a 60-40 split, Pew projects that 22% of the respondents will be under 35. Nationally, this group comprises 31% of the adult population. Pew will continue to make statistical adjustments to correct for the undercoverage.

A more serious source of bias in most sample surveys is nonresponse, which occurs when a selected individual cannot be contacted or refuses to cooperate. Nonresponse to sample surveys often reaches 50% or more, even with careful planning and several callbacks. Because nonresponse is higher in urban areas, most sample surveys substitute other people in the same area to avoid favoring rural areas in the final sample. if the people who respond to the survey hold views different from those who are rarely at home or who refuse to answer questions, some bias remains.

image

Undercoverage DEFINITION

Undercoverage occurs when some groups in the population are left out of the process of choosing the sample.

As Spotlight 7.2 shows, major research institutions, such as the Pew Research Center, take undercoverage into account when designing their polls.

Nonresponse DEFINITION

Nonresponse occurs when an individual chosen for the sample can’t be contacted or refuses to participate.

EXAMPLE 6 How Bad Is Nonresponse?

The CPS (Example 1, page 293) has the lowest nonresponse rate of any poll we know. Only about 4% of the households in the CPS sample refuse to take part, and another 3 or 4% can’t be contacted. People are more likely to respond to a government survey such as the CPS, and the CPS contacts its sample in person before doing later interviews by phone. (On a related note, the national mail participation rate for the 2010 Census was 74%.)

304

What about polls done by the media and by market research and opinion polling firms? We don’t know their rates of nonresponse because they won’t say. That nondisclosure is a bad sign. The Pew Research Center imitated a telephone survey and published the results: out of 2879 households called, 1658 were never at home, refused the interview, or would not finish it. That’s a nonresponse rate of , or about 58%.

Algebra Review Appendix

Fractions, Percents, and Percentages

Self Check 4

The campus food service wants to know how students feel about their food. They hand out a survey during Friday morning breakfast between 7 am and 9 am. Are the survey results more likely to be affected by undercoverage or nonresponse? Explain.

  • This is a retrospective study.

Another danger is that when people do respond to a survey question, we can’t always rely on them to tell the truth. For example, will students answer truthfully when asked: On how many occasions (if any) have you taken cocaine during the last 30 days? Consider another example. People know that they should take the trouble to vote. Therefore, many who didn’t vote in the last election will tell a pollster that they did. Fortunately, there are strategies to help determine the level of participants’ truthfulness and to improve accuracy.

EXAMPLE 7 Validating Truthfulness and Encouraging Honesty

In a major national study, researchers were concerned about the truthfulness of student answers to questions about illicit drug use. Researchers compared data among twelfth-graders’ responses to questions about their own drug use, their friends’ drug use, and their own exposure to drug use. in any given year, comparisons across these three measures tended to be consistent from drug to drug. Because respondents should have little reason to answer untruthfully about their friends or their general exposure to drugs, the researchers considered this consistency as evidence of the truthfulness of student responses about their own drug use.

Another strategy for encouraging honesty with sensitive topics is called randomized response, invented by sociologist S. L. Warner in 1965 and explored in Exercises 20 and 21 (page 331). By introducing randomness into the responses in a structured way, researchers use their knowledge of probability distributions to get reasonably accurate information about the overall group while allowing each potentially embarrassing answer to be “camouflaged.” Because the interviewee knows that, for example, the researcher has no way to distinguish which “yes” answers are real and which are simply introduced by the random mechanism, they will feel safe answering honestly a question of the form “Have you ever done [some embarrassing or illegal action]?”

Finally, the wording of questions strongly influences the answers given to a sample survey. Confusing or leading questions can introduce strong bias, and even minor changes in wording or order can change a survey’s outcome. Here are some examples.

305

EXAMPLE 8 Watch That Wording!

How do Americans feel about government help for the poor? only 13% think we are spending too much on “assistance to the poor,” but 44% believe we are spending too much on “welfare.” How did the Scots feel about the movement to become independent from England? Well, 51% voted for “independence for Scotland,” but only 34% supported “an independent Scotland separate from the United Kingdom.” it seems that assistance to the poor and independence are nice, hopeful terms, whereas welfare and separate are negative words. other topics that have produced survey results that vary greatly depending on the wording of the questions include abortion, immigration, gay rights, gun control, and affirmative action.

Self Check 5

Consider the two questions below. Do you believe that either of these questions is designed to elicit a particular response? If so, explain how.

Q1. Do you think that smokers should have the freedom to smoke in a hotel room in which they are staying? Yes No

Q2. Given the recent reports on the dangers of second-hand smoke, do you think that companies should allow their employees to smoke in the workplace? Yes No

  • (a) In a sample of size 2, it is possible to get 0 “yes” responses, 1 “yes” response, or 2 “yes” responses. Hence, the possible values for would be 0, 0.5, and 1.0.

    (b) Since , it is least likely that both responses are “no.” Hence, will occur least frequently.

The design of sample surveys is a science, but this science is only part of the art of sampling. Because of nonresponse, false responses, and the difficulty of posing clear and neutral questions, you should analyze critically before fully trusting reports about complicated issues based on surveys of large human populations. Insist on knowing the exact questions asked, the rate of nonresponse, and the date and method of the survey before you trust a poll result.