On January 15, 2014, the Pew Research Center announced that it was changing how it surveyed Americans by telephone. Starting in 2014, 60% of interviews in its national polls would be conducted via cellphones and 40% on landline phones. This represents a change from the 50-50 split used in 2013. Although cellphone surveys are more expensive to conduct than landline surveys, Pew believes that this change will improve the quality of its surveys. For example, with the 50-50 split used in 2013, only 19% of the respondents to their surveys were under 35. By making the shift to a 60-40 split, Pew projects that 22% of the respondents will be under 35. Nationally, this group comprises 31% of the adult population. Pew will continue to make statistical adjustments to correct for the undercoverage.

A more serious source of bias in most sample surveys is nonresponse, which occurs when a selected individual cannot be contacted or refuses to cooperate. Nonresponse to sample surveys often reaches 50% or more, even with careful planning and several callbacks. Because nonresponse is higher in urban areas, most sample surveys substitute other people in the same area to avoid favoring rural areas in the final sample. if the people who respond to the survey hold views different from those who are rarely at home or who refuse to answer questions, some bias remains.

Undercoverage DEFINITION

Undercoverage occurs when some groups in the population are left out of the process of choosing the sample.

Nonresponse DEFINITION

Nonresponse occurs when an individual chosen for the sample can’t be contacted or refuses to participate.

EXAMPLE 6 How Bad Is Nonresponse?

The CPS (Example 1, page 293) has the lowest nonresponse rate of any poll we know. Only about 4% of the households in the CPS sample refuse to take part, and another 3 or 4% can’t be contacted. People are more likely to respond to a government survey such as the CPS, and the CPS contacts its sample in person before doing later interviews by phone. (On a related note, the national mail participation rate for the 2010 Census was 74%.)

304

What about polls done by the media and by market research and opinion polling firms? We don’t know their rates of nonresponse because they won’t say. That nondisclosure is a bad sign. The Pew Research Center imitated a telephone survey and published the results: out of 2879 households called, 1658 were never at home, refused the interview, or would not finish it. That’s a nonresponse rate of , or about 58%.

The campus food service wants to know how students feel about their food. They hand out a survey during Friday morning breakfast between 7 am and 9 am. Are the survey results more likely to be affected by undercoverage or nonresponse? Explain.

EXAMPLE 7 Validating Truthfulness and Encouraging Honesty

In a major national study, researchers were concerned about the truthfulness of student answers to questions about illicit drug use. Researchers compared data among twelfth-graders’ responses to questions about their own drug use, their friends’ drug use, and their own exposure to drug use. in any given year, comparisons across these three measures tended to be consistent from drug to drug. Because respondents should have little reason to answer untruthfully about their friends or their general exposure to drugs, the researchers considered this consistency as evidence of the truthfulness of student responses about their own drug use.

Another strategy for encouraging honesty with sensitive topics is called randomized response, invented by sociologist S. L. Warner in 1965 and explored in Exercises 20 and 21 (page 331). By introducing randomness into the responses in a structured way, researchers use their knowledge of probability distributions to get reasonably accurate information about the overall group while allowing each potentially embarrassing answer to be “camouflaged.” Because the interviewee knows that, for example, the researcher has no way to distinguish which “yes” answers are real and which are simply introduced by the random mechanism, they will feel safe answering honestly a question of the form “Have you ever done [some embarrassing or illegal action]?”

EXAMPLE 8 Watch That Wording!

How do Americans feel about government help for the poor? only 13% think we are spending too much on “assistance to the poor,” but 44% believe we are spending too much on “welfare.” How did the Scots feel about the movement to become independent from England? Well, 51% voted for “independence for Scotland,” but only 34% supported “an independent Scotland separate from the United Kingdom.” it seems that assistance to the poor and independence are nice, hopeful terms, whereas welfare and separate are negative words. other topics that have produced survey results that vary greatly depending on the wording of the questions include abortion, immigration, gay rights, gun control, and affirmative action.

Consider the two questions below. Do you believe that either of these questions is designed to elicit a particular response? If so, explain how.

Q1. Do you think that smokers should have the freedom to smoke in a hotel room in which they are staying? Yes No

Q2. Given the recent reports on the dangers of second-hand smoke, do you think that companies should allow their employees to smoke in the workplace? Yes No