EXAMPLE 4 How bad is nonresponse?

The U.S. Census Bureau’s American Community Survey (ACS) is a monthly survey of almost 300,000 housing units and replaced the U.S. Census Bureau’s “long form” that, in the past, was sent to some households in the every-10-years national census. Participation in the ACS is mandatory, and the U.S. Census Bureau follows up by telephone and then in person if a household fails to return the mail questionnaire.

The ACS has the lowest nonresponse rate of any poll we know: in 2013, only about 1.3% of the households in the sample refused to respond; the overall nonresponse rate, including “never at home” and other causes, was 10.1%. This is a stark contrast in nonresponse from previous years—in 2012, the total nonresponse rate was only 2.7%. In October 2013, there was a government shutdown. During that time, “the ACS did not have a second mailing, a telephone followup, or a person followup operation for the October 2013 housing unit panel. . . . This caused a drop in the annual housing unit response rate of about 7 percentage points.” If October 2013 is excluded from the ACS when considering nonresponse, the total nonresponse rate for 2013 was only 2.9%, similar to previous years.

Another survey that has a remarkable response rate is the University of Chicago’s General Social Survey (GSS), the nation’s most important social survey. The GSS (Example 7 in Chapter 1) contacts its sample in person, and it is run by a university. Despite these advantages, its recent surveys have a 30% rate of nonresponse.

What about polls done by the media and by market research and opinion-polling firms? We often don’t know their rates of nonresponse because they won’t say. That itself is a bad sign. The Pew poll we looked at in the Case Study suggests how bad things are. Pew got 1221 responses (of whom 1000 were in the population they targeted) and 1658 who were never at home, refused, or would not finish the interview. That’s a nonresponse rate of 1658 out of 2879, or 58%. The Pew researchers were more thorough than many pollsters. Insiders say that nonresponse often reaches 75% or 80% of an opinion poll’s original sample.

The situation investigated in the Case Study was more promising than a recent Pew survey. In a December 2013 survey on social media use, the Pew Research Center provided a full disposition of the sampled phone numbers. Here are the details. Initially, 40,985 landline and 27,000 cell phones were dialed, with 11,260 of the landlines and 15,758 of the cell numbers being working numbers. Of these 27,018 working numbers, they were able to contact 17,335, or about 64%, as a large portion of the calls went to voicemail. Among those contacted, about 15% cooperated. Of those cooperating, some numbers were ineligible due to language barriers or contacting a child’s cell phone, and some calls were eventually broken off without being completed. To sum it up, the 27,018 working numbers dialed resulted in a final sample of 1801, giving a response rate of about 7%, an estimate of the fraction of all eligible respondents in the sample who were ultimately interviewed.