Wording questions
70
A final influence on the results of a sample survey is the exact wording of questions. It is surprisingly difficult to word questions so that they are completely clear. A survey that asks “Do you enjoy watching football?” will generate different answers based on the respondent’s understanding of “football” (American football or soccer).
EXAMPLE 5 Words make a big difference
In May 2013, the Pew Research Center and the Washington Post/ABC News conducted polls asking whether people felt the U.S. Department of Justice had the right to subpoena Associated Press reporters’ phone records. Each survey phrased the question differently, and each survey found different results.
When the Pew Research Center asked, “Do you approve or disapprove of the Justice Department’s decision to subpoena the phone records of AP journalists as part of an investigation into the disclosure of classified information?” 36% of respondents approved. The Washington Post/ABC News survey said, “The AP reported classified information about U.S. anti-terrorism efforts and prosecutors have obtained AP’s phone records through a court order. Do you think this action by federal prosecutors is or is not justified?” Fifty-two (52%) of respondents to this survey said that the action of federal prosecutors was justified.
Before asking about approval or disapproval of the Justice Department’s actions, Pew asked how closely respondents had followed the issues that led up to the subpoena, and 64% reported not following the news story too closely. Additionally, the difference in wording—“decision to subpoena” in the Pew Research survey versus “obtained . . . through a court order” used by the Washington Post/ABC News survey—could have led to legitimizing the actions of the Department of Justice and thus a higher “approval” for the Justice Department’s actions. Or, was the higher justification based on the inclusion of the mention of U.S. antiterrorism efforts? Or, perhaps the difference is due to Pew using “the Justice Department” and the Washington Post/ABC News using “federal prosecutors.” We cannot begin to determine which part of the wording affected the responses. As the Pew Research Center said in its article comparing the results of these two surveys, “each polling organization made good-faith efforts to describe the facts of the situation as accurately as possible, but the word choices and context make it impossible to identify one particular phrase or concept that tipped the public's thinking.”
71
The wording of questions always influences the answers. If the questions are slanted to favor one response over others, we have another source of nonsampling error. A favorite trick is to ask if the subject favors some policy as a means to a desirable end: “Do you favor banning private ownership of handguns in order to reduce the rate of violent crime?” and “Do you favor imposing the death penalty in order to reduce the rate of violent crime?” are loaded questions that draw positive responses from people who are worried about crime. Here is another example of the influence of question wording.
EXAMPLE 6 Paying taxes
In April 2015, a Gallup Poll asked two questions about the amount one pays in federal income taxes. Here are the two questions:
Do you consider the amount of federal income tax you have to pay as too high, about right, or too low?
Do you regard the income tax which you will have to pay this year as fair?
The first question had 42% of respondents say “about right” (51% said “too high”), while the second question resulted in 56% of respondents saying that the taxes they paid were “fair.” There appears to be a definite difference in opinions about taxes when people are asked about the amount they have to pay or whether it is fair.
72
NOW IT’S YOUR TURN
Question 4.1
4.1 Should we recycle? Is the following question slanted toward a desired response? If so, how?
In view of escalating environmental degradation and incipient resource depletion, would you favor economic incentives for recycling of resourceintensive consumer goods?