4.8 What kind of error? Which of the following are sources of sampling error and which are sources of nonsampling error? Explain your answers.

4.9 What kind of error? Each of the following is a source of error in a sample survey. Label each as sampling error or nonsampling error, and explain your answers.

4.10 Not in the margin of error. According to a February 2008 USA Today/Gallup Poll, 43% of Americans identify themselves as baseball fans. That is low by recent standards, as an average of 49% of Americans have said they were fans of the sport since Gallup started tracking this measure in 1993. The high point came in 1998, when Sammy Sosa and Mark McGwire pursued (and ultimately surpassed) Roger Maris’s single-season home run record, at which time 56% of Americans considered themselves baseball fans. The Gallup press release says:

For results based on this sample, one can say with 95% confidence that the maximum error attributable to sampling and other random effects is ±5 percentage points.

Give one example of a source of error in the poll result that is not included in this margin of error.

4.11 Not in the margin of error. According to a June 2015 Gallup Poll, more American adults ages 18 to 29 report being single and never married—the percentage has risen from 52% in 2004 to 64% in 2014. For American adults aged 30 to 39, this same percentage has risen modestly from 15% in 2004 to 19% in 2014. The Gallup press release says:

For results based on the total sample of Americans ages 18 to 29 or 30 to 39 in any given year, the maximum margin of sampling error is ±3 percentage points at the 95% confidence level.

Give one example of a source of error in the poll result that is not included in this margin of error.

4.12 College parents. An online survey of college parents was conducted during February and March 2007. An email was sent to 41,000 parents who were listed in either the College Parents of America database or the Student Advantage database. Parents were invited to participate in the online survey. Out of those invited, 1727 completed the online survey. The survey protected the anonymity of those participating in the survey but did not allow more than one response from an individual IP address.

One of the survey results was that 33% of mothers communicate at least once a day with their child while at school.

4.13 Polling customers. An online store chooses an SRS of 100 customers from its list of all people who have bought something from the store in the last year. It asks those selected how satisfied they are with the store’s website. If it selected two SRSs of 100 customers at the same time, the two samples would give somewhat different results. Is this variation a source of sampling error or of nonsampling error? Would the survey’s announced margin of error take this source of error into account?

4.14 Ring-no-answer. A common form of nonresponse in telephone surveys is “ring-no-answer.” That is, a call is made to an active number but no one answers. The Italian National Statistical Institute looked at nonresponse to a government survey of households in Italy during the periods January 1 to Easter and July 1 to August 31. All calls were made between 7 and 10 P.M., but 21.4% gave “ring-noanswer” in one period versus 41.5% “ring-no-answer” in the other period. Which period do you think had the higher rate of no answers? Why? Explain why a high rate of nonresponse makes sample results less reliable.

4.15 Race relations. Here are two opinion poll questions asked about race relations in the United States.

We’d like to know how you would rate relations between various groups in the United States these days. Would you say relations between whites and blacks are very good, somewhat good, somewhat bad, or very bad?

Do you think race relations in the U.S. are generally good or generally bad?

In response to the first question, 72% of non-Hispanic whites and 66% of blacks answered relations between blacks and whites are “very good” or “somewhat good.” Sixty-one percent (61%) of those answering the second question responded “generally bad.”

The first question came from a poll that was conducted in March 2015. The second question came from a poll that was conducted between April 30 and May 3 of that same year. Between the two polls, a man named Freddie Gray died after being in the custody of the Baltimore police. In what ways do you think this event may have affected the responses to the two different polls? Do you think the results would be different if the question from the second poll had been worded like the question from the first poll?

4.16 The environment and the economy. Here are two opinion poll questions asked in December 2009 about protecting the environment versus protecting the economy.

Often there are trade-offs or sacrifices people must make in deciding what is important to them. Generally speaking, when a trade-off has to be made, which is more important to you: stimulating the economy or protecting the environment?

Which worries you more: that the U.S. will NOT take the actions necessary to prevent the catastrophic effects of global warming because of fears those actions would harm the economy or that the U.S. WILL take actions to protect against global warming and those actions will cripple the U.S. economy?

In response to the first question, 61% said stimulating the economy was more important. But only 46% of those asked the second question said they were afraid that the United States will take actions to protect against global warming and that those actions will cripple the U.S. economy. Why do you think the second wording discouraged more people from expressing more concern about the economy than about the environment?

4.17 Amending the Constitution. You are writing an opinion poll question about a proposed amendment to the Constitution. You can ask if people are in favor of “changing the Constitution” or “adding to the Constitution” by approving the amendment.

4.18 Right to refuse services. The Supreme Court of the United States released a ruling on same-sex marriage in June 2015. Prior to the release of this ruling, many agencies conducted polls about same-sex marriage. In April 2015, a Quinnipiac University poll asked two questions about businesses and services to gays and lesbians. Here are the two questions:

Do you think businesses should or should not be allowed to refuse services to gays and lesbians?

What if the business says homosexuality violates its owners’ religious beliefs? In that case, do you think the business should or should not be allowed to refuse services to gays and lesbians?

One of these questions drew 35% saying the businesses should be allowed to refuse services; the other drew 26% with the same response. Which wording produced the higher percentage? Why?

4.19 Wording survey questions. Comment on each of the following as a potential sample survey question. Is the question clear? Is it slanted toward a desired response? (Survey questions on issues that one might regard as inflammatory are often prone to slanted wording.)

4.20 Bad survey questions. Write your own examples of bad sample survey questions.

4.21 Appraising a poll. In January 2015, The Wall Street Journal published an article on satisfaction with the U.S. economy and other issues. The article noted 45% of Americans were very or somewhat satisfied with the state of the U.S. economy, even though 49% felt America is in a state of decline. News articles tend to be brief in describing sample surveys. Here is part of The Wall Street Journal’s description of this poll:

The . . . poll was based on nationwide telephone interviews of 800 adults, including 280 respondents who use only a cellphone. . . . Individuals were selected proportionate to the nation’s population in accordance with a probability sample. . . . The data’s margin of error is plus or minus 3.46 percentage points.

Page 80 lists several “questions to ask” about an opinion poll. What answers does The Wall Street Journal give to each of these questions?

4.22 Appraising a poll. A June 2015 New York Times article about the inequality of wealth and income discussed the results of a sample survey that found, for example, that 67% of those surveyed think the gap between the rich and the poor in the United States is getting larger. Here is part of the Times’s statement “How the Poll Was Conducted”:

The latest New York Times/CBS News Poll is based on telephone interviews conducted May 28 to 31 with 1,022 adults throughout the United States . . .

The sample of land-line telephone exchanges called was randomly selected by a computer . . .

The Times’s statement goes on to say that cell phone numbers were also randomly selected by a computer and that the samples of landline and cell phone responses were combined and appropriately adjusted. Page 80 lists several “questions to ask” about an opinion poll. What answers does the Times give to each of these questions?

4.23 Closed versus open questions. Two basic types of questions are closed questions and open questions. A closed question asks the subject for one or more of a fixed set of responses. An open question allows the subject to answer in his or her own words; the interviewer writes down the responses and classifies them later. An example of an open question is

What do you believe about the afterlife?

An example of a closed question is

What do you believe about the afterlife? Do you believe

What are the advantages and disadvantages of open and closed questions?

4.24 Telling the truth? Many subjects don’t give honest answers to questions about activities that are illegal or sensitive in some other way. One study divided a large group of white adults into thirds at random. All were asked if they had ever used cocaine. The first group was interviewed by telephone: 21% said Yes. In the group visited at home by an interviewer, 25% said Yes. The final group was interviewed at home but answered the question on an anonymous form that they sealed in an envelope. Of this group, 28% said they had used cocaine.

4.25 Did you vote? When the Current Population Survey asked the adults in its sample of 56,000 households if they voted in the 2012 presidential election, 61.8% said they had. The margin of error was less than 0.3%. In fact, only 58.2% of the adult population voted in that election. Why do you think the CPS result missed by 12 times the margin of error?

4.26 A party poll. At a party, there are 20 students over age 21 and 40 students under age 21. You choose, at random, two of those over 21 and separately choose at random four of those under 21 to interview about attitudes toward alcohol. You have given every student at the party the same chance to be interviewed: what is that chance? Why is your sample not an SRS?

4.27 A stratified sample. A club has 30 student members and 10 faculty members. The students are

The faculty members are

The club can send three students and two faculty members to a convention. It decides to choose those who will go by random selection.

4.28 A stratified sample. A state university has 4900 in-state students and 2100 out-of-state students. A financial aid officer wants to poll the opinions of a random sample of students. In order to give adequate attention to the opinions of out-of-state students, the financial aid officer decides to choose a stratified random sample of 200 in-state students and 200 out-of-state students. The officer has alphabetized lists of in-state and out-of-state students.

4.29 Sampling by accountants. Accountants use stratified samples during audits to verify a company’s records of such things as accounts receivable. The stratification is based on the dollar amount of the item and often includes 100% sampling of the largest items. One company reports 5000 accounts receivable. Of these, 100 are in amounts over $50,000; 500 are in amounts between $1000 and $50,000; and the remaining 4400 are in amounts under $1000. Using these groups as strata, you decide to verify all of the largest accounts and to sample 5% of the midsize accounts and 1% of the small accounts. How would you label the two strata from which you will sample? Use the Simple Random Sample applet, other technology, or Table A, starting at line 115, to select only the first five accounts from each of these strata.

4.30 A sampling paradox? Example 8 compares two SRSs of a university’s undergraduate and graduate students. The sample of undergraduates contains a smaller fraction of the population, 1 out of 90, versus 1 out of 15 for graduate students. Yet sampling 1 out of 90 undergraduates gives a smaller margin of error than sampling 1 out of 15 graduate students. Explain to someone who knows no statistics why this happens.

4.31 Appraising a poll. Exercise 4.22 gives part of the description of a sample survey from The New York Times. It appears that the sample was taken in several stages. Why can we say this? The first stage no doubt used a stratified sample, though the Times does not say this. Explain why it would be bad practice to use an SRS from all possible telephone numbers rather than a stratified sample of landline-only, cell-phone-only, and dual-phone users.

4.32 Multistage sampling. An article in the journal Science looks at differences in attitudes toward genetically modified foods between Europe and the United States. This calls for sample surveys. The European survey chose a sample of 1000 adults in each of 17 European countries. Here’s part of the description: “The Eurobarometer survey is a multistage, random-probability face-to-face sample survey.”

4.33 Online courses in high schools? What do adults believe about requiring online courses in high schools? Are opinions different in urban, suburban, and rural areas? To find out, researchers wanted to ask adults this question:

It has become common for education courses after high school to be taken online. In your opinion, should public high schools in your community require every student to take at least one course online while in high school?

Because most people live in heavily populated urban and suburban areas, an SRS might contain few rural adults. Moreover, it is too expensive to choose people at random from a large region. We should start by choosing school districts rather than people. Describe a suitable sample design for this study, and explain the reasoning behind your choice of design.

4.34 Systematic random samples. The last stage of the Current Population Survey (Example 7) uses a systematic random sample. An example will illustrate the idea of a systematic sample. Suppose that we must choose four rooms out of the 100 rooms in a dormitory. Because 1004 = 25, we can think of the list of 100 rooms as four lists of 25 rooms each. Choose one of the first 25 rooms at random, using Table A. The sample will contain this room and the rooms 25, 50, and 75 places down the list from it. If 13 is chosen, for example, then the systematic random sample consists of the rooms numbered 13, 38, 63, and 88. Use Table A to choose a systematic random sample of five rooms from a list of 200. Enter the table at line 120.

4.35 Systematic isn’t simple. Exercise 4.34 describes a systematic random sample. Like an SRS, a systematic sample gives all individuals the same chance to be chosen. Explain why this is true, then explain carefully why a systematic sample is nonetheless not an SRS.

4.36 Planning a survey of students. The student government plans to ask a random sample of students their opinions about on-campus parking. The university provides a list of the 20,000 enrolled students to serve as a sampling frame.

4.37 Sampling students. You want to investigate the attitudes of students at your school toward the school’s policy on extra fees for lab courses. You have a grant that will pay the costs of contacting about 500 students.

4.38 Mall interviews. Example 1 in Chapter 2 (page 23) describes mall interviewing. This is an example of a convenience sample. Why do mall interviews not produce probability samples?

4.39 Partial-birth abortion? Here are three opinion poll questions on the same issue, with the poll results:

As you may know, the Supreme Court recently upheld a law that makes the procedure commonly known as a partial birth abortion illegal. Do you favor or oppose this ruling by the Supreme Court? Result: 53% favor; 34% oppose.

[T]he Supreme Court . . . upheld a law that makes . . . partial birth abortion illegal. [This] procedure [is] performed in the late-term of pregnancy, when in some cases the baby is old enough to survive. . . . The court’s ruling . . . [doesn’t] make an exception for the [mother’s] health. . . . Do you favor or oppose this ruling . . . ? Result: 47% favor; 43% oppose.

. . . I would like to ask your opinion about a specific abortion procedure known as a “late-term” . . . or “partial-birth” abortion, which is sometimes performed . . . during the last few months of pregnancy. Do you think . . . the government should make this procedure illegal, or do you think . . . the procedure should be legal? Result: 66% illegal; 28% legal.

Using this example, discuss the difficulty of using responses to opinion polls to understand public opinion.