Introduction
(Slide 1 of 38)

Chapter 17. Getting More Responses to Emails: Everything Up to Chi-Square

Which Test Is Best?
true
true
You must read each slide, and complete any questions on the slide, in sequence.

By Warren Fass, University of Pittsburgh Bradford, and Susan A. Nolan, Seton Hall University

A young woman sitting at her desk and using her laptop
© Radius Images/Corbis

In this activity, we will consider various findings from the Boomerang article, “7 Tips for Getting More Responses to Your Emails (With Data!)” (Moore, 2016). Then you will identify statistical analyses that could be used to support the various conclusions.

Guidelines for choosing the appropriate hypothesis test

A young woman using her smartphone
Denis Kuvaev/Shutterstock

Most of us have sent an email – maybe even an important one – and never received a response. Now, maybe a technological tool can help you change that.
Boomerang is an add-on for email that helps manage the seemingly endless stream of messages most of us get. One of its services is a reminder for when a particular email doesn’t receive a response by a certain date, which gives the email sender a chance to follow up. Just tell Boomerang to turn on this service for a particular email and provide the date on which the reminder is desired.
Boomerang, the company that makes the tool of the same name, analyzed data from 40 million emails in which people used this reminder service. From this analysis, they developed six tips that lead to a higher email response rate (Moore, 2016).

Boomerang reported that their “most surprising finding was that the reading grade level of your emails has a dramatic impact on response rates.” For example, they reported that emails were more likely to get a response when they were written at a 3rd-grade level than at a college level. The bar graph here shows the response rates for four different grade levels. Bottom line: Keep it simple.

The bar graph showing the e-mail response rates for four different grade levels.
Data from Moore (2016)

Email Response Rate, Image Long Description

The bar graph showing the e-mail response rates for four different grade levels. In the table below, the left column is the grade level, and the right column is the rate.

The bar graph showing the e-mail response rates for four different grade levels.
Kindergarten 46
Third Grade 53
High School 45
College 39

Which statistical test could be used to determine if there was a relation between the two variables – reading level and response category? Note that these response rates were calculated by first counting how many emails in each category received responses and how many did not receive responses. Also, remember that ordinal data can be treated as nominal when they represent categories.
Correct! The researchers could have used the chi-square test for independence because there are two variables and both variables can be treated as nominal variables. The first nominal variable is reading level (kindergarten, third grade, high school, or college), and the second nominal variable is response category (received a response or did not receive a response). There are no scale variables.
 
Now skip ahead to the next example by clicking here. Or, for more practice walking through the flowchart questions, simply click the Next button in the bottom right corner of the screen.
Actually, that’s not the correct statistical analysis. Let’s walk through the questions on the flowchart in Appendix E to determine what analysis сould be used in this case.

In which of the following four categories does this situation fall? Click to see the data again. And click on the flowchart button to see the overview for choosing the best test.

Boomerang reported that their "most surprising finding was that the reading grade level of your emails has a dramatic impact on response rates." For example, they reported that emails were more likely to get a response when they were written at a 3rd grade level than at a college level.
Correct! There are only nominal variables.
Actually, there are only nominal variables.

How many nominal variables are there?
Correct! There are two nominal variables, reading level and response category.
Actually, there are two nominal variables, reading level and response category.

Based on the answers to these questions, which statistical test сould be used to determine if there was a relationship between the two variables, reading level and response rate?
Correct! The researchers could have used the chi-square test for independence because there are two variables and both variables can be treated as nominal variables. The first nominal variable is reading level (kindergarten, third grade, high school, or college), and the second nominal variable is response category (received a response or did not receive a response). There are no scale variables.
Actually, that’s not the correct statistical analysis. The researchers could have used the chi-square test for independence because there are two variables and both variables can be treated as nominal variables. The first nominal variable is reading level (kindergarten, third grade, high school, or college), and the second nominal variable is response category (received a response or did not receive a response). There are no scale variables.

A young man sitting on a sofa and using his laptop.
sturti/Getty Images

Boomerang also reported that emails that received responses tended to be positive or negative in tone, rather than neutral in tone. All emails received a rating ranging from 0, which represented a neutral email with no emotion, to 1, which represented an emotional email that was either positive or negative in tone. Boomerang reported an example of a neutral email with a 0 emotion rating: “Hey, I was thinking about you earlier. Do you want to get pizza?” And of a positive email with a 0.55 emotion rating: “Hey, it would be really great to see you and catch up? Do you want to get pizza?” A negative email example is “I had an awful experience at your store today. The clerk was very rude.” This negative email received an emotion rating of 0.35. Emails that did not receive responses were more likely to receive neutral ratings, and emails that did receive responses were more likely to receive emotional ratings, either positive or negative.

What statistical analysis could be used to determine whether there was a significant difference between the ratings from the two groups—emails that received responses and emails that did not receive responses?
Correct! The researchers could have used an independent-samples t test, because there is one nominal independent variable, response group, with two levels or groups: response and no response. There is a scale dependent variable, emotional ratings. And participants are only in one of the two groups, so it is a between-groups design.
 
Now skip ahead to the next example by clicking here. Or, for more practice walking through the flowchart questions, simply click the Next button in the bottom right corner of the screen.
Actually, that's not the correct statistical analysis. Let’s walk through the questions on the flowchart in Appendix E to determine what analysis could be used in this case.

In which of the following four categories does this situation fall? Click to see the data again. And click on the flowchart button to see the overview for choosing the best test.

Boomerang also reported that emails that received responses tended to be positive or negative in tone, rather than neutral in tone. All emails received a rating ranging from 0, which represented a neutral email with no emotion, to 1, which represented an emotional email that was either positive or negative in tone. Boomerang reported an example of a neutral email with a 0 emotion rating: “Hey, I was thinking about you earlier. Do you want to get pizza?” And of a positive email with a 0.55 emotion rating: “Hey, it would be really great to see you and catch up? Do you want to get pizza?” A negative email example is “I had an awful experience at your store today. The clerk was very rude.” This negative email received an emotion rating of 0.35. Emails that did not receive responses were more likely to receive neutral ratings, and emails that did receive responses were more likely to receive emotional ratings, either positive or negative.





Correct! There is at least one nominal independent variable and a scale dependent variable.
Actually, there is at least one nominal independent variable and a scale dependent variable.

How many nominal independent variables are there?
Correct! There is one nominal independent variable, response group. (The dependent variable, emotional ratings, is scale.)
Actually, there is only one nominal independent variable, response group. (The dependent variable, emotional ratings, is scale.)

How many levels does the independent variable have?
Correct! There are two levels or groups, response and no response.
Actually, there are two levels or groups, response and no response.

How many samples are there?
Correct! There are two samples, one consisting of emails that received responses and one consisting of emails that did not receive responses.
Actually, there are two samples, one consisting of emails that received responses and one consisting of emails that did not receive responses.

What type of design is this?
Correct! This is a between-groups design. Each email appears in only one of the two groups.
Actually, this is a between-groups design. Each email appears in only one of the two groups.

Based on the answers to these questions, what statistical analysis could be used to determine whether there was a significant difference in emotional ratings between the two groups?
Correct! The researchers could have used an independent-samples t test, because there is one nominal independent variable, response group, with two levels or groups: response and no response. There is a scale dependent variable, emotional ratings. And participants are only in one of the two groups, so it is a between-groups design.
Actually, that’s not the correct statistical analysis. The researchers could have used an independent-samples t test, because there is one nominal independent variable, response group, with two levels or groups: response and no response. There is a scale dependent variable, emotional ratings. And participants are only in one of the two groups, so it is a between-groups design.

Boomerang also reported that the frequency of responding to emails was related to the length of the emails. Emails containing 75-100 words were responded to more frequently than emails containing 25 or fewer words. Imagine that we recruited 20 adult participants. We ask each participant how frequently they respond, on a scale of 1-7 (1 = not very, 7 = very), to emails containing 25 or fewer words. We ask those same participants how often they respond to emails containing 75-100 words.

Which statistical test could be used to determine if there was a significant difference between the participants’ ratings of the two types of email length?
Correct! The researchers could have used a paired-samples t test. There is one nominal independent variable, length of email, with two levels or groups, 25 or fewer words and 75-100 words. There is a scale dependent variable, frequency ratings on a scale of 1-7. All participants rated both types of email, so it is a within-groups design.
 
Now skip ahead to the next example by clicking here. Or, for more practice walking through the flowchart questions, simply click the Next button in the bottom right corner of the screen.
That’s not the correct statistical analysis. Let’s walk through the questions on the flowchart in Appendix E to determine what analysis could be used in this case.

In which of the following four categories does this situation fall? Click to see the data again. And click on the flowchart button to see the overview for choosing the best test.

Boomerang also reported that the frequency of responding to emails was related to the length of the emails. Emails containing 75-100 words were responded to more frequently than emails containing 25 or fewer words. Imagine that we recruited 20 adult participants. We ask each participant how frequently they respond, on a scale of 1-7 (1 = not very, 7 = very), to emails containing 25 or fewer words. We ask those same participants how often they respond to emails containing 75-100 words.
Correct! There is at least one nominal independent variable and a scale dependent variable.
Actually, there is at least one nominal independent variable and a scale dependent variable.

How many nominal independent variables are there?
Correct! There is one nominal independent variable, length of email. (The dependent variable, frequency ratings, is scale.)
Actually, there is one nominal independent variable, length of email. (The dependent variable, frequency ratings, is scale.)

How many levels does the independent variable have?
Correct! The independent variable has two levels or groups: 25 words or fewer and 75-100 words.
Actually, the independent variable has two levels or groups: 25 or fewer words and 75-100 words.

How many samples are there?
Correct! There are two samples, one consisting of emails with 25 or fewer words, and one consisting of emails with 75-100 words.
Actually, there are two samples, one consisting of emails with 25 or fewer words, and one consisting of emails with 75-100 words.

What type of design is this?
Correct! This is a within-groups design. Each participant rated both types of email.
Actually, this is a within-groups design. Each participant rated both types of email.

Based on the answers to these questions, which statistical test could be used to determine if there was a significant difference between the participants’ ratings of the two types of email length?
Correct! The researchers could have used a paired-samples t test. There is one nominal independent variable, length of email, with two levels or groups, 25 or fewer words and 75-100 words. There is a scale dependent variable, frequency ratings on a scale of 1-7. All participants rated both types of email, so it is a within-groups design.
Actually, that’s not the correct statistical analysis. The researchers could have used a paired-samples t test. There is one nominal independent variable, length of email, with two levels or groups, 25 or fewer words and 75-100 words. There is a scale dependent variable, frequency ratings on a scale of 1-7. All participants rated both types of email, so it is a within-groups design.

Boomerang reported that the number of questions asked in an email was also related to the likelihood of responding: “We found that emails that asked 1-3 questions are 50% more likely to get a response than emails asking no questions.” However, asking too many questions also lowers the response rate: Boomerang’s results indicated that “an email with 3 questions is 20% more likely to get a response than an email with 8 or more!” Imagine that we recruited 30 undergraduates, and randomly assigned them to one of three groups: The first group would be shown an email containing 1-3 questions; the second group would be shown an email containing 4-7 questions; and the third group would be shown an email containing 8 or more questions. After viewing their respective emails, the participants would be asked to rate the likelihood, on a scale of 1-7 (1 = not very, 7 = very), that they would respond to the emails.

Which statistical test could be used to determine if there was a significant difference among the three groups’ ratings?












Correct! The researchers could have used a one-way between-groups ANOVA because there is one nominal independent variable, number of questions, with three levels or groups: 1-3, 4-7, and 8 or more. There is a scale dependent variable, likelihood-of-responding ratings. And the participants are only in one of the three groups, so it is a between-groups design.
 
Now skip ahead to the next example by clicking here. Or, for more practice walking through the flowchart questions, then simply click the Next button in the bottom right corner of the screen.
That’s not the correct statistical analysis. Let’s walk through the questions on the flowchart in Appendix E to determine what analysis could be used in this case.

In which of the following four categories does this situation fall? Click to see the data again. And click on the flowchart button to see the overview for choosing the best test.

Boomerang reported that the number of questions asked in an email was also related to the likelihood of responding: “We found that emails that asked 1-3 questions are 50% more likely to get a response than emails asking no questions.” However, asking too many questions also lowers the response rate: Boomerang’s results indicated that “an email with 3 questions is 20% more likely to get a response than an email with 8 or more!” Imagine that we recruited 30 undergraduates, and randomly assigned them to one of three groups: The first group would be shown an email containing 1-3 questions; the second group would be shown an email containing 4-7 questions; and the third group would be shown an email containing 8 or more questions. After viewing their respective emails, the participants would be asked to rate the likelihood, on a scale of 1-7 (1 = not very, 7 = very), that they would respond to the emails.
Correct! There is at least one nominal independent variable and a scale dependent variable.
Actually, there is at least one nominal independent variable and a scale dependent variable.

How many nominal independent variables are there?
Correct! There is one nominal independent variable, number of questions. (The dependent variable, likelihood rating, is scale.)
Actually, there is one nominal independent variable, number of questions. (The dependent variable, likelihood rating, is scale.)

How many levels does the independent variable have?
Correct! There are three levels or groups: 1-3, 4-7, 8 or more.
Actually, there are three levels or groups: 1-3, 4-7, 8 or more.

What type of design is this?
Correct! This is a between-groups design. Each participant is only in one of the groups.
Actually, this is a between-groups design. Each participant is only in one of the groups.

Based on the answers to these questions, which statistical test could be used to determine if there was a significant difference in likelihood of responding ratings among the three variables: 1-3 questions, 4-7 questions, 8 or more questions?
Correct! The researchers could have used a one-way between-groups ANOVA because there is one nominal independent variable, number of questions, with three levels or groups: 1-3, 4-7, 8 or more. There is a scale dependent variable, likelihood ratings. And the participants are only in one of the three groups, so it is a between-groups design.
Actually, that’s not the correct statistical analysis. The researchers could have used a one-way between-groups ANOVA because there is one nominal independent variable, questions, with three levels or groups: 1-3, 4-7, 8 or more. There is a scale dependent variable, likelihood ratings. And the participants are only in one of the three groups, so it is a between-groups design..

Boomerang also presented a finding about the relation between the number of words in the subject line of an email and the likelihood of responding to that email. According to Boomerang, subject lines containing 3 or 4 words received more responses than emails with a greater number of words in the subject line. Imagine that we recruited 30 people. We present each person with an email in which the number of words in the subject line varied. We also ask each person to rate the likelihood, on a scale of 1-10 (1 = not very, 10 = very) of responding to that email.

Which statistical test could be used to determine if there was a relation between the two variables – number of words in subject line and likelihood-of-responding rating?












Correct! The researchers could have used the Pearson correlation coefficient. There are only scale variables. The first scale variable is number of words in subject line, and the second scale variable is likelihood-of-responding ratings. The question is about association rather than prediction. The finding as described would be a negative correlation. As the number of words in the subject line increases, the likelihood of responding tends to be lower.
 
Now skip ahead to the next example by clicking here. Or, for more practice walking through the flowchart questions, simply click the Next button in the bottom right corner of the screen.
That’s not the correct statistical analysis. Let’s walk through the questions on the flowchart in Appendix E to determine what analysis could be used in this case.

In which of the following four categories does this situation fall? Click to see the data again. And click on the flowchart button to see the overview for choosing the best test.

Imagine that we recruited 30 people. We present to each person an email in which the number of words in the subject line varied. We also ask each person to rate the likelihood (1 = not very, 10 = very) of responding to that email.
Correct! There are only scale variables—number of words in subject line and likelihood-of-responding rating.
Actually, there are only scale variables – number of words in subject line and likelihood-of-responding rating.

Is the research question about association or prediction?
Correct! The research question is about association. We want to know if two variables are related, not whether one predicts the other.
Actually, the research question is about association. We want to know if two variables are related, not whether one predicts the other.

Based upon the answers to these questions, which statistical test could be used to determine if there was a relation between the two variables – number of words in subject line and number of responses?
Correct! The researchers could have used the Pearson correlation coefficient. There are only scale variables, number of words in the subject line and likelihood-of-responding rating. The research questions is about association rather than prediction. The finding as described would be a negative correlation. As the number of words in the subject line increases, the likelihood of responding tends to be lower.
Actually, that’s not the correct statistical analysis. The researchers should have used the Pearson correlation coefficient. There are only scale variables, number of words in the subject line and likelihood-of-responding rating. The research questions is about association rather than prediction. The finding as described would be a negative correlation. As the number of words in the subject line increases, the likelihood of responding tends to be lower.

Let’s look at one last finding from the Boomerang article. According to Boomerang, when writing an email, people should include “more opinions and more subjectivity!” Boomerang indicated “the more opinionated the content of the email, the higher the response rate climbed.” They reported that subjective emails were more likely to receive a response than objective emails. But let’s say we didn’t have these data yet, and only knew that there was a 42% response rate for objective emails.
So, we recruit 20 people to receive subjective emails that are otherwise identical to the objective emails used to find the 42% rate. We wait 48 hours and then indicate whether there was a response or no response to that email.

What statistical analysis could be used to determine whether there was a significant difference in the response rate that we observed for subjective emails compared to the known Boomerang response rate for objective emails? Note that the response category was calculated by counting how many emails received a response and how many did not receive a response.












Correct! The researchers could have used a chi-square test for goodness of fit. There is one nominal variable, response category (received a response or did not receive a response). There are no scale variables.
 
Now skip ahead to the end of the activity by clicking here. Or, for more practice walking through the flowchart questions, simply click the Next button in the bottom right corner of the screen.
Actually, that’s not the correct statistical analysis. Let’s walk through the questions on the flowchart in Appendix E to determine what analysis could be used in this case.

In which of the following four categories does this situation fall? Click to see the data again. And click on the flowchart button to see the overview for choosing the best test.

Let’s look at one last finding from the Boomerang article. According to Boomerang, when writing an email, people should include “more opinions and more subjectivity!” Boomerang indicated “the more opinionated the content of the email, the higher the response rate climbed.” They reported that subjective emails were more likely to receive a response than objective emails. But let’s say we didn’t have these data yet, and only knew that there was a 42% response rate for objective emails.
So, we recruit 20 people to receive subjective emails that are otherwise identical to the objective emails used to find the 42% rate. We wait 48 hours and then indicate whether there was a response or no response to that email.
What statistical analysis could be used to determine whether there was a significant difference in the response rate that we observed for subjective emails compared to the known Boomerang response rate for objective emails? Note that the response category was calculated by counting how many emails received a response and how many did not receive a response.





Correct! There is only a nominal variable, response category. There are no scale variables.
Actually, there is only one nominal variable, response category. There are no scale variables.

How many nominal variables are there?
Correct! There is one nominal variable, response category.
Actually, there is only one nominal variable, response category.

Based on the answers to these questions, what statistical test could be used to determine whether there was a significant difference in the response rate that we observed compared to the known Boomerang response rate?
Correct! The researchers could have used a chi-square test for goodness of fit. There is one nominal variable, response category (received a response or did not receive a response). There are no scale variables.
Actually, that’s not the correct statistical analysis. The researchers could have used a chi-square for goodness of fit test. There is one nominal variable, response category (received a response or did not receive a response). There are no scale variables.

Congratulations! You have completed the activity and gained some good experience in choosing the best hypothesis test.