Introduction

Chapter 4. The Psychologist’s Building Blocks for Better Design

Introduction

Chapter 4
true
true
You must read each slide, and complete any questions on the slide, in sequence.
Self Report Measure
A measure that involves directly asking participants to express how they feel or what they think about a particular topic.
Behavioral Measure
A measure that assesses the actions a person took, how the person responds to a request, the person’s decisions, or simply what some researchers call “actual behavior.”
Behavioral Trace
A measure that uses evidence left behind by a person who is no longer present to determine a behavior occurred.
Behavioral Observation
A measure that relies on directly seeing or observing behavior as it occurs.
Behavioral Choice
A measure that assesses individual’s selection among a variety of options.
Raw Score
The score that a measure provides that is the combination of the true score and error.
True Score
The part of the raw score that is an accurate depiction of your actual ability, or what your score would be on a perfect test, free of extraneous factors.
Error
The part of the raw score that represents those extraneous influences that will cause the raw score to deviate (up or down) from the true score.
Random Error
Measurement error that is variation from one’s true score due to unsystematic or chance factors that occurs in a haphazard, disorganized, or arbitrary way.
Bias or Systematic Error
Measurement error that varies systematically or consistently pushes scores higher or lower in a way that leads to inaccurate or misleading conclusions.
Standardization
keeping the experimental situation the same for everyone
sensitivity
the range of data a researcher can gather from a particular instrument
Reliability
the stability or consistency of a measure
Validity
the degree to which a tool measures what it claims to measure
Random Sampling
selecting participants using an unpredictable method that give everyone an equal chance of selection in order to avoid potential bias
Convenience Sampling
non-randomly recruiting participants from a known, readily available population
Nonresponse Bias
those who do not respond to a survey may differ in meaningful ways related to the research question from those who do.
Volunteer Subject Problem
those who self-select or chose to be in a study may differ in some meaningful way related to the research question from those who make the choice to not participate
College Sophomore Problem
using a sample that relies on college students from a participant pool who may not adequate represent other populations as they tend to be all around the same age
Demand Characteristics
cues that make participants potentially aware of what the experimenter expects.
Social Desirability Concerns
the tendency for respondents to give answers that make them look good.

Got Swagger?
Evaluating the Swagger-Meter

Online Activity
Chapter 4: The Psychologist’s Building Blocks for Better Design

Something to Think About

Scenario: The mall is a popular place for the 16-25 year-old demographic. Companies know this and continually seek to develop new products that target this key segment of the market. In fact, a new company named YOLO Incorporated has designed a product that, according to their website and press release, “provides user-specific, science-based insights into a key aspect of young adults’ personality.” In addition, “the Swagger-Meter uniquely capitalizes on people’s desire for self-knowledge and data-based social comparison information.” Specifically, YOLO Inc. wants to place their new product the “Swagger-Meter 1.0™” in mall food courts throughout the United States.

Your job is to determine whether you should invest your money in this company.

Something to Think About (continued)

If you are thinking like a scientist, you are probably a bit skeptical about the “Swagger-Meter 1.0™.” To decide whether this product will be successful and ultimately worth investing your money in, you need to determine whether the “Swagger Meter 1.0™” is a quality measure.

Something to Think About (continued)

Before you can know whether something is a good measure, you first need to know exactly what it aims to measure. That is, you need to clearly define key concepts with an operational definition that describes exactly what a variable means.

Question 4.1

In this case, the key variable is “swagger.” How should you go about defining it?

A.
B.
C.
3
Correct.
Try again.
Incorrect.

Something to Think About (continued)

According to the YOLO Inc. press release, “Swagger is a mental state of being in which a person exudes a high level of confidence and composure. Swagger can border on arrogance, but is more about a sense of positive self-assuredness that makes the person charismatic and admired.”

Now that you know how YOLO Inc. is defining what is meant by “swagger,” the next question is to understand how they are actually measuring “swagger.” Yolo Inc. is doing this by having individuals place their right hand on a mental plate. This plate measures their skin conductance and indicates the corresponding degree of swagger. This is what the “Swagger Meter 1.0™” looks like:

Swagger Meter 1.0™

Our Question

Now that you know what the “Swagger Meter 1.0™” looks like, and what it claims to measure, you can begin to answer two questions:

  1. Is the “Swagger Meter 1.0™” a quality measure?
  2. Should you invest your money in this company and product?

Picking the Best Measurement

Before you can know whether something is a good measure, you first need to know exactly what it aims to measure. That is, you need to clearly define key concepts with an operational definition that describes exactly what a variable means.

Let’s start by evaluating the nature of the “Swagger Meter 1.0™” measurements.

  • Self Report Measure
  • Behavioral Measure

Question 4.2

What type of measurement does the “Swagger Meter 1.0™” use?

A.
B.
3
Correct.
Try again.
Incorrect.

Picking the Best Measurement

Question 4.3

Is it a good idea for the “Swagger Meter 1.0™” to avoid using self-report measurements?

A.
B.
C.
3
Correct.
Try again.
Incorrect.

Picking the Best Measurement

Using one of the types of behavioral measure was a good idea.

  • Behavioral Trace
  • Behavioral Observation
  • Behavioral Choice

Question 4.4

What type of behavioral measurement does the “Swagger Meter 1.0™” use?

A.
B.
C.
3
Correct.
Try again.
Incorrect.

Evaluating the Measure

Measurement is inherently imperfect. In fact, every measurement consists of three parts:

  • Raw Score
  • True Score
  • Error

Evaluating the Measure

If all measures contains some type of error, the “Swagger Meter 1.0™” will be no exception. The key will be to identify where error can creep in and determine how much of problem it may be. There are two types of measurement error.

  • Random Error
  • Bias or Systematic Error

Evaluating the Measure

For each of the following potential sources of measurement error, please indicate whether it represents random error or systematic bias.

Question 4.5

The “Swagger Meter 1.0™” has a glitch that is causing it to routinely overestimate users’ swagger.

A.
B.
2
Correct.
Try again.
Incorrect.

Question 4.6

Three of the “Swagger Meter 1.0™” sensors are faulty and only register measurements intermittently.

A.
B.
2
Correct.
Try again.
Incorrect.

Question 4.7

One of the legs on the “Swagger Meter 1.0™” machine is short, causing it to sometimes wobble, which makes it hard for users to properly place their hand on the machine.

A.
B.
2
Correct.
Try again.
Incorrect.

Evaluating the Measure

To determine if the “Swagger Meter 1.0™” is a high quality measure, you need to determine if YOLO Inc. has developed a consistent process for measuring a person’s swagger. In other words, you need to know if there is standardization in the administration of the “Swagger Meter 1.0™.”

  • Standardization

Evaluating the Measure

Question 4.8

Which of the following is something that that YOLO Inc. cannot standardize about the administration of the “Swagger Meter 1.0™?”

A.
B.
C.
3
Correct.
Try again.
Incorrect.

Evaluating the Measure: A Checklist

checklist_activity
checklist_question_1

Question 4.9

Based on the what YOLO Inc. can control in measuring swagger, do you think the “Swagger Meter 1.0™” has adequate standardization? Please indicate your selection by clicking on one of the boxes.

A.
B.
checklist_question_1
2
Correct.
Try again.
Incorrect.
Adequate Poor
Standardization
Sensitivity
Reliability
Validity

Evaluating the Measure

If the “Swagger Meter 1.0™” is a high quality measure, it will have adequate sensitivity or a good range of possible measurement levels.

  • sensitivity

Question 4.10

Which of the following is the best range of measurement levels for the “Swagger Meter 1.0™?”

A.
B.
C.
3
Correct.
Try again.
Incorrect.

Evaluating the Measure: A Checklist

checklist_activity
checklist_question_2

Question 4.11

Based on the number of options YOLO Inc. currently provides for the “Swagger Meter 1.0™,” decide whether this measure has appropriate sensitivity. Please indicate your selection by clicking on one of the boxes.

A.
B.
checklist_question_2
2
Correct.
Try again.
Incorrect.
Adequate Poor
Standardization
Sensitivity
Reliability
Validity

Evaluating the Measure

If the “Swagger Meter 1.0™” is a high quality measure, it must be able to consistently or reliably assess users’ swagger.

  • Reliability

Question 4.12

Which of the follow scenarios depicts a reliable measure?

A.
B.
C.
3
Correct.
Try again.
Incorrect.

Evaluating the Measure

Based on what you know about reliability, you decide that you need to test the “Swagger Meter 1.0™” prototype during an investor visit to YOLO Inc.

Question 4.13

Which of the follow is the better test of the Swagger Meter’s reliability?

A.
B.
2
Correct.
Try again.
Incorrect.

Evaluating the Measure: A Checklist

checklist_activity
checklist_question_3

Question 4.14

Assuming that the measurements you made of your friends’ swagger at YOLO Inc. were consistent, do you think the “Swagger Meter 1.0™” is a reliable measure? Please indicate your selection by clicking on one of the boxes.

A.
B.
checklist_question_3
2
Correct.
Try again.
Incorrect.
Adequate Poor
Standardization
Sensitivity
Reliability
Validity

Evaluating the Measure

So far you have determined that the “Swagger Meter 1.0™” has adequate standardization, good sensitivity, and good reliability. Next you need to evaluate whether it actually measures what it claims to measure. That is, does it actually measure “swagger?”

  • Validity

Evaluating the Measure

One way to check a measure’s validity is to compare it to other measures of swagger. The YOLO Inc.’s press release states, “Our Research and Development Department placed a beta version of the Swagger Meter in the nearby mall where mall patrons of all ages could choose to test their swagger. Our test included 490 patrons with equal numbers of males and females ranging in age from 18 – 87… All test participants reported the Swagger Meter was uncanny in its accuracy.”

  • Random Sampling
  • Convenience Sampling

Question 4.15

Based on this information, what type of sample did they use to evaluate the validity of the Swagger Meter 1.0™?

A.
B.
C.
3
Correct.
Try again.
Incorrect.

Evaluating the Measure

Anytime someone uses data to support a claim being made on TV, in a magazine, an advertisement, or in this case on a business press release, you need to use your psychological literacy skills to critically evaluate that claim. After all, you don’t want to make the wrong choice based on potentially faulty information. A good place to start is to look at the source of the data, namely the sample used. Some common problems with samples that can bias the data include:

  • Nonresponse Bias
  • Volunteer Subject Problem
  • College Sophomore Problem

Question 4.16

Which of the following is a potential problem with the sample YOLO Inc. used?

A.
B.
C.
3
Correct.
Try again.
Incorrect.

Evaluating the Measure

YOLO Inc.’s press release states, “Based on field test data, the R & D department’s analysis demonstrates that the “Swagger Meter 1.0™” capably measures a user’s swagger based on user’s ‘hotness’ and the ‘electricity of their personality.’ The Swagger Meter accurately measures these constructs using a set of strategically placed titanium sensors that assess ‘tactile temperature’ and ‘skin conductance/galvanic skin response’ on the person’s palm. All test participants reported the Swagger Meter was uncanny in its accuracy.”

For each of the following, please indicate whether this finding undermines the validity of the “Swagger Meter 1.0™ as a measure of swagger.

Question 4.17

Independent testing shows that the “Swagger Meter 1.0™” does accurately measure a user’s temperature and skin conductance.

A.
B.
2
Correct.
Try again.
Incorrect.

Question 4.18

Users can influence their swagger rating by trying tricks such as warming their hands before placing them on the “Swagger Meter 1.0™.”

A.
B.
2
Correct.
Try again.
Incorrect.

Evaluating the Measure

  • Demand Characteristics
  • Social Desirability Concerns

Question 4.19

The fact that “all test participants reported the Swagger Meter was uncanny in its accuracy” supports YOLO Inc.’s claim that the “Swagger Meter 1.0™” is a valid measure of a person’s swagger.

A.
B.
3
Correct.
Try again.
Incorrect.

Evaluating the Measure: A Checklist

checklist_activity
checklist_question_4

Question 4.20

Based on your evaluation, do you think the “Swagger Meter 1.0™” is a valid measure? Please indicate your selection by clicking on one of the boxes.

A.
B.
checklist_question_4
2
Correct.
Try again.
Incorrect.
Adequate Poor
Standardization
Sensitivity
Reliability
Validity

The Decision

As Warren Buffet advised, “Never invest in a business you cannot understand.” Now that you have spent time studying the “Swagger Meter 1.0™,” you now must make a final determination about whether the “Swagger Meter 1.0™” is a quality measure, and if so, whether you should invest your money in this company.

Question 4.21

Based on your evaluation of the measure, what is the best conclusion?

A.
B.
C.
3
Correct.
Try again.
Incorrect.

The Decision

Because you contacted YOLO Inc. for a copy of its press release, someone from the Investor Relations Department gives you a call as a follow-up to answer any questions you may have. During that call you explain your evaluation of the measure and concerns about its lack of validity. The company representative freely admits that the “Swagger Meter 1.0™” may not actually measure swagger. But, he goes on to explain “Users are confident that it measures swagger. Besides, if people believe it is true, who cares whether it is. People are free to believe what they want and believe all sorts of things that aren’t true all the time.”

Question 4.22

Does this change your mind?

A.
B.
C.
3
Correct.
Try again.
Incorrect.

Take Home Message

Now that you have decided to not invest in YOLO Inc.’s new product, it is useful to explain what your findings mean in everyday terms so that others can benefit from your evaluation.

Question 4.23

How would you explain what you found about the “Swagger Meter 1.0™” to your Aunt or Uncle? Select the best option.

A.
B.
C.
D.
4
Correct.
Try again.
Incorrect.

Next Step

Question

Consider each of the following and provide an open ended response.

• What additional evidence would you want to consider the “Swagger Meter 1.0™” before determining that it is a valid measure?
• How might you improve reliability and validity of the “Swagger Meter 1.0™?”
• How could you measure someone’s swagger using behavioral traces?
• How could you measure someone's swagger using a non-physiological behavioral observations?
• How could you assess a person’s swagger using a self-report measure? How might you evaluate this measure’s reliability and validity?

Your Turn

Before you can know whether something is a good measure, you first need to know exactly what it aims to measure. That is, you need to clearly define key concepts with an operational definition that describes exactly what a variable means.

Question 4.24

1. Which of the following is an example of validity?

A.
B.
C.
3
Correct.
Try again.
Incorrect.

Question 4.25

2. Fred notices that people tend to underestimate the number of minutes it takes them to complete a task. It appears that these time estimates are subject to

A.
B.
C.
3
Correct.
Try again.
Incorrect.

Question 4.26

3. Olivia is debating whether to use 3, 7, or 25 response options for a self-report measure she is developing. Olivia is trying to decide the measure’s

A.
B.
C.
3
Correct.
Try again.
Incorrect.

Your Turn

Question 4.27

4. Jaynden notices that there appears to be an inconsistency in how researchers are using a new instrument to measure cognitive ability. He is concerned about the instrument's apparent lack of

A.
B.
C.
3
Correct.
Try again.
Incorrect.

Question 4.28

5. Ava is conducting a web-based survey to assess student attitudes toward a proposed closing of the College Tutoring Center. She sends all 2000 students enrolled in the college an email invitation to complete the survey. Of the 100 students who responded, 85% strongly believed that tutoring should be eliminated to save money on tuition. Before Ava concludes that the students in general support this closing, she should be concerned with which of the following potential problems with her sample?

A.
B.
C.
3
Correct.
Try again.
Incorrect.

Activity Completed!

Activity results are being submitted.