Chapter 4. Is the Swagger-Meter 1.0 a Quality Measure?

Introduction

This activity requires you to critically evaluate whether a device called the Swagger-Meter is a quality measure in terms of validity, reliability, and demand characteristics, and asks you to consider how to best administer the measure.

Dr. Natalie J. Ciarocco, Monmouth University

Dr. David B. Strohmetz, Monmouth University

Dr. Gary W. Lewandowski, Jr., Monmouth University

Something to Think About...

Scenario: The mall is a popular place for the 16- to 25-year-old demographic. Companies know this and continually seek to develop new products that target this key segment of the market. In fact, a new company named YOLO Incorporated has designed a product that, according to their website and press release, “provides user-specific, science-based insights into a key aspect of young adults’ personalities.” In addition, “the Swagger-Meter uniquely capitalizes on people’s desire for self-knowledge and data-based social comparison information.” Specifically, YOLO Inc. wants to place their new product, the Swagger-Meter 1.0, in mall food courts across the United States.

Your job is to determine whether you should invest your money in this company.

Something to Think About...

If you are thinking like a scientist, you are probably feeling skeptical about the Swagger-Meter 1.0. To decide whether this product will be successful and worth your investment, you need to determine whether the Swagger-Meter 1.0 is a quality measure.

John Giustina/Iconica/Getty Images

Something to Think About...

Before you can know whether something is a quality measure, you need to clearly define the variable that it aims to measure.

Something to Think About...

According to the YOLO Inc. press release, “Swagger is a mental state of being in which a person exudes a high level of confidence and composure. Swagger can border on arrogance, but is more about a sense of positive self-assuredness that makes the person charismatic and admired.”

Now that you know how YOLO Inc. defines “swagger,” it is time to find out how the company attempts to measure “swagger.” First, YOLO Inc. asks individuals to place their right hand on a metal plate. The company then measures their skin conductance, which corresponds to “degree of swagger.” This is what the Swagger-Meter 1.0 looks like:

Swagger-Meter 1.0

Our Research Question

Jesse Kunerth/iStock/Getty Images

Now that you know what the Swagger-Meter 1.0 looks like, and what it claims to measure, you can begin to answer two questions:

1. Is the Swagger-Meter 1.0 a quality measure?
2. Should you invest your money in YOLO Inc. and its product?

Picking the Best Measurement

Let’s start by evaluating the nature of the Swagger-Meter 1.0 measurements.

• Self-report Measure

• Behavioral Measure

Lev Mei/iStock/Getty Images

Picking the Best Measurement

Lev Mei/iStock/Getty Images

Picking the Best Measurement

Now let’s evaluate the different types of behavioral measures.

• Behavioral Trace

• Behavioral Observation

• Behavioral Choice

Lev Mei/iStock/Getty Images

Evaluating the Measure

Keep in mind that all measurements are approximate. In fact, every measurement consists of three parts:

• Raw Score

• True Score

• Error

Evaluating the Measure

Since all measures contain some form of error, the Swagger-Meter 1.0 will be no exception. We must identify where error can creep in and determine how much of problem it might prove to be. First, let’s review the two types of measurement error:

• Random Error

• Bias or Systematic Error

Evaluating the Measure

Now it’s your turn. Indicate whether the following potential sources of measurement error represent random error or systematic bias.

Evaluating the Measure

To decide whether or not the Swagger-Meter 1.0 is a high-quality measure, you need to determine if YOLO Inc. has developed a consistent process for measuring people’s swagger. In other words, you need to know if there is standardization in the administration of the Swagger-Meter 1.0.

• Standardization

Evaluating the Measure

Evaluating the Measure: A Checklist

Evaluating the Measure

If the Swagger-Meter 1.0 is truly a high-quality measure, it will have adequate sensitivity, or a good range of possible measurement levels.

• Sensitivity

Evaluating the Measure: A Checklist

Evaluating the Measure

If the Swagger-Meter 1.0 is a high-quality measure, it must be able to consistently and reliably assess users’ swagger.

• Reliability

Evaluating the Measure

Based on what you know about reliability, you decide that you need to test the Swagger-Meter 1.0 prototype during an investor visit to YOLO Inc.

Evaluating the Measure: A Checklist

Evaluating the Measure

So far you have determined that the Swagger-Meter 1.0 has adequate standardization, good sensitivity, and good reliability. Next you need to evaluate whether it actually measures what it claims to measure. That is, does it actually measure “swagger”?

• Validity

Evaluating the Measure

One way to check a measure’s validity is to compare it to other measures of swagger. The YOLO Inc. press release states, “Our Research and Development Department placed a beta version of the Swagger-Meter in the nearby mall where mall patrons of all ages could choose to test their swagger. Our test included 490 patrons with equal numbers of males and females ranging in age from 18–87… All test participants reported the Swagger-Meter was uncanny in its accuracy.”

• Random Sampling

• Convenience Sampling

Evaluating the Measure

Anytime someone uses data to support a claim being made on TV, in a magazine, in an advertisement, or, in this case, on a business press release, you need to use your psychological literacy skills to critically evaluate their claim. After all, you don’t want to make the wrong choice based on faulty information. A good place to start is to look at the source of the data, namely the sample used. Some common problems with samples that can bias data include:

• Nonresponse Bias

• Volunteer Subject Problem

• College Sophomore Problem

Evaluating the Measure

YOLO Inc.’s press release states, “Based on field test data, the R & D department’s analysis demonstrates that the Swagger-Meter 1.0 capably measures a user’s swagger based on user’s ‘hotness’ and the ‘electricity of their personality.’ The Swagger-Meter accurately measures these constructs using a set of strategically placed titanium sensors that assess ‘tactile temperature’ and ‘skin conductance/galvanic skin response’ on the person’s palm. All test participants reported the Swagger-Meter was uncanny in its accuracy.”

For each of the following, indicate whether this finding undermines the validity of the Swagger-Meter 1.0 as a measure of swagger.

Evaluating the Measure

• Demand Characteristics

• Social Desirability Concerns

Evaluating the Measure: A Checklist

The Decision

As successful investor Warren Buffett advised, “Never invest in a business you cannot understand.” Now that you have spent time studying the Swagger-Meter 1.0, you must make a final determination about whether the Swagger-Meter 1.0 is a quality measure, and, if so, whether you should invest your money in this company.

The Decision

Because you contacted YOLO Inc. for a copy of its press release, someone from the Investor Relations Department gives you a call as a follow-up to answer any questions you may have. During that call you explain your evaluation of the measure and concerns about its lack of validity. The company representative freely admits that the Swagger-Meter 1.0 may not actually measure swagger. But, he goes on to explain, “Users are confident that it measures swagger. Besides, if people believe it is true, who cares whether it is? People are free to believe what they want and believe all sorts of things that aren’t true all the time.”

Take Home Message

Now that you have decided not to invest in YOLO Inc.’s new product, it is useful to explain what your findings mean in everyday terms so that others can benefit from your evaluation.

RapidEye/iStockphoto/Getty Images

Congratulations! You have successfully completed this activity.

KMGS Photography/Moment/Getty Images