Chapter 1. What's Wrong With This Study?

Synopsis

Human Development Video Activity

true

You must read each slide, and complete any questions on the slide, in sequence.

control condition

In scientific research and experimentation, the standard to which manipulated subjects, conditions, or states are compared.

correlational approach

A research approach that assesses different variables to see if there is a predictable relationship between them.

descriptive approach

A research approach in which straightforward and accurate descriptions are provided of the characteristics of something in a study.

double-blind procedure

A scientific experiment in which both the subjects of the study and those administering the study are prevented from knowing critical details that might bias them and compromise the study’s results.

experimental approach

A research approach for assessing cause and effect relationships in which subjects of study are manipulated in order to observe any corresponding changes.

experimental condition

The condition or treatment that is introduced for testing allowing experimenters to observe its effects.

norms

A marker of average development or achievement.

population

A group of people, animals, or organisms that are of a particular concern in a scientific study.

random assignment

In a research study, the relying on chance procedures, such as coin flipping, for the assignment of participants to control or experimental groups. This random approach insures that each participant has an equal opportunity of being assigned to a particular group.

random sample

A study sample that is selected using the random assignment method that insures that all members of a group or population have the same chance of being selected.

representative sample

A small research group with characteristics that resemble those of the larger population from which it is drawn.

sample

A portion of a population or group that is selected to represent the group as a whole.

Student looking through documents in a library

What's Wrong With This Study?

Author

Thomas E. Ludwig, Hope College
S. Stavros Valenti, Hofstra University

Synopsis

Putting together a research study is challenging work. In this activity, you will explore some of the major pitfalls of the process of designing a research study and then read descriptions of three hypothetical studies. Each of these hypothetical studies has one major flaw and several minor problems. You will be asked to identify the major flaw in each study.

REFERENCES

Baltes, P. B., Reese, H., W., & Nesselroade, J. R. (1977). Life-span developmental psychology Introduction to research methods. Hillsdale, NJ: Lawrence Erlbaum.

Brown, K. W., Cozby, P. C., Kee, D. W., & Worden, P. E. (1999). Research methods in human development (2nd ed.). Mountain View, CA: Mayfield.

Gelfand, D. M. (2000). Developmental science: What do we know and how do we know it? Human Development, 43, 252–256.

Hartmann, D. P., & George, T. P. (1999). Design, measurement, and analysis in developmental research. In M. H. Bornstein & M. E. Lamb (Eds.), Developmental psychology: An advanced textbook (4th ed.). Mahwah, NJ: Lawrence Erlbaum

Miller, S. A. (1998). Developmental research methods (2nd ed.). Upper Saddle River, NJ: Prentice-Hall.

Rogers, A. G. (2000). When methods matter: qualitative research issues in psychology. Harvard Educational Review, 70, 75–85.

Zablotsky, D. (2001). Why do I have to learn this if I’m not going to graduate school? Teaching research methods in a social psychology of aging course. Educational Gerontology, 27, 609–622.

Thinking About Developmental Research

Students in lab coats in a lab having a discussion

Planning and conducting a good research project is no small task. There are many complexities to consider when developing a research study, and there is always a chance that the results may not support the researcher’s hypothesis. While this activity will test your knowledge of some of the common pitfalls in developmental research, let’s review some of the basic concepts in research methodology.

Research Goals

Depending on the goal of their study, researchers generally adopt one of three methods when designing a study.

• If the goal is to describe or document a particular behavior, researchers rely on the descriptive approach, which is a method of investigation used to observe, measure, and describe behavior.

• If the goal is to attempt to predict the occurrence of a particular behavior, researchers use the correlational approach, which is a method that measures two different types of behavior and evaluates the relationship between them.

• If the goal is to explain a behavior or understand why it occurs, researchers turn to the experimental approach, which is a method of investigation used to demonstrate cause-and-effect relationships by purposely manipulating one aspect of a situation that is thought to produce a change in a particular behavior.

Samples and Populations

Five people of different ages, sex, and races and ethnicities as if representing a sample for a research experiment

Most developmental research involves testing a relatively small group of people, known as the sample, and then trying to apply the results of that test to a much larger group of people, known as the population. In order to generalize their findings, researchers must select a representative sample, one that closely parallels the population on all the relevant characteristics. For example, if the population of interest is evenly divided between men and women and you planned to survey 100 individuals, you would aim to include 50 men and 50 women in your sample.

Usually, researchers begin by choosing a random sample, which is a group chosen from the population by random selection such that every member of the larger group, or population, has an equal chance of being selected. For example, picking every tenth name from a college phone directory demonstrates random sampling. This method of sampling generally produces a representative sample.

The size of the sample is also a factor. Generally, large representative samples are preferred to small representative samples because they are more likely to reflect the patterns of the greater population. However, a small representative sample may well be better than a large sample that is not representative of the population.

Experimental Control

A student with a computer collecting data in the field

While descriptive and correlational studies provide useful information, only actual experiments enable the investigator to explore cause and effect. Because an experiment allows for the controlling of all relevant factors, the researcher can begin to isolate and explain behavior. In many experiments, control is achieved through the random assignment of the participants to different conditions or experiences. Typically, a researcher randomly assigns participants either to an experimental condition, in which they are exposed to a particular treatment, or to a control condition, in which they are not exposed to the treatment.

How much control researchers actually have over their experiments is debatable. For example, if participants know which group (either control or experimental) that they have been assigned to, their expectations about the treatment might influence their behavior. Another example of an influence might be if the researchers who interact with the participants are aware of which participants are in the experimental group and which ones are in the control group, they may inadvertently influence the results of the experiment. They might, for example, treat the groups differently or score people’s behavior differently because they expect the groups to differ. A preferred approach is to use a double-blind procedure. With this procedure, neither the participants nor the researchers who collect the data know in which group any particular participant is.

Now, we are ready to test your knowledge. The following pages contain brief descriptions of three studies, each with a major methodological flaw. After reading the descriptions of the studies, your task is to identify the flaw.

Study 1: Vocabulary Growth

A young girl around 4 or 5 years old reading with her father

Professor Jones wants to study vocabulary growth in preschool children. He contacts all of his faculty colleagues who have children, and from this group, he assembles a sample of 20 faculty children ranging from two to five years of age. To test each child’s word knowledge, Professor Jones reads each child a list of simple words that he has gathered from children’s storybooks and records whether or not the child understands each word. He then uses the results to publish norms (standard or expected levels of achievement) for vocabulary mastery for each year from two to five years of age.

Study 1: Vocabulary Growth (continued)

A young girl around 2 or 3 years old looking at a book with a woman

Question 1.1

TE8wEXz9wLtCuAWSgj7PKV0CGdH4hPneZ37sdNevJ1JG6E/sNdYRF0TgUFTej23dRhF8k1mzA4P7sSGsc9fcw/+US6JPxyd0X0EyZFdoaF6CLF43Jf18WG+hm9yM1SbY/DG60nu6w1WVhLXwriSvTeVRSLNPR0VW1LmVHrA2G61E2MUr952MwpHRrk48O+fBSrxjAGdcFKVOJ0C+ZQLit3fMyBbcVUxuPe88fea8yi60YADUB8phZ8Lbhha6u3Ox9zlQ4jOrbOpm3DWeBzlRqj5WeD3YTgbSljAnzj+Y2wtcxR8srjOu+g5l5hedWL06sj71T0GRAKp5e+j59k5siO/B7zDA/O7Bsokd4DvUkXNbg7Jgy/giYrbXcl+KVzIyLDWZUgeEDPEkA7tuwIF2obftHkN47cYVjW9TVpADfJGW6EoB

Correct. Professor Jones tested only the children of college professors. This limited group is an example of a biased or nonrepresentative sample. The genetic inheritance and environmental experiences of faculty children could differ from those of the general population in important ways that would influence the rate of language development.

Incorrect. Professor Jones tested only the children of college professors. This limited group is an example of a biased or nonrepresentative sample. The genetic inheritance and environmental experiences of faculty children could differ from those of the general population in important ways that would influence the rate of language development.

Study 2: Learning to Share

Professor Okamoto is interested in the effects of prosocial TV on sharing behavior among preschool children. For clarification, prosocial actions are meant to benefit others rather than oneself. She assembles a sample of 100 children and then assigns each child at random to either the prosocial TV group or the neutral TV group. The prosocial TV group views 30 minutes of TV with a prosocial theme while the neutral group views 30 minutes of TV with a neutral theme. After the children finish their TV session, Professor Okamoto assigns them to “play groups” of five children from the same TV group. She observes the small groups at play and rates each child on the tendency to share toys with other children.

Study 2: Learning to Share (continued)

Question 1.2

YmFyHXGwaDp2FOluTnavIE9178uAsqS9hVH4sRGNz2eu8AuFkNXCYv+jVTX8UkD/LkyagctGOMhwQyPZNB6JzUY3K8Woh9YpoG/dFxntEcZpvtw+Ek3CMCbVsYWz6Eb4Iy1L8U5Gf8wonD8MlMvAetlD98pQnM317ZfXEsRN2mLnqKFRxdf5iNk7g1DTfhR/GBBQVWk4cZix4WGMgB6c9jGWrLaZoIHmTL7gUfioueSfPY8i8eA+otSX/TyiysEEb/t/46NDLKUoT4uWfJg8yXKP6d5gSe2nlRocrOxgUnz3uX+dppl7exVJydYPFHRwLxSt4MVodhoFbV52qViuaZCnQ/8iiaCL1M/uJuHz4/xHA5dr2g1/pCUlVIxwI+YCI1ViRAbVjGvMeiFpZKHodIvpqhLnBsljqxAm4//tRsMohfRD

Correct. The major flaw in this study is the fact that while Professor Okamoto did assign children randomly to the prosocial or neutral groups, she was actively involved in putting children into their play groups as well as observing their behaviors. This is an example of a possible experimenter bias because Professor Okamoto’s knowledge of the type of television that each child had watched might influence her ratings of each child’s behavior.

Incorrect. The major flaw in this study is the fact that while Professor Okamoto did assign children randomly to the prosocial or neutral groups, she was actively involved in putting children into their play groups as well as observing their behaviors. This is an example of a possible experimenter bias because Professor Okamoto’s knowledge of the type of television that each child had watched might influence her ratings of each child’s behavior.

Study 3: Language Instruction

A language teacher standing in front of a chalkboard

Professor Knudsen wanted to test the effect of a new method of second language instruction for young adults speaking only one language. She introduced the new method in all three of her spring semester German classes. At the end of the semester, she averaged the student grades. Since the overall class average was a B, she concluded that the new method enhanced acquisition of the second language.

Study 3: Language Instruction (continued)

Students in a language lab with headphones

Question 1.3

SDObFe7DTNpM5FdIoLc0YBKWGdhyXfMi8Sm4TDUGWtKVTPcbUxtRjskpd7cAoL63WPD9uEbI/3NkPLZdODawzHj9QgqJLXMFIWSIaLD5WW2TlxZpz2aSVYQfIr2ytsSgpBXiXZM7WIz6Jp0klIuVKlwowNLwJmxW+JBuYMPlCUX50I1OMTV06YzAJlVrwdFT1FxC20FRr3+LzMofrAxuwc3Yjcnm/1UfSohCI4+D9dhOrRXsPrmXN9Ybn+rsECtZy17rpktBYN8oszQd+xxZyGkzJlTerhG6Qdjwq+aPwqZx9W+LnaR1omrC9rjlnvN2kmlKY15tCcJyccrJ/0XcT3BqhiWN3tQKvJM3iExtRd3MT+gX1yna8+o1Hh5wXkA990xEvgFZ4AS3A7EdTNSXzCNmDUJrEt7qUEOn3nLm9OVXLXj5zpllUw==

Correct. The major flaw in this study is lack of a control group. Professor Knudsen administered the new teaching method to all of her students so there is absolutely no way to tell whether her method made any difference in her students’ acquisition of the second language. Without a comparison group, we do not know whether a B average is better or worse than student performance under Professor Knudsen’s old teaching method or even under alternative teaching methods used by other instructors.

Incorrect. The major flaw in this study is lack of a control group. Professor Knudsen administered the new teaching method to all of her students so there is absolutely no way to tell whether her method made any difference in her students’ acquisition of the second language. Without a comparison group, we do not know whether a B average is better or worse than student performance under Professor Knudsen’s old teaching method or even under alternative teaching methods used by other instructors.

Some General Conclusions About Research

What can you conclude from these research study examples? We hope you do not become discouraged about the research process or the prospects of learning anything from developmental research. Instead, we hope that you have gained a greater appreciation for the need to plan carefully before beginning a research project and the need to follow commonly accepted procedures to avoid pitfalls in research design and in the interpretation of results.

Assessment: Check Your Understanding

Question 1.4

yalxJEkcZDssPi+Tf9SSJsmQDDIUIYj5pLLslcF2WDHh1M4mpfswQuSyozzmSr39J49Sysga4PAPZMjQq2Hbd3udx5Gnv1Y5Zqtl+IcUKTWovIDjc/MebCg6cbKW/4Z1WJqlLmS1pkh7wtjVtKnaaWoFWWiR6Wn5U6RSxBLDejv30hgYGZ5QGRUvdRcycZJu5qcq5Ns4qtY3XTAeT8RXsX0c7BYXmlmaK3mudyQf954fB88OFUv0Kw==

Correct. This statement is true. If participants were allowed to choose in which group they wanted to be, it is likely that the groups would differ in important ways before the experiment even begins. However, random assignment should minimize any differences.

Incorrect. This statement is true. If participants were allowed to choose in which group they wanted to be, it is likely that the groups would differ in important ways before the experiment even begins. However, random assignment should minimize any differences.

Assessment: Check Your Understanding

Question 1.5

4p2kMzbbP0VY8sbFhP7/8grudaIz6+tNmxPZ9v5b8d74d1r3idl4N6SzvjKaFo1fiber+L2Xy+a9LwmEj5K46aioKPIanKi7b75ruyV55cxmKKvqMCA6ky8P6Eq2yxLgEzb1xe7lpvpNfDoTbiHg6tHyTUxi94GvZfSvrFoVgbmHlewH3NEA1oeeTFBQXDejT4xYP8JggG+CaTWfTQBARj1Pga1sGNg+8B7csfDupipJ0lKOQQYjyvOHGAJKt6kH

Correct. This statement is false. Researchers study a sample and then generalize to a population — not the other way around.

Incorrect. This statement is false. Researchers study a sample and then generalize to a population — not the other way around.

Assessment: Check Your Understanding

Question 1.6

z7VRMoiF6kGw2IMkrbyFe3iOjzKhp6AnF73bAt/6HWJVRK1vgOFrnx94i6J9ekJ9zBZGLzBtzGVdWqsM5LCoTw2FttLt6scZDb573DvYwCRPOiJ08vojEl/D2X1qRFH/1o5jBJl6XKZBb4DUxJ+nJRmg/gg03rOhFYXA08MJv0aa5zkk2vbfeC4fIqiGskkNKVNZxd30QA75uGDHwipCB7NKluYMHcB5OcCWBp8HQGkQtFej

Correct. This statement is true. Double-blind procedures minimize the risk that the expectations of either the participants or the researchers will influence the outcome.

Incorrect. This statement is true. Double-blind procedures minimize the risk that the expectations of either the participants or the researchers will influence the outcome.

Assessment: Check Your Understanding

Students in a lab coat in a lab having a discussion

Question 1.7

VYBhM5U1/ONGaVCki7hM6l1ewrzUHdlTLdt1W5bqonrjPbLIOrMzKwuKT+cYpXNwWyRzpkAIi2Y9FS2pBwFe7EEE8OaxguPVYyjATCWbVdCqihyeOYMhHRTzX1rYFCn9nQjJ+GGga1E2l1lgXEDcrd3dd16e+Y0ZDL1wKpA+kCDtLoeSzFz8Kw==

Correct. This statement is false. A basic principle of research is that correlation does not imply causation. Only an experiment in which all relevant factors have been controlled and one variable has been manipulated can provide evidence of cause and effect.

Incorrect. This statement is false. A basic principle of research is that correlation does not imply causation. Only an experiment in which all relevant factors have been controlled and one variable has been manipulated can provide evidence of cause and effect.

Congratulations! You have completed this activity. Total Score: x out of x points (x%) You have received a provisional score for your essay answers, which have been submitted to your instructor.