18 Behavioral and Experimental Economics

18.3 Testing Economic Theories with Data: Experimental Economics

The unorthodox results emanating from behavioral economics research in recent years have placed even more significance on the issue of how economists can test which models are correct. All economic models generate predictions that, in principle, can be tested and compared. Some predictions are fairly basic, such as the prediction from demand theory that demand curves slope downward because people buy less of a good when its price increases. This sounds like just about the easiest thing to test in the world. But, real-world complications often get in the way of testing even the most basic predictions from the models. Even if you had data on the prices and quantities of, say, chicken nuggets across different cities, you couldn’t really say that supply-and-demand theory predicts that the markets with higher prices will have lower quantities demanded. The theory predicts that holding all else equal, when prices are higher, consumers will want less of a good. A comparison of two markets only sheds light on this question if all else is equal between the two markets, and in the real world, that’s a big “if.”

econometrics

Field that develops and uses mathematical and statistical techniques to test economic theory.

To get around the problem of messiness in the real world, economists have developed a large number of data, statistical, and analytical techniques to help test economic theories—techniques such as how to use shifts in a market’s supply curve to statistically trace out its associated demand curve and so on. These techniques are part of the field of econometrics, the use of mathematics and statistics to describe numerical relationships between key economic variables. Even the best econometric techniques still rely on some strong assumptions to get around the potential complications of the real world, to say nothing of the messiness inherent in most data.

704

experimental economics

Branch of economics that relies on experiments to illuminate economic behavior.

Dissatisfied with empirical data methods like econometrics, some economists have turned to experimental economics, in which they can hold everything else equal and directly test economic hypotheses by conducting explicit experiments.

Lab Experiments

lab experiment

Test of an economic theory in a laboratory setting.

Lab experiments allow an economist to control all aspects of a test and explain precisely what was done. Typically, the experimenter has groups of subjects take part in the same activity, but with one element of the activity altered for one group. For instance, to test if demand curves slope downward, an experimenter might bring students into a lab, give them each $30, and then randomly assign each of them to one of three groups. Each group would then be given the chance to buy coffee mugs embossed with their school’s mascot, but one group would face a price of $3 per mug, the second $5 per mug, and the third $10 per mug. Everything but the price is the same in this experiment, so a comparison of the quantities purchased by the groups traces out a demand curve and allows the experimenter to test whether it slopes downward (which, fortunately for economics, the experiments overwhelmingly verify).

Experiments that randomly assign participants to a treatment group allow researchers to test for influences such as mental accounting errors, overconfidence, the endowment effect, and the other biases we talked about earlier in the chapter.

As helpful as economic laboratory experiments are, they are not without shortcomings or controversy. Economic experiments suffer from some important weaknesses that are missing in most scientific applications of the experimental method. First, unlike chemical compounds or lab mice, humans know that when they participate in an experiment, their behavior is being watched. Not surprisingly, critics believe that participants in economic experiments tend to act differently because they are being observed. One factor that tends to distort participants’ behavior is that subjects in lab experiments exhibit a strong tendency to act in a manner that they believe will please or impress the experimenter or the other participants. For example, college students repeatedly make choices in experiments that are more socially desirable, cooperative, or morally “proper” and shy away from things that make them look selfish or that simply maximize their personal gains in the experiment. This distortion leads some critics of such economic experiments to question whether lab experiments really tell us about people’s behavior in actual markets and whether the experiments truly show that individuals make systematic mistakes in their decision making.

A second issue with the typical lab experiment is that the stakes of the experiment are often much different than in real life. It doesn’t take much money to entice undergraduates to participate in experiments (which is fortunate because research budgets are usually small). Consequently, the stakes at risk in the typical economics lab experiment are low. Low stakes make it less costly for participants to act in ways that are against their own self-interest. Further, sometimes experiments involve really complex or tedious tasks that hardly seem worth doing for so little, whereas in the real world, if you could make a million dollars completing some tedious calculations, you would be much more likely to complete them.

A third issue with lab experiments is that students are often asked to do tasks in the lab that are foreign to them, such as participating in auctions with randomized and strange rules like requiring the top two bidders to pay but with only the highest bidder receiving the prize. One might expect students to do poorly at this task compared to experts who have studied and practiced such tasks, or even people who can do a Web search to learn how to effectively participate in an auction or who can call a more auction-savvy friend for advice.

705

Finally, although an experimenter can control the experiment, he cannot control the personal baggage brought by the participants. For instance, experimenters often design experiments as one-shot games so that the choice made by a lab subject should be dictated exclusively by the direct payoff and not by any future considerations. But in real life, people do not seem to be able to assume away their cultural norms. The Freakonomics box on the next page describes how differently people from varying cultures play the same lab games, highlighting the difficulties of designing experiments that truly hold everything else equal.

Despite the potential pitfalls of lab experiments, they remain a useful economic tool. After all, the analysis of non-experimental data suffers from many shortcomings as well. But, be wary of thinking that what happens in the lab will automatically happen in real markets. Instead, when faced with data generated from a lab experiment, consider the various biases we’ve discussed and contemplate how they might distort any interpretation of the lab results.

Natural Experiments and Field Experiments

Responding to some of the criticisms of lab experiments, economists have tried using experiments in the real world instead of the lab to test their theories, in the hope that they can merge some of the control features of lab experiments with the real stakes and context of the marketplace.

natural experiment

A randomization or near-randomization that arises by happenstance.

One type of real-world experiment is known as a natural experiment. A natural experiment is a situation in which, by chance, something happens that allows the researcher to learn about an economic question of interest. When the United States passed the Clean Air Act and its amendments, for example, it set up pollution thresholds for counties in the United States. If a county had a pollution level above the threshold, it faced stringent anti-pollution requirements. If the county fell below the threshold, even by a small amount, the anti-pollution requirements did not apply. For counties just a tiny bit above and just a tiny bit below the threshold, the Clean Air Act provided a natural experiment on the impact of environmental regulation and researchers documented effects on pollution, industrial activity, health, infant mortality, and so on. An obvious problem with natural experiments is that they are hard to find. A second weakness is that it is not always straightforward to generalize from a specific natural experiment to other settings of economic interest.

field experiment

Research method in which randomizations are carried out in real-world settings.

The other type of real-world experiment is the field experiment. A field experiment uses randomization, just as in the lab, but does it in real-world settings, ideally using research subjects who are making decisions as part of their everyday lives, unaware that they are part of an experiment. Consequently, it is possible to make strong inferences about behavior from field experiments without having to worry about how the artificiality of the lab might be distorting the results. The study we discussed in the previous section in which researchers sent volunteers to people’s homes to ask for donations is a classic type of field experiment.

Clever firms have long engaged in field experiments, and you have unknowingly been part of these experiments throughout your life. Often when you receive a catalog or a credit card offer in the mail, the firm sending it is actually conducting a field experiment, by changing around the order in which the goods are presented for some customers, altering the descriptions of goods to see which lead to more sales, and perhaps playing with the prices offered to different recipients. Economists have increasingly turned to field experiments to answer questions as diverse as whether paying high school students for good grades leads to better outcomes, the best way to incentivize people to lose weight, and which movies will be box-office hits.

706

FREAKONOMICS

Going to the Ends of the World (Literally) to Test Economic Theory

The Lamalera cooperate in whaling and in games, while the Hadza, a hunting and gathering tribe, display more self-interest and less cooperation in games.

The great majority of economics lab experiments are done with college students. This is a perfect example of the principles of economics at work: Students are plentiful and willing to participate in experiments for low wages. This makes it cheap and easy to run experiments on campus, and economic researchers have responded by running lots of these experiments. When researchers have looked beyond students to other subject pools like CEOs, professional soccer players, and dealers at flea markets, the results have generally been similar to those obtained from students. It is also the case that Americans and Europeans tend to make similar choices in many lab experiments.

The fact that lab results are robust (i.e., they hold up) across different groups of subjects is good news for lab experiments because it means that what we find in one particular experiment is more likely to generalize to other settings. Researchers can conduct experiments using students and be reasonably confident that, if they replicate the study using stay-at-home moms or movie stars, the results will be similar.

One characteristic that is shared by virtually all of the subjects who have been part of experiments, however, is their culture. Americans, whether they are college students, CEOs, or pro athletes, have grown up with a shared culture. European culture differs from American culture, but both of them are closer to each other than to many non-Western cultures. Do the lab experiments on economic behavior yield similar results when conducted in other places?

^*Joseph Henrich, Robert Boyd, Samuel Bowles, Colin Camerer, Ernst Fehr, Herbert Gintis, and Richard McElreath, “In Search of Homo Economicus: Behavioral Experiments in 15 Small-Scale Societies,” American Economic Review Papers and Proceedings 91, no. 2 (May 2001): 73–78.

A collaborative team of economists and anthropologists set out to answer this question.^* The team gathered together anthropologists who had spent their careers living with and studying indigenous societies in many different parts of the world. These were small-scale societies that have virtually no contact with the modern world. These groups included hunter-gatherers in Tanzania, nomadic herders in Mongolia, and whale hunters in Indonesia. The anthropologists learned how to carry out these experimental games and then scattered to the far reaches of the world to conduct them. The results obtained from these different groups were all over the map. Some groups behaved more like Homo economicus than Westerners; others deviated even further from economic predictions than had American college students. Members of some cultures exhibited remarkable altruism, whereas others were the epitome of selfishness.

And yet, from this wild mish-mash of findings, a simple yet profound pattern emerged: How people played these experimental games was systematically related to the norms that existed in their cultures. For instance, the Lamalera of east Indonesia subsist by hunting whales, an activity that requires high degrees of cooperation across large numbers of hunters to succeed. Not surprisingly, the Lamalera played the games in a cooperative way, with players frequently passing up chances to act in a way that would benefit them to the detriment of the other players.

In contrast, the Hadza, hunters and gatherers in Tanzania, were very self-interested and uncooperative toward one another. It is said that the Hadza abandoned traditional garb for Western-style blue jeans because the deep pockets of the jeans allowed them to more effectively hide the spoils of their hunting and gathering from the prying eyes of their neighbors. The Hadza in the lab experiments did not cooperate much with the other players and instead looked out for themselves.

Perhaps the most remarkable example of culture permeating the lab came from the Au and Gnau peoples of Papua New Guinea. When playing a lab game that involved splitting a pie between themselves and another player, the Au and Gnau often offered the bigger slice of the pie to their opponent (something that virtually never happens with American students), and the opponent frequently refused to accept this bigger slice (again, not likely with the American students). It turns out that the Au and Gnau are competitive gift-giving societies in which accepting a gift today obligates the recipient to provide a more generous gift to the giver in the future. Thus, a gift is a mixed blessing in this culture. Even though the lab experiment was explicitly described as a one-shot game and played anonymously, so there was no chance that the recipient of the big slice of the experimental pie could be obligated to make a more generous repayment in the future, the Au and Gnau apparently brought such powerful cultural norms into the lab that they participated in the experiment as if it were a typical, real-life gift exchange.

These experiments teach us an important lesson about lab experiments. Although it might seem as if the experimenter gets to choose the game that is played and the context of the experiment, in practice, experimental subjects bring their own contexts with them when they enter the lab. In particular, when people enter a lab, they are not able to instantly abandon life lessons about what society expects and the rules of thumb that serve them well in the real world.

707