763

In this interview from July 30, 2014, with Christian Rudder, one of the co-founders of the dating site OkCupid and author of Dataclysm: Who We Are (When We Think No One Is Looking) (2014), Todd Zwillich, Washington correspondent for The Takeaway, a Public Radio International program of news and commentary, asks probing questions about experiments OkCupid ran on users without their foreknowledge or permission. Zwillich has been a journalist in Washington for fifteen years, working first in print; he has been in public radio since 2006. In contrast to the first three selections in this chapter, this piece focuses on industry, not government, and it concerns exactly the sorts of issues raised by boyd and Crawford in the previous selection, Six Provocations for Big Data.

If you listen to this interview (at http://bit.ly/1HK3VHb), you’ll be reminded that speakers have resources to communicate that differ in important ways from those writers have and that spoken and written language — at least in academic contexts — can be quite distinct. For example, you’ll notice that Rudder frequently uses intonation in interesting ways: his pitch will often go up when he is emphasizing new or important information. A writer would have to find other ways to show this emphasis. Similarly, you’ll hear Rudder using hedges like “kind of” or “sort of.” Linguists have demonstrated that speakers often use such expressions to minimize the force of what follows, that is, to try to lessen its impact. Pay attention to where they occur; as you’ll see, it’s no surprise that the title of this interview in the original URL (rather than the shortened one given above) is “are-okcupids-experiments-users-ethical.” You’ll get to be the judge of whether they are or aren’t as we continue exploring issues of privacy and surveillance.

It’s Not OK Cupid: Co-Founder Defends User Experiments

TODD ZWILLICH AND CHRISTIAN RUDDER

Note on the transcript: We’ve tried to stick as close as possible to the interview in transcribing this interaction. In other words, we haven’t cleaned up the um’s, the repeated words, or the occasions when a speaker corrects himself to encourage you to think about the complex relationship between spoken and written language. In the transcript, overlap refers to two people talking at once; as you’ll hear, neither speaker seems to treat the overlap that occurs as interruption.

764

TODD ZWILLICH: Dating online can feel like a shot in the dark when you’re looking for love, but for those in the business of selling matchmaking services online, there’s not a whole lot of mystery to it. It’s not about magic; it’s all about data and algorithmic calculations. But this week, online dating giant OkCupid came under a little bit of fire for taking the scientific approach to match-making to the next level when it revealed it had conducted experiments on its own users. Ricardo Sanchez sent us this tweet: “Not cool,” he writes. “The participants were drafted without their permission.”

And it’s not the first time OkCupid has done research like this on the behavior of its members. In the past, the site’s team has used its trove of data to identify things like whether you should use a smiley or a flirty face in your profile to get more response, how much skin to show in a photo, even what kind of camera takes the best headshot. But what makes the online-dating company’s latest research a little different is that it actually involved manipulating the information that users received about potential matches. Did they learn anything by doing this? Christian Rudder, co-founder of OkCupid, joins me now. He’s also author of the forthcoming book Dataclysm. Christian, welcome.

CHRISTIAN RUDDER: Thank you, glad to be here.

TZ: Everybody heard about Facebook and . . .

765

CR: Right. (CR overlaps TZ in a low voice.)

TZ: Everybody heard about the experiment that Facebook was conducting on its users. Explain how your manipulations of your users are different.

CR: You know, we’re a dating site, so we took groups of people who, uh, we thought were bad matches and told them they were good — for each other, and we took a group of people who were people we thought would be good for each other, and told them that we were bad, that they were bad — uh, for each other. And the goal was kind of to test our estimation of how compatible they were against like the kind of like, uh, null hypothesis, you know, from the scientific method, essentially. We had our method, and then we had essentially nonsense — a random recommendation — and we wanted to see how we did against that.

TZ: And that’s all you did? What else did you do?

CR: Um, the other stuff is fairly straightforward Web design stuff. Like, we showed some profiles, um, with the picture and the text that the user entered, and then sometimes just with the picture, and kind of tracked the difference in how people judged that particular person. It came out about 90 percent the same, so your picture is 90 percent of the, uh, of that first impression.

TZ: I think that surprises nobody.

CR: No, of course not! Yeah. (laughs) Um, and then, one was kind of an implied experiment: There was a day in 2013 when we turned off all the photos on the site, for, uh, about seven hours and kind of, uh, let users use OkCupid without being able to see who they were talking to. And then so if you compare that against a normal day, uh, you see that the site is actually a lot healthier in terms of how people reply to messages, and kind of like length and sort of velocity of conversations is actually a lot better when there’s no photos, except that people just don’t want to use a dating site with no photos, so we would be a better and infinitely smaller site.

TZ: Getting back to the first thing that you mentioned . . .

CR: Uh-hm.

TZ: The power of suggestion: telling people that they weren’t good matches when maybe your algorithm thought that they were, or vice versa.

CR: Exactly.

TZ: This is the part that a lot of people had a problem with, primarily, I think, because somebody might have thought, “Somebody who might have been a good match for me, I might have missed them because I was told they weren’t, and I’m not, uh, you know, I might have missed my, ‘the one,’ ” I guess.

766

CR: Sure, yeah, I definitely understand people’s misgiving. But first of all, like every user who sent a message under this kind of test “match-percentage” was notified after the fact of the experiment. You know, say, if we were matched together: “Hey, Christian, you know, your match percentage with Todd was misstated because of a test. Uh, the correct percentage is 30” or whatever. Also, it’s — we see it as just a part of the continuum of experiments that we are always doing on the site. I mean, just for match-percentage alone, we might change or reweight a variable, and inevitably, people are going to see two different numbers. It’s a continuum. It’s kind of at the end of it, but it’s still part of . . . normal business.

TZ: Um, I’m not quite sure I understand how this manipulation of users, which I’m sure was allowed under your user agreement . . .

CR: Right.

TZ: I’m not quite sure I understand — what that tells you about your own algorithm. It certainly tells you the power of suggestion, which is, if you tell somebody that somebody’s no good for them, they’re likely to believe it. That I get. But what did you learn about your own business? Because it’s, uh, not obvious to me.

767

CR: Well, I think the reason it’s not obvious is that we found that the algorithm actually kind of worked. But imagine it in the other way around; like, what if we had tested our best guess about compatibility against kind of a nonsense measure, which is what we did, and the nonsense one was better, that our best guess was actually worse than random, you know. Then obviously, we got to go back to the drawing board, and our whole reason for existence is called into question at that point. ’Cause I mean, the other option, if we just did nothing, you know, we obviously misled a small number of users as part of this experiment. If we don’t test it, the possibility that we’re misleading the four million people who come to the site every month, every day, uh, that’s the other kind of thing that hangs in the balance.

TZ: So we should be clear: When people write stories about what OkCupid did, or even what Facebook did, social scientists are commenting on how these types of experiments are important. You didn’t do this for any redeeming social value whatsoever; you did this for your own purposes and to test your own algorithms and your own model.

CR: Yeah, for sure. This, these particular experiments were kind of part of the normal course of our business. Unlike Facebook’s, which, I mean, um, I’m a Facebook user like anyone else — I don’t own shares or anything like this — and the one thing that has kind of gone under the radar is that they have put together like a world-class scholarship academic team to do, basically, public research for the public good.

TZ: Right. But yours was different.

CR: No, no, ours was definitely different. (Overlaps with TZ)

TZ: This, this was not for the public good.

CR: No, no. (Continued overlap with TZ)

TZ: This was for your good.

CR: That’s right.

TZ: And people need to understand that.

CR: Of course, yeah, yeah. And, and, look, if there’s kind of a public-facing part of what we did, it’s to point out to every person that uses the Internet, that every site does these experiments. It’s not just OkCupid. Google, you know, their search algorithm gets better because they test version A against version B, and, you know, it sounds a little bit trivial or boring, but that means that for the people in the various groups, they kind of like, truth in the world is reordered by those search results, you know. And so, all of these things are always happening all the time.

768

TZ: Let’s talk about the trust factor a little bit.

CR: Sure.

TZ: Um, it’s probably clear, again, that in your user agreement that somebody clicked, buried in there somewhere, it says, you can do this, . . .

CR: Sure.

TZ: So it’s not like . . .

CR: You bet.

TZ: There. Right. It’s not like there wasn’t disclosure. Not that anybody reads those things.

CR: Yeah. Uh.

TZ: That’s their own problem. But what about the trust factor? It is the case that, uh, when people are on a site like yours, they are trying to meet someone, whether it’s just for a casual date or to find the one for them.

CR: Absolutely.

TZ: If they feel that not everything is necessarily straightforward, and that the people they are seeing or meeting or talking to has something else behind it besides love (maybe), um, how do you answer them?

CR: Well, I would say that people come to us expecting us to do a good job finding them sex, a date, a marriage, whatever: uh, it’s totally right. And, uh, they trust us to do that, and part of us delivering on that trust is actually making sure that what we do, works. Like, so, we try to prove it. And of course, that involves — an experiment. Um, and we take a small group of users and test any idea — you know, this is one particular test, but we test it. And then so those small groups of users bear a sort of cost for keeping the site good for everyone else and for themselves. As soon as we end the experiment, we make the site better. So, it’s part of delivering on the promise rather than betraying it.

TZ: Well, Christian Rudder, uh, let’s sum it up a bit. After all this, what did you learn?

CR: Well, um, you know, the company learned a few banal things: you know, pictures matter and, and the power of suggestion works, like you said. But I guess for us institutionally, um, or for me personally, I learned there’s a long way to go until the public sort of understands the way Web sites work, and I think, obviously, we, we sensationalized it a little bit, but I think it’s a discussion that needs to continue.

769

TZ: And that maybe everybody, every time they go on a Web site, should have it in their mind that they are being watched.

CR: Absolutely. Not only being watched, but, you know, there’s someone that’s looking at that exact same Web page that’s seeing something different.

TZ: Christian Rudder is the co-founder and president of OkCupid. He’s the author of the forthcoming book Dataclysm.