Well, I Guess I’m an Idiot, Too.
Jason Kuznicki on Feb 27th 2008
It seems like a flawed experiment to me. From a New York Times article by John Tierney:
In a series of experiments, hundreds of students could not bear to let their options vanish, even though it was obviously a dumb strategy…
The experiments involved a game that eliminated the excuses we usually have for refusing to let go. In the real world, we can always tell ourselves that it’s good to keep options open.
You don’t even know how a camera’s burst-mode flash works, but you persuade yourself to pay for the extra feature just in case. You no longer have anything in common with someone who keeps calling you, but you hate to just zap the relationship.
Your child is exhausted from after-school soccer, ballet and Chinese lessons, but you won’t let her drop the piano lessons. They could come in handy! And who knows? Maybe they will.
In the M.I.T. experiments, the students should have known better. They played a computer game that paid real cash to look for money behind three doors on the screen. (You can play it yourself, without pay, at tierneylab.blogs.nytimes.com.) After they opened a door by clicking on it, each subsequent click earned a little money, with the sum varying each time.
As each player went through the 100 allotted clicks, he could switch rooms to search for higher payoffs, but each switch used up a click to open the new door. The best strategy was to quickly check out the three rooms and settle in the one with the highest rewards.
Even after students got the hang of the game by practicing it, they were flummoxed when a new visual feature was introduced. If they stayed out of any room, its door would start shrinking and eventually disappear.
They should have ignored those disappearing doors, but the students couldn’t. They wasted so many clicks rushing back to reopen doors that their earnings dropped 15 percent. Even when the penalties for switching grew stiffer — besides losing a click, the players had to pay a cash fee — the students kept losing money by frantically keeping all their doors open.
Yes, yes, I switched doors a few times the first time through (a total of eleven switches, to be honest). But it wasn’t because I was trying to keep my options open (not, in itself, obviously irrational, but I digress). It was simply that I was looking for a pattern, and this is a quite rational thing to do. Life is full of patterns.
I thought that finding a pattern could pay off really big, big enough to make up for the lost opportunities. It seemed reasonable to me that the numbers behind the doors would not be random. Perhaps that one of the doors would start to pay more if I clicked in the proper sequence. Or maybe there was some pattern to be found in the previous numbers in the reward totals. Or… something.
It seems to me that the experimenters are neglecting the extraordinarily small knowledge base that each player starts with. It’s hardly fair to conclude that ignorant people must also be irrational whenever they don’t behave optimally: For rationality to manifest, it has to have at least some data to work with.
In the end, the numbers just seemed random to me. I decided one of two hypotheses must be correct: a) there was no pattern or b) I couldn’t find it. After that I just clicked like mad on one door and racked up the points. But I guess because I didn’t choose this strategy from the very first click, I’m irrational.
How silly: If there had been a pattern, and if I hadn’t looked for it, I could easily see some other behavioral economist calling me irrational for not seeking out opportunities in the market. Or somesuch.
The underlying problem is not irrationality at all — only an acute lack of data, coupled with an inability to judge the worth of each marginal datum: If I click once more, I’ll gain some information (of how much value?). But if I don’t click once more, I’ll gain some points (of how much value? and what am I giving up in exchange?).
No one can answer these questions before they play. What if, on the third change of doors, the program were to reward the player with a million points, while if he never changes, he just gets a few dozen? Who would be the “irrational” one then? What if you got the most points for “rescuing” a closing door? What if it would have helped to change doors when the last reward total you got was even, or divisible by five, or prime? These hypotheses could be easily programmed into this experiment, along with an infinity of others. As a kid, I played computer math games with some of these features.
I’m not alone in thinking this way. Here’s a comment from John Tierney’s blog post on the experiment:
The first time I played, it took a fair amount of door switching (I chose 4 clicks on each door as optimal) to determine that the payoffs behind each door were approximately similar (and remained approximately similar), and thus minimizing switching was the best strategy.
The second time I played, it took less switching to determine that this payoff structure remained, so I switched less — but that was a path-dependent result not connected to any irrationality about the emotional loss of disappearing choices.
Yep. If anything, we didn’t do enough experimenting. There may still be a pattern, for all I know. Or maybe not.
In any case, I know the real point of the experiment: Paternalism. The point is to establish that people are irrational (as defined by men in lab coats). Then the men in lab coats get to make all the important decisions for us. The point of this experiment — if one can even call it that — is to convince everyone, from ordinary joes to policy wonks to legislators — that it’s perfectly okay to take away other people’s choices. My colleague Will Wilkinson reaches a similar conclusion here, with several direct quotes in support.
Yes, we’ve discussed recently some of the reasons why paternalism isn’t a rational response to irrationality: If these experiments prove anything, they show that the people in lab coats are irrational just like the people out of them.
Yet irrationality in the market is at least voluntary. It’s noncoercive. It also produces incentives to find more rational behavior, and it can be curbed with a secondary market in decisionmaking skills (lawyers, financial planners, and sommeliers are just three examples, but nearly all white collar professions have some element of this to them).
Legislated irrationality — the kind belonging to friendly gentlemen in lab coats and not-so-friendly ones in jackboots — well, that kind lasts a lot longer. It doesn’t provide the incentives for its own demise. And it can’t be kept in check by paying smart people to help us figure things out.
So even if we grant the premise that “people are irrational,” the state isn’t necessarily the answer. To return to the experiment at hand, ultimately what it shows is how weak the support is for the premise itself.
Filed in The Biosphere
On the online version, the directions read “you will receive a certain number of points drawn from a given distribution. Each door has a different distribution.” As I understand statistical terminology, that is implying that each door has a fixed probability distribution, and the rewards are independent samples from that distribution. If they wanted to permit the sort of more complex rewards you are describing, I think they would have to say that the points are “computed according to a certain algorithm” or something like that.
Of course, it is not clear from the NYTimes description of the story that this aspect of the directions was made clear to the subjects.
Very interesting, Jason—and also where you took it.
Barack Obama and Howie Mandel fit in here somewheres, although I know not where.
Tom — See Will Wilkinson for where Barack Obama fits in. I’m not sure if you were kidding, but I’m not.
DavidS — I suppose you’re right, though it did not occur to me at the time. Yet another possibility, fully consistent with the more rigorous interpretation of the instructions, is that door 1 has a normal distribution, door 2 has a uniform distribution, and door 3 has a Zipf distribution, with numbers ranked from lowest to highest.
Each is consistent with the more rigorous meaning of “distribution.” Yet door 3 would be preferable, and it would not immediately be clear that this was the case without some amount of switching.
The real test here, though, was whether you switched more times when faced with the possibility of losing doors. Whatever internal algorithm you followed for optimizing your score when the doors didn’t shrink, you ought to have followed that same algorithm when the doors did shrink. Jason’s points about the possibility that saving doors actually gets you more points notwithstanding, it would be irrational to change your strategy if it seemed successful the first time.
I think it is certainly worthwhile to point out that we shouldn’t legislate rational economic decision-making just because individuals may be ignorant or irrational about certain aspect of their economic lives. But, the value to be taken from this is not necessarily that the state would do better. The message I take from it is that there are some irrational behaviors to which humans can be susceptible, and learning and avoiding these behaviors can make you a more effective economic actor.
Each of the doors has a different distribution one is tight in the middle, like roughly 40-60 one was medium 30-70, and one was very variable 20-80. I don’t think it matters what door you pick the average score will be the same (but with more chance of both low or high scores in 20-80). What matters is the number of times you switch doors in each run. I had shrinking doors first and actually switched more the second time without shrinking doors. I guess because I figured I could afford to look at the distributions more while in the first run I thought I better pick one and stick with it.
The real test here, though, was whether you switched more times when faced with the possibility of losing doors.
I may never know the answer: I faced the non-shrinking-doors test second, and by then I was pretty sure I’d figured it out.
Whatever internal algorithm you followed for optimizing your score when the doors didn’t shrink, you ought to have followed that same algorithm when the doors did shrink.
Not necessarily. Chessplayers use different heuristics when they are in time pressure — They may choose a strategy where each of the moves are obvious, for example, rather than one where the moves may be better, but will take longer to find. There’s possibly quite a lot to say here about how we think and how that changes depending on time constraints, and there is much empirical research to do.
Jason’s points about the possibility that saving doors actually gets you more points notwithstanding, it would be irrational to change your strategy if it seemed successful the first time.
Ah, but how do you measure success when you have exactly one datum? It’s not like you can compare your score to anything else at all.
I think it is certainly worthwhile to point out that we shouldn’t legislate rational economic decision-making just because individuals may be ignorant or irrational about certain aspect of their economic lives. But, the value to be taken from this is not necessarily that the state would do better. The message I take from it is that there are some irrational behaviors to which humans can be susceptible, and learning and avoiding these behaviors can make you a more effective economic actor.
Definitely agreed, but see Will Wilkinson’s comments, too. Our take-home messages may vary.
Wait, some get shrinking doors first, and some second? Ah, that makes the results mean something different. And what they mean depends heavily upon how the populations who got shrinking doors first behaved differently from those who got it second.
I’ve run a fair number of these types of experiments, and I have run into some pretty misguided (possibly irrational) behavior.
But in this case I wouldn’t be surprised if the additional switching comes from the fact that it’s harder to keep track of the draws from a given door when you have to worry about how many draws you can take before the other doors disappear. I knew the optimal strategy already (from the article and the instructions), and I switched 3 times in the non-shrinking treatment and 5 times in the shrinking treatment. That’s a 67% increase.
Oh, and allow me to say what everyone else is thinking: “There was significant shrinkage.”