The Drunkard’s Walk

August 14, 2014

We’re naturally poor judges of probability. Correcting our flawed intuitions about chance would leave us feeling less guilty, more grateful, and generally happier.

The Drunkard’s Walk is a well-exampled jaunt through the many ways we tend to misjudge probabilities. While the premise may be somewhat discouraging, the author’s goal is actually to make us happier. By correcting the ways in which we underestimate the role of chance in our fates, we should feel more gratitude when things go our way, less frustration when they don’t, and more humanity toward our fellow recipients of bad luck. Ultimately,

A lot of what happens to us—success in our careers, in our investments, and in our life decisions, both major and minor—is as much the result of random factors as the result of skill, preparedness, and hard work.

Of course, we don’t like to think about this. It’s much more gratifying to take full credit when things go our way. And while we’re quick to blame our stumbles on mere bad luck, we rarely extend others that same courtesy. (This tendency is so central to our outlook that it’s called The Fundamental Attribution Error.) But what’s at stake isn’t just out pride but our sanity. In the 1960s, social psychologist Melvin Lerner was investigating the cause of society’s negative attitudes toward the poor, and:

Realizing that “few people would engage in extended activity if they believed that there were a random connection between what they did and the rewards they received,” Lerner concluded that “for the sake of their own sanity,” people overestimate the degree to which ability can be inferred from success.

There must be some middle ground between denying the role of chance in our lives and resigning ourselves wholly to it. But to get there, we must first understand how randomness and probability work, a task for which our intuitions leave us gravely unprepared.

We Don’t Believe Randomness is Really Random

Curiously, we tend to think of randomness as far more evenly distributed than it really is. Were I to flip a coin ten times, the odds of it coming up heads every time are the same as the odds of the particular sequence tails-heads-heads-tails-heads-tails-tails-tails-heads-tails, and yet the first seems somehow “less random”. (There is good reason for this. Humans’ early survival depended on noticing patterns, so our brains are wired to overestimate their importance.) But outside our heads, true randomness should exhibit this kind of streakiness.

Mlodinow proposes an example of ten perfectly incompetent Hollywood executives who simply flip a coin when deciding which films to greenlight. Given the randomness at play, our natural inclination is to assume all 10 will be pretty close to 50% successful, when in fact:

…if each of 10 Hollywood executives tosses 10 coins, although each has an equal chance of being the winner or the loser, in the end there will be winners and losers. In this example, the chances are 2 out of 3 that at least 1 of the executives will score 8 or more heads or tails.

In effect,

Such issues are not discussed in corporate boardrooms, in Hollywood, or elsewhere, and so the typical patterns of randomness—apparent hot or cold streaks or the bunching of data into clusters—are routinely misinterpreted and, worse, acted on as if they represented a new trend.

And so,

When we look at extraordinary accomplishments in sports—or elsewhere—we should keep in mind that extraordinary events can happen without extraordinary causes. Random events often look like nonrandom events, and in interpreting human affairs we must take care not to confuse the two.

We Ignore Regression to the Mean

“Regression to the mean” refers to the tendency of outlier behavior naturally to correct itself. When things go exceptionally well, they’ll tend to go back to being average, and likewise when they go exceptionally badly. This can lead to a perverse pattern when trying to change outcomes. It can make us think the positive things we do to encourage good outcomes are useless (as good results regress back to the average) and that negative measures to discourage bad outcomes are effective, (as bad results regress back to the average), when in fact neither has a better-than-chance effect. This situation can be especially toxic with regards to punishment, as a story from psychologist Daniel Kahneman’s classroom illustrates:

In this way an apparent pattern would emerge: student performs well, praise does no good; student performs poorly, instructor compares student to lower primate at high volume, student improves. The instructors in Kahneman’s class had concluded from such experiences that their screaming was a powerful educational tool. In reality it made no difference at all.

More precisely, it made no difference at all to the overall outcome. The teachers’ behavior most certainly made a difference, a cruel and useless one, to how students felt.

We Equate More Detail with Higher Probability

If the details we are given fit our mental picture of something, then the more details in a scenario, the more real it seems and hence the more probable we consider it to be—even though any act of adding less-than-certain details to a conjecture makes the conjecture less probable. [emphasis mine]

Even lawyers, who may well exploit this cognitive bias in order to convince a jury of the likelihood of a client’s guilt or innocence, are themselves susceptible to it:

Researchers found that lawyers assign higher probabilities to contingencies that are described in greater detail.

As are doctors, at our peril:

Kahneman and Tversky found that 91 percent of the doctors believed a clot was less likely to cause just a rare symptom than it was to cause a combination of the rare symptom and a common one.

Furthermore, the skewing effect of additional, irrelevant detail is enhanced the more time passes before we act on the information, presumably because sound reasoning fades from memory more quickly than a vivid mental picture.

When the jurors were asked to produce guilt/innocence ratings, the side with the more vivid presentation of the evidence always prevailed, and the effect was enhanced when there was a forty-eight-hour delay before rendering the verdict (presumably because the recall gap was even greater).

We Gravely Misinterpret Low Probabilities

In an especially troubling story, the author was denied insurance as a result of a blood screening. Upon calling his doctor, he learned that he had tested positive for HIV, and that the test’s false-positive rate is only 1-in-1000. That means that there’s a 999-in-1000 chance he has HIV, right? Wrong.

The HIV test produced a positive result when the blood was not infected with the AIDS virus in only 1 in 1,000 blood samples. That might sound like the same message he passed on, but it wasn’t. My doctor had confused the chances that I would test positive if I was not HIV-positive with the chances that I would not be HIV-positive if I tested positive.

Even after reading that last sentence several times, I still find it difficult to intuit the difference. Thankfully, Mlodinow explains by asking us to consider a sample space of 10,000 heterosexual, non-IV-drug-abusing, white male Americans (his cohort). According to 1989 statistics from the CDC, about 1 out of those 10,000 will test positive because they have HIV. His doctor was right about the false-positive rate being 1 in 1,000, but that needs to be considered in comparison to the true-positive rate of only 1 in 10,000:

Now let’s prune the sample space to include only those who tested positive. We end up with 10 people who are false positives and 1 true positive. In other words, only 1 in 11 people who test positive are really infected with HIV. My doctor told me that the probability that the test was wrong—and I was in fact healthy—was 1 in 1,000. He should have said, “Don’t worry, the chances are better than 10 out of 11 that you are not infected.”

A false-positive rate of 1 in 1000 does not mean that there’s a 999-in-1000 chance the test result is correct. That chance can only be arrived at by comparing the false-positive rate with the true-positive rate. This is why it’s important in such cases to know if you’re in a “high-risk group,” since the true-positive rate will be higher, and thus the test will be relative more reliable:

Let’s suppose now that I had been homosexual and tested positive. Assume that in the male gay community the chance of infection among those being tested in 1989 was about 1 percent. That means that in the results of 10,000 tests, we would find not 1 (as before), but 100 true positives to go with the 10 false positives. So in this case the chances that a positive test meant I was infected would have been 10 out of 11. That’s why, when assessing test results, it is good to know whether you are in a high-risk group

This “high risk” factor plays into another of Mlodinow’s examines, this one from the O.J. Simpson trial. The prosecution argued that Simpson’s acknowledged history of abusing his then-wife Nicole made it statistically likely that he murdered her. The defense in turn cited statistics that only 1 in 2500 battered women are killed by their abusers. While true, this is merely statistical sleight-of-hand, and entirely irrelevant:

The relevant number is not the probability that a man who batters his wife will go on to kill her (1 in 2,500) but rather the probability that a battered wife who was murdered was murdered by her abuser. According to the Uniform Crime Reports for the United States and Its Possessions in 1993, the probability Dershowitz (or the prosecution) should have reported was this one: of all the battered women murdered in the United States in 1993, some 90 percent were killed by their abuser. That statistic was not mentioned at the trial.

In other words, that Nicole was murdered puts her in a kind of “high risk” group, for which the probabilities are dramatically different. The defense’s statistics ignored this very-relevant fact.

When we hear statistics, we tend not to consider how the odds can vary drastically depending on intentionally-withheld or seemingly insignificant facts about the situation. For some particularly baffling examples of this at work, consider the following.

A Girl Named Florida

Say you know a family has two children, and further that at least one of them is a girl. What is the probability that they have two girls?

On first glance, the probability appears to be 50%: either the unknown child is a boy or a girl, right? It’s always useful to count the sample space, or the list of all possibilities, when determining probability. They are:

  1. The first child is a boy, and the second child is a boy.
  2. The first child is a boy, and the second child is a girl.
  3. The first child is a girl, and the second child is a boy.
  4. The first child is a girl, and the second child is a girl.

The information given lets us eliminate only option 1, so the probability is in fact 33% that they have two girls, option 4. An initial guess of 50% is wrong because it assumes a piece of information—namely, which child is a girl—which the question didn’t actually provide.

Clearly, it’s easy to read into a situation more information that it actually contains. But we also often misjudge the effect of information that a situation does contain. Consider this:

Say you know a family has two children, and further that at least one of them is a girl named Florida. What is the probability that they have two girls?

(The relatively rare name of Florida is chosen to justify the extra assumption that no family with two daughters would name both Florida.) Intuitively, knowing the girl’s name should change nothing, right? But let’s count the possibilities (shortened for clarity) and see:

  1. boy, boy
  2. boy, girl named Florida
  3. boy, girl not named Florida
  4. girl named Florida, boy
  5. girl not named Florida, boy
  6. girl named florida, girl named Florida
  7. girl not named florida, girl named Florida
  8. girl named florida, girl not named Florida
  9. girl not named florida, girl not named Florida

There are nine possibilities in the sample space, and the information we have lets us eliminate 5 of them (1, 3, 5, 6, and 9). Of the four possibilities left, two are families with two girls (7 and 8), so the probability that a family with two children one of whom is a girl named Florida is 50%.

At first I thought this result was some kind of trickery, but as Mlodinow explains, the differing probabilities between this example and the initial two-daughter problem are explained by the fact that the additional information of knowing the girl’s name does not apply equally to all of the possibilities, and can skew the probabilities for that reason.

One way to understand this, if it still seems puzzling, is to imagine that we gather into a very large room 75 million families that have two children, at least one of whom is a girl. As the two-daughter problem taught us, there will be about 25 million two-girl families in that room and 50 million one-girl families (25 million in which the girl is the older child and an equal number in which she is the younger). Next comes the pruning: we ask that only the families that include a girl named Florida remain. Since Florida is a 1 in 1 million name, about 50 of the 50 million one-girl families will remain. And of the 25 million two-girl families, 50 of them will also get to stay, 25 because their firstborn is named Florida and another 25 because their younger girl has that name. It’s as if the girls are lottery tickets and the girls named Florida are the winning tickets. Although there are twice as many one-girl families as two-girl families, the two-girl families each have two tickets, so the one-girl families and the two-girl families will be about equally represented among the winners.

Let’s Make a Deal

Another puzzling example of the unexpected effect of additional information on probabilities is “the Monty Hall problem”. Monty Hall was the host of Let’s Make a Deal, a game show whose dramatic finale involved a contestant asked to choose one of three doors—behind only one of which was the grand prize. The odds of winning in this case are of course 1 in 3, but here’s the catch. After the contestant made their choice, Monty Hall would reveal one of the doors which didn’t contain the grand prize, and offer the contestant the chance to change their guess. Many contestants didn’t take the offer, intuiting that with two doors left and a prize behind one of them, their chances were 50% either way. But looking back, those who changed their initial guess won twice as often as those who didn’t. How can this be?

It isn’t obvious, but by eliminating one of the doors that doesn’t hide the grand prize and offering the chance to change your guess, Monty Hall is effectively allowing you to bet against your initial guess. Again, let’s look at the possibilities:

  1. Either your initial guess was correct, and you win if you don’t switch doors.
  2. Or your initial guess was wrong, and you win if you do switch doors.

Since the odds were 1-in-3 that your initial guess was correct and thus 2-in-3 that it wasn’t, by switching doors you would double your chances of winning.

We Confuse General Probability with Particular Probability

A great example of this tendency is the Birthday Problem, which asks how many randomly-selected people you’d need in a room for a 50% chance that at least two of them share a birthday? Intuitively, we think about this problem in the first person: if I were in that room, you’d need about 366 (including Feb 29) divided by 2, so 183 people to have a 50% chance that one of them shares my birthday. But the question doesn’t ask the odds of the people in the room sharing a particular birthday, only a birthday. And the answer is that with just 23 people, you’d have a 50% chance that two in the room have the same birthday. If this seems counter-intuitive, it’s not the world at fault, but our intuition.

And lest this problem seem purely academic, our misjudgment in matters like this greatly affects how we perceive our own and others’ success in the world. Mlodinow recalls a mutual fund manager extolled in the newspaper for having a 15-year streak of positive returns. The paper proclaimed the man some kind of wizard whose incredible skill led him to defy the odds. Mlodinow then asks us to consider a field of 1000 mutual fund managers whose yearly results are determined by a random coin flip:

The chances that, after fifteen years, a particular coin tosser would have tossed all heads are then 1 in 32,768.

On this view, the newspaper’s intuition, and ours, would seem right: these are truly long odds to attain the same positive result 15 years in a row, and the manager in question must be some kind of genius to beat them. But as with the Birthday Problem, that’s not the right way to think about the situation. We should be thinking of the odds that anyone would attain this result, for surely if it weren’t this particular manager who’d attained the streak, the newspaper would be writing about and we’d be celebrating whoever did. And those odds are significantly better:

…the chances that someone among the 1,000 who had started tossing coins in 1991 would have tossed all heads are much higher, about 3 percent.

But it’s also not right to only think about this particular 15-year period, for such a streak could have happened at any time in the four-decade history of modern mutual funds, and the newspaper would have written about it at that time.

Finally, there is no reason to consider only those who started tossing coins in 1991—the fund managers could have started in 1990 or 1970 or any other year in the era of modern mutual funds. Since the writers for The Consilient Observer used forty years in their discussion, I calculated the odds that by chance some manager in the last four decades would beat the market each year for some fifteen-year period. That latitude increased the odds again, to the probability I quoted earlier, almost 3 out of 4.

In summary, we ascribe preternatural skill to someone who had a 15-year streak of picking winning stocks, but in any 40-year period, it’d be unlikely (1:4) for someone not to have such a streak due merely to random chance, in which case the person it does happen to is just lucky to be that someone. Our misjudgment in these matters might persist because it feels good: we want to believe that chance has little to do with success, for then if we just work hard enough, we should achieve it ourselves. But alas:

The cord that tethers ability to success is both loose and elastic. It is easy to see fine qualities in successful books or to see unpublished manuscripts, inexpensive vodkas, or people struggling in any field as somehow lacking. It is easy to believe that ideas that worked were good ideas, that plans that succeeded were well designed, and that ideas and plans that did not were ill conceived. And it is easy to make heroes out of the most successful and to glance with disdain at the least. But ability does not guarantee achievement, nor is achievement proportional to ability. And so it is important to always keep in mind the other term in the equation—the role of chance.

Conclusion

The author, having opened the book with the story of his mother’s survival against all odds of a labor camp in Poland during the Holocaust, concludes with a meditation on luck:

But more important, my mother’s experience has taught me that we ought to identify and appreciate the good luck that we have and recognize the random events that contribute to our success. It has taught me, too, to accept the chance events that may cause us grief. Most of all it has taught me to appreciate the absence of bad luck, the absence of events that might have brought us down, and the absence of the disease, war, famine, and accident that have not—or have not yet—befallen us.