Читать книгу The Success Equation - Michael J. Mauboussin - Страница 9

CHAPTER 2

WHY WE'RE SO BAD AT DISTINGUISHING SKILL FROM LUCK

AS PART OF A LECTURE that he delivers to the general public, Simon Singh, a British author who writes about science and math, plays a short snippet from Led Zeppelin's famous rock song, “Stairway to Heaven.” Most of the people in the audience are familiar with the tune, and some know the lyrics well enough to sing along.

He then plays the same song backward. As you would expect, it sounds like gibberish. He follows by earnestly asking how many heard the following lyrics in the backward version:

It's my sweet Satan. The one

whose little path would make me

sad whose power is Satan.

Oh, he'll give you, give you 666.

There was a little toolshed where

he made us suffer, sad Satan.

The words are a little odd, but the satanic theme is clear. Even so, no one in the audience had heard those words the first time through. But then Singh replays the backward clip, and this time he displays the pseudo lyrics on a screen and highlights them so that everyone can follow along. And sure enough, the audience unmistakably hears the words, where before they had heard nothing. The first time through, the backward version was an incoherent mess. But once Singh told the audience what might be there, the previously unintelligible gibberish was transformed into clear speech.¹

Singh's demonstration provides an important clue to why we have a hard time understanding the roles of skill and luck. Our minds have an amazing ability to create a narrative that explains the world around us, an ability that works particularly well when we already know the answer. There are a couple of essential ingredients in this ability: our love of stories and our need to connect cause and effect. The blend of those two ingredients leads us to believe that the past was inevitable and to underestimate what else might have happened.

Stories, Causality, and the Post Hoc Fallacy

John Lewis Gaddis, a professor of history at Yale, creates a vivid image of how we represent time. He suggests that the future is a zone where skill and luck coexist independently. Almost everyone recognizes that many more things could happen than will happen. A wide range of events might occur, but won't. These possibilities come down a funnel to the present, which fuses skill and luck to create whatever happens. The conversion of a range of alternatives into a single event is the process that makes history.²

For example, you undoubtedly trust your skill at driving well enough to get to the grocery store and back without dying. But when you pull out onto the road, you're facing a wide range of possible histories for this journey. In one of them, the engine falls off of the Boeing 767 going overhead and lands on your car and kills you. In another, you turn in front of a motorcyclist you happen not to see, and you kill him. In yet another, a tractor trailer loses its brakes and plows into you from behind, putting you in the hospital for a month. In fact, in this instance, you drive to the store, buy your groceries, and go home. The history of this event was that you didn't die. Was it your skill as a driver that saved your life? Or was it luck?

If we look into the past, skill and luck appear to be inextricably fixed, even though the history that we lived through was but one of many possible histories that could have occurred. While we are capable of contemplating a future pulsating with possibility, we quickly forget that our experience was one of many that could have been. As a consequence, often we draw lessons from the past that are wrong. For example, you could conclude that you're such a skillful driver that you really stand no chance of being in an accident. That's a very dangerous conclusion.

Humans love stories.³ They are one of the most powerful and emotive ways that we communicate with one another. Our parents told us stories, and we tell them to our children. People tell stories to teach lessons or to codify the past. The oral tradition of storytelling goes back thousands of years and predates writing. All stories have common elements. There is a beginning, some inciting episode that launches a sequence of events. The storyteller explains why events unfolded as they did, though he may be inventing those causes. As the story proceeds, the action rises. Complications occur. Interesting stories have an element of suspense and surprise. We get invested in a story when there is something at stake, when the tension mounts, and when events occur that upset our expectations. And stories have a climax and a resolution: the protagonist wins or loses, and then the tension is released as things settle down once more.

The need to connect cause and effect is deeply ingrained in the human mind.⁴ When we see an effect, we naturally seek the cause. Michael Gazzaniga, a professor of psychology at the University of California, Santa Barbara, has worked with patients who have undergone surgery to sever the corpus callosum, the bundle of nerves that connects the right and left hemispheres of the brain. This surgery is a treatment for severe epilepsy. Gazzaniga and his colleagues were able to learn just how each hemisphere functions because in these patients the two halves of the brain cannot communicate with each other and so must function in isolation.

One of their main conclusions was that the left hemisphere “includes a special region that interprets the inputs we receive every moment and weaves them into stories to form an ongoing narrative of our self-image and our beliefs.”⁵ Gazzaniga calls this region the interpreter. One of the left hemisphere's main jobs is to make sense of the world by finding a cause for every effect, even if the cause is nonsensical.

In one experiment, Gazzaniga showed a split-brain patient two cards with images on them. The patient's left eye (controlled by the right hemisphere) saw a snowy scene. The patient's right eye (controlled by the left hemisphere) saw a chicken's foot. When asked to pick a card that related to what he saw, the patient picked a shovel with his left hand (right hemisphere) and a chicken with his right hand (left hemisphere). In other words, each hemisphere independently came up with an appropriate response. For example, the right hemisphere correctly chose something related to what it had seen: a snow shovel for the snow. However, in most people, the right hemisphere has no ability to express language. And all the left hemisphere knew about was a chicken's foot and the image of a shovel that it inexplicably chose. How could he resolve the conflict? Make up a story. When the researchers asked the patient why he picked what he did, the interpreter in the left brain kicked into gear: “Oh, that's simple. The chicken claw goes with the chicken, and you need a shovel to clean out the chicken shed.” Rather than saying, “I don't know,” the left hemisphere made up a response based on what it knew.⁶

Steven Pinker, a psychologist at Harvard, calls this part of the left hemisphere the baloney-generator. He wrote, “The spooky part is that we have no reason to believe that the baloney-generator in the patient's left hemisphere is behaving any different from ours as we make sense of the inclinations emanating from the rest of our brains. The conscious mind—the self or soul—is a spin doctor, not the commander in chief.”⁷ Gazzaniga's patient simply reveals what's going on in all of our heads.

To explain the past, we also naturally apply the essential elements of stories: a beginning, an end, and a cause.⁸ As events in our world unfold, we don't—really, can't—know what's happening. But once we know the ending, we stand ready to create a narrative to explain how and why events unfolded as they did.⁹ For our purpose, the two critical elements required for analyzing the past are that we already know the ending and that we want to understand the cause of what happened. Those two elements are what get us into trouble. Most of us will readily believe that this happens to others. But we are much more reluctant to admit that we can fall prey to the same bias.

We often assume that if event A preceded event B, then A caused B. Even Nassim Taleb, who has done a great deal to raise the awareness of the role of randomness and luck in our daily lives, points the finger at himself in this regard. He tells this story: Every day, he used to take a cab to the corner of Park Avenue and 53rd Street in New York City and take the 53rd Street entrance to go to work. One day, the driver let him out closer to the 52nd Street entrance and threw Taleb off his routine. But that day, he had great success at his job trading derivatives. So the next day, he had the cab driver drop him off on the corner of Park and 52nd Street so that he could extend his financial success. He also wore the same tie he had worn the day before. He obviously knew intellectually that where he got out of the cab and which tie he wore had nothing to do with trading derivatives, but he let his superstition get the best of him. He admitted that, deep down, he believed that where he entered the building and what tie he wore were causing him to succeed. “On the one hand, I talked like someone with strong scientific standards,” he continued. “On the other, I had closet superstitions just like one of these blue-collar pit traders.”¹⁰ Taleb entered the building from 52nd Street and then made money; therefore entering the building from 52nd Street caused him to make money. That faulty association is known as the post hoc fallacy. The name comes from the Latin, post hoc ergo propter hoc, “after this, therefore because of this.” A lot of the science done in the last two hundred years has been aimed at doing away with that mistaken way of thinking.

Knowing the end of the story also leads to another tendency, one that Baruch Fischhoff, a professor of psychology at Carnegie Mellon University, calls creeping determinism. This is the propensity of individuals to “perceive reported outcomes as having been relatively inevitable.”¹¹ Even if a fog of uncertainty surrounded an event before it unfolded, once we know the answer, that fog not only melts away, but the path the world followed appears to be the only possible one.

Here is how all of this relates to skill and luck: even if we acknowledge ahead of time that an event will combine skill and luck in some measure, once we know how things turned out, we have a tendency to forget about luck. We string together the events into a satisfying narrative, including a clear sense of cause and effect, and we start to believe that what happened was preordained by the existence of our own skill. There may be an evolutionary reason for this. In prehistoric times, it was probably better for survival to take the view that we have some control over events than to attribute everything to luck and give up trying.

John Glavin is a professor of English at Georgetown University who teaches courses in writing for the stage and screen. Glavin spends a great deal of time understanding what makes for a great narrative and emphasizes that stories are vehicles for communicating how to act. We use stories, especially those about history, to learn what to do. “Narrative is deeply connected with ethics,” he notes, “and narratives tell us how we should and should not behave.” But when we try to learn from history, we naturally look for causes even when there may be none. Glavin adds, “For a story to work, someone has to be responsible.”¹² History is a great teacher, but the lessons are often unreliable.

Undersampling and Sony's Miraculous Failure

The most common method for teaching a manager how to thrive in business is to find successful businesses, identify the common practices of those businesses, and recommend that the manager imitate them. Perhaps the best-known book about this method is Jim Collins's Good to Great. Collins and his team analyzed thousands of companies and isolated eleven whose performance went from good to great. They then identified the concepts that they believed had caused those companies to improve—these include leadership, people, a fact-based approach, focus, discipline, and the use of technology—and suggested that other companies adopt the same concepts to achieve the same sort of results. This formula is intuitive, includes some great narrative, and has sold millions of books for Collins.¹³

No one questions that Collins has good intentions. He really is trying to figure out how to help executives. And if causality were clear, this approach would work. The trouble is that the performance of a company always depends on both skill and luck, which means that a given strategy will succeed only part of the time. So attributing success to any strategy may be wrong simply because you're sampling only the winners. The more important question is: How many of the companies that tried that strategy actually succeeded?

Jerker Denrell, a professor of strategy at Oxford, calls this the undersampling of failure. He argues that one of the main ways that companies learn is by observing the performance and characteristics of successful organizations. The problem is that firms with poor performance are unlikely to survive, so they are inconspicuously absent from the group that any one person observes. Say two companies pursue the same strategy, and one succeeds because of luck while the other fails. Since we draw our sample from the outcome, not the strategy, we observe the successful company and assume that the strategy was good. In other words, we assume that the favorable outcome was the result of a skillful strategy and overlook the influence of luck. We connect cause and effect where there is no connection.¹⁴ We don't observe the unsuccessful company because it no longer exists. If we had observed it, we would have seen the same strategy failing rather than succeeding and realized that copying the strategy blindly might not work.

Denrell illustrates the idea by offering a scenario in which firms that pursue risky strategies achieve either high or low performance, whereas those that choose low-risk strategies achieve average performance. A high-risk strategy might put all of a company's resources into one technology, while a low-risk strategy would spread resources across various alternatives. The best performers are those that bet on one option and happen to succeed, and the worst performers are those that make a similar bet but fail. As time passes, the successful firms thrive and the failed firms go out of business or get acquired.

Someone attempting to draw lessons from this observation would therefore see only those companies that enjoyed good performance and would infer, incorrectly, that the risky strategies led to high performance. Denrell emphasizes that he is not judging the relative merits of a high- or low-risk strategy. He's saying that you need to consider a full sample of strategies and the results of those strategies in order to learn from the experiences of other organizations. When luck plays a part in determining the consequences of your actions, you don't want to study success to learn what strategy was used but rather study strategy to see whether it consistently led to success.

In chapter 1, we met Michael Raynor, a consultant at Deloitte. Raynor defines what he calls the strategy paradox—situations where “the same behaviors and characteristics that maximize a firm's probability of notable success also maximize its probability of failure.” To illustrate this paradox, he tells the story of Sony Betamax and MiniDiscs. At the time those products were launched, Sony was riding high on the success of its long string of winning products from the transistor radio to the Walkman and compact disc (CD) player. But when it came to Betamax and MiniDiscs, says Raynor, “the company's strategies failed not because they were bad strategies but because they were great strategies.¹⁵

The case of the MiniDisc is particularly instructive. Sony developed MiniDiscs to replace cassette tapes and compete with CDs. The disks were smaller and less prone to skip than CDs and had the added benefit of being able to record as well as play music. Announced in 1992, MiniDiscs were an ideal format to replace cassettes in the Walkman to allow that device to remain the portable music player of choice.

Sony made sure that the MiniDisc had a number of advantages that put it in a position to be a winner. For example, existing CD plants could produce MiniDiscs, allowing for a rapid reduction in the cost of each unit as sales grew. Furthermore, Sony owned CBS Records, so it could supply terrific music and make even more profit. The strategy behind the MiniDisc reflected the best use of Sony's vast resources and embodied all of the lessons that the company had learned from the successes and failures of past products.

But just as the MiniDisc player was gaining a foothold, seemingly out of nowhere, everyone had tons of cheap computer memory, access to fast broadband networks, and they could swap files of a manageable size that contained all their favorite music essentially for free. Sony had been hard at work on a problem that vanished from beneath their feet. Suddenly, no one needed cassette tapes. No one needed disks either. And no one could possibly have foreseen that seismic shift in the world in the 1990s. In fact, much of it was unimaginable. But it happened. And it killed the MiniDisc. Raynor asserts, “Not only did everything that could go wrong for Sony actually go wrong, everything that went wrong had to go wrong in order to sink what was in fact a brilliantly conceived and executed strategy. In my view, it is a miracle that the MiniDisc did not succeed.”¹⁶

One of the main reasons we are poor at untangling skill and luck is that we have a natural tendency to assume that success and failure are caused by skill on the one hand and a lack of skill on the other. But in activities where luck plays a role, such thinking is deeply misguided and leads to faulty conclusions.

Most Research Is False

In 2005, Dr. John Ioannidis published a paper, titled “Why Most Published Research Findings Are False,” that shook the foundation of the medical research community.¹⁷ Ioannidis, who has a PhD in biopathology, argues that the conclusions drawn from most research suffer from the fallacies of bias, such as researchers wanting to come to certain conclusions or from doing too much testing. Using simulations, he shows that a high percentage of the claims made by researchers are simply wrong. In a companion paper, he backed up his contention by analyzing forty-nine of the most highly regarded scientific papers of the prior thirteen years, based on the number of times those papers were cited. Three-quarters of the cases where researchers claimed an effective intervention (for example, vitamin E prevents heart attacks) were tested by other scientists. His analysis showed a stark difference between randomized trials and observational studies. In a randomized trial, subjects are assigned at random to one treatment or another (or none). These studies are considered the gold standard of research, because they do an effective job of finding genuine causes rather than simple correlations. They also eliminate bias in many cases, because the people running the experiment don't know who is getting which treatment. In an observational study, subjects volunteer for one treatment or another and researchers have to take what is available. Ioannidis found that more than 80 percent of the results from observational studies were either wrong or significantly exaggerated, while about three-quarters of the conclusions drawn from randomized studies proved to be true.¹⁸

Ioannidis's work doesn't touch on skill as we have defined it, but it does address the essential issue of cause and effect. In matters of health, researchers want to understand what causes what. A randomized trial allows them to compare two groups of subjects who are similar but who receive different treatments to see whether the treatment makes a difference. By doing so, these trials make it less likely that the results derive from luck. But observational studies don't make the same distinction, allowing luck to creep in if the researchers are not very careful in their methods. The difference in the quality of the findings is so dramatic that Ioannidis recommends a simple approach to observational studies: ignore them.¹⁹

The dual problems of bias and conducting too much testing are substantial, and by no means limited to medical research.²⁰ Bias can arise from many factors. For example, a researcher who is funded by a drug company may have an incentive to find that the drug works and is safe. While scientists generally believe themselves to be objective, research in psychology shows that bias is most often subconscious and nearly unavoidable. So even if a scientist believes he is behaving ethically, bias can exert a strong influence.²¹ Furthermore, a bit of research that grabs headlines can be very good for advancing an academic's career.

Doing too much testing can cause just as much trouble. There are standard methods to deal with testing too much, but not all scientists use them. In much of academic research, scientists lean heavily on tests of statistical significance. These tests are supposed to indicate the probability of getting a result by chance (more formally, when the null hypothesis is true). There is a standard threshold that allows a researcher to claim that a result is significant. Here's where the trouble starts: if you test enough relationships, you will eventually find a few that pass the test but that are not really related as cause and effect.²²

One example comes from a paper published in The Proceedings of the Royal Society B, a peer-reviewed journal. The article suggests that women who eat breakfast cereal are more likely to give birth to boys than girls.²³ The paper naturally generated a great deal of attention, especially in the media. Stan Young, a statistician at the National Institute of Statistical Sciences, along with a pair of colleagues, reexamined the data and concluded that the finding was likely the product of chance as a result of testing too much. The basic idea is that if you examine enough relationships, some will pass the test of statistical significance by virtue of chance. In this case, there were 264 relationships (132 foods and two time periods), and the plot of expected values of statistical significance between the various relationships was completely consistent with randomness. Young and his collaborators conclude flatly that their analysis “shows that the [findings] claimed as significant by the authors are easily the result of chance.”²⁴

So if we don't consider a sample that is large enough, we can miss the fact that a single strategy can always give rise to unanticipated results, as we saw in the case of the Sony MiniDisc. In contrast, we can comb through lots of possible causes and pick one that really has nothing to do with the effect we observe, such as women eating cereal and having boys as opposed to girls. What's common to the two approaches is an erroneous association between the effect, which is known, and the presumed cause. In each case, researchers fail to appreciate the role of luck.

Where Is the Skill? It's Easier to Trade for Punters Than Receivers

Many organizations, including businesses and sports teams, try to improve their performance by hiring a star from another organization. They often pay a high price to do so. The premise is that the star has skill that is readily transferable to the new organization. But the people who do this type of hiring rarely consider the degree to which the star's success was the result of either good luck or the structure and support of the organization where he or she worked before. Attributing success to an individual makes for good narrative, but it fails to take into account how much of the skill is unique to the star and is therefore portable.

Boris Groysberg, a professor of organizational behavior at Harvard Business School, has studied this topic in depth. His research shows that organizations tend to overestimate the degree to which the star's skills are transferrable. His most thorough study was of analysts at Wall Street firms.²⁵ The primary responsibility of these analysts is to determine whether or not a given stock is attractive within the industry that they follow. (I used to be one of these analysts.) Institutional Investor magazine ranks the analysts annually, which provides a measure of quality.

Groysberg examined all of the moves by ranked analysts over a twenty-year period and found 366 instances of a star analyst moving to another firm. If the skill were associated solely with the analyst, you would expect the star's performance to remain stable when he or she changed jobs. That is not what the data showed. Groysberg writes, “Star analysts who switched employers paid a high price for jumping ship relative to comparable stars who stayed put: overall, their job performance plunged sharply and continued to suffer for at least five years after moving to a new firm.”²⁶ He considered a number of explanations for the deterioration in performance and concluded that the main factor was that they left behind a good fit between their skills and the resources of their employer.

General Electric is a well-known source of managerial talent, and its alumni are disproportionately represented among CEOs in the S&P 500. Groysberg and his colleagues tracked the performance of twenty managers from GE that other organizations hired as chairman, CEO, or CEO-designate between 1989 and 2001. They found a stark dichotomy. Ten of the hiring companies resembled GE, so the skills of the executives were neatly transferable and the companies flourished. The other ten companies were in lines of business different from GE. For example, one GE executive went to a company selling groceries, whereas his experience had been in selling appliances. Even with a GE-trained executive at the helm, those companies delivered poor returns to shareholders. Again, developing skill is a genuine achievement. And skill, once developed, has a real influence on what we can do and how successful we are. But skill is only one factor that contributes to the end result of our efforts. The organization or environment in which a CEO works also has an influence. The evidence shows that employers systematically overestimate the power of an individual's skill and underestimate the influence of the organization in which he or she operates.

Along with some fellow researchers, Groysberg showed this point neatly by analyzing the performance of players who switched teams in the National Football League. They compared wide receivers with punters in the period between 1993 and 2002. Since each team has eleven players on the field at a time, wide receivers rely heavily on the strategy of the team and on interaction with their teammates, factors that can vary widely from team to team. Punters pretty much do the same thing no matter which team they play for, and have more limited interaction with teammates. The contrast in interaction allowed the scientists to separate an individual's skill from the influence of the organization on performance. They found that star wide receivers who switched teams suffered a decline in performance for the subsequent season compared with those who stayed with the team. Their performance then improved as they adjusted to their new team. Whether a punter changed teams or stayed put had no influence on his performance. Punters are more portable than wide receivers.²⁷

As with testing too much or too little, the difficulty in determining the portability of a skill lies in the relationship between cause and effect. Groysberg's work dwells on stars and finds that the organizations that support them contribute meaningfully to their success. Yet we see people consistently overestimate skill in fields as diverse as catching touchdown passes and selling motorcycles.

Stories Can Obscure Skills

We re-create events in the world by creating a narrative that is based on our own beliefs and goals. As a consequence, we often struggle to understand cause and effect, and especially the relative contributions of skill and luck in shaping the events we observe.²⁸ As we've seen, we may make the mistake of drawing conclusions from samples that are too small. We may fail to consider all of the causes that might lead to particular events. We might test too much—so much, in fact, that we wind up finding causes where we're simply seeing the results of chance. Or we may look at high performance and believe we are seeing a star with exceptional skill, when in reality we are seeing the combined effects of skill and the powerful influence that an organization can exert on someone. All of these mistakes are manageable, but it is critical to learn about them and to see where they apply if we are going to overcome them. The effort of untangling skill from luck, even with its practical difficulties, still yields great value when we are trying to improve the way we make decisions.