Читать книгу Inside the Crystal Ball - Harris Maury - Страница 8

Some seers do much better than others in addressing the difficulties cited earlier. But what makes these individuals more accurate? The answer is critical for learning how to make better predictions and for selecting needed inputs from other forecasters. We first review some studies identifying characteristics of both successful and unsuccessful forecasters. That is followed in Chapter 2 by a discussion of my experience in striving for better forecasting accuracy throughout my career.

What Is “Success” in Forecasting?

A forecast is any statement regarding the future. With this broad definition in mind, there are several ways to evaluate success or failure. Statistics texts offer a number of conventional gauges for judging how close a forecaster comes to being right over a number of forecast periods. (See an explanation and examples of these measures in Chapter 2.) Sometimes, as in investing, where the direction of change is more important than the magnitude of change, success can be defined as being right more often than being wrong. Another criteria can be whether a forecaster is correct about outcomes that are especially important in terms of costs of being wrong and benefits of being right (i.e., forecasting the big one.)

Over a forecaster's career, success will be judged by all three criteria – accuracy, frequency of being correct, and the ability to forecast the big one. And, as we see, it's rare to be highly successful in addressing all of these challenges. The sometimes famous forecasters who nail the big one are often neither accurate nor even directionally correct most of the time. On the other hand, the most reliable forecasters are less likely to forecast rare and very important events.

One-Hit Wonders

Reputations often are based on an entrepreneur, marketer, or forecaster “being really right when it counted most.” Our society lauds and rewards such individuals. They may attain a guru status, with hordes of people seeking and following their advice after their “home run.” However, an impressive body of research suggests that these one-hit wonders are usually unreliable sources of advice and forecasts. In other words, they strike out a lot. There is much to learn about how to make and evaluate forecasts from this phenomenon.

In the decade since it was published in 2005, Phillip E. Tetlock's book Expert Political Judgment – How Good Is It? How Can We Know? has become a classic in the development of standards for evaluating political opinion.17 In assessing predictions from experts in different fields, Tetlock draws important conclusions for successful business and economic forecasting and for selecting appropriate decision-making/forecasting inputs. For instance:

“Experts” successfully predicting rare events were often wrong both before and after their highly visible success. Tetlock reports that “When we pit experts against minimalist performance benchmarks – dilettantes, dart-throwing chimps, and assorted extrapolation algorithms, we find few signs that expertise translates into greater ability to make either ‘well-calibrated’ or ‘discriminating’ forecasts.”

The one-hit wonders can be like broken clocks. They were more likely than most forecasters to occasionally predict extreme events, but only because they make extreme forecasts more frequently.

Tetlock's “hedgehogs” (generally inaccurate forecasters who manage to correctly forecast some hard-to-forecast rare event) have a very different approach to reasoning than his more reliable “foxes.” For example, hedgehogs often used one big idea or theme to explain a variety of occurrences. However, “the more eclectic foxes knew many little things and were content to improvise ad hoc solutions to keep pace with a rapidly changing world.”

While hedgehogs are less reliable as forecasters, foxes may be less stimulating analysts. The former encourage out-of-the-box thinking. The latter are more likely to be less decisive, two-handed economists.

Tetlock's findings about political forecasts also apply to business and economic forecasts. Jerker Denrell and Christina Fang have provided such illustrations in their 2010 Management Science article titled “Predicting the Next Big Thing: Success as a Signal of Poor Judgement.”18 They conclude that “accurate predictions of an extreme event are likely to be an indication of poor overall forecasting ability, when judgment or forecasting ability is defined as the average level of forecast accuracy over a wide range of forecasts.”

Denrell and Fang assessed the forecasting accuracy of professional forecasters participating in Wall Street Journal semi-annual forecasting surveys between July 2002 and July 2005. (Every six months at the start of January and July around 50 economists and analysts provided six-month-ahead forecasts of key economic variables, such as GNP, inflation, unemployment, interest rates, and exchange rates.) Their study focused on the overall accuracy of forecasters projecting extreme events, which were defined as results either 20 percent above or below average forecasts. For each forecaster, they compared overall accuracy for all of the forecast variables with the accuracy of each forecaster's projections of the defined extreme events.

Forecasters who were more accurate than the average forecaster in predicting extreme outcomes were less accurate in predicting all outcomes. Also, the prognosticators who were comparatively more accurate in predicting extreme outcomes had extreme outcomes as a higher percentage of their overall forecasts. In the authors' assessment, “Forecasting ability should be based on all predictions, not only a selected subset of extreme predictions.”

What do these results tell us about the characteristics of forecasters with these two different types of success? Denrell and Fang offer the following four observations:

1. Extreme outcomes are, by definition, less likely than all outcomes. Therefore, extreme outcomes are more likely to be successfully predicted by forecasters who rely on intuition or who emphasize a single determinant than by those forecasters extrapolating from a more comprehensive sense of history.

2. Forecasters who happen to be correct about extreme outcomes become overconfident in their judgment. The authors cite research indicating that securities analysts who have been relatively more accurate in predicting earnings over the previous four quarters tend to be less accurate compared to their peers in subsequent quarters.19

3. Forecasters can be motivated by their past successes and failures. Forecasters with relatively bold past forecasts may be tempted to alter their tarnished reputations with more bold forecasts. (Denrell and Fang cite research by Chevalier and Ellison and by Leone and Wu.20,21) On the other hand, successful forecasters might subsequently move closer to consensus expectations in order to avoid the risk of incorrect bold forecasts on their reputations and track records. (The authors cite research by Pendergrast and Stole.22) In other words, in the world of forecasting both successes and failures can feed on each other.

4. Some forecasters may be motivated to go out on a limb if the rewards for being either relatively right or highly visible with such forecasts are worth taking the reputational risk of building up a bad track record with many long shots that don't pan out. This may be especially so if forecasters perceive that they will be more commercially successful with the attention garnered by comparatively unique projections. However, Denrell and Fang cite research suggesting that securities analysts are more likely to be terminated if they make bold and inaccurate forecasts.23

Who Is More Likely to Go Out on a Limb?

It can be dangerous for a forecaster to go out on a limb too often, especially if the proverbial limb usually gets sawed off. But why do some forecasters make that choice? Do we know who they are, so that we can consider the source when hearing their forecasts?

Researchers have examined popular, well-publicized forecast surveys to identify who is most likely to go against the grain and what are the consequences of doing so. Among the studied forecaster surveys have been those appearing in The Blue Chip Economic Indicators, the Wall Street Journal, and Business Week.

Karlyn Mitchell and Douglas Pearce have examined six-month-ahead interest rate and foreign exchange rate forecasts appearing in past Wall Street Journal forecaster surveys.24 They asked whether a forecaster's employment influenced various forecast characteristics. Specifically, they compared forecasters employed by banks, securities firms, finance departments of corporations, econometric modelers, and independent firms.

Their research indicated that economists with firms bearing their own name deviated more from the consensus than did other forecasters of interest rates and foreign exchange rates. In the authors' view, such behavior could have been motivated by the desire to gain publicity. (Note: Large sell-side firms employing economists and analysts have the financial means for large advertising budgets and are presumably not as hungry for free publicity.)

While Mitchell and Pearce studied the Wall Street Journal surveys, Owen Lamont examined participants in Business Week's December surveys of forecasts for GDP growth, unemployment, and inflation in successive years.25 His statistical results indicated that economists who own their consulting firms are more likely to deviate from the consensus and are relatively less accurate in forecasting growth, unemployment, and inflation. His findings indicated to him that some forecasters are “optimizing the market value of their reputations” instead of “acting to minimize mean squared error.”

David Laster, Paul Bennett, and In Sun Geoum reviewed U.S. real GDP forecast behavior of economists participating in Blue Chip Economic Indicator surveys in the 1976 to 1995 period.26 They conclude that “forecasters' deviations from the consensus are related to the types of firms for which they work” and illustrate that “professional forecasting has a strong strategic component.” Studying mean absolute deviation (MAD) from the consensus, they report that “independent forecasters with firms bearing their own names tend to make unconventional forecasts.” (See Table 1.5.)

Table 1.5 Average Forecast Deviations from Consensus GDP Growth Forecasts 1977–1996

Source: David Laster, Paul Bennett, and In Sum Geoum, “Rational Basis in Macroeconomic Forecasts,” Federal Reserve Bank of New York Research Papers, July 1996.

Consensus Copycats

Sticking with the consensus forecast is the opposite of going out on a limb. One motivation for doing so is that there's safety in numbers. Some forecasters might think that they'll never be fired when they are wrong if so many others also were incorrect. But is it wise to copy the consensus?

When the economics profession is criticized for inaccurate forecasts, the judgments reflect comparisons of actual outcomes to consensus forecasts. But the reason for copying consensus economic forecasts is that the consensus average of all forecasts usually outperforms any individual forecaster.27

Why? Because when we consider all forecasts, we incorporate more input information than is represented in any individual pronouncement. Also, the averaging process can iron out individual forecasters' biases. In addition to their relative accuracy, consensus economic forecasts also have the advantage of often being free of charge and time-saving. However, relying on consensus forecasts does have some important drawbacks:

• First, consensus economic growth forecasts almost always fail at critical turning points in the business cycle – especially when heading into a recession.28 For companies and investors, not being prepared for a recession can often turn success into failure.

• Second, if one's objective is to outperform one's peers, sticking with the consensus is not the recipe for such relative success. Students of forecasting approach this issue from the perspective of herding. Consider the results of one study of the characteristics of competing forecasters in the Blue Chip Survey. Two-time winners of the top forecaster award herd to the consensus the least amount, while the most herding is done by nonwinners.29

Although it may be sensible for less skilled or infrequent forecasters to mimic the consensus, they often don't. Laster, Bennett, and Geoum also find that nonwinners in the Blue Chip survey herd more often to the forecasts of winners than to the consensus forecast.