Читать книгу The Failure of Risk Management - Douglas W. Hubbard - Страница 38

The most convincing way—and the hardest way—to measure the effectiveness of risk management is with a large-scale experiment over a long period tracking dozens or hundreds of organizations. This is still time-consuming—for example, waiting for the risk event to occur in your own organization—but it has the advantage of looking at a larger population of firms in a formal study. If risk management is supposed to, for example, reduce the risk of events that are so rare that actual results alone would be insufficient to draw conclusions, then we can't just use the short-term history of one organization. Even if improved risk management has a significant effect on reducing losses from various risks, it may take a large number of samples to be confident that the risk management is working.

To build on the previous pharmaceutical outsourcing example, imagine applying a method that pharmaceutical companies would already be very familiar with in the clinical testing of drugs. Suppose that nearly all of the major health products companies (this includes drugs, medical instruments, hospital supplies, etc.) are recruited for a major risk management experiment. Let's say, in total, that a hundred different product lines that will be outsourced to China are given one particular risk management method to use. Another hundred product lines, again from various companies, implement a different risk management method. For a period of five years, each product line uses its new method to assess risks of various outsourcing strategies. Over this period of time, the first group experiences a total of twelve events resulting in adverse health effects traced to problems related to the overseas source. During the same period, the second group has only four such events without showing a substantial increase in manufacturing costs.

Of course, it would seem unethical to subject consumers to an experiment with potentially dangerous health effects just to test different risk management methods. (Patients in drug trials are at least volunteers.) But if you could conduct a study similar to what was just described, the results would be fairly good evidence that one risk management method was much better than the other. If we did the math (which I will describe more later on as well as show an example on the website www.howtomeasureanything.com/riskmanagement) we would find that it would be unlikely for this result to be pure chance if, in fact, the probability of the events were not different. In both groups, there were companies that experienced unfortunate events and those that did not, so we can infer something about the performance of the methods only by looking at the aggregation of all their experiences.

Although this particular study might be unethical, there were some examples of large studies similar to this that investigated business practices. For example, in July 2003, Harvard Business Review published the results of a study involving 160 organizations to measure the effectiveness of more than two hundred popular management tools, such as TQM, ERP, and so on.¹⁵ Then independent external reviews of the degree of implementation of the various management tools were compared to shareholder return over a five-year period. In an article titled “What Really Works,” the researchers concluded, to their surprise, that “most of the management tools and techniques we studied had no direct causal relationship to superior business performance” That would be good to know if your organization was about to make a major investment in one of these methods.

Another study, which was based on older but more relevant data, did look at alternative methods of risk management among insurance companies. There was a detailed analysis of the performance of insurance companies in mid-nineteenth century Great Britain when actuarial science was just emerging. Between 1844 and 1853, insurance companies were starting up and failing at a rate more familiar to Silicon Valley than the insurance industry. During this period 149 insurance companies formed and after that period just fifty-nine survived. The study determined that the insurance companies who were using statistical methods were more likely to stay in business (more on this study later).¹⁶ Actuarial methods that were at first considered a competitive advantage became the norm.

Again, this is the hard way to measure risk management methods. The best case for organizations would be to rely on research done by others instead of conducting their own studies—assuming they find the relevant study. Or, similar to the insurance industry study, the data are all historical and are available if you have the will to dig all of it up. Fortunately, there are alternative methods of measurement.