Читать книгу Planning and Executing Credible Experiments - Robert J. Moffat - Страница 42

As researchers and instructors, we have been promoting experimental repeatability and uncertainty analysis for more than 60 years. When the work of Dr. J.P.A. Ioannidis brought the Reproducibility Crisis in the medical field to public awareness, we welcomed the positive impact it produced.

Two papers by Dr. Ioannidis in 2005 brought the Reproducibility Crisis to the fore. One was the Journal of the American Medical Association (JAMA) article mentioned in Chapter 1, “Contradicted and Initially Stronger Effects in Highly Cited Clinical Research” (Ioannidis 2005a). The second was “Why Most Published Research Findings Are False” (Ioannidis 2005b).

The two 2005 articles by Dr. Ioannidis appear to be a watershed moment for science. In various scientific disciplines, researchers have produced guidelines adopted by major publishers.

Going deeper into the 2005 JAMA article, Dr. Ioannidis chose a notably high criteria for the publications he evaluated. He considered only:

 “All original clinical research studies published in 3 major general clinical journals or high‐impact‐factor specialty journals

 in 1990–2003 and

 cited more than 1000 times in the literature…”

Dr. Ioannidis then compared these “results of highly cited articles … against subsequent studies of comparable or larger sample size and similar or better controlled designs. The same analysis was also performed comparatively for matched studies that were not so highly cited.”

Although part of the same article, this collection of research studies fared better than those mentioned in our Chapter 1. “Of 49 highly cited original clinical research studies, 45 claimed that the intervention was effective. Of these, 7 (16%) were contradicted by subsequent studies, 7 others (16%) had found effects that were stronger than those of subsequent studies, 20 (44%) were replicated, and 11 (24%) remained largely unchallenged.”⁵

In the same year, Dr. Ioannidis published “Why Most Published Research Findings Are False,” a provocative title. Although the wording appears to encompass all fields, the examples in the article were medical experiments. In order to make his evaluations, he adopted a key metric called the “Positive Predictive Value” (PPV). From this research, Dr. Ioannidis deduced the following “corollaries about the probability that a research finding is indeed true”:

 Corollary 1: The smaller the studies conducted in a scientific field, the less likely the research findings are to be true.

 Corollary 2: The smaller the effect sizes in a scientific field, the less likely the research findings are to be true.

 Corollary 3: The greater the number and the lesser the selection of tested relationships in a scientific field, the less likely the research findings are to be true.

 Corollary 4: The greater the flexibility in designs, definitions, outcomes, and analytical models in a scientific field, the less likely the research findings are to be true.

 Corollary 5: The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true.

 Corollary 6: The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true.

Be aware. Beware small studies (i) and small effects (ii). Beware explaining indiscriminately (iii). Beware lack of standards (iv). Beware political favors and conflicts of interest (v). Beware fashionable science (vi). The corollaries provide fair warning for all research, and do affect the Nature of Experimental Work.

Feynman's words suggest that experiments help science to be self‐correcting. However, Ioannidis (2012) gave the warning “Why Science Is Not Necessarily Self‐Correcting” in a more recent paper concerning psychological science.

Dr. J.P.A. Ioannidis has done more for us than exposing flaws in medical science. He invites better experiments in the biological and medical sciences. He sharpens experimental discernment.

Dr. Ioannidis is not alone. For example, Simera et al. (2010) report in the European Journal of Clinical Investigation how researchers collaborated and top publishers have agreed to guidelines regarding health research studies. There are many similar groups now.

Furthermore, there is incentive to resist bad and fraudulent research. RetractionWatch.com keeps track of research that has been retracted by author, publisher, or sponsor.

Overall, the concerted effort to assure the credibility of experiments keeps us in good company.