Читать книгу Planning and Executing Credible Experiments - Robert J. Moffat - Страница 50

If your type of experiment is a survey or poll, this text is also for you. The same strategies for sampling, statistics, and modeling apply. For a poll to promise any sort of accuracy, the sampling must be random and guided by good statistical practice. The principal challenges are designing nonleading questions and obtaining a representative sample. Otherwise a poll will mislead. Who has not witnessed and pained over a misleading poll?

Surveys of selective scientific audiences have a mixed history. One survey is particularly notable to us as experimentalists. In 2006, a survey of the International Astronomical Union (IAU) in Prague became memorable when the IAU demoted the planet Pluto. Pluto had been discovered and recognized as a planet in 1930. For decades teachers and textbooks across the world taught the nine planets in the solar system. Then the press in 2006 widely reported that the IAU declared that Pluto was no longer a planet. From our viewpoint, as IAU outsiders, it was a decree with huge impact on classrooms and the textbook industry.

As experimentalists, we are interested in the sampling, statistics, accuracy, and credibility of the Pluto survey. These all have relevance to us as we consider “The Nature of Experimental Work” (this chapter). We read that within the IAU there is dispute and controversy. How well did the vote represent the entire population?

Let's review details: The total population of IAU members is about 11,000. In 2012, the IAU reported 10,894 individual members from 93 countries worldwide. At the 2006 Prague conference, 2,400 members had registered. On the last day of the conference, just 424 astronomers remained and were polled on several issues. On some issues, exact numbers of the vote were reported (IAU 2006). However, the definitions that resulted in Pluto's demotion was reportedly decided by voice vote, approved by a majority.

How representative of the IAU population was the poll? For a sample size of 424 randomized members out of 11,000, a typical confidence interval could be the claimed 5%. The confidence interval surrounds an estimate, however a voice vote denies one. With a savvy, engaged population of size 11,000, why not take a vote of the full IAU population? Why rely on a poll to redefine international standards?

The IAU clearly has the right to set its own standards and definitions. Via the 2006 Pluto poll however, the IAU leaders decided for discriminating communication within its own community. How well did the IAU poll represent all science teachers when it demoted Pluto? By its decision, one consequence was to make existing text books obsolete, affecting the budgets of school systems worldwide.

Did the IAU leadership poorly serve its members, other scientists, and students across the world by relying on a poll instead of a full vote? From history, can we find examples where polls with small margins of error failed to predict the vote?

Another survey of note, recently conducted by the journal Nature, hearkened back to the Reproducibility Crisis announced by Ioannidis. Nature published (2016) a survey of researchers across the living and hard sciences asking “Is There a Reproducibility Crisis?” The author, Monya Baker, highlights a contradiction within the survey (Baker 2016): although researchers expected high confidence in their own fields, they admitted an inability to replicate results a majority of the time.

On a related note we may ask: what place have fashion and popularity in science? Fashion does have a notable impact on availability of funding. We defer to Panel 2.2, “Selected Invitations to Experimental Research, Insights from Theoreticians,” because funding encompasses all research, not just experiments.