Читать книгу Planning and Executing Credible Experiments - Robert J. Moffat - Страница 63

Now I raise the following five questions.

If we do this experiment and get all the data we have asked for:

 Q1. What question will we have answered?

 Q2. Is this question worth the cost of answering it?

 Q3. Has it already been answered?

 Q4. Is the proposed experiment the best way to answer it?

 Q5. If we get what we asked for, will that solve the problem that led to this work?

Almost always, trying to answer Q1 honestly and carefully puts the issue in a new light.

Once attention is focused on what we want to accomplish instead of on what we want to do, we can admit the possible existence of other ways to accomplish the same objective. This often leads us to formulate a more important question than we first had in mind and to propose a quite different experiment.

It is a good idea to start by formulating several versions of what seems to be the motivating question, and then start critiquing them. The first versions will leave a lot unsaid, and a good “devil's advocate” approach will make some of them look downright naive.² Don't be reluctant to play devil's advocate – if the question allows a silly answer, the question is poorly formulated. Precision will soon start to emerge.

When you have a few good candidates, the test for identifying a really good motivating question is, “Which of these questions, if it alone is answered, will justify the cost of this experiment?”

There can be only one “top priority question.” During the planning of the experiment, when a trade‐off is required, the motivating question provides the decision criterion. The motivating question is the “mission statement” of the experiment.

Sometimes, the question keeps on being refined, even while the program is in motion.³ No harm in that.