Читать книгу Planning and Executing Credible Experiments - Robert J. Moffat - Страница 56

3 An Overview of Experiment Planning

My background is mainly in research and development experiments in heat transfer and fluid physics. When I think of planning an experiment, I think about wind tunnels, heat exchangers, temperature, and flow control. I have tried to generalize my experience, but my background certainly colors my outlook.

Experimental work is expensive. Although costs vary for different fields and situations, the time costs are all similar: all laboratory work runs in real time, an hour for an hour, and there are no short cuts. It is important that experiments be well conceived, well executed, and well documented.

The purpose of experiments is to produce provably accurate data that answer an agreed‐upon question about the behavior of a system. This is the kind of experiment I wish to discuss. Some of the important ideas will be transferable to other types of experiment, but the main thrust here is to deal with R&D experiments with tangible, numerical objectives.

The planning of such an experiment is necessarily iterative. The process starts with a tentative plan: a first impression of the goal, a plausible experimental approach, a possible suite of instruments, a tentative set of tests to run. This plan must then be challenged: will it produce the desired information with acceptable accuracy? Then the plan is refined, sometimes by improving the statement of objectives, sometimes by selecting a better approach, sometimes by improving the instrumentation.

3.1 Steps in an Experimental Plan

The general steps in an experimental program can be summarized in the following outline:

1 Assignment.

The Iterative Loop

1 Determine the objectives.

2 Select the experimental approach.

3 Parametrically design an apparatus.

4 Design apparatus hardware.

5 Construct and install apparatus.

6 Design analysis software; debug with fabricated sample data.

7 Perform shakedown, debugging, and qualification runs.

The Execution

1 Collect data.

2 Reduce data and analyze.

3 Report.

Seldom is a successful research experiment designed on a once‐through basis. This is not surprising when you think about the amount of scratch paper usually generated in trying to develop an original analysis. Alternative experimental approaches must be investigated and trade‐offs made between accuracy and convenience, range and speed, etc.

3.2 Iteration and Refinement

Research echoes sailing into uncharted territory. Sailboats cannot sail directly into the wind. The skipper must tack into the wind, repeatedly aiming right then left, iteratively correcting course.

This book takes an iterative approach to experiment planning. For example, it may not be clear how to execute some of the steps until later, when their background material has been developed. Some of the steps themselves may not even make sense until the background material has been developed. And some of the background material we ask for won't make sense until the need for it has been established. This circularity is typical of large‐scale projects with interactions: They cannot be studied sequentially, they have to be approached integrating “all at once” and iterating.

We emphasize the iterative nature of experiments. What seems plausible at first may not prove acceptable later. Sometimes a preliminary “exploratory” experiment is a good investment – to test an approach. It may show that the original concept of the experiment is not a good one. You likely will iterate the experiment itself, as in Figure 1.1. Details of the planning steps and their objectives are dealt with in subsequent chapters.

An additional issue of risk assessment warrants early treatment because it affects every decision in the planning process.

3.3 Risk Assessment/Risk Abatement

Consider the following risks:

 The data may not answer the motivating question.

 The data may not be provably accurate.

 The schedule may not be met.

 The budget may be exceeded.

Before the experiment is authorized to proceed, it must be established to the satisfaction of the reviewers that the motivating question can be answered with the desired accuracy, within the time allowed, and within the allowable budget.

3.4 Questions to Guide Planning of an Experiment

Tables 3.1–3.3 elaborate the contents for the rest of this book – the topics that we feel should be addressed during the planning of an experiment.

Before some of the above topics can be addressed directly, some background material must be developed on the topology of experiments and the handling of experimental uncertainties.

Table 3.1 Overview of a research experiment plan.

Set up the experiment log.	Keep a detailed log of your decisions.
Identify:The motivating question.The form of an acceptable answer.The allowable uncertainty.	What question are you trying to answer? What should the answer look like? What accuracy do you need?
Design the data interpretation program (DIP).	What equations provide the answer?
Specify the data you need.	Output data, peripheral data, and control values.
Establish the allowable uncertainties.	How accurately must variables be measured in order to get useful results?
Select the instruments.	Cross‐check with required uncertainties.
Specify the operable domain.	What range of conditions must be covered?
Estimate the shape of the response surface.	What will be the likely outcome?
Select the data trajectories and data‐density distribution.	How should the data points be distributed over the operating surface?
Design the hardware.	The apparatus must create the desired domain.

Table 3.2 Review the program plan. Do risk assessment and plan risk abatement. If satisfactory, go ahead. If not, go back.

Build apparatus.	Watch schedule and cost. Track critical path and critical person lines.
Write the DIP.	Convert equations and measurable inputs to selected response variables. Debug program.
Shake down apparatus.	First, make it repeatable. Then, make it work. Finally, make it work well.
Execute qualification runs and document credibility.	Calibrate. Document uniformity and stability. Certify baseline data.

Table 3.3 Assess the credibility of the program. Do risk assessment and plan risk abatement. If satisfactory, go ahead. If not, go back.

Take production data.	These are the required results.
Interpret the results.	What do the results mean to the client or target audience?
Document the experiment.	Present the results and interpret them. Record the data that support these conclusions and establish credibility of the experiment.

Homework

1 3.1 From your review of prior work and research, what risks might your experiment encounter?

2 3.2 What risks will your client accept?

3 3.3 What risks will your client face by not pursuing this experiment?

4 3.4 How much will your client stake on successful completion of this experiment?

5 3.5 What is the cost of a null answer?

6 3.6 How much risk will your client assume?

7 3.7 Do the Occupational Safety and Health Administration (OSHA) or state laws or national codes limit the direction and extent that your proposed experiment can pursue?

8 3.8 Rinse and repeat from Exercise 3.1.