Читать книгу Business Experiments with R - B. D. McCullough - Страница 30

From the previous examples, we can list the four steps of an experiment:

1 Randomly divide the subjects (e.g. customers) into groups. The researcher does not allow the subjects to pick the group – that would be a form of self‐selection that makes the data observational rather than experimental. Moreover, this division is made randomly – the researcher doesn't assign the groups for that, too, would be a form of selection that would render the data observational.

2 Expose each group to a different treatment. The researcher does not allow the subjects to choose which treatment to receive; this assignment has to made randomly.

3 Measure a response for each group. The outcome of interest has to be chosen before the experiment is conducted, and the method for performing the measurement has to be decided in advance, too.

4 Compare group responses to determine which treatment is better. This is accomplished with a statistical test. It can be something as simple as a two‐sample test of means. Whatever test is applied, the test will tell us whether the difference between the groups could be just random or is more likely due to some systematic effect.

We know that observational data can only give correlations and they can't give causation. When we have a well‐designed experiment, we can answer causal questions: Does changing cause a change in ? Experiments are a systematic way to avoid lurking variables and are the gold standard for causal inference. Thomke Thomke (2020, chapter 2) describes an example of this phenomenon:

[Yahoo did a study] to assess whether display ads for a brand, shown on Yahoo sites, can increase searches for the brand name or related keywords. The observational part of the study estimated that ads increased the number of searches by 871 percent to 1,198 percent. But when Yahoo ran a controlled experiment, the increase was only 5.4 percent. If not for the control, the company might have concluded that the ads had a huge impact and wouldn't have realized that the increases in searches was due to other variables that changed during the observation period.