Читать книгу Business Experiments with R - B. D. McCullough - Страница 10

Preface

Rigorous experiments are rapidly becoming a critical tool for businesses. While experiments have been a popular tool in engineering for decades, driven largely by the quality movement of the 1980s, experiments are rapidly coming out of the engineering lab to be deployed more broadly across business operations. Companies today are using experiments to solve a broad range of problems from determining whether improved training results in better employee productivity and retention to gauging whether store renovations result in higher profitability, to determining which subject lines in a promotional email attract customer attention, to assessing which behavioral nudges are most likely to get utility users to shift energy consumption to nonpeak times, and to deciding which color to use for the checkout buttons on a website. Articles in the business press such as those by Davenport (2009), Anderson and Simester (2011), and Thomke and Manzi (2014) have made the case that experiments are a key component of “data‐driven” business and companies are taking note and incorporate experiments into their day‐to‐day operations. With this rapid adoption of experimentation, there is a need for a comprehensive resource for business leaders and analysts who want to use experiments to understand how business systems work and solve tactical business problems.

Yet, material on business experiments remains siloed within specific business functions. For example, there are dozens of books and articles on website testing that focus intently on experimentation with websites and never mention that the same basic ideas – randomization, measurement, and comparison – can be applied in other business domains. Our goal for this book is to step back from this highly tactical approach and give business students a broad, theoretically grounded understanding of the role of experiments in business, providing them with a practical foundation for designing and analyzing experiments that will be useful to them throughout their careers.

Of course, experiments have long been a foundational tool in engineering, physical sciences, psychology, medicine, and statistics, and excellent books on how to design and analyze experiments are available within each of these fields, yet none of these books provides the right coverage for business students. The literature on experiments in the physical sciences and engineering provides great depth on multivariable test design, yet often focuses on situations where the experimenter is studying a physical process in a lab setting, where everything can be measured and controlled and error can be reduced nearly to zero – conditions that seldom hold in business. The classic textbook on experimental design for engineers by Montgomery (2017) naturally assumes a level mathematical skill that is not common among business students. Psychologists like Campbell and Stanley (1963) focus more on experiments with human subjects, raising many of the same issues that come up in marketing and human resources experiments, yet they focus intently on using experiments to develop theory and so cover issues like construct validity that are less relevant in the types of tactical experiments that business uses to drive day‐to‐day decisions. Useful discussions of the practicalities of running experiments in the field have appeared recently in the economics literature championed by Harrison and List (2004) and Gerber and Green (2012), yet the experiments reported tend to focus on testing economic theory. In many ways, the types of tactical experiments that are valuable in business are much more similar to clinical trials and other biomedical experiments, where there is a tradition of tactical experiments designed to determine, “Should we give this treatment to patients?” and a well‐established culture of randomized controlled trials as the gold standard of evidence. While much of this work from other disciplines is relevant to business experiments, it doesn't provide students with concrete examples of how to apply these methods to answer important business questions.

This book has drawn from all of these literatures selecting the key ideas that business students need to know about experiments. This book approaches experimentation as a skill that one becomes better at over time, like playing a sport or an instrument, which requires equal measures of creativity and technical skill. To help students practice, the book is structured around a series of examples, focusing first on a specific business question, then turning to the design or analysis of an experiment that then will answer that question, and then building up the appropriate statistical methods. This way, the technical material is presented in context. This approach provides an ideal “second course in statistics,” where students can review material they have seen before (specifically two‐sample tests, power, ANOVA, confidence intervals, and regression) in the context of business problems. My hope is that students who work through this book will move beyond the mechanics and develop their abilities to frame ill‐defined problems, determine what data and analysis would provide information about that problem, and examine the evidence for or against a particular business decision.

For clarity in the use of pronouns, I have adopted the convention that the analyst is always female. Other persons are male.

Suggested Courses Using This Book

This book can be used both with general business students, such as MBA or executive education students, and with the growing number of master's programs in business analytics and marketing analytics.

The first four or five chapters (depending on instructor preference) provides students with the tools they need to design and analyze two‐treatment experiments (i.e. A/B tests) to answer business questions, focusing on the strategic and technical issues involved in designing experiments that will truly affect organizations. The book begins with a discussion of the importance of causal analysis in business and the risks of observational data analysis in Chapter 1. By design, the next chapter begins with the analysis of A/B tests, with a review of the statistical methods required for analyzing experiments, focusing on visualization and hypothesis testing. Once students understand how experiments are analyzed, in Chapter 3, they can move on to understanding issues involved in designing experiments including selecting treatments, measuring responses, planning sample sizes, and the importance of randomization for causal interpretation. Chapter 4 returns to the analysis of A/B tests, covering several advanced analysis techniques including analyzing tests with matched pairs, analyzing tests with more than two treatment groups, analyzing subgroups within an A/B test, and determining the minimal detectable effect for an experiment that has been completed. Chapter 5, which covers issues involved in the design of A/B tests when the number of units available for tests is small, as is the case when doing geo‐testing or testing across store locations. This coverage makes Part I ideal for a half‐semester graduate course for second‐year MBA students or strong undergraduate students. These chapters can be covered over five weeks, skipping some of the more technical material and allowing time for the students to apply the material to a project where they plan, execute, and analyze an A/B test.

Part II builds on the foundation in Part I, expanding to multivariable testing. Chapter 6 begins Part II with a review of the primary tool used to analyze multivariate tests: regression analysis. Next comes full and fractional factorial designs in Chapters 7 and 8, which builds students' understanding of the core issues in multivariate design: interactions and confounding. Fractional factorial designs are useful for building understanding. Next comes custom design, which is ideal for designing experiments in constrained business contexts. The development of software for optimal design has made this sophisticated tool accessible to users, and my goal for Chapter 9 is to provide students with a basic introduction to optimal design, while avoiding the algorithmic details that would be more appropriate to a “technical manual.” Parts I and II together form the core of a full‐semester course that is appropriate for more analytically oriented graduate students including MBA students concentrating in analytics and master's of business analytics students.

Not all experiments are technically complicated. There is a great deal of “low‐hanging fruit” in the business world that can be profitably plucked by persons whose knowledge of experimental design does not exceed 2^k designs. The goal of this book is to enable students who have taken business statistics to pluck that fruit. In particular, after finishing the first half of this book, the student should be able to conduct an A/B test. After finishing the second half, the student should be able to design, execute, and analyze a 2^k experiment. To this end, instructors should require students to complete a project where they design, execute, analyze, and report on an experiment. Students benefit greatly from the experience, which helps them see the challenges involved in designing a conclusive experiment and cement the ideas in the book by applying them in practice. A/B tests are easy to dream up, multivariable tests not so much. Hence, Box's helicopter experiment is very important. It is perhaps the only easily executed multivariable experiment available to students that is amenable to screening designs, full factorial designs, and even custom designs. By the end of this book, the reader should be able to execute Box's helicopter experiment Box (1992a) even if the professor does not make it part of the class.

Philadelphia, PA

B. D. McCullough