RCTs - not so much of a "gold standard" after all

14 February, 2013 at 08:51 | Posted in Statistics & Econometrics | 1 Comment

Controlled experiments are the gold standard in science for proving causality. The FDA, for example, requires controlled experiments (randomized clinical trials) for approving drugs. In the software world, online controlled experiments are being used heavily to make data-driven decisions, especially in areas where the forefront of knowledge is being pushed …

The statistical theory of controlled experiments is well understood, but the devil is in the details and the difference between theory and practice is greater in practice than in theory. We have shared five puzzling experiment outcomes, which we were able to analyze deeply and explain …

Generalizing from these puzzles, we see two themes. One is that instrumentation is not as precise as we would like it to be, interacting in subtle ways with experiments …A second theme is that lessons from offline experiments don’t always map well online …

Anyone can run online controlled experiments and generate numbers with six digits after the decimal point. It’s easy to generate p-values and beautiful 3D graphs of trends over time. But the real challenge is in understanding when the results are invalid, not at the sixth decimal place, but before the decimal point, or even at the plus/minus for the percent effect; that’s what these analyses did to the initial results. We hope we’ve managed to shed light on puzzling outcomes and we encourage others to drill deep and share other similar results. Generating numbers is easy; generating numbers you should trust is hard!

Ron Kohavi et al.

h/t Andrew Gelman

About these ads