RCTs — the danger of imposing a hierarchy of evidence

28 Dec, 2019 at 14:37 | Posted in Theory of Science & Methodology | Comments Off on RCTs — the danger of imposing a hierarchy of evidence

The imposition of a hierarchy of evidence is both dangerous and unscientific. Dangerous because it automatically discards evidence that may need to be considered, evidence that might be critical. Evidence from an RCT gets counted even if when the population it covers is very different from the population where it is to be used, if it has only a handful of observations, if many subjects dropped out or refused to accept their assignments, or if there is no blinding and knowing you are in the experiment can be expected to change the outcome. Discounting trials for these flaws makes sense, but doesn’t help if it excludes more informative non-randomized evidence. By the hierarchy, evidence without randomization is no evidence at all, or at least is not “rigorous” evidence. An observational study is discarded even if it is well-designed, has no clear source of bias, and uses a very large sample of relevant people.

Angus Deaton