RCT — a questionable claim of establishing causality

16 Jul, 2021 at 11:33 | Posted in Statistics & Econometrics | 2 Comments

Amazon.com: Randomized Control Trials in the Field of Development: A  Critical Perspective eBook: Bédécarrats, Florent, Guérin, Isabelle,  Roubaud, François: Kindle StoreThe ideal RCT is the special case in which the trial’s treatment status is also assigned randomly (in addition to drawing random samples from the two populations, one treated and one not) and the only error is due to sampling variability … In this special case, as the number of trials increases, the mean of the trial estimates tends to get closer to the true mean impact. This is the sense in which an ideal RCT is said to be unbiased, namely that the sampling error is driven to zero in expectation …

Prominent randomistas have sometimes left out the “in expectation” qualifier, or ignored its implications for the existence of experimental errors. These advocates of RCTs attribute any difference in mean outcomes between the treatment and control samples to the intervention … Many people in the development community now think that any measured difference between the treatment and control groups in an RCT is attributable to the treatment. It is not; even the ideal RCT has some unknown error.

A rare but instructive case is when there is no treatment. Absent any other effects of assignment (such as from monitoring), the impact is zero. Yet the random error in one trial can still yield a non-zero mean impact from an RCT. An example is an RCT in Denmark in which 860 elderly people were randomly and unknowingly divided into treatment and control groups prior to an 18-month period without any actual intervention (Vass, 2010). A statistically significant (prob. = 0.003) difference in mortality rates emerged at the end of the period.

Martin Ravaillon

2 Comments

  1. Shouldn’t we (re)read Fischer Black’s “Noise”?
    .
    “Perhaps most important, research will be seen as a process leading to reliable and relevant conclusions only very rarely, because of the noise that creeps in at every step.”

    • Absolutely 🙂


Sorry, the comment form is closed at this time.

Blog at WordPress.com.
Entries and Comments feeds.