Randomization is no panacea

21 Mar, 2013 at 16:49 | Posted in Statistics & Econometrics, Theory of Science & Methodology | Comments Off on Randomization is no panacea

When it comes to questions of causality, randomized controlled trials (RCTs) are nowadays considered some kind of “gold standard” in social sciences and policies. Everything has to be “evidence based,” and the evidence preferably has to come from randomized experiments.

But randomization is basically – just as e. g. econometrics – a deductive method. Given warranted assumptions (manipulability, transitivity, separability, additivity, linearity etc)  this method delivers deductive inferences. The problem, of course, is that we will never completely know when the assumptions are warranted and a fortiori being able to justify our causal conclusions. Although randomization may contribute to controlling for “confounding,” it does not guarantee it, since genuine ramdomness presupposes infinite experimentation and we know all real experimentation is finite. Even if randomization may help to establish average causal effects, it says nothing of individual effects unless homogeneity is added to the list of assumptions. Real target systems are seldom epistemically isomorphic to our axiomatic-deductive models/systems, and even if they were, we still have to argue for the external validity of  the conclusions reached from within these epistemically convenient models/systems. Causal evidence generated by randomization procedures may be valid in “closed” models, but what we usually are interested in, is causal evidence in the real target system we happen to live in.

So RCTs are not at all the “gold standard” they have lately often been portrayed as. RCTs usually do not provide evidence that their results are exportable to other target systems. RCTs cannot be taken for granted to give generalizable results. That something works somewhere is no warranty for it to work for us or even that it works generally.

Even though I can present evidence for being able to sharpen my pencils with Rube Goldberg’s ingenious construction – mainly becuase flying kites in my windy hometown (Lund, Sweden) is no match – it does not come with a warranted export license. Most people would probably find ordinary pencil sharpeners more efficacious.

Blog at WordPress.com.
Entries and comments feeds.