Natural experiments in economics

28 Apr, 2021 at 15:35 | Posted in Economics | 1 Comment

Natural Experiments in the Social Sciences: A Design-Based Approach  (Strategies for Social Inquiry): Dunning, Thad:  9781107698000: BooksThad Dunning’s book Natural Experiments in the Social Sciences (CUP 2012) is a very useful guide for economists interested in research methodology in general and natural experiments in specific. Dunning argues that since random or as-if random assignment in natural experiments obviates the need for controlling potential confounders, this kind of “simple and transparent” design-based research method is preferable to more traditional multivariate regression analysis where the controlling only comes in ex post via statistical modelling.

But — there is always a but …

The point of making a randomized experiment is often said to be that it ‘ensures’ that any correlation between a supposed cause and effect indicates a causal relation. This is believed to hold since randomization (allegedly) ensures that a supposed causal variable does not correlate with other variables that may influence the effect.

The problem with that simplistic view on randomization is that the claims made are exaggerated and sometimes even false:

• Even if you manage to do the assignment to treatment and control groups ideally random, the sample selection certainly is — except in extremely rare cases — not random. Even if we make a proper randomized assignment, if we apply the results to a biased sample, there is always the risk that the experimental findings will not apply. What works ‘there,’ does not work ‘here.’ Randomization hence does not ‘guarantee ‘ or ‘ensure’ making the right causal claim. Although randomization may help us rule out certain possible causal claims, randomization per se does not guarantee anything!

• Even if both sampling and assignment are made in an ideal random way, performing standard randomized experiments only give you averages. The problem here is that although we may get an estimate of the ‘true’ average causal effect, this may ‘mask’ important heterogeneous effects of a causal nature. Although we get the right answer of the average causal effect being 0, those who are ‘treated’  may have causal effects equal to -100 and those ‘not treated’ may have causal effects equal to 100. Contemplating being treated or not, most people would probably be interested in knowing about this underlying heterogeneity and would not consider the average effect particularly enlightening.

• There is almost always a trade-off between bias and precision. In real-world settings, a little bias often does not overtrump greater precision. And — most importantly — in case we have a population with sizeable heterogeneity, the average treatment effect of the sample may differ substantially from the average treatment effect in the population. If so, the value of any extrapolating inferences made from trial samples to other populations is highly questionable.

• Since most real-world experiments and trials build on performing a single randomization, what would happen if you kept on randomizing forever, does not help you to ‘ensure’ or ‘guarantee’ that you do not make false causal conclusions in the one particular randomized experiment you actually do perform. It is indeed difficult to see why thinking about what you know you will never do, would make you happy about what you actually do.

• And then there is also the problem that ‘Nature’ may not always supply us with the random experiments we are most interested in. If we are interested in X, why should we study Y only because design dictates that? Method should never be prioritized over substance!

Randomization is not a panacea. It is not the best method for all questions and circumstances. Proponents of randomization make claims about its ability to deliver causal knowledge that are simply wrong. There are good reasons to be sceptical of the now popular — and ill-informed — view that randomization is the only valid and the best method on the market. It is not.

1 Comment

  1. It is fascinating and horrifying to see how various mathematical concepts used in economics, like ‘averages’, assume an imperious, privileged and powerful status that is much worse than merely explanatory.
    The concept of average displays the greatly embellished power to impose economic mathematical constructions onto social operations – peoples lives. The concept is elevated to such primacy that it displaces the real world of people, most of whom are struggling to make ends meet. ‘Average’ is then used to dismiss economic criticisms, to assume legitimacy as fulsome explanation and justification, to institutionalize an arbitrary mathematical concept as appropriate for social coercion and to impose an elite, anti-democratic mythology onto the struggles of millions. If a government ‘reduces’ poverty through changing an economic average, instead of vilifying and ending the political economics that creates poverty, it becomes a ‘legitimate’ metric that perpetuates more poverty, well… because the numbers say so, and the numbers are given primary power!
    Changes in the ‘average’ then become established as a tradition that has magical, vested power to explain away and ignore suffering while further empowering the establishment privileges of elite economic operators. If the average goes up or down, depending on the arbitrary frames of reference, elite economic actors ‘adjust’ the imposed economic machinery to maintain classical numbers within the tradition of a hegemonic and autocratic mythology.
    The assumption of this exclusive, undemocratic power for mathematical economic concepts is the fundamental corruption of our politics and economics. The tail wags the dog when governance is operated by classical economic constructions. Instead of economics serving democracy, economics undermines democracy with a pervasive ideological superstition and fetish.
    Unwinding the ball of yarn twisted into a fully resistant, self-replicating, elite construction is the challenge of every age that has faced the imposition of a coercive, unjust and undemocratic social dominance based upon accumulations of wealth by powerful, minority elites.

Sorry, the comment form is closed at this time.

Blog at
Entries and Comments feeds.