What do RCTs reveal about causality?

28 Oct, 2019 at 12:26 | Posted in Economics | 4 Comments

rct-gold-standardThe insight critique contested the proposition that RCTs had revealed significant new facts or provided new understanding of development processes … But closer inspection reveals that they most often merely provide a validation of common sense. Whereas at times randomization seemed to reveal something surprising … in other instances it simply told us what had been long expected … One such finding—that providing preventative public health treatments at low or no cost, or better yet with incentives, leads to an increase in the number of people willing to accept them—is cited by the prize committee as having led to a change in the received wisdom in favor of user fees in primary health. This gets the history quite wrong, since such fees had long before that lost favor …

RCTs cannot reveal very much about causal processes since at their core they are designed to determine whether something has an effect, not how. The randomistas have attempted to deal with this charge by designing studies to interpret whether variations in the treatment have different effects, but this requires a prior conception of what the causal mechanisms are. The lack of understanding of causation can limit the value of any insights derived from RCTs in understanding economic life or in designing further policies and interventions. Ultimately, the randomistas tested what they thought was worth testing, and this revealed their own preoccupations and suppositions, contrary to the notion that they spent countless hours listening to and in close contact with the poor …

If RCTs now “entirely dominate” development economics, or worse, provide the basis for development policymaking, that is no cause for celebration. The roaring success of the randomistas tells us most of all about the historical moment in which they came to prominence: one in which defeatism or cynicism about public initiatives on a larger scale has been replaced by a focus on what works at the level of individuals and communities. But even there, what does work, really, remains an open question. The difficult question of how to fix broken institutions and help societies function better requires going beyond a biomedical metaphor of taking the right pill. Nobel or not, the debate must continue.

Sanjay Reddy

The problem many ‘randomistas’ — like this year’s ‘Nobel prize’ winners in economics; Duflo, Banerjee and Kremer — end up with when underestimating heterogeneity and interaction is not only an external validity problem when trying to ‘export’ regression results to different times or different target populations. It is also often an internal problem to the millions of regression estimates that economists produce every year.

‘Ideally controlled experiments’ tell us with certainty what causes what effects — but only given the right ‘closures.’ Making appropriate extrapolations from (ideal, accidental, natural or quasi) experiments to different settings, populations or target systems, is not easy. ‘It works there’ is no evidence for ‘it will work here.’ Causes deduced in an experimental setting still have to show that they come with an export-warrant to the target population/system. The causal background assumptions made have to be justified, and without licenses to export, the value of ‘rigorous’ and ‘precise’ methods — and ‘on-average-knowledge’ — is despairingly small.

RCTs have very little reach beyond giving descriptions of what has happened in the past. From the perspective of the future and for policy purposes they are as a rule of limited value since they cannot tell us what background factors were held constant when the trial intervention was being made.

RCTs usually do not provide evidence that the results are exportable to other target systems. RCTs cannot be taken for granted to give generalizable results. That something works somewhere for someone is no warranty for us to believe it to work for us here or even that it works generally.

4 Comments

  1. You may be interested in the review of Banerjee & Dufflo’s Poor Economics by one of the respected development macroeconomist AP Thirlwall. Here are some excerpts.

    “The book has received much adulation and many prizes, and there is no doubt that the detail is fascinating, but the question remains: will it radically change attitudes and policies towards poverty reduction in poor countries? This is where doubts start to creep in. Apart from the fact that the results of many of the RCTs could have been predicted in advance, there are well-known limitations of such trials. First they don’t address the general equilibrium effects, or secondary repercussions, of particular programmes. Secondly, the evaluation of a trial itself may cause both the treatment and comparison groups to alter their behaviour for the period of the experiment, leading to false inferences. Thirdly, it may be difficult to generalise the results of RCTs because they are context specific. ….
    As one approaches the end of the book, full of experimental results, case studies, and anecdotes, it is still not entirely clear what works and what doesn’t work to lift people out of poverty, and what the policy implications are. And can changes be implemented within the existing institutional frameworks of countries? The last chapter gives the opposing views of William Easterly and Jeffrey Sachs on these issues. Easterly is a well-known critic of RCTs arguing that they are infeasible for many of the big questions in development economics, such as the economy-wide effects of good institutions or good macroeconomic policies. Rodrik (2012) argues in the same vein that micro anti-poverty programmes often treat the symptoms of poverty rather than their causes, and that poverty is best addressed not by helping the poor to be better at what they are already doing but by getting them to do something different. This may require policy intervention by governments at the macro, rather than micro, level to encourage, for example, the diversification of production. Micro-development economists, he says, ‘need to recognise – – – that while randomised evaluations are tremendously useful, the utility of their results is often restricted by the narrow scope of their application’….
    What is still missing from the book, however, is a summary at the end, for each of the topics discussed, of the way forward. Which anti-poverty policies are most effective and which policies should be discarded?”

    For the full review, see https://www.kent.ac.uk/economics/documents/tt/Poor-economics-review-article.pdf

    • Thanks. Insightful.

  2. Oxfam : “The Randomistas just won the Nobel Economics prize. Here’s why RCTs aren’t a magic bullet.”

    “Lant Pritchett once likened Randomized Controlled Trials (RCTs) to flared jeans. On the way out and soon we’d be wondering what on earth we’d seen in them.

    Not so fast. Yesterday, three of the leading ‘Randomistas’ won the Nobel economics prize (before the pedants jump in, strictly speaking it’s the Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel). Congrats to Esther Duflo, who becomes only the second woman to win the prize (yes really, the other was Elinor Ostrom), with Abhijit Banerjee and Michael Kremer.

    There will be lots of plaudits in the press, and a fair amount of magic-bulletism about RCTs as the alleged ‘gold standard’ for evidence of what works in development. So, channelling Pulp Fiction, ‘allow me to retort’. For the ‘bah humbug’ corner, here (in reverse chronological order) are some FP2P links to various sceptical views of RCTs going back several years.Quote: ‘RCTs need to acknowledge the central role of human agency in enabling or thwarting project objectives at every stage of the processes they study. It is unlikely they will be able to do this by confining themselves to quantitative methods alone.’

    Stefan Dercon (2018) on Duflo and Banerjee: ‘Everything has to be inductive, experimental. Lots of little solutions will move us forward. They have no big theory of what causes low growth, no big questions, just ‘a technocratic agenda of fixing small market failures’. Getting institutions right is not crucial – we can do lots of bad policies in good institutional settings, and lots of good policies in bad institutional settings.

    Lant Pritchett v the Randomistas on the nature of evidence – is a wonkwar brewing? (2012)
    Quote: ‘‘RCTs are a tool to cut funding, not to increase learning.’ ‘Randomization is a weapon of the weak’ – a sign of how politically vulnerable the argument for aid has become since the end of the Cold War. ‘Henry Kissinger wouldn’t have demanded an RCT before approving aid to some country.’

    And a quote from me in that blog: ‘On one side are the ‘best fit’ institutionalists and complexity people, with their focus on path dependence, evolution and trial and error. On the other are the ‘universal law’ experimentalists, offering the illusory certainty of numbers, and (crucially) comfort to the political paymasters seeking to prove to sceptical publics that aid works. It’s hard to see how they can both be right, or happily coexist for long.’
    On “Poor Economics – a rich new book from Abhijit Banerjee and Esther Duflo (2011)
    Quote: ‘But is it a Big Book? Yes in terms of the approach – I think it will leave a lasting impact on its readers in showing the merits of a bottom-up, evidence-driven approach. But not, I think, in terms of content – lots of interesting, surprising facts and analyses, but no one big message. Given their suspicion of grand narratives, I’m sure the authors would be quite happy with that.’

    Well, I got that wrong…..

    And here’s another economics Nobel, Angus Deaton, on why RCTs aren’t all they’re cracked up to be 2017 paper with Nancy Cartwright here.

    In my experience, those at the centre of a hype cycle are often highly aware of the limitations and weaknesses of their product. It’s the acolytes and spin doctors who remove all the caveats and nuances, and a magic bullet is born. Let’s hope we can generate a better discussion about the strengths and weaknesses of RCTs in response to the Nobel prize.”
    https://oxfamblogs.org/fp2p/the-randomistas-just-won-the-nobel-economics-prize-heres-why-rcts-arent-a-magic-bullet/

  3. Oxfam-” How did the Randomistas get so good at influencing Policy?”
    https://oxfamblogs.org/fp2p/how-did-the-randomistas-get-so-good-at-influencing-policy/


Sorry, the comment form is closed at this time.

Blog at WordPress.com.
Entries and comments feeds.