Lessons from the Reinhart-Rogoff debacle

17 Apr, 2013 at 16:53 | Posted in Economics | 4 Comments

open access1) We should always have the data and code for published results …

2) The data and code should be open source …

3) These mistakes are easy to make and they’re mostly not considered mistakes.

Let’s talk about the “mistakes” the authors found. First, they’re excluding certain time periods for certain countries, specifically right after World War II. Second, they chose certain “non-standard” weightings for the various countries they considered. Finally, they accidentally excluded certain rows from their calculation.

Only that last one is considered a mistake by modelers. The others are modeling choices, and they happen all the time. Indeed it’s impossible not to make such choices. Who’s to say that you have to use standard country weightings? Why? How much data do you actually need to consider? Why? …

4) We need to actually consider other modeling possibilities.

It’s not a surprise, to economists anyway, that after you include more post-WWII years of data, which we all know to be high debt and high growth years worldwide, you get a substantively different answer. Excluding these data points is just as much a political decision as a modeling decision.

In the end the only reasonable way to proceed is to describe your choices, and your reasoning, and the result, but also consider other “reasonable” choices and report the results there too. And if you don’t like the answer, or don’t want to do the work, at the very least you need to provide your code and data and let other people check how your result changes with different “reasonable” choices.

Once the community of economists (and other data-centric fields) starts doing this, we will all realize that our so-called “objective results” utterly depend on such modeling decisions, and are about as variable as our own opinions.

5) And this is an easy model.

Think about how many modeling decisions and errors are in more complicated models!

Cathy O’Neil


  1. Ahhh, kvantitativ samhällsvetenskap, denna härliga pseudovetenskap.

    Chasing shadows.

    Vad du gör när du tvingar något multidimensionellt ned i en enda variabel, det är att projicera det multivariata objektet ned på ett plan. Det är skuggan du bildar på en yta.

    Samhällsvetarna tror att de har blicken riktad emot objektet och ser dess form, men de har blicken riktad ifrån objektet emot skuggan på planet, dessutom befinner de sig mellan objektet och dess skugga, utöver detta så kan de heller aldrig låta bli att stoppa sina feta huvuden ivägen för projiceringen. När samhällsvetarna gör kvantitativ samhällsvetenskap så ägnar de sig således bara åt ett skuggspel.

    När de gör sån där “vetenskap”, där de visar ett samband eller trend, så säger de: Om jag tittar på verkligheten, i precis rätt vinkel, om jag håller upp mina armar och skuggar verkligheten på precis det här sättet, då ser det ut som ett samband.

    När man sedan bygger “modeller” och inför styrsystem i samhället baserad på dessa kvantitativa slutsatserna, då försöker man pressa in det N-dimensionella objektet (verkligheten) i ett (N-n) dimensionellt utrymme. De flesta andra lär sig redan på dagis att inte försöka trycka i den fyrkantiga klossen i det runda hålet.

  2. The Colbert Report 04/23/13 Stephen Colbert interviews Thomas Herndon The Colbert Report 04/23/13 Stephen Colbert interviews Thomas Herndon that debunked Carmen Reinhart and Kenneth Rogoff’s 2010 debt study

  3. Stephen Colbert discusses and mocks the problems resulting from basic mistakes made in a study by Reinhart and Rogoff. which excluded relevant data from: Canada, Australia etc. https://www.youtube.com/watch?v=2Oa-Bfdkg3w

Sorry, the comment form is closed at this time.

Blog at WordPress.com.
Entries and Comments feeds.