Econometrics in the 21st century

22 Nov, 2020 at 19:05 | Posted in Statistics & Econometrics | 6 Comments


If you don’t have time to listen to all of the presentations (a couple of them are actually quite uninformative) you should at least scroll forward to 1:39:25 and listen to what Angus Deaton has to say. As so often, he is spot on!

As Deaton notes, evidence-based theories and policies are highly valued nowadays. Randomization is supposed to control for bias from unknown confounders. The received opinion is that evidence based on randomized experiments, therefore, is the best.

More and more economists and econometricians have also lately come to advocate randomization as the principal method for ensuring being able to make valid causal inferences.

Yours truly would however rather argue that randomization, just as econometrics, promises more than it can deliver, basically because it requires assumptions that in practice are not possible to maintain. Just as econometrics, randomization is basically a deductive method. Given the assumptions (such as manipulability, transitivity, separability, additivity, linearity, etc.) these methods deliver deductive inferences. The problem, of course, is that we will never completely know when the assumptions are right. And although randomization may contribute to controlling for confounding, it does not guarantee it, since genuine randomness presupposes infinite experimentation and we know all real experimentation is finite. And even if randomization may help to establish average causal effects, it says nothing of individual effects unless homogeneity is added to the list of assumptions. Real target systems are seldom epistemically isomorphic to our axiomatic-deductive models/systems, and even if they were, we still have to argue for the external validity of the conclusions reached from within these epistemically convenient models/systems. Causal evidence generated by randomization procedures may be valid in ‘closed’ models, but what we usually are interested in, is causal evidence in the real target system we happen to live in.

The point of making a randomized experiment is often said to be that it ‘ensures’ that any correlation between a supposed cause and effect indicates a causal relation. This is believed to hold since randomization (allegedly) ensures that a supposed causal variable does not correlate with other variables that may influence the effect.

The problem with that simplistic view on randomization is that the claims made are both exaggerated and false:

• Even if you manage to do the assignment to treatment and control groups ideally random, the sample selection certainly is — except in extremely rare cases — not random. Even if we make a proper randomized assignment, if we apply the results to a biased sample, there is always the risk that the experimental findings will not apply. What works ‘there,’ does not work ‘here.’ Randomization hence does not ‘guarantee ‘ or ‘ensure’ making the right causal claim. Although randomization may help us rule out certain possible causal claims, randomization per se does not guarantee anything!

• Even if both sampling and assignment are made in an ideal random way, performing standard randomized experiments only give you averages. The problem here is that although we may get an estimate of the ‘true’ average causal effect, this may ‘mask’ important heterogeneous effects of a causal nature. Although we get the right answer of the average causal effect being 0, those who are ‘treated’ may have causal effects equal to -100, and those ‘not treated’ may have causal effects equal to 100. Contemplating being treated or not, most people would probably be interested in knowing about this underlying heterogeneity and would not consider the average effect particularly enlightening.

• There is almost always a trade-off between bias and precision. In real-world settings, a little bias often does not overtrump greater precision. And — most importantly — in case we have a population with sizeable heterogeneity, the average treatment effect of the sample may differ substantially from the average treatment effect in the population. If so, the value of any extrapolating inferences made from trial samples to other populations is highly questionable.

• Since most real-world experiments and trials build on performing one single randomization, what would happen if you kept on randomizing forever, does not help you to ‘ensure’ or ‘guarantee’ that you do not make false causal conclusions in the one particular randomized experiment you actually do perform. It is indeed difficult to see why thinking about what you know you will never do, would make you happy about what you actually do.

Randomization is not a panacea. It is not the best method for all questions and circumstances. Proponents of randomization make claims about its ability to deliver causal knowledge that is simply wrong. There are good reasons to be skeptical of the now popular — and ill-informed — view that randomization is the only valid and best method on the market. It is not.


  1. All of us readers of this blog are greatly indebted to Prof. Syll for his regular, very erudite, challenging and stimulating posts, and for his enlightened tolerance of comments even those which are hostile or off-topic.

    So I must appologise for own comments often being too harsh and even rude according to the following 10 “rules” for netiquette in on-line discussions:

    All caps means that you’re yelling.

    Stay on topic.

    Only post links to your own stuff in signature lines or posts inviting you to do so.

    Practice respectful disagreement, not personal attacks.

    Try to post something that adds value to a conversation (“me, too” doesn’t add value).

    Ignore trolls.

    Be brief and don’t turn every comment into your own personal blog post.

    To avoid redundancy, read previous comments before posting your own.

    It’s not all about you so avoid making every comment a testimony as to how awesome you are.

    You’re a guest, so keep this in mind when commenting.

  2. Prof. Syll is correct in saying that econometrics can’t guarantee correct conclusions from empirical data.
    However, it is a category error to think that this is valid criticism of econometrics.
    Certainty may be possible in deductive logic and maths, but it is not achievable in empirical work.
    Unlike philosophers, econometricians are realistic in seeking “maximum likelihood” etc, not absolute certainty.
    After this blunder, Prof. Syll asserts (in his last sentence) that econometrics/randomisation is not “the only valid and best method on the market”. But he makes no attempt to indicate which methods might be superior.
    Is he claiming he knows a method which guarantees certain knowledge?

    • Consider an options market. You can buy and sell opposite positions in an underlying stock, for a fraction of the price of the stock itself. The profit curve is well-defined. You labor by setting stops or actively monitoring the price movements and getting out of the positions while still in the profit section of the curve. It’s as close to certain knowledge as finance gets, so far, I think, and it is close enough to provide handsome returns when done in volume.
      Because you can hedge options with higher derivatives (delta hedging, gamma hedging, and the like), everyone can probably book a profit off the same series of trades. Speculators exist, but may not be necessary. Money emerges from financial markets.
      Central banks provide a backstop, implicitly insuring against default risk, panic devaluing risk, funding price spike risk. The backstop is so implicit that it sometimes appears explicitly in models from big financial institutional research departments.

      • Robert,
        We should all mortgage our homes to the hilt and open massive derivative positions and wait for the guaranteed big money to roll in.
        I think I’ll start looking over some yacht brochures.

        • Better vision: Give everyone a generous, inflation-proofed basic income. If you want extra yacht money, you can play in financial markets. Or continue business-as-usual, which shouldn’t change significantly. We should disqualify economic predictions of dire, dismal consequences for all the reasons Lars Syll keeps pointing out …

        • Robert,
          “If you want extra yacht money, you can play in financial markets.”
          I have “played” financial markets for 45 years, professionally and personally.
          It’s not quite as straight forward as you make out.

Sorry, the comment form is closed at this time.

Blog at
Entries and Comments feeds.