Three suggestions to ‘save’ econometrics

29 Nov, 2016 at 11:33 | Posted in Economics, Statistics & Econometrics | 6 Comments

Reading an applied econometrics paper could leave you with the impression that the economist (or any social science researcher) first formulated a theory, then built an empirical test based on the theory, then tested the theory. But in my experience what generally happens is more like the opposite: with some loose ideas in mind, the econometrician runs a lot of different regressions until they get something that looks plausible, then tries to fit it into a theory (existing or new) … Statistical theory itself tells us that if you do this for long enough, you will eventually find something plausible by pure chance!

0This is bad news because as tempting as that final, pristine looking causal effect is, readers have no way of knowing how it was arrived at. There are several ways I’ve seen to guard against this:

(1) Use a multitude of empirical specifications to test the robustness of the causal links, and pick the one with the best predictive power …

(2) Have researchers submit their paper for peer review before they carry out the empirical work, detailing the theory they want to test, why it matters and how they’re going to do it. Reasons for inevitable deviations from the research plan should be explained clearly in an appendix by the authors and (re-)approved by referees.

(3) Insist that the paper be replicated. Firstly, by having the authors submit their data and code and seeing if referees can replicate it (think this is a low bar? Most empirical research in ‘top’ economics journals can’t even manage it). Secondly — in the truer sense of replication — wait until someone else, with another dataset or method, gets the same findings in at least a qualitative sense. The latter might be too much to ask of researchers for each paper, but it is a good thing to have in mind as a reader before you are convinced by a finding.

All three of these should, in my opinion, be a prerequisite for research that uses econometrics …

Naturally, this would result in a lot more null findings and probably a lot less research. Perhaps it would also result in fewer attempts at papers which attempt to tell the entire story: that is, which go all the way from building a new model to finding (surprise!) that even the most rigorous empirical methods support it.

Unlearning Economics

Good suggestions, but unfortunately there are many more deep problems with econometrics that have to be ‘solved.’

In econometrics one often gets the feeling that many of its practitioners think of it as a kind of automatic inferential machine: input data and out comes causal knowledge. This is like pulling a rabbit from a hat. Great — but first you have to put the rabbit in the hat. And this is where assumptions come in to the picture. The assumption of imaginary ‘superpopulations’ is one of the many dubious assumptions used in modern econometrics.

Misapplication of inferential statistics to non-inferential situations is a non-starter for doing proper science. And when choosing which models to use in our analyses, we cannot get around the fact that the evaluation of our hypotheses, explanations, and predictions cannot be made without reference to a specific statistical model or framework. The probabilistic-statistical inferences we make from our samples decisively depends on what population we choose to refer to. The reference class problem shows that there usually are many such populations to choose from, and that the one we choose decides which probabilities we come up with and a fortiori which predictions we make. Not consciously contemplating the relativity effects this choice of ‘nomological-statistical machines’ have, is probably one of the reasons econometricians have a false sense of the amount of uncertainty that really afflicts their models.

As economists and econometricians we have to confront the all-important question of how to handle uncertainty and randomness. Should we define randomness with probability? If we do, we have to accept that to speak of randomness we also have to presuppose the existence of nomological probability machines, since probabilities cannot be spoken of – and actually, to be strict, do not at all exist – without specifying such system-contexts. Accepting Haavelmo’s domain of probability theory and sample space of infinite populations – just as Fisher’s ‘hypothetical infinite population,’ von Mises’s ‘collective’ or Gibbs’s ‘ensemble’ – also implies that judgments are made on the basis of observations that are actually never made! Infinitely repeated trials or samplings never take place in the real world. So that cannot be a sound inductive basis for a science with aspirations of explaining real-world socio-economic processes, structures or events. It’s not tenable.

Economists — and econometricians — have (uncritically and often without arguments) come to simply assume that one can apply probability distributions from statistical theory on their own area of research. However, there are fundamental problems arising when you try to apply statistical models outside overly simple nomological machines like coin tossing and roulette wheels.

Of course one could arguably treat our observational or experimental data as random samples from real populations. But probabilistic econometrics does not content itself with that kind of populations. Instead it creates imaginary populations of ‘parallel universes’ and assume that our data are random samples from that kind of populations. But this is actually nothing but hand-waving! Doing econometrics it’s always wise to remember C. S. Peirce’s remark that universes are not as common as peanuts …


  1. […] Three suggestions to ‘save’ econometrics, […]

  2. Peiya,
    That was a direct quote from the introduction of the Reinhart-Rogoff paper, “Growth in a Time of Debt”. You should look into that. The paper received quite a bit of attention.
    I offer it as a high-profile illustrative case of numbers working in service of policy, rather than vice versa. The sparser the data set, naturally, the easier this is to accomplish.
    In many, if not most, cases, the econometric data is so sparse, one can accomplish little else.

  3. Here is what I thought about fundamental limits of econometrics.

    In general, types of relationships between two time-series data X(t) and Y(t) can be categorized as:

    (1) Identity (ID)
    (2) Single-Value Functional Dependency(FD)
    (3) Multi-value Functional Dependency(MFD)
    (4) Causal Relation(Cause)
    (5) Correlation(Corr)
    (6) Independence(Indep)

    Each category is a proper subset of next category. In set notations,
    ID ⊂ FD ⊂ MFD ⊂ Cause ⊂ Corr ⊂ Indep.

    For economic and financial time-series parameters, we only have accounting identities which are valid for all time periods, for example,
    (∀ 𝑡 [S(t) – I(t) = Balance_of_Current_Account(t)]). Accounting identities are based on T-accounting principle (balance_item (t) = income(t) – expense(t)). They are valid for all time periods for both past and future.

    Rigorously speaking, we do not have any other categories of relationships in economics, which are valid for ALL time periods except accounting identities. The reason is that relationships of many economic parameters are based on groups of human decisions, changes at different time periods with infinitely uncountable choices. These relationships are non-ergodic and non-deterministic and they cannot be specified mechanically and algorithmically in current maths and Turing computation model.

    What we can do in econometrics is to find time-specific relationships of FD/MFD/Cause/Corr/Indep between economic parameters as case studies for checking applicability to current or time-specific future economic conditions. But this is abductive reasoning and no guarantee for predication. This approach is useful for economic condition monitoring.

    • “What we can do in econometrics is to find time-specific relationships of FD/MFD/Cause/Corr/Indep between economic parameters”
      If you have enough data, and if the data you have is good enough.
      Which (for the most part) you don’t, and it isn’t.
      One data point every three months does not get you sufficient statistical significance to invalidate competing models before the regime changes. Consequently, at any point in time, you have a large set of divergent models and parameters which are all consistent with observed data at that point in time.
      Any preference of one model over another under such conditions is a political choice, not a mathematical outcome.
      By the way, did you know that median growth rates for countries with public debt over roughly 90 percent of GDP are about one percent lower than otherwise?
      I can show you a model that proves it.

      • Hi Michael,

        What we can prove at most is a temporal theorem with EXIT temporal quantifier.

        ∃ 𝑡 [Growth_Rate_Over90(t) < Growth_Rate_Under90(t)]

        I can assume that your time-specific periods t here include all past time periods and present. But it does not include future.

        But the more predictive theorem is with ALL temporal quantifier.

        ∀ 𝑡 [Growth_Rate_Over90(t) < Growth_Rate_Under90(t)]

        This logic variable t here includes all past, present and future time periods.

        Even in science and physics, we can only falsify the models by using observable facts, but cannot validate the models to be true in the future with more available facts. Einstein’s relativity theory only means that our universe seems to follow mechanical laws and no inconsistency with existing observations.

        One serious issue in current formal economic models is that many supply/demand equilibrium or behavior temporal equations directly contradict accounting identities. If following the scientific principle, the models should be revised first before further testing hypothesis. Otherwise, these models are just economic or scientific fictions.

        A meta theorem is shown below.
        If any economic model contradicts NIPA/FOFA accounting identities, then only two cases can happen:
        (1) this model is self-inconsistent
        (2) faulty economy axiom (my expense = your income) and finance axiom(my liability = your asset).

        All NIPA/FOFA accounting identities can be rigorously derived from these two basic assumptions(axioms) and term definitions without any market assumptions such as sticky price, perfect competition, etc. and without supply/demand equilibrium assumptions such as rational expectation, ex ante/ost, etc. More importantly, accounting identities are consistent with real economic data in past, current and future.

        At last, I would like to make a correction for my previous description
        It should be like this:

        ID ⊂ FD ⊂ MFD ⊂ Cause ⊂ Corr ⊂ Dependency ⊂ ANY

  4. “As economists and econometricians we have to confront the all-important question of how to handle uncertainty and randomness.”
    Let’s just say, for the sake of argument (and also because it happens to be true), that the quantity and quality of available data are insufficient for informing public policy-making on a sound empirical, analytic, and objective basis.
    In that case, on what basis should public policy-making be made? And, most importantly, who will be the likely winners and losers of conducting public policy-making on that basis, vs. the pseudo-empirical, pseudo-analytic, pseudo-objective basis on which it is currently conducted?
    That question, it seems to me, is conspicuous by its absence in your ongoing analysis of shortcomings of economics as it is currently practiced. The current practice of economics did not drop fully formed from the heavens one day. It was constituted over time by human actors. The deficiencies you note today have been present, and evident, from the very beginning, and yet, here it is. How exactly did it end up that way? Cui buono?

Sorry, the comment form is closed at this time.

Blog at
Entries and Comments feeds.