The fundamental flaw of econometrics

30 Sep, 2016 at 18:07 | Posted in Statistics & Econometrics | 8 Comments

It is often said that the error term in a regression equation represents the effect of the variables that were omitted from the equation. This is unsatisfactory …

There is no easy way out of the difficulty. The conventional interpretation for error terms needs to be reconsidered. At a minimum, something like this would need to be said:

The error term represents the combined effect of the omitted variables, assuming that
(i) the combined effect of the omitted variables is independent of each variable included in the equation,
(ii) the combined effect of the omitted variables is independent across subjects,
(iii) the combined effect of the omitted variables has expectation 0.

This is distinctly harder to swallow.

David Freedman

Yes, indeed, that is harder to swallow.

Those conditions on the error term actually means that we are being able to construct a model where all relevant variables are included and correctly specify the functional relationships that exist between them.

But that is actually impossible to fully manage in reality!

The theories we work with when building our econometric regression models are insufficient. No matter what we study, there are always some variables missing, and we don’t know the correct way to functionally specify the relationships between the variables (usually just assuming linearity).

Every regression model constructed is misspecified. There are always an endless list of possible variables to include, and endless possible ways to specify the relationships between them. So every applied econometrician comes up with his own specification and ‘parameter’ estimates. No wonder that the econometric Holy Grail of consistent and stable parameter-values is still nothing but a dream.

overconfidenceIn order to draw inferences from data as described by econometric texts, it is necessary to make whimsical assumptions. The professional audience consequently and properly withholds belief until an inference is shown to be adequately insensitive to the choice of assumptions. The haphazard way we individually and collectively study the fragility of inferences leaves most of us unconvinced that any inference is believable. If we are to make effective use of our scarce data resource, it is therefore important that we study fragility in a much more systematic way. If it turns out that almost all inferences from economic data are fragile, I suppose we shall have to revert to our old methods …

Ed Leamer

A rigorous application of econometric methods in economics really presupposes that the phenomena of our real world economies are ruled by stable causal relations between variables.  Parameter-values estimated in specific spatio-temporal contexts are presupposed to be exportable to totally different contexts. To warrant this assumption one, however, has to convincingly establish that the targeted acting causes are stable and invariant so that they maintain their parametric status after the bridging. The endemic lack of predictive success of the econometric project indicates that this hope of finding fixed parameters is a hope for which there really is no other ground than hope itself.

Real world social systems are not governed by stable causal mechanisms or capacities. As Keynes noticed when he first launched his attack against econometrics and inferential statistics already in the 1920s:

The atomic hypothesis which has worked so splendidly in Physics breaks down in Psychics. We are faced at every turn with the problems of Organic Unity, of Discreteness, of Discontinuity – the whole is not equal to the sum of the parts, comparisons of quantity fails us, small changes produce large effects, the assumptions of a uniform and homogeneous continuum are not satisfied. Thus the results of Mathematical Psychics turn out to be derivative, not fundamental, indexes, not measurements, first approximations at the best; and fallible indexes, dubious approximations at that, with much doubt added as to what, if anything, they are indexes or approximations of.

The kinds of laws and relations that econom(etr)ics has established, are laws and relations about entities in models that presuppose causal mechanisms being atomistic and additive. When causal mechanisms operate in real world social target systems they only do it in ever-changing and unstable combinations where the whole is more than a mechanical sum of parts. If economic regularities obtain they do it (as a rule) only because we engineered them for that purpose. Outside man-made “nomological machines” they are rare, or even non-existant. Unfortunately that also makes most of the achievements of econometrics – as most of contemporary endeavours of economic theoretical modeling – rather useless.

statRegression models are widely used by social scientists to make causal inferences; such models are now almost a routine way of demonstrating counterfactuals. However, the “demonstrations” generally turn out to depend on a series of untested, even unarticulated, technical assumptions. Under the circumstances, reliance on model outputs may be quite unjustified. Making the ideas of validation somewhat more precise is a serious problem in the philosophy of science. That models should correspond to reality is, after all, a useful but not totally straightforward idea – with some history to it. Developing appropriate models is a serious problem in statistics; testing the connection to the phenomena is even more serious …

In our days, serious arguments have been made from data. Beautiful, delicate theorems have been proved, although the connection with data analysis often remains to be established. And an enormous amount of fiction has been produced, masquerading as rigorous science.

The theoretical conditions that have to be fulfilled for regression analysis and econometrics to really work are nowhere even closely met in reality. Making outlandish statistical assumptions does not provide a solid ground for doing relevant social science and economics. Although regression analysis and econometrics have become the most used quantitative methods in social sciences and economics today, it’s still a fact that the inferences made from them are invalid.


  1. Henry,
    I agree with most of your comment. As you say, our brains and cognitive abilities were undoubtedly shaped by “an iterative process of trial and error over eons, conditioned by the real world.”
    Humans have numerous psychological traits which evolved in the environments of our distant ancestors, e.g. instinctive fear of snakes. Our abilities were also shaped by general selective pressures. An important example is language, a biological adaptation enabling humans to communicate information.
    General selective pressures also shaped our memories and information processing mechanisms. Some of the interesting suggestions from Evolutionary Psychologists are:
    — Cosmides & Tooby (1996) – Are humans good intuitive statisticians after all? Cognition, 58, 1-73:
    “the conclusion most common in the literature on judgment under uncertainty – that our inductive reasoning mechanisms do not embody a calculus of probability – will have to be re-examined. From an ecological and evolutionary perspective, humans may turn out to be good intuitive statisticians after all.”
    — Rakoczy et al. (2014) – Apes are intuitive statisticians. Cognition, 131(1):60-8:
    “In a series of 7 experiments, Bonobos, Chimpanzees, Gorillas and Orangutans drew flexible statistical inferences from populations to samples. These inferences, furthermore, were truly based on statistical information regarding the relative frequency distributions in a population,
    and not on absolute frequencies. Intuitive statistics in its most basic form is thus an evolutionarily more ancient rather than a uniquely human capacity.”
    For these and other reasons it is plausible that the intuitive statistical abilities of humans evolved to enable us to survive and reproduce in the environment of our ancestors. Moreover, without these abilities it is unlikely that our ancestors could have survived.
    It is therefore surprising that Prof. Syll suggests that our intuitive statistical abilities are merely the result of “theoretical conditions” which are “nowhere even closely met in reality”.

  2. ” the survival of our species is ample evidence that econometric concepts really work are widely and closely met in reality”

    Right at the last, this is were Kingsley stumbles badly. He takes a long stretch so as to draw the experience of the real world into his conclusion.

    Human beings may have over time learned how to survive. Firstly, they are driven to survive, as seems to be the intent of nature, but it is an iterative process of trial and error over eons, conditioned by the real world. Over the generations, practices are transferred which promote survival – a social phenomenon. There is even evidence that such learning can be transferred genetically.

    But this is not how modern economic “science” works. Its considerations and deliberations begin with abstract premises based on abstract assumptions and axioms (which have actually failed the test of time and reality) and then draws conclusions.

    The way understanding is developed in the two approaches is entirely at odds. One is formed of convention (Keynesian, if you will) and the other in complete abstraction (modern economics).

  3. […] The fundamental flaw of econometrics, […]

  4. KM,
    The authors of the journals which you mention may well be “rigorous thinkers” and “analytic philosophers” but they use too much jargon for me to be able judge the relevance or usefulness of their papers for economics.
    In contrast, as others have observed, it seems likely that the embryonic science of Evolutionary Psychology has a great deal to contribute to economics.
    However, EP is not yet a rigorous science.And I agree that my ideas are speculative.

  5. I had the exact same initial response as the redoubtable Bruce Wilder: I thought for a second this might be pure parody. But reading through it, and having encountered many of Mr. Lewis’s past interventions, I too am inclined to think that while full of sarcasm and what Mr. Lewis no doubt imagines are clever exaggerations, this is actually supposed to represent a sincere argument.
    In fact, I actually think it’s an excellent illustration of how conventional social scientists operate when they are challenged in any fundamental way.
    Social scientists who prate on endlessly about “rigour” and “precision” when “on the offensive”, as it were — vaunting the alleged scientificity of their models — almost *always* display a shocking lack thereof when put on the defensive. And this is a great example of this pattern.
    Mr. Lewis here puts on a classic display with all the usual patterns: wildly loose, ad hoc, purely analogical reasoning (without the slightest acknowledgement of their (purely) analogical nature, nor any justification thereof); a strong dash of just-so evolutionary theory and crazy speculative universal generalisations about human nature (both last refuges of intellectual scoundrels of every kind); wild category mistakes; pure empty assertion after pure assertion (without the slightest logical argumentation or empirical illustration to back them up); and the convenient use of hyper-abstract functional categories to mask the absence of any genuine conceptual precision.
    And the juvenile potshots at “university philosophy departments”? Are you freaking kidding me? I challenge Mr. Lewis to read through the latest issue of Philosophy of Science, Synthese, Noûs or Journal of Economic Methodology and then try to make the case that mainstream economists are more rigorous thinkers than analytic philosophers. Good luck, my pompous friend.

  6. There is an overwhelming refutation of Prof. Syll’s absurd conclusion that “The theoretical conditions that have to be fulfilled for regression analysis and econometrics to really work are nowhere even closely met in reality.”
    The refutation is the survival of Prof. Syll’s ancestors and his own existence today.

    Humans evolved innate econometric abilities during the last 500,000+ years.
    In past eras the main economic activities were hunting, fishing, foraging and primitive agriculture, and conquests through wars with neigbours. Survival required the ability to correlate success in these activities with such factors as location, season, techniques, time devoted to various activities, etc.
    Tribes which were good at econometrics tended to survive. Other tribes tended to become extinct.
    The evolution of related statistical instincts and skills may have begun even earlier, e.g. abilities to assess the dangers and efforts required when running, jumping etc. in different situations.

    As today, not all members of successful tribes were well endowed with statistical and econometric abilities. The less able tended to die early or be assigned to non-leadership tasks, or to the precursors of modern monasteries, mental institutions or university philosophy departments.
    Disregarding those with poor empirical skills, the bulk of the populations of successful tribes developed an innate understanding of how “real world economies are ruled by stable causal relations between variables”.

    Of course, no-one ever supposed that observed empirical relationships are immutable and apply in all conceivable circumstances. And of course, the econometrics of our ancestors was not formulated with mathematical symbols. And yes, modern hyper-advanced econometrics is probably overkill, with little practical use.

    But the essential concepts of econometrics appear to be innate within most of us. Moreover, contrary to Prof, Syll’s metaphysics, the survival of our species is ample evidence that econometric concepts really work are widely and closely met in reality.

    • Is that supposed to be a satiric parody? I’m guessing not.
      A plain reading of what we have been able to observe of human natural and recorded history would suggest that econometrics, a weak technology for developing knowledge at best, also has a singularly bad fit to innate human intuition.

      In fact, humans are prone to learning useless rituals, which are retained in their behavioral repertoire indefinitely and to seeing meaningful patterns with no functional cause. We see constellations in the sky, daemon spirits in nature, and readily mistake elaborate superstition for knowledge. The Romans followed the Etruscans in divining the future from the entrails of sheep; modern economists would be impressed that they had models — elaborate inscribed artifacts representing animal livers have been found suggesting that people were formally trained in the art not just in the Roman world, but as far back as the empire of Akkad.
      Seeking stable correlations as clues to nature is not a particularly efficacious method of investigation in any case. Where econometrics is prone to compressing and homogenizing information from details to general tendencies obscured by a fog of errors and exceptions, methods of active investigation actually go in the opposite direction, seeking to imagine a mechanism and to confirm its practical operation by elaborating an interpretation, extending a powerful explanation to account for more and more observations that are otherwise uncorrelated by passive experience alone. Think about how people come to understand that the earth is a sphere: noticing that ships masts sink away at the horizon or the changing angle of the sun over differing latitudes or the shadow of the earth cast on the moon.

      • Bruce,
        Yes, there have been, and still are, all sorts of weird mental and physical mutations and ideas among humans.
        In the past, traits and ideas which diminished survival prospects tended to be extinguished by the eventual extinction of those having such traits.
        In contrast, traits which enhanced survival prospects tended to persist in our genes and store of communal knowledge. Survival by hunting, fishing, foraging, agriculture and war required the ability to learn how success in these activities was correlated with such factors as location, time of day, season, techniques used, time devoted to various activities, etc.

        And Yes, “seeking stable correlations as clues to nature is not a particularly efficacious method of investigation”.
        It may take millennia of observations and trial and error to distinguish good ideas from “a fog of errors and exception”. But this is what our species achieved.

        And Yes, “methods of active investigation” are important catalysts for the learning process. Leaps of imagination and analysis complement empirical observations rather than going “in the opposite direction”.

        So we are left with the conclusion that the survival of our species is ample evidence that econometric concepts really work and are widely and closely met in reality. Otherwise it is unlikely that our species could have survived predators, resource constraints, climate changes, natural disasters and wars.

Sorry, the comment form is closed at this time.

Blog at
Entries and Comments feeds.