Econometrics — a con art with no relevance whatsoever to real world economics

11 Jul, 2019 at 15:51 | Posted in Economics, Statistics & Econometrics | 17 Comments

Econometrics-as-a-Con-Art-Imad-A-MoosaEconometrics looks “sciency”. Once in a seminar presentation I displayed two equations, one taken from Econometrica and the other from the Journal of Theoretical and Experimental Physics and challenged the audience to tell me which is which. No one volunteered to tell me which is which, including at least one hard-core econometrician. Economics is a social science where the behaviour of decision makers is not governed purely by economic considerations but also by social and psychological factors, which are not amenable to econometric testing. This is why no economic theory holds everywhere all the time. And this is why the results of empirical testing of economic theories are typically a mixed bag. And this is why econometricians use time-varying parametric estimation to account for changes in the values of estimated parameters over time (which means that the underlying relationship does not have the universality of a law). And this is why there are so many estimation methods that can be used to produce the desired results. In physics, on the other hand, a body falling under the force of gravity travels with an acceleration of 32 feet per second per second – this is true anywhere any time. In physics also, the boiling point of water under any level of atmospheric pressure can be predicted with accuracy.

Unlike physicists, econometricians are in a position to obtain the desired results, armed with the arsenal of tools produced by econometric theory. When an econometrician fails to obtain the desired results, he or she may try different functional forms, lag structures and estimation methods, and indulge in data mining until the desired results are obtained (torture produces a confession even when applied to data). If the empirical work is conducted for the purpose of writing an academic paper, the researcher seeks results that are “interesting” enough to warrant publication or results that confirm the view of the orthodoxy. And it is typically the case that the results cannot be replicated. Physicists do not have this luxury – it is unthinkable and easily verifiable that a physicist manipulates data (by using principal components or various econometric transformations) to obtain readings that refute Boyle’s law. Economists study the behaviour of consumers, firms and governments where expectations and uncertainties play key roles in the translation of economic theory into real world economics. These uncertainties mean that econometric modelling cannot produce accurate representation of the working of the economy.

Imad Moosa / RWER

Mainstream economists often hold the view that if you are critical of econometrics it can only be because you are a sadly misinformed and misguided person who dislike and do not understand much of it.

As Moosa’s eminent article shows, this is, however, nothing but a gross misapprehension.

And just as Moosa, Keynes certainly did not misunderstand the crucial issues at stake in his critique of econometrics. Quite the contrary. He knew them all too well — and was not satisfied with the validity and philosophical underpinnings of the assumptions made for applying its methods.

LierKeynes’ critique is still valid and unanswered in the sense that the problems he pointed at are still with us today and ‘unsolved.’ Ignoring them — the most common practice among applied econometricians — is not to solve them.

To apply statistical and mathematical methods to the real-world economy, the econometrician has to make some quite strong assumptions. In a review of Tinbergen’s econometric work — published in The Economic Journal in 1939 — Keynes gave a comprehensive critique of Tinbergen’s work, focusing on the limiting and unreal character of the assumptions that econometric analyses build on:

Completeness: Where Tinbergen attempts to specify and quantify which different factors influence the business cycle, Keynes maintains there has to be a complete list of all the relevant factors to avoid misspecification and spurious causal claims. Usually, this problem is ‘solved’ by econometricians assuming that they somehow have a ‘correct’ model specification. Keynes is, to put it mildly, unconvinced:

istheseptuagintaIt will be remembered that the seventy translators of the Septuagint were shut up in seventy separate rooms with the Hebrew text and brought out with them, when they emerged, seventy identical translations. Would the same miracle be vouchsafed if seventy multiple correlators were shut up with the same statistical material? And anyhow, I suppose, if each had a different economist perched on his a priori, that would make a difference to the outcome.

J M Keynes

Homogeneity: To make inductive inferences possible — and being able to apply econometrics — the system we try to analyse has to have a large degree of ‘homogeneity.’ According to Keynes most social and economic systems — especially from the perspective of real historical time — lack that ‘homogeneity.’ As he had argued already in Treatise on Probability (ch. 22), it wasn’t always possible to take repeated samples from a fixed population when we were analysing real-world economies. In many cases, there simply are no reasons at all to assume the samples to be homogenous. Lack of ‘homogeneity’ makes the principle of ‘limited independent variety’ non-applicable, and hence makes inductive inferences, strictly seen, impossible since one of its fundamental logical premises are not satisfied. Without “much repetition and uniformity in our experience” there is no justification for placing “great confidence” in our inductions (TP ch. 8).

And then, of course, there is also the ‘reverse’ variability problem of non-excitation: factors that do not change significantly during the period analysed, can still very well be extremely important causal factors.

Stability: Tinbergen assumes there is a stable spatio-temporal relationship between the variables his econometric models analyze. But as Keynes had argued already in his Treatise on Probability it was not really possible to make inductive generalisations based on correlations in one sample. As later studies of ‘regime shifts’ and ‘structural breaks’ have shown us, it is exceedingly difficult to find and establish the existence of stable econometric parameters for anything but rather short time series.

Measurability: Tinbergen’s model assumes that all relevant factors are measurable. Keynes questions if it is possible to adequately quantify and measure things like expectations and political and psychological factors. And more than anything, he questioned — both on epistemological and ontological grounds — that it was always and everywhere possible to measure real-world uncertainty with the help of probabilistic risk measures. Thinking otherwise can, as Keynes wrote, “only lead to error and delusion.”

Independence: Tinbergen assumes that the variables he treats are independent (still a standard assumption in econometrics). Keynes argues that in such a complex, organic and evolutionary system as an economy, independence is a deeply unrealistic assumption to make. Building econometric models from that kind of simplistic and unrealistic assumptions risk producing nothing but spurious correlations and causalities. Real-world economies are organic systems for which the statistical methods used in econometrics are ill-suited, or even, strictly seen, inapplicable. Mechanical probabilistic models have little leverage when applied to non-atomic evolving organic systems — such as economies.

originalIt is a great fault of symbolic pseudo-mathematical methods of formalising a system of economic analysis … that they expressly assume strict independence between the factors involved and lose all their cogency and authority if this hypothesis is disallowed; whereas, in ordinary discourse, where we are not blindly manipulating but know all the time what we are doing and what the words mean, we can keep “at the back of our heads” the necessary reserves and qualifications and the adjustments which we shall have to make later on, in a way in which we cannot keep complicated partial differentials “at the back” of several pages of algebra which assume that they all vanish.

Building econometric models can’t be a goal in itself. Good econometric models are means that make it possible for us to infer things about the real-world systems they ‘represent.’ If we can’t show that the mechanisms or causes that we isolate and handle in our econometric models are ‘exportable’ to the real world, they are of limited value to our understanding, explanations or predictions of real-world economic systems.

The kind of fundamental assumption about the character of material laws, on which scientists appear commonly to act, seems to me to be much less simple than the bare principle of uniformity. They appear to assume something much more like what mathematicians call the principle of the superposition of small effects, or, as I prefer to call it, in this connection, the atomic character of natural law. 3The system of the material universe must consist, if this kind of assumption is warranted, of bodies which we may term (without any implication as to their size being conveyed thereby) legal atoms, such that each of them exercises its own separate, independent, and invariable effect, a change of the total state being compounded of a number of separate changes each of which is solely due to a separate portion of the preceding state …

The scientist wishes, in fact, to assume that the occurrence of a phenomenon which has appeared as part of a more complex phenomenon, may be some reason for expecting it to be associated on another occasion with part of the same complex. Yet if different wholes were subject to laws qua wholes and not simply on account of and in proportion to the differences of their parts, knowledge of a part could not lead, it would seem, even to presumptive or probable knowledge as to its association with other parts.

Linearity: To make his models tractable, Tinbergen assumes the relationships between the variables he study to be linear. This is still standard procedure today, but as Keynes writes:

It is a very drastic and usually improbable postulate to suppose that all economic forces are of this character, producing independent changes in the phenomenon under investigation which are directly proportional to the changes in themselves; indeed, it is ridiculous.

To Keynes, it was a ‘fallacy of reification’ to assume that all quantities are additive (an assumption closely linked to independence and linearity).

2014+22keynes%20illo2The unpopularity of the principle of organic unities shows very clearly how great is the danger of the assumption of unproved additive formulas. The fallacy, of which ignorance of organic unity is a particular instance, may perhaps be mathematically represented thus: suppose f(x) is the goodness of x and f(y) is the goodness of y. It is then assumed that the goodness of x and y together is f(x) + f(y) when it is clearly f(x + y) and only in special cases will it be true that f(x + y) = f(x) + f(y). It is plain that it is never legitimate to assume this property in the case of any given function without proof.

J. M. Keynes “Ethics in Relation to Conduct” (1903)

And as even one of the founding fathers of modern econometrics — Trygve Haavelmo — wrote:

What is the use of testing, say, the significance of regression coefficients, when maybe, the whole assumption of the linear regression equation is wrong?

Real-world social systems are usually not governed by stable causal mechanisms or capacities. The kinds of ‘laws’ and relations that econometrics has established, are laws and relations about entities in models that presuppose causal mechanisms and variables — and the relationship between them — being linear, additive, homogenous, stable, invariant and atomistic. But — when causal mechanisms operate in the real world they only do it in ever-changing and unstable combinations where the whole is more than a mechanical sum of parts. Since statisticians and econometricians — as far as I can see — haven’t been able to convincingly warrant their assumptions of homogeneity, stability, invariance, independence, additivity as being ontologically isomorphic to real-world economic systems, Keynes’ critique is still valid. As long as — as Keynes writes in a letter to Frisch in 1935 — “nothing emerges at the end which has not been introduced expressively or tacitly at the beginning,” I remain doubtful of the scientific aspirations of econometrics.

In his critique of Tinbergen, Keynes points us to the fundamental logical, epistemological and ontological problems of applying statistical methods to a basically unpredictable, uncertain, complex, unstable, interdependent, and ever-changing social reality. Methods designed to analyse repeated sampling in controlled experiments under fixed conditions are not easily extended to an organic and non-atomistic world where time and history play decisive roles.

Econometric modelling should never be a substitute for thinking. From that perspective, it is really depressing to see how much of Keynes’ critique of the pioneering econometrics in the 1930s-1940s is still relevant today. And that is also a reason why we — as does Moosa — have to keep on criticizing it.

The general line you take is interesting and useful. It is, of course, not exactly comparable with mine. I was raising the logical difficulties. You say in effect that, if one was to take these seriously, one would give up the ghost in the first lap, but that the method, used judiciously as an aid to more theoretical enquiries and as a means of suggesting possibilities and probabilities rather than anything else, taken with enough grains of salt and applied with superlative common sense, won’t do much harm. I should quite agree with that. That is how the method ought to be used.

Keynes, letter to E.J. Broster, December 19, 1939


RSS feed for comments on this post. TrackBack URI

  1. “In physics, on the other hand, a body falling under the force of gravity travels with an acceleration of 32 feet per second per second – this is true anywhere any time.”
    Except in galaxies, so dark matter had to be invented to maintain the assumption.
    “In physics also, the boiling point of water under any level of atmospheric pressure can be predicted with accuracy.”
    There is no equation of state for water. There are look up tables based on observation and interpolation schemes, which have associated error.
    The point: physics too is really about arbitrary and fickle social consensus.

  2. “In physics, on the other hand, a body falling under the force of gravity travels with an acceleration of 32 feet per second per second – this is true anywhere any time.”

    Actually this is not true anywhere, any time, as on Earth, g varies with both altitude and latitude. The maximum variance is under 1%, but it does vary

  3. At above … its ‘true’ in the sense that it establishes a base line by which variables can be adjusted to local environmental factors, which have no human foibles to reconcile e.g. a meteor entering the atmosphere does not have any agency when choosing its trajectory vs a human interacting in a dynamic environmental context with a plethora of psychological and biological factors playing out.

    Said as an ex Halo and Pathfinder operator.

    • “The solar system is ultimately chaotic. This is particularly the case for asteroids and comets on highly elliptical trajectories. Consider a comet whose orbit takes it from beyond Neptune’s orbit to inside Venus’s orbit. Over the centuries, that comet has ample opportunities to make a close pass with a planet. There are but slight differences between the planet making a minor change in the comet’s orbit, the comet colliding into the planet, the planet sending the comet toward the Sun or another planet, or the planet ejecting the comet from the solar system. This is chaos at its worst.”

      • Fail to see your argument about agency and Newtonian metrics applied to human dynamics …. am I a rock in space nudged by gravitational forces.

        • My point is that you can’t even predict rocks in space. Oumuamua sped up as it got further from the sun.

        • “The universe is not only stranger than you imagine but stranger than you can imagine.”

        • It think you should have a thought to the concept of theory vs other methodology.

          • Economics and Physics are both just human stories that have achieved an arbitrary, fickle social consensus. Data is cherry-picked, outliers are thrown out, so high priests can feel self-important … Feynman warned against this in Cargo Cult Science, noting how researchers replicating Millikan’s oil drop experiment somehow found ways to change their more accurate results to match his mistakes for decades.

            • It seems what you and I consider Science are two different things and I would note “Science-Mart” and what it ensues … own goal as it were …

  4. Prof. Syll,
    This post is superbly eloquent, as usual.
    But there is one important matter which needs explicit clarification, or correction.
    The title and text implies that that we can and must learn about actual economies without any econometrics. The problem with this is that while many of us can’t understand fancy econometrics, but we do nevertheless like to study graphs and even simple regressions. This is crude econometrics, with all the well known dangers which you mention.
    Is such simple “graphical” econometrics “a con art with no relevance whatsoever to real world economics”?
    You appear to be saying that we must learn about actual economies merely by casual observation and intuition, without using data? If so, please reference illustrative examples of this methodology/ontology/epistemology.
    Or are you merely reminding us that empirical estimates are surrounded by confidence intervals, are not universally valid, and rely on judgements in conjunction with the available data?
    If so, some of us are bound to resent such banal advice from a theoretical philosopher who has never published any useful empirical results in the field of economics. (Apologies if I am wrong on this last point).

    • do those also resent banal advice from say Milton Friedman or others, who also have not published any useful empirical results in the field of economics?

      • “You appear to be saying that we must learn about actual economies merely by casual observation and intuition, without using data? ”

        No, you must use primary documentation and other unquantifiable, but nevertheless verifiable, material. Rather like a good historian does. Quantitative data alone, even more so without substantial investigation and discussion, about what the data really means, is unlikely to be meaningful. This should be glaringly obvious, but to too many people it clearly isn’t it. How do you put a number on things like geopolitical power, soft power, or social relations? However, these may be the very things driving, for example, the stability of a currency. Just ignoring such things , or making a casual reference to them because they are unquantifiable or not amenable to neo-classical modelling, should not be considered satisfactory.

        My advice to you is that if you are happy with how the orthodoxy does things and regard unquantifiable evidence and approaches that eschew models as casual you’ve come to the wrong place. The problem we have is that we regard the treatment of non-formalistic approaches and unquantifiable evidence in the mainstream economics profession compared with other professions like history and psychology as absurdly casual (the mirror image of what you say). What we regard as the right way probably won’t get you a job at MIT, the ECB or IMF, but it’s infinitely more interesting – and more likely to get us closer to identifying the real problems so that we can find the right solutions.

        • “do those also resent banal advice from say Milton Friedman or others, who also have not published any useful empirical results in the field of economics?”

          I would argue that Friedman was not the worst offender when it came to this type of abuse. Sure he argued that the cause of the Depression was a rapid steep fall in the money supply (an historian would argue that this was just one of a complex set of interrelated causes), but yet he understood the importance of historical context. Far worse are Sameulson, Sargent, Fama, Prescott, Krugman et al. Even Lucas comes across as relatively open minded compared with the latter.

          • Friedman was the first that came to mind

    • There could have been a shift in meaning, here: it seems to me that econometrics could mean merely measurement in economics, and finding numbers and quantities to describe various things. It seems now to mean the application of tricky mathematics to those quantities.

      There seems to be a parallel with “pasigraphy” which was a notation promoted by people early in the last century to express mathematics in a precise and formal way. Henri Poincaré gave his opinion, sarcastically, in _Science and Method_, Book II, Chapter III, section 7:

      “First we see Burali-Forti define the number 1 as follows:
      1 = ⍳T'{Kon̲̂(u,h)ε(uεuₙ)},
      a definition eminently fitted to give an idea of the number 1 to persons who had never heard speak of it.”

      He didn’t like it. He did, in fairness I guess, mention other people who did.

  5. Because of interest, privatized for the first time ca 3000 bc. and as this privatisation that can only be defended by more and more complicated econometrics, the subject is as respected as astrology was. For similar reasons. But in the interest of the bourgeois rentiers instead of the hereditary astoricracy.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Blog at
Entries and comments feeds.