Watch out for econometric sinning in the basement!

19 October, 2014 at 17:00 | Posted in Statistics & Econometrics | 2 Comments

Brad DeLong wonders why Cliff Asness is clinging to a theoretical model that has clearly been rejected by the data …

Data-Mining
 
There’s a version of this in econometrics, i.e. you know the model is correct, you are just having trouble finding evidence for it. It goes as follows. You are testing a theory you came up with, but the data are uncooperative and say you are wrong. But instead of accepting that, you tell yourself “My theory is right, I just haven’t found the right econometric specification yet. I need to add variables, remove variables, take a log, add an interaction, square a term, do a different correction for misspecification, try a different sample period, etc., etc., etc.” Then, after finally digging out that one specification of the econometric model that confirms your hypothesis, you declare victory, write it up, and send it off (somehow never mentioning the intense specification mining that produced the result).

Too much econometric work proceeds along these lines. Not quite this blatantly, but that is, in effect, what happens in too many cases. I think it is often best to think of econometric results as the best case the researcher could make for a particular theory rather than a true test of the model.

Mark Thoma

Mark touches the spot — and for the sake of balancing the overly rosy picture of econometric achievements given in the usual econometrics textbooks today, it may also be interesting to see how Trygve Haavelmo, with the completion (in 1958) of the twenty-fifth volume of Econometrica, assessed the the role of econometrics in the advancement of economics. Although mainly positive of the “repair work” and “clearing-up work” done, Haavelmo also found some grounds for despair:

We have found certain general principles which would seem to make good sense. Essentially, these principles are based on the reasonable idea that, if an economic model is in fact “correct” or “true,” we can say something a priori about the way in which the data emerging from it must behave. We can say something, a priori, about whether it is theoretically possible to estimate the parameters involved. And we can decide, a priori, what the proper estimation procedure should be … But the concrete results of these efforts have often been a seemingly lower degree of accuracy of the would-be economic laws (i.e., larger residuals), or coefficients that seem a priori less reasonable than those obtained by using cruder or clearly inconsistent methods.

Haavelmo-intro-2-125397_630x210There is the possibility that the more stringent methods we have been striving to develop have actually opened our eyes to recognize a plain fact: viz., that the “laws” of economics are not very accurate in the sense of a close fit, and that we have been living in a dream-world of large but somewhat superficial or spurious correlations.

And as the quote below shows, Frisch also shared some of Haavelmo’s — and Keynes’s — doubts on the applicability of econometrics:

sp9997db.hovedspalteI have personally always been skeptical of the possibility of making macroeconomic predictions about the development that will follow on the basis of given initial conditions … I have believed that the analytical work will give higher yields – now and in the near future – if they become applied in macroeconomic decision models where the line of thought is the following: “If this or that policy is made, and these conditions are met in the period under consideration, probably a tendency to go in this or that direction is created”.

Ragnar Frisch

Advertisements

2 Comments

  1. The philistine’s version of this is to state an economic rule or law or rule of thumb, pick a photogenic example of that law in action, and then quote that example at every opportunity, asked or unasked. When counter examples are presented, dismiss them with a Wolfe-esque “pfui!” and refuse to engage in a detailed testing of that law or rule in the real world.

    Trigger phrases may include “It stands to reason,” and similar, while never in fact pushing to see whether it stands or falls when impacted by reason.

    So, “it stands to reason” that if minimum wages rise, then the number of jobs will fall. But this pfui’s the simple fact that in many minimum wage jobs, the employer is already employing the fewest, cheapest staff they can while still getting the firm’s work accomplished. These employers are a chintzy bunch, why would they keep extras around?

    Noni

  2. As allready Gunnar Myrdal’s pointed out long ago in his stance in favour of economics as a intergrated social theory:

    “I have no illusions that it will ever be possible to fit a
    general theory into a neat econometric model. The relevant variables and the
    relevant relations between them are too many to permit that sort of heroic
    simplification. This does not mean, however, that particular problems could not
    with advantage be treated in this way – provided that the variables and
    assumptions were selected on the basis of such insight into essential facts and
    relations as only a general theory can furnish”
    Gunnar Myrdal, (1957), Economic Theory and Underdeveloped Regions, London: University Paperbacks, Methuen.

    Later on, according to Myrdal there was a proliferation of terminological
    and mathematical innovation, which presented general equilibrium and welfare theories in a new form, that is through a new analytical language. In spite of this kind of escapism, however, this body of neoclassical economic thought still
    incorporates as he wrote:
    “One version or another of the old, discredited rationalistic psychology and
    utilitarian moral philosophy. By implying them – as practical conclusions make
    evident that it does – it becomes unfounded and false”
    Gunnar Myrdal. (1970), Objectivity in Social Research, London: Duckworth


Sorry, the comment form is closed at this time.

Blog at WordPress.com.
Entries and comments feeds.