## Econometric causality

2 Oct, 2013 at 10:51 | Posted in Statistics & Econometrics | 4 CommentsA popular idea in quantitative social sciences is to think of a cause (C) as something that increases the probability of its effect or outcome (O). That is:

P(O|C) > P(O|-C)

However, as is also well-known, a correlation between two variables, say A and B, does not necessarily imply that that one is a cause of the other, or the other way around, since they may both be an effect of a common cause, C.

In statistics and econometrics we usually solve this “confounder” problem by “controlling for” C, i. e. by holding C fixed. This means that we actually look at different “populations” – those in which C occurs in every case, and those in which C doesn’t occur at all. This means that knowing the value of A does not influence the probability of C [P(C|A) = P(C)]. So if there then still exist a correlation between A and B in either of these populations, there has to be some other cause operating. But if *all* other possible causes have been “controlled for” too, and there is still a correlation between A and B, we may safely conclude that A is a cause of B, since by “controlling for” all other possible causes, the correlation between the putative cause A and all the other possible causes (D, E,. F …) is broken.

This is of course a very demanding prerequisite, since we may never actually be sure to have identified *all* putative causes. Even in scientific experiments may the number of uncontrolled causes be innumerable. Since nothing less will do, we do all understand how hard it is to actually get from correlation to causality. This also means that *only* relying on statistics or econometrics is not enough to deduce causes from correlations.

Some people think that randomization may solve the empirical problem. By randomizing we are getting different “populations” that are homogeneous in regards to all variables except the one we think is a genuine cause. In that way we are supposed being able not having to actually know what all these other factors are.

If you succeed in performing an *ideal* randomization with different treatment groups and control groups that is attainable. *But* it presupposes that you really have been able to establish – and not just assume – that the probability of all other causes but the putative (A) have the same probability distribution in the treatment and control groups, and that the probability of assignment to treatment or control groups are independent of all other possible causal variables.

Unfortunately, *real *experiments and *real* randomizations seldom or never achieve this. So, yes, we may do without knowing *all *causes, but it takes *ideal* experiments and *ideal* randomizations to do that, not *real *ones.

As I have argued (here) that means that in practice we do have to have sufficient background knowledge to deduce causal knowledge. Without old knowledge, we can’t get new knowledge – and, no causes in, no causes out.

Just so that you do not think this assertion is some idiosyncrasy of yours truly, let me back up my claim with quotes from two eminent statisticians.

I start with **John Maynard Keynes.** As I have written about earlier (e.g. here and here), Keynes was very critical of the way statistical tools were used in social sciences. In his criticism of the application of inferential statistics and regression analysis in the early development of econometrics, Keynes in a critical review of the early work of Tinbergen, writes:

Prof. Tinbergen agrees that the main purpose of his method is to discover, in cases where the economist has correctly analysed beforehand the qualitative character of the causal relations, with what -strength each of them operates. If we already know what the causes are, then (provided all the other conditions given below are satisfied) Prof. Tinbergen, given the statistical facts, claims to be able to attribute to the causes their proper quantitative importance. If (anticipating the conditions which follow) we know beforehand that business cycles depend partly on the present rate of interest and partly on the birth-rate twenty years ago, and that these are independent factors in linear correlation with the result, he can discover their relative importance. As regards disproving such a theory, he cannot show that they are not verce causce, and the most he may be able to show is that, if they are verce cause, either the factors are not independent, or the correlations involved are not linear, or there are other relevant respects in which the economic environment is not homogeneous over a period of time (perhaps because non-statistical factors are relevant).

Am I right in thinking that the method of multiple correlation analysis essentially depends on the economist having furnished, not merely a list of the significant causes, which is correct so far as it goes, but a

completelist? For example, suppose three factors are taken into account, it is not enough that these should be in fact verce causce; there must be no other significant factor. If there is a further factor, not taken account of, then the method is not able to discover the relative quantitative importance of the first three. If so, this means that the method is only ap-plicable where the economist is able to provide beforehand a correct and indubitably complete analysis of the significant factors. The method is one neither of discovery nor of criticism. It is a means of giving quantitative precision to what, in qualita-tive terms, we know already as the result of a complete theoretical analysis.

This, of course, is absolutely right. Once you include *all* actual causes into the original (over)simple model, it may well be that the causes are no longer independent or linear, and that *a fortiori* the coefficients in the econometric equations no longer are identifiable. And so, since *all* causal factors are not included in the original econometric model, it is not an adequate representation of the real causal structure of the economy that the model is purportedly meant to represent.

My second source is **David Freedman**. In his *Statistical Models and Causal Inference* (2010) Freedman writes:

If the assumptions of a model are not derived from theory, and if predictions are not tested against reality, then deductions from the model must be quite shaky. However, without the model, the data cannot be used to answer the research question …

In my view, regression models are not a particularly good way of doing empirical work in the social sciences today, because the technique depends on knowledge that we do not have. Investigators who use the technique are not paying adequate attention to the connection – if any – between the models and the phenomena they are studying. Their conclusions may be valid for the computer code they have created, but the claims are hard to transfer from that microcosm to the larger world …

Regression models often seem to be used to compensate for problems in measurement, data collection, and study design. By the time the models are deployed, the scientific position is nearly hopeless. Reliance on models in such cases is Panglossian …

Given the limits to present knowledge, I doubt that models can be rescued by technical fixes. Arguments about the theoretical merit of regression or the asymptotic behavior of specification tests for picking one version of a model over another seem like the arguments about how to build desalination plants with cold fusion and the energy source. The concept may be admirable, the technical details may be fascinating, but thirsty people should look elsewhere …

Causal inference from observational data presents may difficulties, especially when underlying mechanisms are poorly understood. There is a natural desire to substitute intellectual capital for labor, and an equally natural preference for system and rigor over methods that seem more haphazard. These are possible explanations for the current popularity of statistical models.

Indeed, far-reaching claims have been made for the superiority of a quantitative template that depends on modeling – by those who manage to ignore the far-reaching assumptions behind the models. However, the assumptions often turn out to be unsupported by the data. If so, the rigor of advanced quantitative methods is a matter of appearance rather than substance.

And in *Statistical Models: Theory and Practice* (2009) Freedman writes:

The usual point of running regressions is to make causal inferences without doing real experiments. On the other hand, without experiments, the assumptions behind the models are going to be iffy. Inferences get made by ignoring the iffiness of the assumptions. That is the paradox of causal inference …

Path models do not infer causation from association. Instead, path models

assumecausation through response schedules, and – using additional statistical assumptions – estimate causal effects from observational data … The problems are built into the assumptions behind the statistical models … If the assumptions don’t hold, the conclusions don’t follow from the statistics.

Econometrics is basically a deductive method. Given the assumptions (such as manipulability, transitivity, Reichenbach probability principles, separability, additivity, linearity etc) it delivers deductive inferences. The problem, of course, is that we will never completely know when the assumptions are right. Real target systems are seldom epistemically isomorphic to axiomatic-deductive models/systems, and even if they were, we still have to argue for the external validity of the conclusions reached from within these epistemically convenient models/systems. Causal evidence generated by statistical/econometric procedures may be valid in “closed” models, but what we usually are interested in, is causal evidence in the real target system we happen to live in.

**Advocates of econometrics want to have deductively automated answers to fundamental causal questions. But to apply “thin” methods we have to have “thick” background knowledge of what’s going on in the real world, and not in idealized models. Conclusions can only be as certain as their premises – and that also applies to the quest for causality in econometrics.**

## 4 Comments

Sorry, the comment form is closed at this time.

Blog at WordPress.com.

Entries and Comments feeds.

[…] As an example of an ultrasocial norm, consider generalized trust. Propensity to trust and help individuals outside of one’s ethnic group has a clear benefit for multiethnic societies, but ethnic groups among whom this ultrasocial norm is wide-spread are vulnerable to free-riding by ethnic groups that restrict cooperation to coethnics (e.g., ethnic mafias). An example of an ultrasocial institution, much discussed by historians and political scientists, is government by professional bureaucracies. Other examples include systems of formal education, with the Mandarin educational system in China as the most famous example, and universalizing religions. (p1) The idea lying behind the paper is then that war breeds higher degrees of social cohesion which leads to the formation of states which in turn generates greater civilisation. The authors lay out the causal chain as such, The conceptual core of the model invokes the following causal chain: spread of military technologies→intensification of warfare→evolution of ultrasocial traits→ rise of large-scale societies. (p2) The authors then use modelling that will be familiar to readers of this blog; that is, they effectively use a variety of regression techniques that are familiar to economists. As readers of this blog know, I am extremely skeptical of these techniques. The above mentioned study indicates that such techniques are now being applied to history under the guise of something called ‘cliodynamics’ and which is often associated with ecology (which I think is a somewhat dubious ‘science’…). The author of a Wired article discussing the study describes cliodynamics as such, Cliodynamics is a field of study created by Peter Turchin in the early 2000s. The idea is to use data as a means of predicting the future, but also as a way of testing theories about what happened in the past. You build a model that seeks to explain history, and then you test this model using real historical data. I assume that this will sound more than a little familiar to economists in general and those that read this blog in particular. There are so many problems with applying these techniques to historical data — which, in my opinion is identical to economic data — that it would take me far too long to enumerate them. But the core problem is that of causality. As can be seen from the causal chain laid out by the authors above the study definitely assumes a rather fixed causality. Yet mathematicians that developed these techniques have long pointed out that they cannot really identify causality. David Freedman, for example, notes that without being able to do real experiments — which we cannot generally do with historical or economic data — any causal inferences drawn from such models are useless. He writes, […]

Pingback by Historians, Avoid the Mistakes We Economists Made! | Fixing the Economists— 26 Oct, 2013 #

Thank you so much for a brilliant article. I am currently reading a book Designing Social Inquiry from Diamond, Keohane and Yerba (1994), and I think they haven´t mentioned this. Thick background is exactly what we need. If we don´t have it, we are only manipulating the reality by our biased researches. This is why I consider modern economy to be in a really bad shape, with its scientism, and no possibility to actually help us improve our thinking about the economical system. For example, if we talk about mortgage fear, we have to keep in mind the problem is not based only on the current interest, liquidity or home values, but also on expectations, unemployment level in the area, family situation of each one of the debtors, etc. Modern social science should, in my point of view, rather focus on the synthesis of the qualitative and quantitative researches.

Comment by Laura S.— 6 Oct, 2013 #

Great piece. Response:

http://fixingtheeconomists.wordpress.com/2013/10/02/the-holy-grail-distractions-in-econometrics-and-economic-modelling/

Comment by pilkingtonphil— 2 Oct, 2013 #

[…] Syll ran an interesting piece today on the “confounder” problem in econometrics. This is basically the problem of how do we […]

Pingback by The Holy Grail: Distractions in Econometrics and Economic Modelling | Fixing the Economists— 2 Oct, 2013 #