LARS P. SYLL

Causal inference — Fisher vs. Neyman (student stuff)

.

Statistical assumptions as empirical commitments

Real probability samples have two great benefits: (i) they allow unbiased extrapolation from the sample; (ii) with data internal to the sample, it is possible to estimate how much results are likely to change if another sample is taken. These benefits, of course, have a price: drawing probability samples is hard work. An investigator who assumes that a convenience sample is like a random sample seeks to obtain the benefits without the costs—just on the basis of assumptions. If scrutinized, few convenience samples would pass muster as the equivalent of probability samples. Indeed, probability sampling is a technique whose use is justified because it is so unlikely that social processes will generate representative samples. Decades of survey research have demonstrated that when a probability sample is desired, probability sampling must be done. Assumptions do not suffice. Hence, our first recommendation for research practice: whenever possible, use probability sampling.

If the data-generation mechanism is unexamined, statistical inference with convenience samples risks substantial error. Bias is to be expected and independence is problematic. When independence is lacking, the p-values produced by conventional formulas can be grossly misleading. In general, we think that reported p-values will be too small; in the social world, proximity seems to breed similarity. Thus, many research results are held to be statistically significant when they are the mere product of chance variation.

Richard Berk & David Freedman

In econometrics one often gets the feeling that many of its practitioners think of it as a kind of automatic inferential machine: input data and out comes casual knowledge. This is like pulling a rabbit from a hat. Great — but first you have to put the rabbit in the hat. And this is where assumptions come into the picture.

The assumption of imaginary ‘super populations’ is one of many dubious assumptions used in modern econometrics and statistical analyses to handle uncertainty. As social scientists — and economists — we have to confront the all-important question of how to handle uncertainty and randomness. Should we define randomness with probability? If we do, we have to accept that to speak of randomness we also have to presuppose the existence of nomological probability machines, since probabilities cannot be spoken of – and actually, to be strict, do not at all exist – without specifying such system-contexts. Accepting a domain of probability theory and sample space of infinite populations also implies that judgments are made on the basis of observations that are actually never made.

Economists who apply econometrics usually react to this kind of critique by telling us they — of course — already know it all. But, if so, why do they just keep on doing what they do without ever discussing the validity of these assumptions?

Fictional analysis does not give us anything but fictional results. Inferences to imaginary populations are — rather unsurprisingly — also imaginary. Convenient fiction? Yes. Of real-world significance? No!

Without knowledge of the real-world data-generating mechanisms, the use of statistics is of questionable value.

Statistical reification — a deadly sin

It would not be an exaggeration to say that many of us see academic statistics as “the sick man of science,” an alarming diagnosis in light of the central role it plays in science in general. The situation is maintained through specious and often byzantine rationales for destructive traditions, which is as expected from another human condition: Blindness to conceptual mistakes when correcting those would threaten the entire foundation on which one’s teaching and research
(which is to say, entire careers) have been built. The central example is how, by themselves, P-values and CI do not test any hypothesis, nor do they measure the significance of results or the confidence we should have in them. The sense otherwise is an ongoing cultural error perpetuated by large segments of the statistical and research community via evocative terminology such as “statistical significance” and “confidence interval” for what are only dichotomous P-value summaries …

P-values and CIs can only become contextually interpretable when derived explicitly from causal stories about the real data generator (such as randomization), and can only become reliable when those stories are based on valid and public documentation of the physical mechanisms that generated the data. Absent these assurances, traditional interpretations of “inferential statistics” become pernicious fictions that need to be replaced by far more circumspect descriptions of data and model relations.

Sander Greenland

Better than Diff-in-Diff

.

How to avoid pernicious fictional statistics

Mathematics is a limited component of solutions to real-world problems, as it expresses only what is expected to be true if all our assumptions are correct, including implicit assumptions that are omnipresent and often incorrect. Statistical methods are rife with implicit assumptions whose violation can be life-threatening when results from them are used to set policy. Among them are that there is human equipoise or unbiasedness in data generation, management, analysis, and reporting. These assumptions correspond to levels of cooperation, competence, neutrality, and integrity that are absent more often than we would like to believe.

Given this harsh reality, we should ask what meaning, if any, we can assign to the P-values, “statistical significance” declarations, “confidence” intervals, and posterior probabilities that are used to decide what and how to present (or spin) discussions of analyzed data. By themselves, P-values and CI do not test any hypothesis, nor do they measure the significance of results or the confidence we should have in them. The sense otherwise is an ongoing cultural error perpetuated by large segments of the statistical and research community via misleading terminology.

So-called “inferential” statistics can only become contextually interpretable when derived explicitly from causal stories about the real data generator (such as randomization), and can only become reliable when those stories are based on valid and public documentation of the physical mechanisms that generated the data. Absent these assurances, traditional interpretations of statistical results become pernicious fictions that need to be replaced by far more circumspect descriptions of data and model relations.

Sander Greenland

All science entails human judgment, and using statistical models does not relieve us of that necessity. Working with misspecified models, the scientific value of ‘significance testing’ is actually zero — even though you’re making valid statistical ‘inferences’! Statistical models and concomitant significance tests are no substitutes for doing real science.

In its standard form, a significance test is not the kind of ‘severe test’ that we are looking for in our search for being able to confirm or disconfirm empirical scientific hypotheses. This is problematic for many reasons, one being that there is a strong tendency to accept the ‘null hypothesis’ since it can’t be rejected at the standard 5% significance level. And as shown over and over again when it is applied, people have a tendency to read ‘not disconfirmed’ as ‘probably confirmed.’

The excessive reliance on significance​ testing in science is disturbing​ and should be fought. But it is also important to put significance testing abuse in perspective. The real problem in today’s social sciences is not significance testing per se. No, the real problem has to do with the often unqualified and mechanistic application of statistical methods to real-world phenomena without having even the slightest idea of how the assumptions behind​ the statistical models condition and severely limit the value of the inferences made.

Econometrics — a mixture of ‘metaphors, metaphysics, and a pinch of bluff’

This effort to create foundations for the probability approach in econometrics finally results in an inconsistent set of claims in its defence.

First, there are vast amounts of experience which warrant a frequency interpretation. This is supported by repetitive discussions of experimental design, but the inability to experiment inspires an epistemological interpretation. Then Haavelmo mentions the futility of bothering with these issues because the probability approach is most of all a useful tool. This would be an instrumentalistic justification for its use if Haavelmo gave supportive evidence for his claim. There is not one example which attempts to do so …

In practice econometricians mostly have to do without experiments. From the early days on, they tried to provide stand-ins for real experimentation. First, it was argued that economic models themselves could substitute as alternatives to experiments. However, this was not intended so much as to justify probabilistic inference, but as, primarily, to level economics with the ‘experimental sciences. Then Haavelmo introduced his version of experimental design in order to make economic theories (statistically) operational. Again, this could not provide the justification for using the probabilistic framework. The founders of econometrics tried to adapt the sampling approach to a non-experimental small sample domain. They tried to justify this with a priori and analytical arguments. However, the ultimate argument for a ‘probability approach in econometrics’ consists of a mixture of metaphors, metaphysics and a pinch of bluff.

As always — if people do not give justifications for the validity of the assumptions they make, well then those assumptions are nothing but assumptions. Just talking about — as did Haavelmo —  “experimental design,” “hoping” that nature takes care of the necessary ceteris paribus conditions, and believing it “fruitful” to treat economic data as if they were a set of truly random data does not take us very far. Treating observational data as if they were experimental is nothing but wishful thinking, based on a totally unwarranted assumption. Bridging from econometric and statistical models to real-world target systems demands much more. Most of all you have to — not just rhetorically — search for the real data-generating mechanisms.

The assumptions of imaginary “super-populations” or “hypothetical infinite populations” are some of the many dubious assumptions used in modern econometrics.

As social scientists — and economists — we have to confront the all-important question of how to handle uncertainty and randomness. Should we define randomness with probability? If we do, we have to accept that to speak of randomness we also have to presuppose the existence of nomological probability machines, since probabilities cannot be spoken of – and actually, to be strict, do not at all exist – without specifying such system-contexts. Accepting a domain of probability theory and sample space of infinite populations also implies that judgments are made on the basis of observations that are actually never made!

Infinitely repeated trials or samplings never take place in the real world. So that cannot be a sound inductive basis for a science with aspirations of explaining real-world socio-economic processes, structures or events. It’s not tenable.

And as if this wasn’t enough, one could, as argued, also seriously wonder what kind of “populations” these statistical and econometric models ultimately are based on. Why should we as social scientists — and not as pure mathematicians working with formal-axiomatic systems without the urge to confront our models with real target systems — unquestioningly accept models based on concepts like the “infinite superpopulations” used in e.g. the potential outcome framework that has become so popular lately in social sciences?

Of course, one could treat observational or experimental data as random samples from real populations. I have no problem with that. At least not as long as one remembers that usually, the random-sampling assumptions do not apply to the kind of ‘convenience samples’ that economic data standardly constitute and that the usual statistical inferences therefore can not be made.

However probabilistic econometrics does not content itself with that kind of population. Instead, it creates imaginary populations of “parallel universes” and assumes that our data are random samples from that kind of  “infinite superpopulations.”

But this is actually nothing else but hand-waving! And it is inadequate for real science. In social sciences — including economics — it’s always wise to ponder C. S. Peirce’s remark that universes are not as common as peanuts …

Nowadays it has almost become a self-evident truism among economists that you cannot expect people to take your arguments seriously unless they are based on or backed up by advanced econometric modelling. So legions of mathematical-statistical theorems are proved — and heaps of fiction are being produced, masquerading as science. The rigour​ of the econometric modelling and the far-reaching assumptions they are built on are frequently not supported by data.

Modelling assumptions made in statistics and econometrics are more often than not made for mathematical tractability reasons, rather than verisimilitude. That is unfortunately also a reason why the methodological ‘rigour’ encountered when taking part in statistical and econometric research to a large degree is nothing but a deceptive appearance. The models constructed may seem technically advanced and very ‘sophisticated,’ but that’s usually only because the problems here discussed have been swept under the carpet. Assuming that our data are generated by ‘coin flips’ in an imaginary ‘superpopulation’ only means that we get answers to questions that we are not asking.