Perilous use of statistics

11 Dec, 2014 at 10:34 | Posted in Economics | 3 Comments

uncertaintyOne thing we’ve been discussing a lot lately is the discomfort many people—many researchers—feel about uncertainty … We see it all over: people find some pattern in their data and they don’t even want to consider the possibility that it might not hold in the general population …

The story seems pretty clear to me (but, admittedly, this is all speculation, just amateur psychology on my part): in general, people are uncomfortable with not knowing and would like to use statistics to create fortresses of certainty in a dangerous, uncertain world.

Along with this is an even more extreme attitude, which is not just to deny uncertainty but to deny variation … Even in regular psychology this attitude comes up, of focusing on similarities between people rather than differences. For example, we learn from Piaget that children can do X at age 3 and Y at age 4 and Z at age 5, not that some children go through one developmental process and others learn in a different order …

But then I recalled an even more extreme example, from a paper by Phoebe Clarke and Ian Ayres that claimed that “sports participation [in high school] causes women to be less likely to be religious . . . more likely to have children . . . more likely to be single mothers.” In my post on this paper a few months ago, I focused on the implausibility of the claimed effect sizes and on the problems with trying to identify individual-level causation from state-level correlations in this example … The article by Clarke and Ayres includes the following footnote:

“It is true that many successful women with professional careers, such as Sheryl Sandberg and Brandi Chastain, are married. This fact, however, is not necessarily opposed to our hypothesis. Women who participate in sports may “reject marriage” by getting divorces when they find themselves in unhappy marriages. Indeed, Sheryl Sandberg married and divorced before marrying her current husband.”

This footnote is a striking (to me) example of what Tversky and Kahneman called the fallacy of “the law of small numbers”: the attitude that patterns in the population should appear in any sample, in this case even in a sample of size 1. Even according to their own theories, Clarke and Ayres should not expect their model to work in every case! The above paragraph indicates that they want their theory to be something it can’t be; they want it to be a universal explanation that works in every example. Framed that way, this is obvious. My point, though, is that it appears that Clarke and Ayres were thinking deterministically without even realizing it.

Andrew Gelman


  1. this is all speculation, just amateur psychology

  2. I agree with Andrew that there is something going on that we would do well to understand, and that if one lacks a clear explanation then it is good to speculate – as long as one is clear that that is what one is doing. I am also somewhat dubious about professional, even Nobel-prize-winning, psychologists’ performance in this area.

    My own view is that most people ‘in a state of nature’ are pretty comfortable with uncertainty most of the time. On the other hand, it seems to me that Western civilization was largely built on people acting as if they were more certain than they had any right to be, so maybe what Andrew says of ‘people’ Andrew applies more to successful Western professionals. (Experiment, anyone?) For example, in Britain successful politicians will often act as if they are more certain than they really are, because to be more ‘honest’ would damage their careers. I speculate that this is cultural / social / institutional rather than more directly psychological, and hence something that could – in principal – be fixed.

  3. “…and on the problems with trying to identify individual-level causation from state-level correlations ”

    No, they are not trying to do that. They instead trying to identify state-level effects on individuals. .

    It seems like one confused person about probability and statistics, causality and sociology amd psychology, is trying to argue against another confused person or persons. This is the state of the art. Shame on the educational establishment.

Sorry, the comment form is closed at this time.

Blog at
Entries and comments feeds.