James Heckman — ‘Nobel prize’ winner gone wrong

20 Aug, 2017 at 11:32 | Posted in Statistics & Econometrics | 1 Comment

Here’s James Heckman in 2013:

“Also holding back progress are those who claim that Perry and ABC are experiments with samples too small to accurately predict widespread impact and return on investment. This is a nonsensical argument. Their relatively small sample sizes actually speak for — not against — the strength of their findings. Dramatic differences between treatment and control-group outcomes are usually not found in small sample experiments, yet the differences in Perry and ABC are big and consistent in rigorous analyses of these data.”

Wow. The “What does not kill my statistical significance makes it stronger” fallacy, right there in black and white … Heckman’s pretty much saying that if his results are statistically significant (and “consistent in rigorous analyses,” whatever that means) that they should be believed—and even more so if sample sizes are small (and of course the same argument holds in favor of stronger belief if measurement error is large).

With the extra special bonus that he’s labeling contrary arguments as “nonsensical” …

Heckman is wrong here. Actually, the smaller sample sizes (and also the high variation in these studies) speaks against—not for—the strength of the published claims …

Andrew Gelman

One of the first things yours truly warns his statistics students against, is to jump to the conclusion that signal-to-noise levels have to be high just because they get statistically significant estimates when running regressions. One would have thought a prize winner should know that too …