Texas sharpshooter fallacy

14 December, 2017 at 09:21 | Posted in Statistics & Econometrics | Comments Off on Texas sharpshooter fallacy

Data clusters are everywhere, even in random data. Someone who looks for an explanation will inevitably find one, but a theory that fits a data cluster is not persuasive evidence. The found explanation needs to make sense, and it needs to be tested with uncontaminated data.

gasmithSimilarly, someone who fires enough bullets at enough targets is bound to hit​ one. A researcher who tests hundreds of theories is bound to find evidence supporting at least one. Such evidence is unconvincing unless the theory is sensible and confirmed with fresh data.

When you hear that the data support a theory, don’t be persuaded until you’ve answered two questions. First, does the theory make sense? If it doesn’t, don’t be easily persuaded that nonsense is sensible. Second, is there​ a Texas sharpshooter​ in the house? Did the person promoting the theory look at the data before coming up with​ the theory​? Were hundreds of theories tested before settling on the theory being promoted? If there is a smoking gun, withhold judgment on the theory until it has been tested with different data.

Advertisements

Create a free website or blog at WordPress.com.
Entries and comments feeds.