Texas sharpshooter fallacy

14 December, 2017 at 09:21 | Posted in Statistics & Econometrics | Leave a comment

Data clusters are everywhere, even in random data. Someone who looks for an explanation will inevitably find one, but a theory that fits a data cluster is not persuasive evidence. The found explanation needs to make sense, and it needs to be tested with uncontaminated data.

gasmithSimilarly, someone who fires enough bullets at enough targets is bound to hit​ one. A researcher who tests hundreds of theories is bound to find evidence supporting at least one. Such evidence is unconvincing unless the theory is sensible and confirmed with fresh data.

When you hear that the data support a theory, don’t be persuaded until you’ve answered two questions. First, does the theory make sense? If it doesn’t, don’t be easily persuaded that nonsense is sensible. Second, is there​ a Texas sharpshooter​ in the house? Did the person promoting the theory look at the data before coming up with​ the theory​? Were hundreds of theories tested before settling on the theory being promoted? If there is a smoking gun, withhold judgment on the theory until it has been tested with different data.

Advertisements

Leave a Comment »

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s

Create a free website or blog at WordPress.com.
Entries and comments feeds.