Beware of Monte Carlo simulations

17 Jun, 2023 at 16:59 | Posted in Statistics & Econometrics | 3 Comments

In some fields—physics, geophysics, climate science, sensitivity analysis, and uncertainty quantification in particular—there is a popular impression that probabilities can be estimated in a ‘neutral’ or ‘automatic’ way by doing Monte Carlo simulations: just let the computer reveal the distribution …

Setting aside other issues in numerical modeling, Monte Carlo simulation is a way to substitute computing for hand calculation. It is not a way to discover the probability distribution of anything; it is a way to estimate the numerical values that result from an assumed distribution. It is a substitute for doing an integral, not a way to uncover laws of Nature.

Monte Carlo doesn’t tell you anything that wasn’t already baked into the simulation. The distribution of the output comes from assumptions in the input (modulo bugs): a probability model for the parameters in the simulation. It comes from what you program the computer to do. Monte Carlo reveals the consequences of your assumptions about randomness. The rabbit goes into the hat when you build the probability model and write the software. The rabbit does not come out of the hat without having gone into the hat first.

Philip B. Stark

Stark’s article is an absolute must-read! One of the best statistics critiques yours truly has read for years.