‘Theoretical cherrypicking’ in economics

12 Jul, 2023 at 14:41 | Posted in Economics | 1 Comment

The proposition that theoretical models are necessary for understanding our economic system does not imply that having some particular theoretical model automatically means that we understand anything useful. If one is creative in choosing the ‘right’ assumptions and reasonably clever, then one can produce all kinds of results … This potentially creates a problem that might be called ‘theoretical cherrypicking.’ In empirical work it is well understood that biased and misleading results are obtained if one cherry picks the data, that is, if one selects data that generally support a desired result and excludes those data that do not. Understandably, this is viewed by careful empiricists as a mortal sin.

Analogously, in theoretical work it is often possible to cherry pick assumptions to produce a given result. Essentially it is a matter of reverse engineering: what do I need to assume to obtain the result that configuration X is optimal or that increasing Y will increase Z? … If the assumptions critical to the result are patently false, then the model will not be taken seriously. In many other cases, however, it will not be so transparent that the model is fatally disconnected from the real world …

My reason for introducing the notion of theoretical cherry picking is to emphasize that since a given result can almost always be supported by a theoretical model, the existence of a theoretical model that leads to a given result in and of itself tells us nothing definitive about the real world. Though this is obvious when stated baldly like this, in practice various claims are often given credence—certainly more than they deserve simply because there are theoretical models in the literature that ‘back up’ these claims. In other words, the results of theoretical models are given an ontological status that they do not deserve. In my view this occurs because models—and specifically their assumptions—are not always subjected to the critical evaluation that is necessary to see whether and how they apply to the real world

Paul Pfleiderer

Pfleiderer’s perspective on ‘theoretical cherrypicking’ may be applied to many of the issues involved when modelling complex and dynamic economic phenomena. Let me take just one example — simplicity.

When it comes to modelling I do see the point often emphatically made for simplicity among economists and econometricians — but only as long as it doesn’t impinge on our truth-seeking. “Simple” macroeconom(etr)ic models may of course be an informative heuristic tool for research. But if practitioners of modern macroeconom(etr)ics do not investigate and make an effort of providing a justification for the credibility of the simplicity assumptions on which they erect their building, it will not fulfil its tasks. Maintaining that economics is a science in the “true knowledge” business, I remain a sceptic of the pretences and aspirations of  “simple” macroeconom(etr)ic models and theories. So far, I can’t really see that e. g. “simple” microfounded models have yielded very much in terms of realistic and relevant economic knowledge.

All empirical sciences use simplifying or unrealistic assumptions in their modelling activities. That is not the issue – as long as the assumptions made are not unrealistic in the wrong way or for the wrong reasons.

Models do not only face theory. They also have to look to the world. Being able to model a “credible world,” a world that somehow could be considered real or similar to the real world, is not the same as investigating the real world. Even though — as Pfleiderer acknowledges — all theories are false, since they simplify, they may still possibly serve our pursuit of truth. But then they cannot be unrealistic or false in any way. The falsehood or unrealisticness has to be qualified.

Explanation, understanding and prediction of real-world phenomena, relations and mechanisms therefore cannot be grounded on simpliciter assuming simplicity. If we cannot show that the mechanisms or causes we isolate and handle in our models are stable, in the sense that when we export them from are models to our target systems they do not change from one situation to another, then they – considered “simple” or not – only hold under ceteris paribus conditions and a fortiori are of limited value for our understanding, explanation and prediction of our real world target system.

The obvious ontological shortcoming of a basically epistemic – rather than ontological – approach, is that “similarity” or “resemblance” tout court does not guarantee that the correspondence between model and target is interesting, relevant, revealing or somehow adequate in terms of mechanisms, causal powers, capacities or tendencies. No matter how many convoluted refinements of concepts made in the model, if the simplifications made do not result in models similar to reality in the appropriate respects (such as structure, isomorphism etc), the surrogate system becomes a substitute system that does not bridge to the world but rather misses its target.

Constructing simple macroeconomic models somehow seen as “successively approximating” macroeconomic reality, is a rather unimpressive attempt at legitimising using fictitious idealisations for reasons more to do with model tractability than with a genuine interest of understanding and explaining features of real economies. Many of the model assumptions standardly made by neoclassical macroeconomics – simplicity being one of them – are restrictive rather than harmless and could a fortiori anyway not in any sensible meaning be considered approximations at all.

If economists aren’t able to show that the mechanisms or causes that they isolate and handle in their “simple” models are stable in the sense that they do not change when exported to their “target systems”, they do only hold under ceteris paribus conditions and are a fortiori of limited value to our understanding, explanations or predictions of real economic systems.

That Newton’s theory in most regards is simpler than Einstein’s is of no avail. Today Einstein has replaced Newton. The ultimate arbiter of the scientific value of models cannot be simplicity.