The problem of extrapolation

14 February, 2018 at 00:01 | Posted in Theory of Science & Methodology | 8 Comments

steelThere are two basic challenges that confront any account of extrapolation that seeks to resolve the shortcomings of simple induction. One challenge, which I call extrapolator’s circle, arises from the fact that extrapolation is worthwhile only when there are important limitations on what one can learn about the target by studying it directly. The challenge, then, is to explain how the suitability of the model as a basis for extrapolation can be established given only limited, partial information about the target … The second challenge is a direct consequence of the heterogeneity of populations studied in biology and social sciences. Because of this heterogeneity, it is inevitable there will be causally relevant differences between the model and the target population.

In economics — as a rule — we can’t experiment on the real-world target directly.  To experiment, economists therefore standardly construct ‘surrogate’ models and perform ‘experiments’ on them. To be of interest to us, these surrogate models have to be shown to be relevantly ‘similar’ to the real-world target, so that knowledge from the model can be exported to the real-world target. The fundamental problem highlighted by Steel is that this ‘bridging’ is deeply problematic​ — to show that what is true of the model is also true of the real-world target, we have to know what is true of the target, but to know what is true of the target we have to know that we have a good model  …

Most models in science are representations of something else. Models “stand for” or “depict” specific parts of a “target system” (usually the real world). A model that has neither surface nor deep resemblance to important characteristics of real economies ought to be treated with prima facie suspicion. How could we possibly learn about the real world if there are no parts or aspects of the model that have relevant and important counterparts in the real world target system? The burden of proof lays on the theoretical economists thinking they have contributed anything of scientific relevance without even hinting at any bridge enabling us to traverse from model to reality. All theories and models have to use sign vehicles to convey some kind of content that may be used for saying something of the target system. But purpose-built tractability assumptions — like, e. g., invariance, additivity, faithfulness, modularity, common knowledge, etc., etc. — made solely to secure a way of reaching deductively validated results in mathematical models, are of little value if they cannot be validated outside of the model.

All empirical sciences use simplifying or unrealistic assumptions in their modeling activities. That is (no longer) the issue – as long as the assumptions made are not unrealistic in the wrong way or for the wrong reasons.

Theories are difficult to directly confront with reality. Economists therefore build models of their theories. Those models are representations that are directly examined and manipulated to indirectly say something about the target systems.

There are economic methodologists and philosophers that argue for a less demanding view on modeling and theorizing in economics. And to some theoretical economists it is deemed quite enough to consider economics as a mere “conceptual activity” where the model is not so much seen as an abstraction from reality, but rather a kind of “parallel reality”. By considering models as such constructions, the economist distances the model from the intended target, only demanding the models to be credible, thereby enabling him to make inductive inferences to the target systems.

But what gives license to this leap of faith, this “inductive inference”? Within-model inferences in formal-axiomatic models are usually deductive, but that does not come with a warrant of reliability for inferring conclusions about specific target systems. Since all models in a strict sense are false (necessarily building in part on false assumptions) deductive validity cannot guarantee epistemic truth about the target system. To argue otherwise would surely be an untenable overestimation of the epistemic reach of surrogate models.

Models do not only face theory. They also have to look to the world. But being able to model a credible world, a world that somehow could be considered real or similar to the real world, is not the same as investigating the real world. Even though all theories are false, since they simplify, they may still possibly serve our pursuit of truth. But then they cannot be unrealistic or false in any way. The falsehood or unrealisticness has to be qualified (in terms of resemblance, relevance etc). At the very least, the minimalist demand on models in terms of credibility has to give away to a stronger epistemic demand of appropriate similarity and plausibility. One could of course also ask for a sensitivity or robustness analysis, but the credible world, even after having tested it for sensitivity and robustness, can still be a far way from reality – and unfortunately often in ways we know are important. Robustness of claims in a model does not per se give a warrant for exporting the claims to real world target systems.

Questions of external validity — the claims the extrapolation inference is supposed to deliver — are important. It can never be enough that models somehow are regarded as internally consistent. One always also has to pose questions of consistency with the data. Internal consistency without external validity is worth nothing.

Advertisements

8 Comments »

RSS feed for comments on this post. TrackBack URI

  1. Stephen J. Gould in his Magnus Opus “The Structure of Evolutionary Theory” speaks directly to the issue of extrapolation regarding the role of natural selection in creating novelty (new form) and how new discoveries in comparative genomics and evo-devo (evolutionary developmental biology) have “discombobulated” theoretical expectations.

    “The discovery [evo-devo] that has so discombobulated the confident expectations of orthodox theory can be stated briefly and baldly: the extensive “deep homology” now documented in both the genetic structure and developmental architecture of phyla separated at least since the Cambrian explosion (ca. 530 million years ago) should not, and cannot, exist under conventional concepts of natural selection as the dominant cause of evolutionary change…. Any wider hold of homology [which has already occurred] would have to inspire suspicions that the central tenet of orthodox Darwinism can no longer be sustained: the control of rates and directions of evolutionary change by the functional force of natural selection. In a particularly revealing quote within the greatest summary document of the Modern Synthesis, for example, Mayr … formulated the issue in a forthright manner. After all, he argued, more than 500 million years of independent evolution must erase any extensive genetic homology among phyla if natural selection holds such power to generate favorable change [novelty]. Adaptive evolution, over these long intervals, must have crafted and recrafted every genetic locus, indeed every nucleotide position, time and time again to meet the constantly changing selective requirements of continually varying environments. At this degree of cladistic separation, any independently evolved phenotypic similarity in basic adaptive architecture must represent the selective power of separate shaping by convergence, and cannot record conserved influence of retained genetic sequences, or common generation by parallelism: “In the early days of Mendelism there was much search for homologous genes that would account for such similarities. Much that has been learned about gene physiology makes it evident that the search for homologous genes is quite futile except in very close relatives.” But we now know that extensive genetic homology for fundamental features of development does hold across the most disparate animal phyla. (Gould 2002: 1065-1066) ”

    We have learned much more since then. Up in Canada they are trying to use these genetic toolkits (some of which are just switched off) to turn a chicken into a mini dinosaur; feathers to scales, teeth in beak; extended tail vertebrae, and such. Or resurrecting a baby mammoth.

    Essentially, and here is the irony, they have found and are learning to manipulate ancient genetic toolkits that are silenced by cis-regulatory markers, that either silence or enable downstream cascades for developmental pathways (fin to leg, etc.). Deep homology is very real and we are learning how to control it.

    How to Build a Dinosaur: The New Science of Reverse Evolution
    by Jack Horner et al.
    Link: http://a.co/imv0uJZ

    See: PAX-6 AND THE HOMOLOGY OF DEVELOPMENTAL PATHWAYS IN HOMOPLASTIC EYES OF SEVERAL PHYLA

  2. Genomics

  3. Reference List

    1. Horner, Jack. How to Build a Dinosaur [Extinction Doesn’t Have to be Forever]. New York: DUTTON; 2009; pp. 1-5.

    Notes: Stephen Jay Gould, one of the best-known evolutionary biologists of his time, wrote in Wonderful Life, his book on the weird and wonderful fossils of a rock formation known as the Burgess Shale, that you can’t go home again, evolutionary, unless you want to risk not being here when you come back. What he was saying was that evolution is a chance business, contingent on many influences and events. You can’t rewind it and run it over and hope to get the same result. The second time through Homo sapiens might not appear. Primates might not appear. (Horner 2009: 1-2)

    The dinosaurs had tails, some quite remarkable. Birds, the descendants of dinosaurs, now almost universally described by scientists as avian dinosaurs, do not have tails…. Some of the first birds had long tails, and some later birds had short tails. But there is no modern bird with a tail. (Horner 2009: 2)

    Is there a way to re-create that evolutionary change and see how it happened, right down to the molecules involved in directing, or stopping, tail growth? (Horner 2009: 2)

    I think the answer is yes. I think we can rewind the tape of bird evolution to the point before feathers or a tail emerged, or teeth disappeared. Then we can watch it run forward, and then rewind again, and try to play it without the evolutionary change, reverting to the original process. I am not suggesting we can do this on a grand scale, but we can pick a species, study its growth as an embryo, learn how it develops, and learn how to change that development. (Horner 2009: 3)

    Why couldn’t we take a chicken embryo and biochemically nudge it this way and that, until what hatched was not a chicken but a small dinosaur, with teeth, forearms with claws, and a tail? No reason at all. We haven’t done it yet. But we are taking the first small steps. (Horner 2009: 4-5)

  4. What meaning is being assigned the term, “directly”? As in “what one can learn about the target by studying it directly” in the quoted passage or “we can’t experiment on the real-world target directly” in the commentary. What would it mean to study a phenomenon “directly” or experiment on a subject “directly”? As opposed to ??? “indirectly”???
    .
    This contrast would make some sense if actual differences of method or approach could be explicitly made out. But, if such contrast cannot be adduced maybe it is time to give up the metaphor altogether.
    .
    Perhaps, it would be possible to talk of observation, measurement and — brace yourself! — interpretation? These pedestrian activities depend on using analytic theories and operational models to interact methodically with the phenomenon that is the subject of study.
    .
    Talking about export warrants and target populations make it seem as if there are separable realms of . . . what? thought? experience? . . . from which we launch idea missiles. Maybe that is a model of knowledge generation: if so, I would like to see it illustrated by example of — if not formally operationalized and tested against — some undoubted example of learning, to see if it adds anything to our understanding of what it is that we do to learn something about the world.
    .
    Extrapolation is an estimation of a value based on extending or projecting a known sequence of values or facts beyond the area or range that is certainly known. As such, it seems to me that is in the nature of hypothesis rather than inference, per se. The first doubt arises in me from the presumption that some measured value is really suitable to be treated as a “fact” about the world. In economics, it is common for economists to reason about stylized “facts” that are really just vague generalizations grounded in misconceptions and the accidents of local or recent experience. Should we treat “elasticity of demand” or macroeconomic “multiplier” as if an estimate is a fact and not just an ephemeral accounting artifact? The pseudo-physics that infects economics gave the name, elasticity, to suggest a fact akin to physical facts of metallurgy, where the elasticity of a metal alloy can be measured methodically and the estimate reflects a property of the metal. When the engineer relies on elasticity estimates for a metal, he is treating as fact not the measurement per se, but the implied property of the metal.

    I think an economist would be a fool to treat any estimate of the elasticity of demand or a multiplier for fiscal stimulus spending as if a reliable and stable property of an object in nature were being represented. The implied ontology is wrong. In metallurgy, the metal alloy being measured exists. It is not clear to me that demand for widgets or aggregate demand in a macroeconomic sense even “exists” in any sense faintly analogous to the existence of a particular metal alloy. To be sure, our understanding of what it means to be a particular metal alloy was itself a product of the study of metallurgy; the facts of an alloy and its properties is more than a singular estimate.

    This problem of extrapolation in social science hides a problem of ontology.

    • “This problem of extrapolation in social science hides a problem of ontology.” That is so true Bruce, yet sometimes it seems like a dirty secret no one wants to talk about. Is it just me, or is this observation real?

  5. “Teleology is like a mistress to a biologist: he cannot live without her but he’s unwilling to be seen with her in public.” J. B. S Haldane

    • Robert, that was a keeper. Thanks 😉


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.
Entries and comments feeds.