Does it — really — take a model to beat a model? No!

8 Sep, 2020 at 21:37 | Posted in Economics | 9 Comments

Many economists respond to criticism by saying that ‘all models are wrong’ … But the observation that ‘all models are wrong’ requires qualification by the second part of George Box’s famous aphorism — ‘but some are useful’ … The relevant  criticism of models in macroeconomics and finance is not that they are ‘wrong’ but that they have not proved useful in macroeconomics and have proved misleading in finance.

kaykingWhen we provide such a critique, we often hear another mantra to which many economists subscribe: ‘It takes a model to beat a model.’ On the contrary, we believe that it takes facts and observations to beat a model … If a model fails to answer the problem to which it is addressed, it should be put back in the toolbox … It is not necessary to have an alternative tool available to know that the plumber who arrives armed only with a screwdriver is not the tradesman we need.

A similar critique yours truly sometimes encounters is that as long as I cannot come up with some own alternative model to the failing mainstream models, I shouldn’t expect people to pay attention.

This is, however, not only wrong for the reasons given by Kay and King, but is also to utterly misunderstand the role of philosophy and methodology of economics!

As John Locke wrote in An Essay Concerning Human Understanding:

19557-004-21162361The Commonwealth of Learning is not at this time without Master-Builders, whose mighty Designs, in advancing the Sciences, will leave lasting Monuments to the Admiration of Posterity; But every one must not hope to be a Boyle, or a Sydenham; and in an Age that produces such Masters, as the Great-Huygenius, and the incomparable Mr. Newton, with some other of that Strain; ’tis Ambition enough to be employed as an Under-Labourer in clearing Ground a little, and removing some of the Rubbish, that lies in the way to Knowledge.

That’s what philosophy and methodology can contribute to economics — clearing obstacles to science by clarifying limits and consequences of choosing specific modelling strategies, assumptions, and ontologies.

unnameadIt takes a model to beat a model has to be one of the stupider things, in a pretty crowded field, to come out of economics. … I don’t get it. If a model is demonstrably wrong, that should surely be sufficient for rejection. I’m thinking of bridge engineers: ‘look I know they keep falling down but I’m gonna keep building them like this until you come up with a better way, OK?’

Jo Michell


  1. Lars said:
    “That’s what philosophy and methodology can contribute to economics”
    Fair enough, every argument needs to be critiqued.
    But this is also a facile point to make.
    At least those building models are willing to have a go. And some of these models might be utterly incommensurate with reality – some may be ideologically driven.
    The Truth is not writ large on a grand neon sign sitting on top of a mountain, a path to which up the mountain has to be found.
    Most times, all that can be seen is mountains. It’s difficult enough to choose which mountain to ascend let alone find a path up that mountain.

  2. ” It is not necessary to have an alternative tool available to know that the plumber who arrives armed only with a screwdriver is not the tradesman we need.”
    There are times when all a plumber needs is a screwdriver. Truly. So it is presumptive to say the plumber is ill equipped.
    It has been said many times in this blog that what is needed is to assemble the facts and proceed from there.
    The problem is, how do we know which facts are relevant. It’s not so straight forward.
    It seems to me that a model can direct an intellectual project to the relevant facts. But it’s not that simple either.
    I would say there is an iterative process which needs to be worked through.
    The question is, which is the horse and which is the cart, the “facts” or the model?


  3. I would go even further than John Kay, models are not only useless, they are a massive distraction and actually harmful in trying to really understand what is going on.

    It is interesting to read the mainstream’s response to Kay:

    What’s Wrong with Economic Models? A Response to John Kay
    Woodford, Michael

    ‘John Kay’s thought-provoking essay argues that economists have been led astray by excessive reliance on formal models derived from assumptions that bear too little similarity to the world we live in. And it is surely true that at least at times, disastrous decisions have been made through reliance on models that proved to be incorrect. Some of the statistical models used by financial institutions to value derivative securities based on mortgages just before the recent financial crisis provide a case in point.

    But I do not believe that the route to sounder economic reasoning will involve an abandonment of economists’ penchant for reasoning with the use of models. Models allow the internal consistency of a proposed argument to be checked with greater precision; they allow more finely-grained differentiation among alternative hypotheses, and they allow longer and more subtle chains of reasoning to be deployed without both author and reader becoming hopelessly tangled in them. Nor do I believe it is true that economists who are more given to the use of formal mathematical analysis are generally more dogmatic in their conclusions than those who customarily rely upon more informal styles of argument. Often, reasoning from formal models makes it easier to see how strong are the assumptions required for an argument to be valid, and how different one’s conclusions may be depending on modest changes in specific assumptions. And whether or not any given practitioner of economic modeling is inclined to honestly assess the fragility of his conclusions, the use of a model to justify those conclusions makes it easy for others to see what assumptions have been relied upon, and hence to challenge them. As a result, the resort to argumentation based on models facilitates the general project of critical inquiry that represents, in my view, our best hope for some eventual approach toward truth.’

    • “Often, reasoning from formal models makes it easier to see how strong are the assumptions required for an argument to be valid, and how different one’s conclusions may be depending on modest changes in specific assumptions.”

      The problem with Woodford’s argument is of course is that the modest changes in specific assumptions they make are not interesting because the assumptions they are based on are highly controversial philosophical foundations or completely way out in the first place. This is just rearranging the deckchairs.

      Even the basic assumptions of rational choice and constrained optimisation, for example, are questionable. And that’s just the beginning. You need far more than tinkering to acquire true understanding.

      But what is important is the facts, knowing them, and accounting for them – that is where 90 per cent of your efforts should be. The other 10 per cent goes in a technical appendix. If it is just conducting thought experiments in a world of abstraction and make belief that you do, you should be in a different department.

    • I tried (with a lot of professional help) to read Michael Woodford’s New Keynesian bible, and did not understand half of it. I blame Woodford. I can report Michael Woodford does indeed rely excessively on formal models with unrealistic assumptions, and shows precious little respect for the hazard to any approach toward truth, however approximate, that shows little interest in models that have no definite or stable solution. This is a terrible bias for a would-be truth-teller to suffer and lack the self-awareness to confront. It is quite possible, even likely, that the actual economy is a jumble of still disputed bargains and unsolved, possibly insoluble problems. If your method insists instead on a presumption that a collective will seeks to solve an intertemporal optimization problem — surely a very, very difficult problem and not obviously solvable — I think it would be fair to conclude that the author is profoundly stupid.
      The economics scholars who follow Woodford are fed by a proliferation of models, and Woodford patiently instructs them in how to cross the “t” and dot the “i” in dynamic stochastic general equilibrium models that have no referents in the actual, institutional economy.
      I am not about to join Nanikore in endorsing a new Dust bowl empiricism. Woodford would claim empiricism is alive and well, but fed by a theory of economics that gives no account of money, while purporting to advise on monetary policy strikes me as lacking in both sense and good faith. The numerous “frictions” introduced to support a claim to accurately account for facts undermines all the rest of the claims of “rigor”.

      • Kudos for ploughing through Woodford’s profound analysis of monetary policy without money Bruce.

        If anyone thinks there has been a change in the macro-economics profession or any introspection since the GFC, they are greatly mistaken. The zero-lower bound, if ever there was a red-herring, seems to have been there saviour – Model has been right all along.

    • “Some of the statistical models used by financial institutions to value derivative securities based on mortgages just before the recent financial crisis provide a case in point.”
      The financial models did not account for a wide, spreading, irrational panic among traders. Or one might say that they implicitly counted on the Fed to act as ultimate insurer against such panics. The Fed should sell panic insurance upfront so finance firms can explicitly hedge against panics …
      Panics are irrational and arbitrary, not a physical “fact”. 4rth-degree derivatives such as volatility indices give you some insight into trader moods; finance models use VIX contango to generate trading signals. This is a constantly evolving model …

      • To clarify the penultimate sentence in the above post:
        Say we call real economy widgets “the underlying” and derive some physical production function for them, using energy, materials, and labor inputs. We might call stock in the widget-producing company a first-order derivative of the underlying. (Already, the production function for company stock is removed from actual production: Tesla, for example produces an order of magnitude fewer cars than Toyota, yet its market capitalization, or value of stock, is greater than Toyota’s.) A stock index, such as the S&P 500, is a second-order derivative (an ergodic ensemble average of the individual stocks in the index). Futures contracts on the S&P 500 index constitute a third-order derivative. A volatility index such as the VIX is a fourth-order derivative of the underlying widget production function …
        Traders today are trading such fourth-order derivative products, and such trades have a feedback effect on real prices. House buyers see house prices go up, yet are still able to afford them because their investment income from fourth-order derivative trading is rising even faster.
        Production functions for derivatives involve a lot of subjective, arbitrary valuations by parties with clearly conflicting interests. Yet they rely on the Fed as ultimate insurer, and we can too in the service of justice …
        Note: An article recently said that defaulted mortgages are up near levels that resulted in a financial panic in 2008, but interest rates are at all-time lows and refinances are climbing. The financial insurance on mortgage backed securities has learned since 2008. The system today can tolerate much higher default rates without straining liquidity. Of course the Fed’s implicit promise to backstop MBS and other market-created securities plays a big role in stabilizing the financial insurance industry. Again, we should use the Fed’s power of unlimited liquidity to finance public spending such as on basic income …

  4. I’ve constructed a mathematical model of the instructions that need to be sent from a dog’s brain to it’s legs and jaw to enable it to catch a frisbee. The model takes into account dozens of relevant factors: gravity, speed of the frisbee, wind speed and direction, atmospheric pressure, etc. To my amazement, dogs are not interested in my model and manage to catch frisbees without even looking at my model.

    I’m very disappointed.

Sorry, the comment form is closed at this time.

Blog at
Entries and Comments feeds.