A low-powered study is only going to be able to see a pretty big effect. But sometimes you know that the effect, if it exists, is small. In other words, a study that accurately measures the effect … is likely to be rejected as statistically insignificant, while any result that passes the p < .05 test is either a false positive or a true positive that massively overstates the … effect.
A conventional boundary, obeyed long enough, can be easily mistaken for an actual thing in the world. Imagine if we talked about the state of the economy this way! Economists have a formal definition of a ‘recession,’ which depends on arbitrary thresholds just as ‘statistical significance’ does. One doesn’t say, ‘I don’t care about the unemployment rate, or housing starts, or the aggregate burden of student loans, or the federal deficit; if it’s not a recession, we’re not going to talk about it.’ One would be nuts to say so. The critics — and there are more of them, and they are louder, each year — say that a great deal of scientific practice is nuts in just this way.
If anything, this underlines how important it is not to equate science with statistical calculation. All science entail human judgement, and using statistical models doesn’t relieve us of that necessity. Working with misspecified models, the scientific value of significance testing is actually zero — even though you’re making valid statistical inferences! Statistical models and concomitant significance tests are no substitutes for doing real science. Or as a noted German philosopher once famously wrote:
There is no royal road to science, and only those who do not dread the fatiguing climb of its steep paths have a chance of gaining its luminous summits.
Statistical significance doesn’t say that something is important or true. Since there already are far better and more relevant testing that can be done (see e. g. here and here), it is high time to consider what should be the proper function of what has now really become a statistical fetish. Given that it anyway is very unlikely than any population parameter is exactly zero, and that contrary to assumption most samples in social science and economics are not random or having the right distributional shape – why continue to press students and researchers to do null hypothesis significance testing, testing that relies on a weird backward logic that students and researchers usually don’t understand?
In its standard form, a significance test is not the kind of “severe test” that we are looking for in our search for being able to confirm or disconfirm empirical scientific hypothesis. This is problematic for many reasons, one being that there is a strong tendency to accept the null hypothesis since it can’t be rejected at the standard 5% significance level. In their standard form, significance tests bias against new hypotheses by making it hard to disconfirm the null hypothesis.
As shown over and over again when it is applied, people have a tendency to read “not disconfirmed” as “probably confirmed.” And — most importantly — we should of course never forget that the underlying parameters we use when performing significance tests are model constructions. Our p-values mean next to nothing if the model is wrong. As David Freedman writes in Statistical Models and Causal Inference:
I believe model validation to be a central issue. Of course, many of my colleagues will be found to disagree. For them, fitting models to data, computing standard errors, and performing significance tests is “informative,” even though the basic statistical assumptions (linearity, independence of errors, etc.) cannot be validated. This position seems indefensible, nor are the consequences trivial. Perhaps it is time to reconsider.
Are you tired of people like walked-out Harvard economist Greg Mankiw and their repeated attempts at defending the 1 % by invoking Adam Smith’s invisible hand and arguing that a market economy is some kind of moral free zone where, if left undisturbed, people get what they “deserve”?
Then I suggest you listen to this great conversation on inequality:
Listening to Solow and Krugman is a healthy antidote to unashamed neoliberal inequality apologetics.
The outstanding faults of the economic society in which we live are its failure to provide for full employment and its arbitrary and inequitable distribution of wealth and incomes … I believe that there is social and psychological justification for significant inequalities of income and wealth, but not for such large disparities as exist to-day.
John Maynard Keynes General Theory (1936)
A society where we allow the inequality of incomes and wealth to increase without bounds, sooner or later implodes. The cement that keeps us together erodes and in the end we are only left with people dipped in the ice cold water of egoism and greed.
One of the most effective ways of clearing up this most serious of all semantic confusions is to point out that private debt differs from national debt in being external. It is owed by one person to others. That is what makes it burdensome. Because it is interpersonal the proper analogy is not to national debt but to international debt…. But this does not hold for national debt which is owed by the nation to citizens of the same nation. There is no external creditor. We owe it to ourselves.
A variant of the false analogy is the declaration that national debt puts an unfair burden on our children, who are thereby made to pay for our extravagances. Very few economists need to be reminded that if our children or grandchildren repay some of the national debt these payments will be made to our children or grandchildren and to nobody else. Taking them altogether they will no more be impoverished by making the repayments than they will be enriched by receiving them.
Abba Lerner The Burden of the National Debt (1948)
Touring again. Conference in Stockholm and guest appearence in the Swedish Parliament and the National Institute of Economic Research. Regular blogging to be resumed during the weekend.
Neoclassical economic theory today is in the story-telling business whereby economic theorists create make-believe analogue models of the target system – usually conceived as the real economic system. This modeling activity is considered useful and essential. Since fully-fledged experiments on a societal scale as a rule are prohibitively expensive, ethically indefensible or unmanageable, economic theorists have to substitute experimenting with something else. To understand and explain relations between different entities in the real economy the predominant strategy is to build models and make things happen in these “analogue-economy models” rather than engineering things happening in real economies.
Formalistic deductive “Glasperlenspiel” can be very impressive and seductive. But in the realm of science it ought to be considered of little or no value to simply make claims about the model and lose sight of reality. As Julian Reiss writes:
There is a difference between having evidence for some hypothesis and having evidence for the hypothesis relevant for a given purpose. The difference is important because scientific methods tend to be good at addressing hypotheses of a certain kind and not others: scientific methods come with particular applications built into them … The advantage of mathematical modelling is that its method of deriving a result is that of mathemtical prof: the conclusion is guaranteed to hold given the assumptions. However, the evidence generated in this way is valid only in abstract model worlds while we would like to evaluate hypotheses about what happens in economies in the real world … The upshot is that valid evidence does not seem to be enough. What we also need is to evaluate the relevance of the evidence in the context of a given purpose.
Neoclassical economics has since long given up on the real world and contents itself with proving things about thought up worlds. Empirical evidence only plays a minor role in economic theory, where models largely function as a substitute for empirical evidence. Hopefully humbled by the manifest failure of its theoretical pretences, the one-sided, almost religious, insistence on axiomatic-deductivist modeling as the only scientific activity worthy of pursuing in economics will give way to methodological pluralism based on ontological considerations rather than formalistic tractability. To have valid evidence is not enough. What economics needs is sound evidence.
Discussing Paul Romer’s “mathiness” concept, Peter Dorman yesterday criticized economists’ belief that theories and models being “consistent with” data somehow make the theories and models a success story. And Chris Dillow elaborates on the weakness of this “consistent with” error in a post today:
If a man has no money, this is “consistent with” the theory that he has given it away. But if in fact he has been robbed, that theory is grievously wrong. Mere consistency with the facts is not sufficient.
This is a point which some defenders of inequality miss. Of course, you can devise theories which are “consistent with” inequality arising from reasonable differences in choices and marginal products. Such theories, though, beg the question: is that how inequality really emerged?** And the answer, to put it mildly, is: only partially. It also arose from luck, inefficient selection, rigged markets, rent-seeking and outright theft …
The Duhem-Quine thesis warns us that facts under-determine theory: they are “consistent with” multiple theories. This is perhaps especially true when those facts are snapshots. For example, a Gini coefficient – being a mere snapshot of inequality – tells us nothing about how the inequality emerged.
So, how can we guard against the “consistent with” error? One thing we need is history: this helps tell us how things actually happened. And – horrific as it might seem to some economists – we also need sociology: we need to know how people actually behave and not merely that their behaviour is “consistent with” some theory. Economics, then, cannot be a stand-alone discipline but part of the social sciences and humanities – a point which is lost in the discipline’s mathiness.
Yes indeed, history helps. And if we’re not to ‘busy’ doing the things we do, but once in a while take a brake and do some methodological reflection on why we do what we do — well, that takes us a long way too.
Modern economics is sick. Economics has increasingly become an intellectual game played for its own sake and not for its practical consequences for understanding the economic world. Economists have converted the subject into a sort of social mathematics in which analytical rigour is everything and practical relevance is nothing. To pick up a copy of The American Economic Review or The Economic Journal these days is to wonder whether one has landed on a strange planet in which tedium is the deliberate objective of professional publication. Economics was once condemned as “the dismal science” but the dismal science of yesterday was a lot less dismal than the soporific scholasticism of today … If there is such a thing as “original sin” in economic methodology, it is the worship of the idol of the mathematical rigour invented by Arrow and Debreu in 1954 and then canonized by Debreu in his Theory of Value five years later, probably the most arid and pointless book in the entire literature of economics. The result of all this is that we now understand almost less of how actual markets work than did Adam Smith or even Léon Walras. We have forgotten that markets require market-makers, that middlemen have to hold inventories to allow markets to function, that markets need to be organized and that property rights need to be defined and enforced if markets are to get started at all. We have even forgotten that markets adjust as often in terms of quantities rather than prices, as in labour markets and customer commodity markets, as Alfred Marshall knew very well but Walras overlooked; so well have we forgotten that fact that a whole branch of economics sprang up in the 1960s and 70s to provide “microfoundations” for Keynesian macroeco- nomics, that is, some ad hoc explanation for the fact that a decline in aggregate demand causes unemployment at the same real wage and not falling real wages at the same level of employment … Indeed, much of modern microeconomics might be fairly described as a kind of geography that consists entirely of images of cities but providing no maps of how to reach a city either from any other city or from the countryside.
Mark Blaug (1927-2011) did more than any other economist to establish the philosophy and methodology of economics a respected subfield within economics. His path-breaking The methodology of economics (1980) is still a landmark. Highly recommended reading — even for ‘busy’ mainstream economists of today …
About math: I have an undergraduate degree in physics. I’ve seen clear evidence that math can facilitate scientific progress toward the truth.
If you think that math is worthless or dangerous, I’m sure that there are people who will be happy to discuss this with you. I’m not interested. I’m busy.
If you do not accept this premise, I’m sure that there are people who would be happy to debate it with you. I’m not interested. I’m busy.
To me this sounds more like a person afraid of methodological self-reflection, rather than an open-minded and pluralist person.
Where does this methodology-aversion come from?
As far as yours truly can see it all grinds down to a misplaced belief in deductivist mathematical reasoning being the only kind of scientific economics around. If economics isn’t performed as a mathematical modeling it’s not really science in Romer’s world-view. There is no problem with that view — as long as you have done some ontological and methodological reflection and presented arguments for the appropriateness of insisting on deductivist-mathematical modeling being the preferred scientific procedure in economics. No such argumentation is presented.
When applying deductivist thinking to economics, Romer and other mainstream economists usually set up “as if” models based on a set of tight axiomatic assumptions from which consistent and precise inferences are made. The beauty of this procedure is of course that if the axiomatic premises are true, the conclusions necessarily follow. The snag is that if the models are to be relevant, we also have to argue that their precision and rigour still holds when they are applied to real-world situations. They often don’t. When addressing real economies, the idealizations necessary for the deductivist machinery to work, simply don’t hold.
So how should we evaluate the search for ever greater precision and the concomitant arsenal of mathematical and formalist models? To a large extent, the answer hinges on what we want our models to perform and how we basically understand the world.
The world in which we live is inherently uncertain and quantifiable probabilities are the exception rather than the rule. To every statement about it is attached a “weight of argument” that makes it impossible to reduce our beliefs and expectations to a one-dimensional stochastic probability distribution. If “God does not play dice” as Einstein maintained, I would add “nor do people”. The world as we know it, has limited scope for certainty and perfect knowledge. Its intrinsic and almost unlimited complexity and the interrelatedness of its organic parts prevent the possibility of treating it as constituted by “legal atoms” with discretely distinct, separable and stable causal relations. Our knowledge accordingly has to be of a rather fallible kind.
To search for precision and rigour in such a world is self-defeating, at least if precision and rigour are supposed to assure external validity. The only way to defend such an endeavour is to take a blind eye to ontology and restrict oneself to prove things in closed model-worlds. Why we should care about these and not ask questions of relevance is hard to see. We have to at least justify our disregard for the gap between the nature of the real world and our theories and models of it.
Now, if the real world is fuzzy, vague and indeterminate, then why should our models build upon a desire to describe it as precise and predictable? Even if there always has to be a trade-off between theory-internal validity and external validity, we have to ask ourselves if our models are relevant.
Models preferably ought to somehow reflect/express/correspond to reality. I’m not saying that the answers are self-evident, but at least you have to do some methodological and philosophical under-labouring to rest your case. Too often that is wanting in modern economics, where methodological justifications of chosen models and methods as a rule are non-existent.
“Human logic” has to supplant the classical, formal, logic of deductivism if we want to have anything of interest to say of the real world we inhabit. Logic is a marvellous tool in mathematics and axiomatic-deductivist systems, but a poor guide for action in real-world systems, in which concepts and entities are without clear boundaries and continually interact and overlap. In this world I would say we are better served with a methodology that takes into account that “the more we know the more we know we don’t know”.
The models and methods we choose to work with have to be in conjunction with the economy as it is situated and structured. Epistemology has to be founded on ontology. Deductivist closed-system theories, as all the varieties of the Walrasian general equilibrium kind, could perhaps adequately represent an economy showing closed-system characteristics. But since the economy clearly has more in common with an open-system ontology we ought to look out for other theories – theories who are rigorous and precise in the meaning that they can be deployed for enabling us to detect important causal mechanisms, capacities and tendencies pertaining to deep layers of the real world.
Rigour, coherence and consistency have to be defined relative to the entities for which they are supposed to apply. Too often they have been restricted to questions internal to the theory or model. But clearly the nodal point has to concern external questions, such as how our theories and models relate to real-world structures and relations. Applicability rather than internal validity ought to be the arbiter of taste.
But obviosly Paul Romer doesn’t want to talk about these scary methodological-philosophical issues. He is ‘busy’ …
Paul Romer inquired why I did not endorse his following Krusell and Smith (2014) in characterizing Piketty and Piketty and Zucman as a canonical example of what Romer calls “mathiness”. Indeed, I think that, instead, it is Krusell and Smith (2014) that suffers from “mathiness”–people not in control of their models deploying algebra untethered to the real world in a manner that approaches gibberish.
My objection to Krusell and Smith (2014) was that it seemed to me to suffer much more from what you call “mathiness” than does Piketty or Piketty and Zucman.
Recall that Krusell and Smith began by saying that they:
do not quite recognize… k/y=s/g”…
But k/y=s/g is Harrod (1939) and Domar (1946). How can they fail to recognize it?
And then their calibration–n+g=.02, δ=.10–not only fails to acknowledge Piketty’s estimates of economy-wide depreciation rate as between .01 and .02, but leads to absolutely absurd results:
For a country with a k/y=4, δ=.10 -> depreciation is 40% of gross output.
For a country like Belle Époque France with a k/y=7, δ=.10 -> depreciation is 70% of gross output.
It seemed to me that Krusell and Smith had no control whatsoever over the calibration of their model at all.
Note that I am working from notes here, because http://aida.wss.yale.edu/smith/piketty1.pdf no longer points to Krusell and Smith (2014). It points, instead, to Krusell and Smith (2015), a revised version.
In the revised version, the calibration differs. It differs in:
1. raising (n+g) from .02 to .03,
2. lowering δ from .10 or .05 (still more than twice Piketty’s historical estimates), and
3.changing the claim that as n+g->0 k/y increases “only very marginally” to “only modestly”
(The right thing to do would be to take economy-wide δ=.02 and say that k/y increases “substantially”.)
If Krusell and Smith (2015) offers any reference to Piketty’s historical depreciation efforts, I missed it.
If it offers any explanation of why they decided to raise their calibration of n+g when they lowered their δ, I missed that too.
Piketty has flaws, but it does not seem to me that working in a net rather than a gross production function framework is one of them. And Krusell and Smith’s continued attempts to demonstrate otherwise seem to me to suffer from “mathiness” to a high degree …
The issue of interpreting economic theory is, in my opinion, the most serious problem now facing economic theorists. The feeling among many of us can be summarized as follows. Economic theory should deal with the real world. It is not a branch of abstract mathematics even though it utilizes mathematical tools. Since it is about the real world, people expect the theory to prove useful in achieving practical goals. But economic theory
has not delivered the goods. Predictions from economic theory are not nearly as accurate as those offered by the natural sciences, and the link between economic theory and practical problems … is tenuous at best. Economic theory lacks a consensus as to its purpose and interpretation. Again and again, we find ourselves asking the question “where does it lead?”
In those cases where economists do focus on questions of market or competitive equilibrium etc., the formulators of the models in question are often careful to stress that their theorising has little connection with the real world anyway and should not be used to draw conclusions about the latter, whether in terms of efficiency or for policy or whatever.
In truth in those cases where mainstream assumptions and categories are couched in terms of economic systems as a whole they are mainly designed to achieve consistency at the level of modelling rather than coherence with the world in which we live.
This concern for a notion of consistency in modelling practice is true for example of the recently fashionable rational expectations hypothesis, originally formulated by John Muth (1961), and widely employed by those that do focus on system level outcomes. The hypothesis proposes that predictions attributed to agents (being theorised about) are treated as being essentially the same as (consistent with)
those generated by the economic model within which the same agents are theorised. As such the proposal is clearly no more than a technique for (consistency in) modelling, albeit a bizarre one. Significantly any assertion that the expectations held (and so model in which they are imposed) are essentially correct, is a step that is additional to assuming rational expectations.
It is a form of modelling consistency (albeit a different one) that underpins the notion of equilibrium itself. In modern mainstream economics the category equilibrium has nothing to do with the features of the real economy … Economic models often comprise not single, but sets of, equations, each of which is notoriously found to have little relation to what happens in the real world. One question that nevertheless keeps economists occupied with such unrealistic models is whether the equations formulated are mutually consistent in the sense that there ‘exists’ a vector of values of some variable, say one labelled ‘prices’, that is consistent with each and all the equations. Such a model ‘solution’ is precisely the meaning of equilibrium in this context. As such the notion is not at all a claim about the world but merely a (possible) property that a set of equations may or may not be found to possess …In short, when mainstream economists question whether an equilibrium ‘exists’ they merely enquire as to whether a set of equations has a solution.
Modern economics has become increasingly irrelevant to the understanding of the real world. Tony Lawson traces this irrelevance to the failure of economists to match their deductive-axiomatic methods with their subject.
It is — sad to say — a fact that within mainstream economics internal validity is everything and external validity nothing. Why anyone should be interested in that kind of theories and models is beyond my imagination. As long as mainstream economists do not come up with any export-licenses for their theories and models to the real world in which we live, they really should not be surprised if people say that this is not science, but autism.
Studying mathematics and logics is interesting and fun. It sharpens the mind. In pure mathematics and logics we do not have to worry about external validity. But economics is not pure mathematics or logics. It’s about society. The real world. Forgetting that, economics is really in dire straits.