The dangers of randomization idolatry

30 Jun, 2022 at 10:49 | Posted in Statistics & Econometrics | 1 Comment

Natural Experiments in the Social Sciences: A Design-Based Approach  (Strategies for Social Inquiry): Dunning, Thad:  9781107698000: BooksHow, then, can social scientists best make inferences about causal effects? One option is true experimentation … Random assignment ensures that any differences in outcomes between the groups are due either to chance error or to the causal effect … If the experiment were to be repeated over and over, the groups would not differ, on average, in the values of potential confounders. Thus, the average of the average difference of group outcomes, across these many experiments, would equal the true difference in outcomes … The key point is that randomization is powerful because it obviates confounding …

Thad Dunning’s book Natural Experiments in the Social Sciences (CUP 2012) is a very useful guide for economists interested in research methodology in general and natural experiments in specific. Dunning argues that since random or as-if random assignment in natural experiments obviates the need for controlling potential confounders, this kind of “simple and transparent” design-based research method is preferable to more traditional multivariate regression analysis where the controlling only comes in ex-post via statistical modelling.

The point of making a randomized experiment is often said to be that it ‘ensures’ that any correlation between a supposed cause and effect indicates a causal relation. This is believed to hold since randomization (allegedly) ensures that a supposed causal variable does not correlate with other variables that may influence the effect.

The problem with that simplistic view of randomization is that the claims made are exaggerated and sometimes even false:

• Even if you manage to do the assignment to treatment and control groups ideally random, the sample selection certainly is — except in extremely rare cases — not random. Even if we make a proper randomized assignment, if we apply the results to a biased sample, there is always the risk that the experimental findings will not apply. What works ‘there,’ does not work ‘here.’ Randomization hence does not ‘guarantee ‘ or ‘ensure’ making the right causal claim. Although randomization may help us rule out certain possible causal claims, randomization per se does not guarantee anything!

• Even if both sampling and assignment are made in an ideal random way, performing standard randomized experiments only gives you averages. The problem here is that although we may get an estimate of the ‘true’ average causal effect, this may ‘mask’ important heterogeneous effects of a causal nature. Although we get the right answer of the average causal effect being 0, those who are ‘treated’  may have causal effects equal to -100 and those ‘not treated’ may have causal effects equal to 100. Contemplating being treated or not, most people would probably be interested in knowing about this underlying heterogeneity and would not consider the average effect particularly enlightening.

• There is almost always a trade-off between bias and precision. In real-world settings, a little bias often does not overtrump greater precision. And — most importantly — in case we have a population with sizeable heterogeneity, the average treatment effect of the sample may differ substantially from the average treatment effect in the population. If so, the value of any extrapolating inferences made from trial samples to other populations is highly questionable.

• Since most real-world experiments and trials build on performing single randomization, what would happen if you kept on randomizing forever, does not help you to ‘ensure’ or ‘guarantee’ that you do not make false causal conclusions in the one particular randomized experiment you actually do perform. It is indeed difficult to see why thinking about what you know you will never do, would make you happy about what you actually do.

• And then there is also the problem that ‘Nature’ may not always supply us with the random experiments we are most interested in. If we are interested in X, why should we study Y only because design dictates that? Method should never be prioritized over substance!

Nowadays many mainstream economists maintain that ‘imaginative empirical methods’ — especially ‘as-if-random’ natural experiments and RCTs — can help us to answer questions concerning the external validity of economic models. In their view, they are, more or less, tests of ‘an underlying economic model’ and enable economists to make the right selection from the ever-expanding ‘collection of potentially applicable models.’

It is widely believed among mainstream economists that the scientific value of randomization — contrary to other methods — is more or less uncontroversial and that randomized experiments are free from bias. When looked at carefully, however, there are in fact few real reasons to share this optimism on the alleged ’experimental turn’ in economics. Strictly seen, randomization does not guarantee anything.

‘Ideally’ controlled experiments tell us with certainty what causes what effects — but only given the right ‘closures.’ Making appropriate extrapolations from (ideal, accidental, natural or quasi) experiments to different settings, populations or target systems, is not easy. Causes deduced in an experimental setting still have to show that they come with an export warrant to the target population. The causal background assumptions made have to be justified, and without licenses to export, the value of ‘rigorous’ and ‘precise’ methods — and ‘on-average-knowledge’ — is despairingly small.

The almost religious belief with which its propagators — including ‘Nobel prize’ winners like Duflo, Banerjee and Kremer  — portray it, cannot hide the fact that RCTs cannot be taken for granted to give generalizable results. That something works somewhere is no warranty for us to believe it to work for us here or that it works generally.

Leaning on an interventionist approach often means that instead of posing interesting questions on a social level, the focus is on individuals. Instead of asking about structural socio-economic factors behind, e.g., gender or racial discrimination, the focus is on the choices individuals make.  Esther Duflo is a typical example of the dangers of this limiting approach. Duflo et consortes want to give up on ‘big ideas’ like political economy and institutional reform and instead go for solving more manageable problems ‘the way plumbers do.’ Yours truly is far from sure that is the right way to move economics forward and make it a relevant and realist science. A plumber can fix minor leaks in your system, but if the whole system is rotten, something more than good old fashion plumbing is needed. The big social and economic problems we face today are not going to be solved by plumbers performing interventions or manipulations in the form of RCTs.

the-right-toolThe present RCT idolatry is dangerous. Believing randomization is the only way to achieve scientific validity blinds people to searching for and using other methods that in many contexts are better. Insisting on using only one tool often means using the wrong tool.

Randomization is not a panacea. It is not the best method for all questions and circumstances. Proponents of randomization make claims about its ability to deliver causal knowledge that is simply wrong. There are good reasons to be sceptical of the now popular — and ill-informed — view that randomization is the only valid and the best method on the market. It is not.

Statistical significance and effect size (student stuff)

30 Jun, 2022 at 09:28 | Posted in Statistics & Econometrics | Comments Off on Statistical significance and effect size (student stuff)


Economics as ideology

29 Jun, 2022 at 20:41 | Posted in Economics | 3 Comments

capitalism-works-bestAlthough I never believed it when I was young and held scholars in great respect, it does seem to be the case that ideology plays a large role in economics. How else to explain Chicago’s acceptance of not only general equilibrium but a particularly simplified version of it as ‘true’ or as a good enough approximation to the truth? Or how to explain the belief that the only correct models are linear and that the von Neuman prices are those to which actual prices converge pretty smartly? This belief unites Chicago and the Classicals; both think that the ‘long-run’ is the appropriate period in which to carry out analysis. There is no empirical or theoretical proof of the correctness of this. But both camps want to make an ideological point. To my mind that is a pity since clearly it reduces the credibility of the subject and its practitioners.

Frank Hahn

Jalāl ad-Dīn Muhammad Rūmī

28 Jun, 2022 at 18:23 | Posted in Varia | Comments Off on Jalāl ad-Dīn Muhammad Rūmī


The shocking truth about econometric ‘precision’ and ‘rigour’

28 Jun, 2022 at 09:39 | Posted in Statistics & Econometrics | Comments Off on The shocking truth about econometric ‘precision’ and ‘rigour’

Mixtape | Scott Cunningham Leverage is a measure of the degree to which a single observation on the right-hand-side variable takes on extreme values and is influential in estimating the slope of the regression line. A concentration of leverage in even a few observations can make coefficients and standard errors extremely volatile and even bias robust standard errors towards zero, leading to higher rejection rates.

To illustrate this problem, Young (2019) went through a simple exercise. He collected over fifty experimental (lab and field) articles from the American Economic Association’s flagship journals: American Economic ReviewAmerican Economic Journal: Applied, and American Economic Journal: Economic Policy. He then reanalyzed these papers, using the authors’ models, by dropping one observation or cluster and reestimating the entire model, repeatedly. What he found was shocking:

With the removal of just one observation, 35% of 0.01-significant reported results in the average paper can be rendered insignificant at that level. Conversely, 16% of 0.01-insignificant reported results can be found to be significant at that level.

The Gray Ghost (personal)

27 Jun, 2022 at 17:32 | Posted in Varia | Comments Off on The Gray Ghost (personal)

For my daughter Tora and son David — with whom, when they were just little kids, yours truly spent hours and hours watching this​ series back in the 1990s.
You were my heroes then.
You still are.

How to tame inflation

27 Jun, 2022 at 13:20 | Posted in Economics | 7 Comments


Regeringen och Riksbanken hanterar inflationen fel

26 Jun, 2022 at 11:31 | Posted in Economics | Comments Off on Regeringen och Riksbanken hanterar inflationen fel

Vad beror den ökande inflationen på?

Is inflation on its way? – Swiss Life GroupMax Jerneck (MJ): Den beror till stor del på ökade priser på energi och livsmedel, som orsakas av saker som hur elmarknaden är reglerad och torka samt kriget i Ukraina. Även priserna på möbler och andra varor, och grundläggande komponenter och insatsvaror som halvledare och stål spelar in. Under pandemin minskade efterfrågan på tjänster samtidigt som den snabbt ökade på varor, vilket tillverkare och leverantörskedjor hade svårt att hantera. Och det skedde i en världsekonomi som under ett årtionde hade präglats av låg efterfrågan och av bristande investeringar i ny kapacitet. Problemen kan nog väntas kvarstå ett tag till, inte minst för att Kina fortfarande stänger ned fabriker och hamnar med hårda covid­restriktioner …

Vilket är enligt dig det bästa sättet att komma till rätta med inflationen?

MJ: Det bästa vore att behandla alla orsaker med skräddarsydda lösningar: om dyra drivmedel är problemet borde man se till att folk åkte mindre bil, att man sänkte hastigheterna på vägarna, och gjorde kollektivtrafiken billigare, och på längre sikt påskyndade elektrifiering, vilket i sin tur kräver investeringar i batterier, gruvor, och så vidare. Det går också att använda mer allmänna verktyg såsom höjda skatter och räntor, men de fungerar bäst mot inflation som uppstår på grund av överhettad efterfrågan, snarare än utbudsproblem …

Vad anser du om regeringens och Riksbankens hantering av problemet hittills?

MJ: Regeringens åtgärder har mestadels bestått av kortsiktiga nödlösningar, som kanske minskar kostnaderna på kort sikt men inte gör något åt grundproblemet utan kanske snarare förvärrar det. Jag tänker på bidrag till bil- och husägare så att de kan fortsätta driva upp priserna på el och bensin. Riksbanken vill höja räntan. Man är öppen med att det inte åtgärdar inflationens orsaker, såsom ökade energipriser, utan skälet sägs vara att man vill hindra inflationen från att bita sig fast. Den typen av analys lägger stor vikt vid idén om inflationsförväntningar, som gör inflationen självförstärkande. Man vill undvika en löne- och pris­spiral … Allmänna räntehöjningar är ett trubbigt verktyg som minskar inflationen genom att exempelvis öka kostnaderna för att bo, så man kan fråga sig hur klokt det är. De dämpar också efterfrågan genom att minska investeringarna. Jag tror att mer traditionell kreditstyrning, bort från överhettade branscher och till de branscher som behöver investeringar, är ett mer träffsäkert verktyg.


Och precis som i övriga världen handlar dagens svenska inflation också delvis om att en del företag och kapitalägare vill passa på och öka sina vinstmarginaler utan att det i någon egentlig mening föreligger reala kostnadsökningar som ‘berättigar’ detta. Den typen av i ‘smyg’ höjda priser blir allt lättare att genomföra i takt med att inflationsförväntningarna nu stiger. Som alltid på marknaden är det de resurssvaga som i sista hand får stå för notan …

Riksbanken har i princip bara ett enda verktyg att ta till — räntan. Och visst kan man använda den ‘hammaren’ för att försöka råda bot på prisstegringsspiralen. Men är det något historien har lärt oss så är det att det sannolikt bara lyckas till priset av ökad arbetslöshet och försämrad välfärd. Inflationen i dag är inte i första hand ett efterfrågeproblem. Det handlar mer om utbudet. Och ett förfelat användande av medicin kan tyvärr leda till att patienten bara blir ännu sjukare …

The Law of Demand

23 Jun, 2022 at 11:45 | Posted in Economics | 2 Comments

Mainstream economics is usually considered to be very ‘rigorous’ and ‘precise.’ And yes, indeed, it’s certainly full of ‘rigorous’ and ‘precise’ statements like “the state of the economy will remain the same as long as it doesn’t change.” Although ‘true,’ this is, however — like most other analytical statements — neither particularly interesting nor informative.

As is well known, the law of demand is usually tagged with a clause that entails numerous interpretation problems: the ceteris paribus clause. In the strict sense this must thus at least be formulated as follows to be acceptable to the majority of theoreticians: ceteris paribus – that is, all things being equal – the demanded quantity of a consumer good is a monotone-decreasing function of its price …

Wissenschaftstheorie: Hans Albert feiert 100. Geburtstag - Forschung & LehreIf the factors that are to be left constant remain undetermined, as not so rarely happens, then the law of demand under question is fully immunized to facts, because every case which initially appears contrary must, in the final analysis, be shown to be compatible with this law. The clause here produces something of an absolute alibi, since, for every apparently deviating behavior, some altered factors can be made responsible. This makes the statement untestable, and its informational content decreases to zero.

One might think that it is in any case possible to avert this situation by specifying the factors that are relevant for the clause. However, this is not the case. In an appropriate interpretation of the clause, the law of demand that comes about will become, for example, an analytic proposition, which is, in fact, ​true for logical reasons, but which is thus precisely for this reason not informative …

Various widespread formulations of the law of demand contain an interpretation of the clause that does not result in a tautology, but that has another weakness. The list of the factors to be held constant includes, among other things, the structure of the needs of the purchasing group in question. This leads to a difficulty connected with the identification of needs. As long as there is no independent test for the constancy of the structures of needs, any law that is formulated in this way has an absolute ‘alibi’. Any apparent counter case can be traced back to a change in the needs, and thus be discounted. Thus, in this form, the law is also immunized against empirical facts. To counter this situation, it is in fact necessary to dig deeper into the problem of needs and preferences; in many cases, however, this is held to be unacceptable, because it would entail crossing the boundaries into social psychology.

Hans Albert

In mainstream economics there’s — still — a lot of talk about ‘economic laws.’ The crux of these laws — and regularities — that allegedly do exist in economics, is that they only hold ceteris paribus. That fundamentally means that these laws/regularities only hold when the right conditions are at hand for giving rise to them. Unfortunately, from an empirical point of view, those conditions are only at hand in artificially closed nomological models purposely designed to give rise to the kind of regular associations that economists want to explain. But, really, since these laws/regularities do not exist outside these ‘socio-economic machines,’ what’s the point in constructing thought experimental models showing these non-existent laws/regularities? When the almost endless list of narrow and specific assumptions necessary to allow the ‘rigorous’ deductions are known to be at odds with reality, what good do these models do?

Deducing laws in theoretical models is of no avail if you cannot show that the models — and the assumptions they build on — are realistic representations of what goes on in real-life.

Conclusion? Instead of restricting our methodological endeavours to building ever more rigorous and precise deducible models, we ought to spend much more time improving our methods for choosing models!

Propensity scores — bias-reduction gone awry

23 Jun, 2022 at 08:28 | Posted in Statistics & Econometrics | Comments Off on Propensity scores — bias-reduction gone awry


Hegel in 60 minuten

22 Jun, 2022 at 18:14 | Posted in Theory of Science & Methodology | Comments Off on Hegel in 60 minuten


The inflation lie

21 Jun, 2022 at 14:02 | Posted in Economics | 4 Comments


Mainstream economics — the art of building fantasy worlds

20 Jun, 2022 at 15:50 | Posted in Economics | 3 Comments

Mainstream macroeconomic models standardly assume things like rational expectations, Walrasian market clearing, unique equilibria, time invariance, linear separability and homogeneity of both inputs/outputs and technology, infinitely lived intertemporally optimizing representative household/ consumer/producer agents with homothetic and identical preferences, etc., etc. At the same time, the models standardly ignore complexity, diversity, uncertainty, coordination problems, non-market clearing prices, real aggregation problems, emergence, expectations formation, etc., etc.

How to Build a Fantasy Economy - YouTubeBehavioural and experimental economics — not to speak of psychology — show beyond doubt that “deep parameters” — peoples’ preferences, choices and forecasts — are regularly influenced by those of other economic participants. And how about the homogeneity assumption? And if all actors are the same — why and with whom do they transact? And why does economics have to be exclusively teleological (concerned with intentional states of individuals)? Where are the arguments for that ontological reductionism? And what about collective intentionality and constitutive background rules?

These are all justified questions — so, in what way can one maintain that these models give workable microfoundations for macroeconomics? Science philosopher Nancy Cartwright gives a good hint at how to answer that question:

Our assessment of the probability of effectiveness is only as secure as the weakest link in our reasoning to arrive at that probability. We may have to ignore some issues to make heroic assumptions about them. But that should dramatically weaken our degree of confidence in our final assessment. Rigor isn’t contagious from link to link. If you want a relatively secure conclusion coming out, you’d better be careful that each premise is secure going on.

Avoiding logical inconsistencies is crucial in all science. But it is not enough. Just as important is avoiding factual inconsistencies. And without showing — or at least warranted arguing — that the assumptions and premises of their models are in fact true, mainstream economists aren’t really reasoning, but only playing games. Formalistic deductive ‘Glasperlenspiel’ can be very impressive and seductive. But in the realm of science, it ought to be considered of little or no value to simply make claims about the model and lose sight of reality.

Instead of making the model the message, I think we are better served by economists who more than anything else try to contribute to solving real problems. And then the motto of John Maynard Keynes is more valid than ever:

It is better to be vaguely right than precisely wrong

Bayesian absurdities

20 Jun, 2022 at 15:23 | Posted in Statistics & Econometrics | 1 Comment

Making Decisions 2e: Lindley, Dennis V.: 9780471908081: BooksIn other words, if a decision-maker thinks something cannot be true and interprets this to mean it has zero probability, he will never be influenced by any data, which is surely absurd.

So leave a little probability for the moon being made of green cheese; it can be as small as 1 in a million, but have it there since otherwise an army of astronauts returning with samples of the said cheese will leave you unmoved.

To get the Bayesian probability calculus going you sometimes have to assume strange things — so strange that you should perhaps start wondering if maybe there is something wrong with your theory …

Added: For those interested in these questions concerning the reach and application of statistical theories, do read Sander Greenland’s insightful comment:

My take is that the quoted passage is a poster child for what’s wrong with statistical foundations for applications. Mathematics only provides contextually void templates for what might be theories if some sensible mapping can be found between the math and the application context. Just as with frequentist and all other statistical “theories”, Bayesian mathematical theory (template) works fine as a tool when the problem can be defined in a very small world of an application in which the axioms make contextual sense under the mapping and the background information is not questioned. There is no need for leaving any probability on “green cheese” if you aren’t using Bayes as a philosophy, for if green cheese is really found, the entire contextual knowledge base is undermined and all well-informed statistical analyses sink with it.

The problems often pointed out for econometrics are general ones of statistical theories, which can quickly degenerate into math gaming and are usually misrepresented as scientific theories about the world. Of course, with a professional sales job to do, statistics has encouraged such reification through use of deceptive labels like “significance”, “confidence”, “power”, “severity” etc. for what are only properties of objects in mathematical spaces (much like identifying social group dynamics with algebraic group theory or crop fields with vector field theory). Those stat-theory objects require extraordinary physical control of unit selection and experimental conditions to even begin to connect to the real-world meaning of those conventional labels. Such tight controls are often possible with inanimate materials (although even then they can cost billions of dollars to achieve, as with large particle colliders). But they are infrequently possible with humans, and I’ve never seen them approached when whole societies are the real-world target, as in macroeconomics, sociology, and social medicine. In those settings, at best our analyses only provide educated guesses about what will happen as a consequence of our decisions.

That’s life

19 Jun, 2022 at 22:20 | Posted in Varia | Comments Off on That’s life


Next Page »

Blog at
Entries and Comments feeds.