## The problems we face when using instrumental variables (student stuff)

14 May, 2021 at 23:26 | Posted in Statistics & Econometrics | Leave a comment.

## Instrumentalvariabler och heterogenitet

13 May, 2021 at 19:25 | Posted in Statistics & Econometrics | Leave a commentAnvändandet av instrumentalvariabler används numera flitigt bland ekonomer och andra samhällsforskare. Inte minst när man vill försöka gå bakom statistikens ‘korrelationer’ och också säga något om ‘kausalitet.’

Tyvärr brister det ofta rejält i tolkningen av de resultat man får med hjälp av den vanligaste metoden som används för detta syfte — statistisk regressionsanalys.

Ett exempel från skolområdet belyser detta väl.

Ibland hävdas det bland skoldebattörer och politiker att friskolor skulle vara bättre än kommunala skolor. De sägs leda till bättre resultat. Alltså: om vi tänker oss att man skulle låta elever från friskolor och kommunala skolor genomföra gemensamma prov så skulle friskoleelever prestera bättre (fler rätt på provräkningar e d).

För argumentets skull antar vi att man för att ta reda på om det verkligen förhåller sig på detta sätt även i Malmö, slumpmässigt väljer ut högstadieelever i Malmö och låter dem skriva ett prov. Resultatet skulle då i vanlig regressionsanalytisk form kunna bli

**Provresultat = 20 + 5*T**,

där T=1 om eleven går i friskola, och T=0 om eleven går i kommunal skola. Detta skulle innebära att man får bekräftat antagandet — friskoleelever har i genomsnitt 5 poäng högre resultat än elever på kommunala skolor i Malmö.

Nu är ju politiker (förhoppningsvis) inte dummare än att de är medvetna om att detta *statistiska* resultat inte kan tolkas i *kausala* termer eftersom elever som går på friskolor typiskt inte har samma bakgrund (socio-ekonomiskt, utbildningsmässigt, kulturellt etc) som de som går på kommunala skolor (relationen skolform-resultat är ‘confounded’ via ‘selection bias.’)

För att om möjligt få ett bättre mått på skolformens kausala effekter väljer Malmös politiker föreslå att man via lottning gör det möjligt för 1000 högstadieelever att bli antagna till en friskola. ‘Vinstchansen’ är 10%, så 100 elever får denna möjlighet. Av dessa antar 20 erbjudandet att gå i friskola. Av de 900 lotterideltagare som inte ‘vinner’ väljer 100 att gå i friskola.

Lotteriet uppfattas ofta av skolforskare som en ’instrumentalvariabel’ och när man så genomför regressionsanalysen med hjälp av denna visar sig resultatet bli

**Provresultat = 20 + 2*T**.

Detta tolkas standardmässigt som att man nu har fått ett *kausalt* mått på hur mycket bättre provresultat högstadieelever i Malmö *i genomsnitt* skulle få om de istället för att gå på kommunala skolor skulle välja att gå på friskolor.

Men stämmer det? Nej!

Om inte alla Malmös skolelever har exakt samma provresultat (vilket väl får anses vara ett rätt långsökt ‘homogenitetsantagande’) så gäller den angivna genomsnittliga kausala effekten bara de elever som väljer att gå på friskola om de ’vinner’ i lotteriet, men som annars inte skulle välja att gå på en friskola (på statistikjargong kallar vi dessa ’compliers’). Att denna grupp elever skulle vara speciellt intressant i det här exemplet är svårt att se med tanke på att den genomsnittliga kausala effekten skattad med hjälp av instrumentalvariabeln inte säger någonting alls om effekten för majoriteten (de 100 av 120 som väljer en friskola utan att ha ‘vunnit’ i lotteriet) av de som väljer att gå på en friskola.

Slutsats: forskare måste vara mycket mer försiktiga med att tolka vanliga statistiska regressionsanalyser och deras ‘genomsnittsskattningar’ som kausala. Verkligheten uppvisar en hög grad av heterogenitet. Och då säger oss regressionsanalysens konstanta ‘genomsnittsparametrar’ i regel inte ett smack!

## Introduction to instrumental variables

13 May, 2021 at 19:23 | Posted in Statistics & Econometrics | Leave a comment.

Great presentation, but maybe Angrist should also have pointed out the mistake many economists do when they use instrumental variables analysis and think that their basic identification assumption is empirically testable. It is not. And just swapping an assumption of residuals being uncorrelated with the independent variables with the assumption that the same residuals are uncorrelated with an instrument doesn’t solve the endogeneity problem or improve our causal analysis.

## Which causal inference method is the best one?

4 May, 2021 at 17:32 | Posted in Statistics & Econometrics | Leave a comment.

## Graphical causal models and collider bias

3 May, 2021 at 16:53 | Posted in Statistics & Econometrics | Leave a commentWhy would two independent variables suddenly become dependent when we condition on their common effect? To answer this question, we return again to the definition of conditioning as filtering by the value of the conditioning variable. When we condition on Z, we limit our comparisons to cases in which Z takes the same value. But remember that Z depends, for its value, on X and Y. So, when comparing cases where Z takes some value, any change in value of X must be compensated for by a change in the value of Y — otherwise, the value of Z would change as well.

The reasoning behind this attribute of colliders — that conditioning on a collision node produces a dependence between the node’s parents — can be difficult to grasp at first. In the most basic situation where Z = X + Y, and X and Y are independent variables, we have the follow- ing logic: If I tell you that X = 3, you learn nothing about the potential value of Y, because the two numbers are independent. On the other hand, if I start by telling you that Z = 10, then telling you that X = 3 immediately tells you that Y must be 7. Thus, X and Y are dependent, given that Z = 10.

Students usually find this collider attribute rather perplexing. Why? My guess is the reason is most students — wrongly — think there can be no correlation without causation.

## Econometrics — science based on whimsical assumptions

22 Apr, 2021 at 14:36 | Posted in Statistics & Econometrics | 3 CommentsIt is often said that the error term in a regression equation represents the effect of the variables that were omitted from the equation. This is unsatisfactory …

There is no easy way out of the difficulty. The conventional interpretation for error terms needs to be reconsidered. At a minimum, something like this would need to be said:

The error term represents the combined effect of the omitted variables, assuming that

(i) the combined effect of the omitted variables is independent of each variable included in the equation,

(ii) the combined effect of the omitted variables is independent across subjects,

(iii) the combined effect of the omitted variables has expectation 0.This is distinctly harder to swallow.

Yes, indeed, that *is* harder to swallow.

Those conditions on the error term actually mean that we are being able to construct a model where all relevant variables are included and correctly specify the functional relationships that exist between them.

But that is actually impossible to fully manage in reality!

The theories we work with when building our econometric regression models are insufficient. No matter what we study, there are always some variables missing, and we don’t know the correct way to functionally specify the relationships between the variables (usually just *assuming* linearity).

*Every* regression model constructed is misspecified. There is always an endless list of possible variables to include, and endless possible ways to specify the relationships between them. So every applied econometrician comes up with his own specification and ‘parameter’ estimates. No wonder that the econometric Holy Grail of consistent and stable parameter-values is still nothing but a dream.

In order to draw inferences from data as described by econometric texts, it is necessary to make whimsical assumptions. The professional audience consequently and properly withholds belief until an inference is shown to be adequately insensitive to the choice of assumptions. The haphazard way we individually and collectively study the fragility of inferences leaves most of us unconvinced that any inference is believable. If we are to make effective use of our scarce data resource, it is therefore important that we study fragility in a much more systematic way. If it turns out that almost all inferences from economic data are fragile, I suppose we shall have to revert to our old methods …

A rigorous application of econometric methods in economics really presupposes that the phenomena of our real-world economies are ruled by stable causal relations between variables. Parameter-values estimated in specific spatio-temporal contexts are *presupposed* to be exportable to totally different contexts. To warrant this assumption one, however, has to convincingly establish that the targeted acting causes are stable and invariant so that they maintain their parametric status after the bridging. The endemic lack of predictive success of the econometric project indicates that this hope of finding fixed parameters is a hope for which there really is no other ground than hope itself.

Real-world social systems are not governed by stable causal mechanisms or capacities. As Keynes noticed when he first launched his attack against econometrics and inferential statistics already in the 1920s:

The atomic hypothesis which has worked so splendidly in Physics breaks down in Psychics. We are faced at every turn with the problems of Organic Unity, of Discreteness, of Discontinuity – the whole is not equal to the sum of the parts, comparisons of quantity fails us, small changes produce large effects, the assumptions of a uniform and homogeneous continuum are not satisfied. Thus the results of Mathematical Psychics turn out to be derivative, not fundamental, indexes, not measurements, first approximations at the best; and fallible indexes, dubious approximations at that, with much doubt added as to what, if anything, they are indexes or approximations of.

The kinds of laws and relations that econom(etr)ics has established, are laws and relations about entities in models that presuppose causal mechanisms being atomistic and additive. When causal mechanisms operate in real-world social target systems they only do it in ever-changing and unstable combinations where the whole is more than a mechanical sum of parts. If economic regularities obtain they do it (as a rule) only because we engineered them for that purpose. Outside man-made “nomological machines” they are rare, or even non-existent. Unfortunately, that also makes most of the achievements of econometrics – as most of the contemporary endeavours of economic theoretical modelling – rather useless.

Regression models are widely used by social scientists to make causal inferences; such models are now almost a routine way of demonstrating counterfactuals.

However, the “demonstrations” generally turn out to depend on a series of untested, even unarticulated, technical assumptions …Developing appropriate models is a serious problem in statistics; testing the connection to the phenomena is even more serious …In our days, serious arguments have been made from data. Beautiful, delicate theorems have been proved, although the connection with data analysis often remains to be established. And

an enormous amount of fiction has been produced, masquerading as rigorous science.

The theoretical conditions that have to be fulfilled for regression analysis and econometrics to really work are nowhere even closely met in reality. Making outlandish statistical assumptions does not provide a solid ground for doing relevant social science and economics. Although regression analysis and econometrics have become the most used quantitative methods in social sciences and economics today, it’s still a fact that most of the inferences made from them are invalid.

## A tragedy of statistical theory

19 Apr, 2021 at 16:05 | Posted in Statistics & Econometrics | Comments Off on A tragedy of statistical theoryMethodologists (including myself) can at times exhibit poor judgment about which of their new discoveries, distinctions, and methods are of practical importance, and which are charitably described as ‘academic’ … Weighing the costs and benefits of proposed formalizations is crucial for allocating scarce resources for research and teaching, and can depend heavily on the application …

Much benefit can accrue from thinking a problem through within these models, as long as the formal logic is recognized as an allegory for a largely unknown reality. A tragedy of statistical theory is that it pretends as if mathematical solutions are not only sufficient but ‘‘optimal’’ for dealing with analysis problems when the claimed optimality is itself deduced from dubious assumptions … Likewise, we should recognize that mathematical sophistication seems to imbue no special facility for causal inference in the soft sciences, as witnessed for example by Fisher’s attacks on the smoking-lung cancer link.

## Econometrics — formal modelling that has failed miserably

16 Apr, 2021 at 14:23 | Posted in Statistics & Econometrics | Comments Off on Econometrics — formal modelling that has failed miserablyAn ongoing concern is that excessive focus on formal modeling and statistics can lead to neglect of practical issues and to overconfidence in formal results … Analysis interpretation depends on contextual judgments about how reality is to be mapped onto the model, and how the formal analysis results are to be mapped back into reality. But overconfidence in formal outputs is only to be expected when much labor has gone into deductive reasoning. First, there is a need to feel the labor was justified, and one way to do so is to believe the formal deduction produced important conclusions. Second, there seems to be a pervasive human aversion to uncertainty, and one way to reduce feelings of uncertainty is to invest faith in deduction as a sufficient guide to truth. Unfortunately, such faith is as logically unjustified as any religious creed, since a deduction produces certainty about the real world only when its assumptions about the real world are certain …

Unfortunately, assumption uncertainty reduces the status of deductions and statistical computations to exercises in hypothetical reasoning – they provide best-case scenarios of what we could infer from specific data (which are assumed to have only specific, known problems). Even more unfortunate, however, is that this exercise is deceptive to the extent it ignores or misrepresents available information, and makes hidden assumptions that are unsupported by data …

Despite assumption uncertainties, modelers often express only the uncertainties derived within their modeling assumptions, sometimes to disastrous consequences. Econometrics supplies dramatic cautionary examples in which complex modeling has failed miserably in important applications …

Much time should be spent explaining the full details of what statistical models and algorithms actually assume, emphasizing the extremely hypothetical nature of their outputs relative to a complete (and thus nonidentified) causal model for the data-generating mechanisms. Teaching should especially emphasize how formal ‘‘causal inferences’’ are being driven by the assumptions of randomized (‘‘ignorable’’) system inputs and random observational selection that justify the ‘‘causal’’ label.

Yes, indeed, econometrics fails miserably over and over again. One reason why it does — besides those discussed by Greenland — is that the error term in the regression models used are thought of as representing the effect of the variables that were omitted from the models. The error term is somehow thought to be a ‘cover-all’ term representing omitted content in the model and necessary to include to ‘save’ the assumed deterministic relation between the other random variables included in the model. Error terms are usually assumed to be orthogonal (uncorrelated) to the explanatory variables. But since they are unobservable, they are also impossible to empirically test. And without justification of the orthogonality assumption, there is as a rule nothing to ensure identifiability.

Without sound justification of the assumptions made, the formal models used in econometric analysis is of questionable value. Failing to take unmodelled uncertainty (not stochastic risk) into serious consideration has made most econonometricians ridiculously overconfident in the reach of the (causal) inferences they make.

## The biggest problem in science

15 Apr, 2021 at 08:24 | Posted in Statistics & Econometrics | Comments Off on The biggest problem in scienceIn 2016, Vox sent out a survey to more than 200 scientists, asking, “If you could change one thing about how science works today, what would it be and why?” One of the clear themes in the responses: The institutions of science need to get better at rewarding failure.

One young scientist told us, “I feel torn between asking questions that I know will lead to statistical significance and asking questions that matter.”

The biggest problem in science isn’t statistical significance. It’s the culture. She felt torn because young scientists need publications to get jobs. Under the status quo, in order to get publications, you need statistically significant results. Statistical significance alone didn’t lead to the replication crisis. The institutions of science incentivized the behaviors that allowed it to fester.

As shown over and over again when significance tests are applied, people have a tendency to read ‘not disconfirmed’ as ‘probably confirmed.’ Standard scientific methodology tells us that when there is only say a 5 % probability that pure sampling error could account for the observed difference between the data and the null hypothesis, it would be more ‘reasonable’ to conclude that we have a case of disconfirmation. Especially if we perform many independent tests of our hypothesis and they all give about the same 5 % result as our reported one, I guess most researchers would count the hypothesis as even more disconfirmed.

We should never forget that the underlying parameters we use when performing significance tests are *model constructions*. Our p-values mean nothing if the model is wrong. And most importantly — statistical significance tests DO NOT validate models!

In journal articles a typical regression equation will have an intercept and several explanatory variables. The regression output will usually include an F-test, with p – 1 degrees of freedom in the numerator and n – p in the denominator. The null hypothesis will not be stated. The missing null hypothesis is that all the coefficients vanish, except the intercept.

If F is significant, that is often thought to validate the model. Mistake. The F-test takes the model as given. Significance only means this:

ifthe model is rightandthe coefficients are 0, it is very unlikely to get such a big F-statistic. Logically, there are three possibilities on the table:

i) An unlikely event occurred.

ii) Or the model is right and some of the coefficients differ from 0.

iii) Or the model is wrong.

So?

## Statistical power (student stuff)

7 Apr, 2021 at 16:59 | Posted in Statistics & Econometrics | Comments Off on Statistical power (student stuff).

## Econometrics — a critical-realist perspective

6 Apr, 2021 at 20:39 | Posted in Statistics & Econometrics | 11 CommentsMainstream economists often hold the view that criticisms of econometrics are the conclusions of sadly misinformed and misguided people who dislike and do not understand much of it. This is a gross misapprehension. To be careful and cautious is not equivalent to dislike.

The ordinary deductivist ‘textbook approach’ to econometrics views the modelling process as foremost an estimation problem since one (at least implicitly) assumes that the model provided by economic theory is a well-specified and ‘true’ model. The more empiricist, general-to-specific-methodology (often identified as the ‘LSE approach’) on the other hand views models as theoretically and empirically adequate representations (approximations) of a data generating process (DGP). Diagnostics tests (mostly some variant of the F-test) are used to ensure that the models are ‘true’ – or at least ‘congruent’ – representations of the DGP. The modelling process is here more seen as a specification problem where poor diagnostics results may indicate a possible misspecification requiring re-specification of the model. The objective is standardly to identify models that are structurally stable and valid across a large time-space horizon. The DGP is not seen as something we already know, but rather something we discover in the process of modelling it. Considerable effort is put into testing to what extent the models are structurally stable and generalizable over space and time.

Although yours truly has some sympathy for this approach in general, there are still some unsolved ‘problematics’ with its epistemological and ontological presuppositions. There is, e. g., an implicit assumption that the DGP fundamentally has an invariant property and that models that are structurally unstable just have not been able to get hold of that invariance. But one cannot just presuppose or take for granted that kind of invariance. It has to be argued and justified. Grounds have to be given for viewing reality as satisfying conditions of model-closure. It is as if the lack of closure that shows up in the form of structurally unstable models somehow could be solved by searching for more autonomous and invariable ‘atomic uniformity.’ But if reality is ‘congruent’ to this analytical prerequisite has to be argued for, and not simply taken for granted.

A great many models are compatible with what we know in economics — that is to say, do not violate any matters on which economists are agreed. Attractive as this view is, it fails to draw a necessary distinction between what is assumed and what is merely proposed as hypothesis. This distinction is forced upon us by an obvious but neglected fact of statistical theory: the matters ‘assumed’ are put wholly beyond test, and the entire edifice of conclusions (e.g., about identifiability, optimum properties of the estimates, their sampling distributions, etc.) depends absolutely on the validity of these assumptions. The great merit of modern statistical inference is that it makes exact and efficient use of what we know about reality to forge new tools of discovery, but it teaches us painfully little about the efficacy of these tools when their basis of assumptions is not satisfied.

Even granted that closures come in degrees, we should not compromise on ontology. Some methods simply introduce improper closures, closures that make the disjuncture between models and real-world target systems inappropriately large. ‘Garbage in, garbage out.’

Underlying the search for these immutable ‘fundamentals’ is the implicit view of the world as consisting of entities with their own separate and invariable effects. These entities are thought of as being able to be treated as separate and addible causes, thereby making it possible to infer complex interaction from a knowledge of individual constituents with limited independent variety. But, again, if this is a justified analytical procedure cannot be answered without confronting it with the nature of the objects the models are supposed to describe, explain or predict. Keynes thought it generally inappropriate to apply the ‘atomic hypothesis’ to such an open and ‘organic entity’ as the real world. As far as I can see these are still appropriate strictures all econometric approaches have to face. Grounds for believing otherwise have to be provided by the econometricians.

Continue Reading Econometrics — a critical-realist perspective…

## Econometrics and the problem of unjustified assumptions

4 Apr, 2021 at 19:11 | Posted in Statistics & Econometrics | 4 CommentsThere seems to be a pervasive human aversion to uncertainty, and one way to reduce feelings of uncertainty is to invest faith in deduction as a sufficient guide to truth. Unfortunately, such faith is as logically unjustified as any religious creed, since a deduction produces certainty about the real world only when its assumptions about the real world are certain …

Unfortunately, assumption uncertainty reduces the status of deductions and statistical computations to exercises in hypothetical reasoning – they provide best-case scenarios of what we could infer from specific data (which are assumed to have only specific, known problems). Even more unfortunate, however, is that this exercise is deceptive to the extent it ignores or misrepresents available information, and makes hidden assumptions that are unsupported by data …

Econometrics supplies dramatic cautionary examples in which complex modelling has failed miserably in important applications …

Yes, indeed, econometrics fails miserably over and over again.

One reason why it does, is that the error term in the regression models used is thought of as representing the effect of the variables that were omitted from the models. The error term is somehow thought to be a ‘cover-all’ term representing omitted content in the model and necessary to include to ‘save’ the assumed deterministic relation between the other random variables included in the model. Error terms are usually assumed to be orthogonal (uncorrelated) to the explanatory variables. But since they are unobservable, they are also impossible to empirically test. And without justification of the orthogonality assumption, there is, as a rule, nothing to ensure identifiability:

Distributional assumptions about error terms are a good place to bury things because hardly anyone pays attention to them. Moreover, if a critic does see that this is the identifying assumption, how can she win an argument about the true expected value the level of aether? If the author can make up an imaginary variable, “because I say so” seems like a pretty convincing answer to any question about its properties.

Nowadays it has almost become a self-evident truism among economists that you cannot expect people to take your arguments seriously unless they are based on or backed up by advanced econometric modelling. So legions of mathematical-statistical theorems are proved — and heaps of fiction are being produced, masquerading as science. The rigour of the econometric modelling and the far-reaching assumptions they are built on is frequently not supported by data.

Econometrics is basically a deductive method. Given the assumptions, it delivers deductive inferences. The problem, of course, is that we almost never know when the assumptions are right. Conclusions can only be as certain as their premises — and that also applies to econometrics.

Econometrics doesn’t establish the truth value of facts. Never has. Never will.

## Ergodicity — an intuitive introduction

3 Apr, 2021 at 11:08 | Posted in Statistics & Econometrics | 6 CommentsErgodicity is a difficult concept that many students of econom(etr)ics have problems with understanding. Trying to explain it, you often find yourself getting lost in mathematical-statistical subtleties difficult for most students to grasp.

In the video below, Luca Dellanna has made an admirably simplified and pedagogical exposition of what it means for probability structures of stationary processes and ensembles to be ergodic.

## Berkson’s paradox (student stuff)

25 Mar, 2021 at 17:09 | Posted in Statistics & Econometrics | Comments Off on Berkson’s paradox (student stuff).

## Wie Statistiken manipuliert werden

16 Mar, 2021 at 08:50 | Posted in Statistics & Econometrics | Comments Off on Wie Statistiken manipuliert werden.

Blog at WordPress.com.

Entries and Comments feeds.