RCT — no guide to the future

10 Feb, 2016 at 17:42 | Posted in Statistics & Econometrics | 8 Comments

To see what RCTs show, let me define the Cartesian product of X2, X3, . . . , Xn by Z. What RCTs show is that there exists some z ∈ Z, such that if we have the world in state (x, z) ∈ X instead of (y, z) ∈ X, the world in the next period will be in state a ∈ X instead of state b ∈ X. This is like saying, other things being the same (that is, z), if you vaccinate people, in the next period, there will be no influenza. But if you do not vaccinate them, there will be influenza. If we accept the determinist axiom, as many do, then this demonstration means that whenever we switch from (y, z) to (x, z), the world will switch in the next period from b to a. It is the “whenever” that makes this a causal claim. This is what I am referring to as “circumstantial causality”. Given a certain set of circumstances, changing y to x has a predictable consequence.

UnknownThe discovery of circumstantial causal connections, as has happened with the rise of RCT studies, is valuable and, at the same time, of limited consequence, more so than the proponents believe. On the one hand, RCTs have given us numerous valuable descriptions of what happened in the past and numerous instances of causes in the past (provided of course that one is willing to accept the determinist axiom). On the other, what they show is very limited. This is because when they show that it was the switch from y to x that caused the switch from b to a, what they are saying is that this was true under certain historical conditions (z), but they cannot tell you what those historical conditions are. RCT discoveries never graduate from something “was a cause” of something else to something “is a cause”. RCTs give us no insight into universal causality because they cannot tell us what it was that was being held constant (z in the above example), when we switched some intervention b to a. For Bengal, in a certain period, electing a woman leader of the local government caused water provisioning to be better. This is no guide to the future because we do not fully know what Bengal in a certain period is like. Henceforth, a reference to causality without a qualifying epithet should be taken to be a reference to universal causality because for policy purposes, that is what is of essence.

Kaushik Basu

8 Comments

  1. For a start I should acknowledge my error in writing Bangladesh rather than Bengal, but I don’t think that changes the rest of the discussion.

    Let me also say that I don’t disagree with the reasoning of Basu as far as it goes, and accepting the assumptions he gives. However, I don’t think that line of reasoning is fruitful. To me Baku seems rather over-optimistic with regard to the capacities of unsystematic observation/reasoning (intuition), and the examples he gives against RCTs are (without going into exhaustive detail) unconvincing. The problem in general (in medicine, economics, and many other fields) is one of decision-making under uncertainty, and the question is which of the possible procedures on offer is to be preferred. Now intuition, argument from authority (such as religious authority, as has happened in the case of vaccination, the case Baku himself introduced), and theory-guided empirical studies (of which well-designed RCTs are an example) are all possibilities. In many, many cases, an RCT is not feasible. In some cases (such as in deeply reflexive systems) they are not possible even in principle, but where they exist, they should be given more weight than alternative sources of information.

    Reading the hypothetical examples Baku proposes, the problems in economics research do not seem so different from those of medicine, which I am more familiar with, and I have to say RCTs are currently very popular for a reason. It is extremely easy to fool yourself, and uncontrolled observational studies are an excellent way of confirming researcher prejudices, to say nothing of the less systematic forms of gaining knowledge.

    Comment by dnm— 12 Feb, 2016 #

  2. It was directed at both of you. Because both of you invent absurd extrapolations from arguments that say no such thing and then mock the figments of your own imagination. The quotation you cite is an excellent example. Please tell me how the following is entailed by that quite reasonable statement:

    A parent’s intuition, on the other hand, (or given that Bangladesh was mentioned, the intuition of the local religious leader), is surely not to be doubted…
    Let alone the silliness of your fellow traveller:

    Yep, I think we should explain to the millions of peoples whose lives been saved by vaccines against smallpox, diphtheria, tetanus, yellow fever, polio, and measles, that their health has no sound ontological grounding.

    And you should know that although circumstances do change and viruses do adapt, the problems with extrapolating automatically from RCT results are not only historical ones.

    Comment by KM— 11 Feb, 2016 #

  3. Of course, my reply was sarcastic. But I disagree that I was straw-manning.
    Right there it says in the quoted text: “The discovery of circumstantial causal connections, as has happened with the rise of RCT studies, is valuable and, at the same time, of limited consequence”.
    Well the discovery of causal connections is important, and we live in an uncertain world. Circumstances change. Viruses adapt. Until that chance occurs our discovery is useful, whether it comes from an RCT or some other kind of study. Universal causality will have to wait until we find a way of determining facts about the world with absolute certainty.
    And of course, if KM’s argument was directed at me, the reference to clinical medicine (or at least vaccination) was in the original text.

    Comment by dnm— 11 Feb, 2016 #

  4. I’d like to award extra points to whoever can come up with the most histrionic “extrapolation” of the arguments they don’t like. If you can’t, you know, deal with the content of an argument, try a silly reductio.

    And I do love the unargued-for premise that whatever goes for clinical medicine will go for experimental economics, too.

    Comment by KM— 11 Feb, 2016 #

  5. I can make strawman arguments, too. Apparently most of those vaccines had no sound ontological grounding before Austin Bradford Hill!

    Comment by KM— 11 Feb, 2016 #

  6. Oh, how clever the responses!

    Comment by KM— 11 Feb, 2016 #

  7. Yep, I think we should explain to the millions of peoples whose lives been saved by vaccines against smallpox, diphtheria, tetanus, yellow fever, polio, and measles, that their health has no sound ontological grounding. Just because it has worked for millions of people in different countries over decades, what validity does that really carry for other people in the future. We simply do not know!
    .
    RCT tested vaccines should buy lottery tickets more often!

    Comment by pontus— 11 Feb, 2016 #

  8. This argument works against all forms of empirical reasoning. Yesterday the mass of my electron was measured as 9.10938188 × 10^-31 kilograms. In a billion years time, or tomorrow, who can say?
    Plus, I award extra points for using vaccination as a nice uncontroversial example of the problems of RCTs. Sure the results are not certain. A parent’s intuition, on the other hand, (or given that Bangladesh was mentioned, the intuition of the local religious leader), is surely not to be doubted…

    Comment by dnm— 11 Feb, 2016 #


Sorry, the comment form is closed at this time.

Blog at WordPress.com.
Entries and Comments feeds.