This is the third of a series of posts on Angus Deaton and Nancy Cartwright’s working paper, “Understanding and Misunderstanding Randomized Controlled Trials.” See the first post and second post.

One of Deaton and Cartwright’s best critiques of experimental research relates to the use of “blinding” or disguising treatment status from participants in a study. In essence, the lack of blinding (awareness of treatment status) opens pathways between treatment-assignment to outcome other than the posited treatment, which is a violation of the exclusion restriction. It may be that knowing I am being treated is what causes my outcome to change, rather than the treatment I receive as a result of being assigned to treatment. Similarly, expecting that I will be treated (as a result of being assigned as such) may change my behavior in other ways - including by leaving the study, attempting to convert my assignment status, or compensating for (non)treatment through means outside the scope of the study.

These placebo effect and compliance dynamics are well-known to anyone trained in experimental methods. Indeed, much of experimental methods courses is consumed with how to design studies to minimize these issues and how to analyze the resulting data when these dynamics manifest. Deaton and Cartwright discuss for example, the three common analytic approaches in situations of non-compliance: intention-to-treat (ITT), as-treated, and LATE/CATE analysis. This may be (perhaps painfully) familiar to anyone who has gone through experimental methods training. Blinding is meant to help avoid the choice between these methods and underlying ability of participants to modify their treatment status. Deaton and Cartwright, however, to point out that blinding as a design technique is used much less often in social science contexts than in medical studies.

(As a brief aside, it is worth noting that social science experiments - unlike medical trials - often engage in a form of blindness that medical trials do not. In a medical RCT, a participant knows they will be randomized. In a social science experiment, participants are often blind to the fact that a study is experimental. While knowing what treatment they receive opens paths between treatment status and outcome assign from treatment per se, blinding to the randomization device - or even to the presence of other treatment conditions - means that the kinds of non-compliance and placebo dynamics that threaten medical RCTs may not manifest at all because participants are unaware that they have the option to noncomply in ways that require knowledge of other conditions.)

Yet there is an important shortcoming in Deaton and Cartwright’s discussion. Their characterization of blinding - and the selection problems it introduces - focuses only on noncompliance, or what I might call “treatment self-selection” as a problem, as something to be eliminated. This is ironically quite typical of the view taken by most experimentalists. The advantage of experiments is to assign treatment, so anything that gives participants choice over treatment moves us outside of an experimental world back into one of observational data. If the researcher does not perfectly dictate treatment status, then why bother even experimenting!?

I have long taken issue with this view (by “long,” I mean since circa-2010 when I started writing my PhD dissertation). Self-selection of treatment is actually an interesting and understudied empirical puzzle that is worth studying in its own right. Indeed, much of my research has been focused on trying to understand how people behave when they have the ability to self-select their treatment status. And the experimental method has been critical to answering questions about self-selection. Let me explain.

Experiments are typically understood as assigning treatment versus control (versus other treatments). But what precisely is the treatment we want to understand the effect of? Is it, in a study of public attitudes, a media message? Or is a set of media messages? What is the control? Is it some alternative message? Is it nothing? Sometimes treatment is a message, but sometimes treatment is also access to a message. Sometimes control is nothing, but sometimes it is the choice to not be treated.

For example, in a 2012 paper, my coauthors and I were interested in opinion dynamics over time. We randomly assigned people to either receive a message once and then receive nothing, or to receive the same message frame repeated multiples times, or to receive a message and then be given the choice of subsequent messages. Among other things we found that peoples’ self-selection of messages (when given the opportunity to choose) reflected the initial message they received and then the opinions they held at the end of the study were similar to those of individuals who had simply had the message repeated. Self-selection, rather than freeing people from treatment, actually reinforced earlier messaging. We wouldn’t have been able to learn this (or any of the paper’s other findings) without both randomizing people into different conditions and defining some of those conditions as spaces where participants were able to engage in self-selection.

As another example, one of my forthcoming articles, shows that the effects of a randomly assigned treatment actually differ for those who want to receive it compared to those who would prefer something else. Through some clever analysis developed by Brian Gaines and Jim Kuklinski (and accessible as an R package that I wrote), it was possible to provide some participants with the opportunity to choose what treatment they received in order to assess whether effects of a randomly assigned treatment in other experimental conditions had different effects on these subsets of the population. The randomization and blindness of some participants had to be forsaken in order to learn something from others.

As a final example from my own work, I have used very subtle violations of self-selection to study how people learn from the news. In what has come to be known as a “patient preference trial,” I allowed participants to choose what kinds of news they wanted to receive and then randomly manipulated a small portion of their chosen content and then measured every participant’s level of post-treatment knowledge. The non-blinded nature of the study was critical - I wanted participants to believe they were receiving the information they chose to receive. If they were randomly assigned to all content, it would defeat the study’s goal of studying effect heterogeneity. Randomization was indeed secondary to self-selection in this design; it was merely an easy analytic device to understand causal effects among those with different content preferences. If participants were blind, the experiment would have been meaningless; the entire point was that they were not blind.

These examples from my own work demonstrate cases where randomly allocated opportunities for self-selection that were completely non-blind to participants proved incredibly useful for understanding the influence of messages on political outcomes. Self-selection - who selects which treatment and with what effect - is actually a valuable research puzzle. Yet studying it by definition requires research designs where participants are not blind to treatment status. The demand for blindness may be useful in some contexts, but I think Deaton and Cartwright - and many experimentalists - overlook the many situations where non-blindness is useful and where experiments involving structured opportunities for self-selection can be particularly enlightening.

---