When I gave a talk and workshop about reproducible research at Aarhus University’s Interacting Minds Centre, one of the most provocative audience questions came from the center’s director, Andreas Roepstorff. His question, in brief, was whether the interpretations of scientific evidence made by scientists could ever be considered reproducible. It’s a question that’s stuck with me and despite thinking about it for over a year, I still do not have an answer.

In the broader debate on research reproducibility, almost all attention is paid to computational considerations (e.g., whether a computer environment can be fully described and preserved) and transparency (e.g., is data made public and code sufficiently clear and complete). The issue of human interpretation is rarely discussed. As an example, the recent book Implementing Reproducible Research focuses entirely on software tools and legal considerations. What is the place for humans in a scientific process increasingly focused on transparency and reproducibility?

Lincoln Mullen, taking history as a disciplinary examples, writes:

historians do not expect that even if they read exactly the same sources in exactly the same order as some other historian, that they would come to the same conclusion as she did. There is an irreducible component of interpretation in every historical study. This is not to suggest that works of history are unverifiable. Historians do hold one another to standards of evidence and argumentation, just not to a standard of replication. History-even computational history-remains a humanistic method.

Mullen goes on to say that “reproducible research does not require any human input or intervention”. Seen in this way, reproducible research is about trying to remove human intervention from the analysis and reporting of data, such that other forms of scholarship must be held to a completely distinct standard of transparency. I suspect this notion of distinct standards is part (but definitely not the only part) of why Data Access and Research Transparency (DA-RT) policies in political science are becoming so controversial.

Human interpretation is, of course, present in all forms of research. To describe a result as “novel,” “exciting,” “unexpected,” “interest,” “significant”, “unsurprising,” and so forth is to embed a (possibly reproducible) scientific process with a layer of interpretation. But some forms of research pivot heavily or perhaps entirely upon human interpretation. Interpretivist social scientists would likely argue that the human interpretation of observations of the social world are likely not at all reproducible according to the common standards of reproducible research, but that this lack of reproducibility is an inherent (and perhaps even desirable) feature of social research.

The interpretivists might be right about the limited ability to reproduce human interpretations. Well-established psychological principles, such as selective perception and confirmation bias, suggest that different individuals will find distinct meanings from the same observed reality. Reproducibility of those observations and interpretations may be unlikely unless the individuals come from relatively similar motivation and attitudinal perspectives and indeed encounter very, very similar versions of reality. It may even be that the same individual will arrive at distinct interpretations of the same information, depending on mood and contextual factors.

The question then remains: Can these complicated processes of human interpretation ever be reproducible? And is that a realistic, or even desirable, objective? If interpretation is critical to the research process and we hold reproducibility in high esteem, then researchers need to do considerably more work to understand how to document and reproduce these interpretive processes. An alternative, of course, is to set aside either human interpretation or the desire for reproducibility.

At this point, I remain committed to the idea that reproducibility is desirable in and of itself, so I am left thinking that we need to address the challenge of reproducing complicated processes of interpretation. Unfortunately much of the current debate about reproducible research is waged by statisticians, computer scientists, and those working in the physical sciences, where there may be little place for subjectivity (or little belief that subjectivity is at work). If the social sciences are going to play a leading role in research transparency, then we need to discuss if and how we can reproduce what are ultimately subjective interpretations of social reality. I believe that is something we can achieve, but I do not yet have definitive ideas for how to do it.

---