The issue of Data Access and Research Transparency (DA-RT) is in the news again this week, after Jeffrey Isaac blogged about his opposition to the DA-RT statement. The post is largely a terse reiteration of points he made in a much more verbose editorial (gated) in the June issue of Perspectives on Politics, the journal he edits for the American Political Science Association. In brief, Isaac is opposed to DA-RT for three reasons:

(1) The statement seemed to have major, and arguably controversial and troubling, implications, and represented a significant departure from established practice. It also seemed to require much serious deliberation – both among editors and editorial boards and among other disciplinary groupings and associations – before moving forward.

(2) In my experience as editor of Perspectives, DA-RT is a solution in search of a problem. We have done quite well with our existing processes of rigorous peer review – which often involve reviewer critiques of data or requests for more data – and editorial scrutiny and judgment; we have always encouraged authors to share data in appendices consistent with their own best judgments; and there was thus no reason to adopt the DA-RT principles.

(3) Perspectives on Politics has a distinctive mission: to promote a broad and pluralistic political science public sphere. I was concerned that the DA-RT framework, principles, and potential rules would privilege certain kinds of methods and formats over others, and would they would have a broader “chilling” effect, producing anxieties and concerns among many scholars that their work was no longer welcome or considered properly “scientific.”

His opposition has now found fairly widespread support among a very large group of mostly - though not exclusively - scholars adopting qualitative methods in the study of politics, who call for a delay in the implementation of DA-RT policies, which have been adopted by 27 political science journals including all the ones you can probably name off the top of your head (except Perspectives).

So, in 2015, we find the discipline of political science yet again rehasing the debates found in and around “A Tale of Two Cultures” (2006), Perestroika (2000), “Replication, Replication” (1995), KKV’s Designing Social Inquiry (1994), and so forth.

Honestly, I am surprised we are still having the conversation. I get it, a methodologically plural discipline requires pluralistic evidentiary standards, diverse publication venues, and research standards that respect distinct practices. I actually suspect those issues are less controversial than they would seem given how frequently political science has been forced to deliberate upon what we are as a discipline. Certainly there are individuals, departments, and publications that favor particular types of work or seem hostile to one or another variety of research, but I genuinely feel that political science is extremely pluralistic.

Further, I do not see DA-RT as especially hostile to any particular form of research. The DA-RT statement is extremely general and, unsurprisingly, leave a considerable amount of flexibility for editors to decide how best to implement DA-RT policies at their respective journals. Basically, it says:

• Data transparency: Authors need to share their data or justify their decision not to do so.

• Analytic transparency: Authors need to describe what they did to arrive at their conclusions.

• Data citation: Journals need to have standardized methods of citing data (independent of citing publications using those data).

I think the third point is fully uncontroversial, so I won’t discuss it. The points about data and analytic transparency, I guess, are the reason for the current debate. If we look at Isaac’s three points above, he seems to imply that requiring data and analytic transparency go against the status quo (1), is a solution to a non-problem (2), and both imposes one methodological perspective on all the others and prioritizes methods over public engagement (3).

Frankly, Isaac’s first point is simply status quo bias. Changes occur. We used to not have to peer review articles before publishing them, now we do. We used to not have to even say where we got our data, now we do. We used to publish on paper, now we publish on the internet. And so on. Science changes; it is not a credible argument to argue that present practices are better without showing why. This leaves Isaac’s second and third points.

Is DA-RT a solution in search of a problem? Clearly not. Reproducibility is a problem; as a simple example: a recent paper showing 50% of psychology articles report p-values inconsistent with their associated statistical estimates. Peer review (as Isaac claims) cannot solve these problems alone without data and analytic transparency, which is precisely what DA-RT aims to solve. He claims Perspectives does not have these problems; without transparency, there would be no way to know, so that claim is simply unfalsifiable.

Isaac’s final, two-part point is that DA-RT is both a prioritization of quantitative perspectives and a distancing from political relevance. These claims, too, are plainly false. DA-RT is methodologically neutral. It asks for greater transparency, in essence to show a more complete research effort rather than just the glances at data, evidence, and interpretation that can be fit into a relatively short journal article. The DA-RT statements says nothing about what types of evidence, analysis, or interpretation are valuable or compliant with the policy - presumably all are.

Here, again, Isaac also tries to create conflict where there is none by suggesting that transparency impedes relevance. I have never understood this argument. Methodological choices seem orthogonal to relevance, given that relevance is about science communication, partnerships, and impact rather than the way that one conducts their research.

In short, then, I do not understand Isaac’s opposition to DA-RT. The three reasons highlighted in his blog post seem irrelevant to the DA-RT statement. I suspect we should all take some advice from Chris Skovron’s grandmother: “Nothing is ever as good or as bad as you think it will be.”

That said, many people clearly share Isaac’s concerns (in opinion and/or judgment). Jarrod Hayes, writing at Duck of Minerva, is particularly concerned about the idea of sharing details of interviews, a concern reiterated by nearly all the contributors to Spring 2015 issue of the APSA Qualitative & Multi-Methods Research newsletter. The DA-RT statement is therefore widely seen as a problematic policy, independent of its specific policy content.

Interviews are clearly a special type of data, but they are one that seems easily accommodated within DA-RT’s explicit accommodation of restricted data: “(e.g., classified, require confidentiality protections, were obtained under a non-disclosure agreement, or have inherent logistical constraints)”. Scholars of all persuasions frequently use restricted data and it would seem that most journals would likely implement DA-RT through something as simple as requiring a one-sentence acknowledgement: “Data in this study are drawn in part from face-to-face interviews with [description of interviewees], the details of which are reported in the paper but the transcripts of which are not publicly available for reasons of confidentially.”

For analytic transparency, clarity and completeness of methodoligical explanation is a standard that applies equally to all research. If you cannot explain how your methods allow you to arrive at conclusions based upon data, then your work should not be published. That is a rule that applies universally. DA-RT does not seem to expect more than that. It rather suggests that additional data availability, such as placing transcripts, field notes, or other details in a public archive, would be valuable, but DA-RT does not aim to violate ethical rules or demand transparency where it is inappropriate or dangerous. It encourages a more open style of research, which might not only lend credibility to researchers’ claims but also facilitate the progression of science through the provision of materials that other researchers might be able to use to make new discoveries. In this way, I think the opponents to DA-RT misunderstand it; or the proponents of DA-RT have masterfully produced a document that hides an ethically dangerous agenda. I find it hard to believe the latter is true.

Speaking somewhat autobiographically, I suspect the DA-RT opponents are feeling a bit the way that I did when the discipline began discussing experimental pre-registration. Everything I read about pre-registration seemed to imply that certain individuals knew best how to conduct experimental research and they wanted to bind the hands of everyone else, forcing them into a particular way of thinking. At APSA 2015, however, I attended a panel describing the joint work of several APSA sections to create a multi-method study registry. What I learned was that registrationw was not about prioritizing particular forms of research at the expense of others, but rather making a clearer mapping between intended research and the actual research products that are produced by scholars. It was not about making sure outputs matched inputs perfectly, but instead to be transparent about their discrepancies so that the outputs are better understood in the context of a researcher’s initial vision for the project. I had long been a registration skeptic, but learning that the registration advocates actually and genuinely just wanted transparency (just as the DA-RT advocates do), convinced me that it might be quite valuable.

I think there’s plenty of space within DA-RT for all types of political science research, just as I strongly feel that there is plenty of space with the discipline for a broad plurality of questions, viewpoints, and approaches. This applies even in my own heavily quantitative, especially experimental field of public opinion research. To quote a scholar who I deeply respect and indeed admire, but appears to be on the opposite side of the debate surrounding DA-RT:

My interpretive work seeks to complement and be in dialogue with positivist studies of public opinion and political behavior. Its purpose is to illuminate the meaning people give to their worlds so that we can better understand the political preferences and actions that result. Understanding public opinion requires listening to the public. Transparency in this work is essential to make my methods clear to other scholars in my field who typically are unfamiliar with this approach, so that they can understand and judge my arguments. But I do not think that mandated transparency should extend to providing my transcripts and fieldnotes. My transcripts and fieldnotes are not raw data. The raw data exist in the act of spending time with and listening to people. That cannot be archived. The expectation for interpretive work should be that scholars thoroughly communicate their methods of data collection and analysis and provide rich contextual detail, including substantial quoting of the dialogue observed. There are many excellent models of such transparency in interpretive ethnographic work already in political science, which we can all aspire to replicate. (Cramer 2015, 20)

I completely agree, and that’s precisely why I think DA-RT is applicable for all forms of research.

---