Peer review is something of a false dichotomy. We are often quick to withhold judgment about studies that have “not yet been peer reviewed” - that phrase being an implicit argument that we should not trust studies that have not yet survived the sometimes mysterious gauntlet that comes after uploading a PDF to Editorial Manager. Similarly, such statements are implicit claims that we should trust research that has passed peer review. These successful studies are deemed more trustworthy, perhaps even more “true” than those that have not yet been revised in response to the comments of (approximately) three anonymous reviewers.

I say peer review is a false dichotomy because we know of course that it draws a distinction between two classes of research claims when there in fact is little that distinguishes studies on either side of the divide. Studies not yet peer reviewed have often suffered through various forms of informal peer review: studies may have been vetted by funding agencies or partnering organizations, drafts of research reports are circulated to colleagues, and research is very often presented at conferences and workshops. From reviews expressed in all of these informal settings, revisions are made in preparation for submission to a peer reviewed academic outlet. Of course, many papers are submitted without such informal processes, but it is just as often the case that a study has crossed the desks (or at least eyes and ears) of far more researchers before submission to a journal than it will pass between the time of submission and publication.

Similarly, those studies that have successfully passed formal peer review can be found in journals (or other similar outlets). We imbue these studies with a high degree of credibility, despite a seemingly endless stream of scientific retractions, reproducibility failures like the Reinhart-Rogoff data glitch, p-hacking, the “Replication Crisis” in psychology, “crappy, or just plain pointless” peer-reviewed research, a suspicion that “Most Published Research Findings Are False”, and numerous cases of misconduct or outright scientific fraud (including in political science). We all know that studies that have passed peer review are not necessarily true, yet we perpetuate a collective belief in a sharp distinction between studies that have been and have not been peer reviewed.

Perhaps peer review is not what it seems. There is much research we should trust that has not yet passed that threshold and there is must research we should not trust that has. Does this mean we should abandon peer review entirely (because, apparently, it does little to help us judge what is good from what is bad)?

No. Peer review is still useful because it indicates that studies at least appear to show a modicum of scientific integrity, but there is an ever-more-apparent need for reform. One step I think might be quite helpful is a separation, indeed a bifurcation, of the peer review and publication processes. At present, “peer reviewed” is a seal of approval only applied to published research, when in reality much research has survived the informal peer review processes described above and most peer-reviewed journals publish a mix of reviewed and invited content (though typically much less of the latter variety).

I would propose, if only as a thought experiment, a bifurcated peer review process wherein research is peer reviewed through a system unattached to any particular publication outlet. Studies would enter the peer review system early (perhaps as part of a pre-registration process) and peer reviewers would be recruited to follow the research through to its conclusion but render no judgment about the “publishability” or “impact” of the research. Research publications having survived the peer review process can then be shopped around to publication venues (as academic books are to publishers) and editors can make final decisions about whether to publish a given article from this universe of centrally peer-reviewed studies. The bifurcation of peer review and publication means that reviewers should focus on peer review true to its name: assessing, evaluating, and (to the extent possible) improving the scientific quality of the research without regard to where that publication will eventually “land” and whether a study is “good enough” for such-and-such journal. Publication then serves not a gate-keeping function to the wonderful world of peer reviewed status but as a recommendation engine for particular topics and quality of research. Bifurcated peer review separates the editorial decision to publish from the scientific assessment of research; reviewers are unattached to any outlet, so their assessments are structured around science rather than passing judgment on fit for a given journal.

The system of bifurcated peer review allows research to formally exist (as it already informally does) in a world between peer review and publication. (The term “bifurcation” comes from the legal notion of separating portions of trial, for example judgment and sentencing.) It also brings some potential other benefits to lingering problems in peer review:

1. It may reduce reviewer burden because an article will only enter one peer review process, rather than “making the rounds” through many journals where the same reviewers within a given subfield are likely to see the same article more than once.
2. It may speed the time between research implementation and research dissemination, again because there is no process of repeated resubmission for peer review after rejection from a given outlet.
3. It offers a centralized system of registering and thus attaching value to peer review service, something largely impossible to track under the current decentralized system.
4. It draws a clear distinction between scientific evaluation (i.e., peer review) and editorial taste that are largely conflated in current peer review processes.

There are probably other benefits. The system may be problematic in several ways. First, it transfers burden of managing peer review of (often for-profit) publishers to some kind of centralized entity (e.g., a scholarly association) that must finance the system. Second, it could create a bottleneck for research dissemination. I’m sure there are others.

I have never seen anyone propose a system of bifurcated peer review. The closest model might be that of PLOS, which explicitly rejects the assessment of “impact” in its peer review process. But, it seems like it should be part of the broader conversation about how best to create, assess, and disseminate scientific research. I would love to hear your feedback.

---