Every time I have published or tried to publish a paper that included so-called “null” results (i.e., evidence that does not indicate a causal effect statistically or substantively distinguishable from zero), reviewers ask for post-hoc explanations. As an exmaple from a recent paper of mine, I had a simple hypothesis, supported by widely used theory, that there should be an effect and I failed to find one in a variety of data sources. My conclusion was that this was surprising, particularly in light of the widely cited theory for the effect and this being the most comprehensive search for that effect to-date, and thus merited both further study and a revision of our understanding of the phenomenon.

Here’s an excerpt of part of the review:

As Reviewer 2 points out, even if you had offered an account, it would be challenging to examine empirically, because your contribution rests entirely on a null effect. Null effects are notoriously hard to interpret with any confidence as they could derive from multiple sources. Maybe there really is no effect (…) maybe there is an effect but measurement error masked it (…) maybe your decision to control for (REDACTED) obscured the relation (REDACTED) would have had with (REDACTED) (Reviewer 1); maybe (REDACTED) are unique in that they are higher in (REDACTED), etc. and these factors kept (REDACTED) relatively (REDACTED) regardless of (REDACTED) (Reviewer 2); or maybe you simply didn’t have enough variance in (REDACTED) to pick up the expected relation…

(Note: I’ve redacted some things to hopefully mask the paper and journal, but also to highlight what are comments that could essentially be made about any “null” effect.) In that set of reviews, the editor and reviewers asked for what (in my view) were a series of fishing expeditions in desperate search for a “positive” result. There were other weaknesses in the paper, certainly, but the apparently unsatisfactory standing of the null result was prominently noted by reviewers and the editor. This craving for an explanation of the null reminded me a lot of a post (now removed, but available on the Internet Archive) by Harvard University’s Jason Mitchell, in essence arguing that there is nothing to be learned from a “null” effect.

I come from a much different position. In my view there is almost always something to be learned from any piece of evidence, even if the change in beliefs or certainty about those beliefs is very small. The frustration of encountering so much concern about “null” results, however, prompted me to think about what it would be like if scientists (and especially reviewers) were more skeptical of “positive” results (i.e., evidence that indicates a causal effect statistically or substantively distinguishable from zero) in the same way that they are of “null” results.

What if my results had been different and a reviewer or editor had written the following:

As Reviewer 2 points out, even if you had offered an account, it would be challenging to examine empirically, because your contribution rests entirely on a positive effect. Positive effects are notoriously hard to interpret with any confidence as they could derive from multiple sources. Maybe there really is no effect (…) maybe there is not an effect but measurement error masked it (…) maybe your decision to control for (REDACTED) obscured the null relation (REDACTED) would have had with (REDACTED) (Reviewer 1); maybe (REDACTED) are unique in that they are higher in (REDACTED), etc. and these factors kept (REDACTED) relatively (REDACTED) regardless of (REDACTED) (Reviewer 2); or maybe you simply didn’t have enough variance in (REDACTED) to pick up the expected relation…

In essence, a few changed words and the same comments are turned on their head. Yet we almost never encounter criticisms of that sort. We rarely voice and rarely hear comments that criticize “positive” results in the same way that “null” results are criticized. Maybe that should tell us something.

---