There is considerable debate in psychology and related fields this week, as Harvard psychologist Daniel Gilbert and some coauthors published a strongly worded critique of the Open Science Collaborations’s finding last year that much of the research in social psychology could not be replicated in a broad follow-up effort. The specific critiques in this specific dialogue aside, I think the questions of replication and reproducibility fundamentally evoke a larger question about what science is, what good science looks like, and what makes someone a great scientist.

I will start with reference to other fields. Clyde Tombaugh is seen by many as a great astronomer, for his discovery of Pluto in 1930. A team of anthropologists, including a certain graduate student who did the actual discovering, found Lucy in 1974. These are acclaimed as some of the most important discoveries of their respective fields in their respective temporal contexts. Yet what did these scientists do exactly and what are we rewarding them for?

The rising concerns about publication biases suggest that we too often publish and thereby give credit to those who happen upon statistically significant or otherwise surprising results. If so, what distinguishes the scientific fame of these anthropologists from that of the tourists that happened across “the Iceman” in 1991 while walking through the Alps or the work of astronomer Alan Hale from that of factory manager Thomas Bopp? What is different about rewarding anthropologists for unexpected findings in the dirt or astronomers for chance sightings in the sky compared with rewarding social psychologists for freak p-values?

I would love a world where we are praised for diligently toiling away on an important social scientific problem regardless of what we find. But how would we actually recognize that when we see it? What defines a great scientist in a world where findings do not matter? Unfortunately, I do not yet have answers but I think it is something we need to start talking about.

---