Re-Thinking Reproducibility as a Criterion for Research Quality
Reason for embargo
Currently under an indefinite embargo pending publication by Emerald. No embargo required on publication.
A heated debate surrounds the significance of reproducibility as an indicator for research quality and reliability, with many commentators linking a "crisis of reproducibility" to the rise of fraudulent, careless and unreliable practices of knowledge production. Through the analysis of discourse and practices across research fields, I point out that reproducibility is not only interpreted in different ways, but also serves a variety of epistemic functions depending on the research at hand. Given such variation, I argue that the uncritical pursuit of reproducibility as an overarching epistemic value is misleading and potentially damaging to scientific advancement. Requirements for reproducibility, however they are interpreted, are one of many available means to secure reliable research outcomes. Furthermore, there are cases where the focus on enhancing reproducibility turns out not to foster high-quality research. Scientific communities and Open Science advocates should learn from inferential reasoning from irreproducible data, and promote incentives for all researchers to explicitly and publicly discuss (1) their methodological commitments, (2) the ways in which they learn from mistakes and problems in everyday practice, and (3) the strategies they use to choose which research component of any project needs to be preserved in the long term, and how.
This research was funded by the European Research Council grant award 335925 (“The Epistemology of Data-Intensive Science”), the Australian Research Council Discovery Project “Organisms and Us” and the UK Economic and Social Research Council award ES/P011489/1.
This is the accepted author manuscript
In: Research in the History of Economic Thought and Methodology [complete citation and DOI to be added on publication]