Methods Matter: P-Hacking and Publication Bias in Causal Analysis in Economics

Abstract

The credibility revolution in economics has promoted causal identification using randomized control trials (RCT), difference-indifferences (DID), instrumental variables (IV) and regression discontinuity design (RDD). Applying multiple approaches to over 21,000 hypothesis tests published in 25 leading economics journals we find that the extent of p-hacking and publication bias varies greatly by method. IV (and to a lesser extent DID) are particularly problematic. We find no evidence that: (1) Papers published in the ‘Top 5’ journals are different to others; (2) The journal ‘revise and resubmit’ process mitigates the problem; (3) Things are improving through time