The plausibility of alternative data-generating mechanisms: Comment on and attempt at replication of Dishop (2022)

Abstract

Dishop (2022) identifies the consensus emergence model (CEM) as a useful tool for future research on emergence but argues that autoregressive models with positive autoregressive effects are an important alternative data generating mechanism that researchers need to rule out. Here, we acknowledge that alternative data generating mechanisms are possibility for most, if not all, non-experimental designs and appreciate Dishop’s attempts to identify cases where the CEM could provide misleading results. However, in a series of independent simulations, we were unable to replicate two of three key analyses, and the results for the third analysis did not support the earlier conclusions. The discrepancies appear to originate from Dishop’s simulation code and what appear to be inconsistent model specifications that neither simulate the models described in the paper nor include notable positive autoregressive effects. We contribute to the wider literature by suggesting four key criteria that researchers can apply to evaluate the possibility of alternative data generating mechanisms: Theory, parameter recovery, fit to real data, and context. Applied to autoregressive effects and emergence data, these criteria reveal that (a) theory in psychology would generally suggest negative instead of positive autoregressive effects for behavior, (b) it is challenging to recover true autoregressive parameters from simulated data, and (c) that real datasets across a number of different contexts show little to no evidence for autoregressive effects. Instead, our analyses suggest that CEM results are congruent with the temporal changes occurring within groups and that autoregressive effects do not lead to spurious CEM results.