Measuring the impact of national guidelines: what methods can be used to uncover time-varying effects for healthcare evaluations?

Abstract

We examine the suitability of three methods using patient-level data to evaluate the time-varying impacts of national healthcare guidelines. Such guidelines often codify progressive change and are implemented gradually; for example, National Institute for Health and Care Excellence (NICE) suspectedcancer referral guidelines. These were revised on 23/06/2015, to include more cancer symptoms and test results (“features”), partly reflecting changing practice. We explore the time-varying impact of guideline revision on time to colorectal cancer diagnosis, which is linked to improved outcomes in decisionanalytic models. We included 11,842 patients diagnosed in 01/01/2006–31/12/2017 in the Clinical Practice Research Datalink with England cancer registry data linkage. Patients were classified by whether their first pre-diagnostic cancer feature was in the original guidelines (NICE-2005) or was added during the revision (NICE-2015-only). Outcome was diagnostic interval: time from first cancer feature to diagnosis. All analyses adjusted for age and sex. Two difference-in-differences analyses used either a Pre (01/08/2012–31/12/2014, n=2,243) and Post (01/08/2015–31/12/2017, n=1,017) design, or event-study cohorts (2006–2017 vs 2015) to estimate change in diagnostic interval attributable to official implementation of therevised guidelines. A semiparametric varying-coefficient model analysed the difference in diagnostic interval between the NICE groups over time. After model estimation, primary and broader treatment effects of guideline content and implementation were measured. The event-study difference-indifferences and the semiparametric varying-coefficient methods showed that shorter diagnostic intervals were attributable to official implementation of the revised guidelines. This impact was only detectable by pre-to-post difference-in-differences when the pre/post periods were selected according to the estimation results from the varying-coefficient model. Formal tests of the parametric models, which are special cases of the semiparametric model, suggest that they are misspecified. We conclude that the semiparametric method is well suited to explore the time-varying impacts of guidelines codifying progressive change.