Using qualitative assessment data to inform decisions about medical student performance: scoping review and secondary analysis

Abstract

Introduction Failings in patient care have led to scrutiny of how medical education ensures the competence of graduates. Limitations of quantitative assessment methods necessitate exploring different approaches to ensure assessment outcomes reflect high quality patient care. Routinely collected qualitative assessment data may enhance decisions about undergraduate medical students. This study explored how Objective Structured Clinical Examination (OSCE) examiners’ comments can inform decisions through two research questions: “What is known about how assessors make sense of comments and make decisions in qualitative assessment?” and “In what ways do examiners’ written comments offer insights about competence that could inform decision-making?” Methods This interpretivist study comprised a scoping review following Arksey and O'Malley’s framework and reflexive thematic analysis of a 50,000-word dataset of comments written by examiners about a full cohort of fourth-year medical students in an Objective Structure Clinical Examination (OSCE) at a UK medical school. Results The scoping review included five studies and found an iterative decision-making process in which assessors built a mental picture by seeking patterns and resolving discrepancies. Divergent interpretation strategies arose from assessors’ values and conceptual frameworks. Reflexive thematic analysis of OSCE examiners’ comments showed that comments provided novel insights not yet captured in assessment outcomes by explaining marking decisions, discriminating between levels of performance, revealing how students enacted the characteristics of good medical practice valued by examiners, and including multiple perspectives on student performance.

Conclusion OSCE examiners comments provide rich insights with potential to enhance decisions about competence by enabling assessment outcomes to be patient-focused, incorporating subjective assessment and promoting fairness. Comments have potential to inform pass/fail decisions in mixed methods assessment designs and could contribute evidence for the quality assurance of OSCEs. Assessor training should explore the subjectivist epistemology of Qualitative Assessment and encourage reflexivity to align practice with the constructivist paradigm in which qualitative assessment is situated.

Key words: assessment, competence, OSCE, decisions, comments