Can GPT-3.5 generate and code discharge summaries?
dc.contributor.author | Falis, M | |
dc.contributor.author | Gema, AP | |
dc.contributor.author | Dong, H | |
dc.contributor.author | Daines, L | |
dc.contributor.author | Basetti, S | |
dc.contributor.author | Holder, M | |
dc.contributor.author | Penfold, RS | |
dc.contributor.author | Birch, A | |
dc.contributor.author | Alex, B | |
dc.date.accessioned | 2024-09-17T08:54:30Z | |
dc.date.issued | 2024-09-13 | |
dc.date.updated | 2024-09-16T16:30:42Z | |
dc.description.abstract | Objectives The aim of this study was to investigate GPT-3.5 in generating and coding medical documents with International Classification of Diseases (ICD)-10 codes for data augmentation on low-resource labels. Materials and Methods Employing GPT-3.5 we generated and coded 9606 discharge summaries based on lists of ICD-10 code descriptions of patients with infrequent (or generation) codes within the MIMIC-IV dataset. Combined with the baseline training set, this formed an augmented training set. Neural coding models were trained on baseline and augmented data and evaluated on an MIMIC-IV test set. We report micro- and macro-F1 scores on the full codeset, generation codes, and their families. Weak Hierarchical Confusion Matrices determined within-family and outside-of-family coding errors in the latter codesets. The coding performance of GPT-3.5 was evaluated on prompt-guided self-generated data and real MIMIC-IV data. Clinicians evaluated the clinical acceptability of the generated documents. Results Data augmentation results in slightly lower overall model performance but improves performance for the generation candidate codes and their families, including 1 absent from the baseline training data. Augmented models display lower out-of-family error rates. GPT-3.5 identifies ICD-10 codes by their prompted descriptions but underperforms on real data. Evaluators highlight the correctness of generated concepts while suffering in variety, supporting information, and narrative. Discussion and Conclusion While GPT-3.5 alone given our prompt setting is unsuitable for ICD-10 coding, it supports data augmentation for training neural models. Augmentation positively affects generation code families but mainly benefits codes with existing examples. Augmentation reduces out-of-family errors. Documents generated by GPT-3.5 state prompted concepts correctly but lack variety, and authenticity in narratives. | en_GB |
dc.description.sponsorship | UKRI | en_GB |
dc.description.sponsorship | Engineering and Physical Sciences Research Council (EPSRC) | en_GB |
dc.description.sponsorship | Wellcome Trust | en_GB |
dc.description.sponsorship | Legal and General PLC | en_GB |
dc.description.sponsorship | National Institute for Health and Care Research (NIHR) | en_GB |
dc.identifier.citation | Published online 13 September 2024 | en_GB |
dc.identifier.doi | https://doi.org/10.1093/jamia/ocae132 | |
dc.identifier.grantnumber | EP/S02431X/1 | en_GB |
dc.identifier.grantnumber | EP/V050869/1 | en_GB |
dc.identifier.grantnumber | 223499/Z/21/Z | en_GB |
dc.identifier.grantnumber | NIHR202639 | en_GB |
dc.identifier.uri | http://hdl.handle.net/10871/137473 | |
dc.identifier | ORCID: 0000-0001-6828-6891 (Dong, Hang) | |
dc.language.iso | en | en_GB |
dc.publisher | Oxford University Press (OUP) / American Medical Informatics Association | en_GB |
dc.relation.url | https://physionet.org/about/citi-course/ | en_GB |
dc.relation.url | https://doi.org/10.13026/bnc2-1a81 | en_GB |
dc.rights | © The Author(s) 2024. Published by Oxford University Press on behalf of the American Medical Informatics Association. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited. | en_GB |
dc.subject | ICD coding | en_GB |
dc.subject | data augmentation | en_GB |
dc.subject | large language model | en_GB |
dc.subject | clinical text generation | en_GB |
dc.subject | evaluation by clinicians | en_GB |
dc.title | Can GPT-3.5 generate and code discharge summaries? | en_GB |
dc.type | Article | en_GB |
dc.date.available | 2024-09-17T08:54:30Z | |
dc.identifier.issn | 1067-5027 | |
dc.description | This is the final version. Available on open access from Oxford University Press via the DOI in this record | en_GB |
dc.description | Data availability: The synthetic discharge summary data generated as part of this study will be shared on reasonable request to the corresponding author upon presenting a certificate of completion of the CITI Data or Specimens Only Research course from the Collaborative Institutional Training Initiative program (https://physionet.org/about/citi-course/). The data has been accepted for publication and will be made available via PhysioNet (https://doi.org/10.13026/bnc2-1a81). | en_GB |
dc.identifier.eissn | 1527-974X | |
dc.identifier.journal | Journal of the American Medical Informatics Association | en_GB |
dc.rights.uri | https://creativecommons.org/licenses/by/4.0/ | en_GB |
dcterms.dateAccepted | 2024-05-22 | |
dcterms.dateSubmitted | 2024-01-02 | |
rioxxterms.version | VoR | en_GB |
rioxxterms.licenseref.startdate | 2024-09-13 | |
rioxxterms.type | Journal Article/Review | en_GB |
refterms.dateFCD | 2024-09-17T08:50:07Z | |
refterms.versionFCD | VoR | |
refterms.dateFOA | 2024-09-17T08:55:13Z | |
refterms.panel | B | en_GB |
refterms.dateFirstOnline | 2024-09-13 | |
exeter.rights-retention-statement | No |
Files in this item
This item appears in the following Collection(s)
Except where otherwise noted, this item's licence is described as © The Author(s) 2024. Published by Oxford University Press on behalf of the American Medical Informatics Association. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited.