From Theory to Practice

Background: Manual record abstraction is the standard method for data collection in observational studies but is labor-intensive, error-prone, and difficult to scale, particularly when information is embedded in unstructured electronic health records. Large language models (LLMs) may streamline this process, yet evidence from real-world clinical research remains limited. Objectives: This study evaluated the performance of a generative pre-trained transformer (GPT)-based LLM in extracting sociodemographic, procedural, and outcome variables from free-text electronic health records of patients undergoing transcatheter aortic valve replacement. Methods: We conducted a retrospective analysis of medical and nursing records for all transcatheter aortic valve replacement procedures performed at Ca’ Foncello Hospital (Treviso, Italy) between January and June 2024. Manual abstraction by 2 reviewers served as the reference standard. Accuracy, sensitivity, and specificity with 95% CIs were calculated. Agreement for continuous variables was assessed using Bland-Altman analyses. Results: A total of 108 cases were included. GPT achieved accuracy ranging from 0.657 (valve brand) to 1.00 (gender, Barthel index, procedure timings, and several intraoperative complications). Sensitivity reached 1.00 for rare but clinically important events, including intraoperative neurological complications, whereas specificity exceeded 0.90 for most variables. For vital parameters, Bland-Altman analyses demonstrated minimal bias and narrow limits of agreement. Conclusions: GPT-based data extraction showed high accuracy across a broad range of variables, particularly continuous repeated measurements and rare intraoperative outcomes. Performance was lower for some infrequent postoperative events, reflecting sparse true positives. These findings support the feasibility of integrating LLM-assisted extraction into observational research workflows, with further validation needed in larger or multicenter cohorts.