Validation, bias assessment, and optimization of the UNAFIED 2-year risk prediction model for undiagnosed atrial fibrillation using national electronic health data

Abstract

Background: Prediction models for atrial fibrillation (AF) may enable earlier detection and guideline-directed treatment decisions. However, model bias may lead to inaccurate predictions and unintended consequences. Objective: The purpose of this study was to validate, assess bias, and improve generalizability of "UNAFIED-10," a 2-year, 10-variable predictive model of undiagnosed AF in a national data set (originally developed using the Indiana Network for Patient Care regional data). Methods: UNAFIED-10 was validated and optimized using Optum de-identified electronic health record data set. AF diagnoses were recorded in the January 2018-December 2019 period (outcome period), with January 2016-December 2017 as the baseline period. Validation cohorts (patients with AF and non-AF controls, aged ≥40 years) comprised the full imbalanced and randomly sampled balanced data sets. Model performance and bias in patient subpopulations based on sex, insurance, race, and region were evaluated. Results: Of the 6,058,657 eligible patients (mean age 60 ± 12 years), 4.1% (n = 246,975) had their first AF diagnosis within the outcome period. The validated UNAFIED-10 model achieved a higher C-statistic (0.85 [95% confidence interval 0.85-0.86] vs 0.81 [0.80-0.81]) and sensitivity (86% vs 74%) but lower specificity (66% vs 74%) than the original UNAFIED-10 model. During retraining and optimization, the variables insurance, shock, and albumin were excluded to address bias and improve generalizability. This generated an 8-variable model (UNAFIED-8) with consistent performance. Conclusion: UNAFIED-10, developed using regional patient data, displayed consistent performance in a large national data set. UNAFIED-8 is more parsimonious and generalizable for using advanced analytics for AF detection. Future directions include validation on additional data sets.