Browsing by Author "Magoc, Tanja"
Now showing 1 - 9 of 9
Results Per Page
Sort Options
Item A large language model for electronic health records(Springer Nature, 2022-12-26) Yang, Xi; Chen, Aokun; PourNejatian, Nima; Shin, Hoo Chang; Smith, Kaleb E.; Parisien, Christopher; Compas, Colin; Martin, Cheryl; Costa, Anthony B.; Flores, Mona G.; Zhang, Ying; Magoc, Tanja; Harle, Christopher A.; Lipori, Gloria; Mitchell, Duane A.; Hogan, William R.; Shenkman, Elizabeth A.; Bian, Jiang; Wu, Yonghui; Health Policy and Management, Richard M. Fairbanks School of Public HealthThere is an increasing interest in developing artificial intelligence (AI) systems to process and interpret electronic health records (EHRs). Natural language processing (NLP) powered by pretrained language models is the key technology for medical AI systems utilizing clinical narratives. However, there are few clinical language models, the largest of which trained in the clinical domain is comparatively small at 110 million parameters (compared with billions of parameters in the general domain). It is not clear how large clinical language models with billions of parameters can help medical AI systems utilize unstructured EHRs. In this study, we develop from scratch a large clinical language model—GatorTron—using >90 billion words of text (including >82 billion words of de-identified clinical text) and systematically evaluate it on five clinical NLP tasks including clinical concept extraction, medical relation extraction, semantic textual similarity, natural language inference (NLI), and medical question answering (MQA). We examine how (1) scaling up the number of parameters and (2) scaling up the size of the training data could benefit these NLP tasks. GatorTron models scale up the clinical language model from 110 million to 8.9 billion parameters and improve five clinical NLP tasks (e.g., 9.6% and 9.5% improvement in accuracy for NLI and MQA), which can be applied to medical AI systems to improve healthcare delivery. The GatorTron models are publicly available at: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/clara/models/gatortron_ogItem Development and validation of computable social phenotypes for health-related social needs(Oxford University Press, 2025-01-07) Gregory, Megan E.; Kasthurirathne, Suranga N.; Magoc, Tanja; McNamee, Cassidy; Harle, Christopher A.; Vest, Joshua R.; Health Policy and Management, Richard M. Fairbanks School of Public HealthObjective: Measurement of health-related social needs (HRSNs) is complex. We sought to develop and validate computable phenotypes (CPs) using structured electronic health record (EHR) data for food insecurity, housing instability, financial insecurity, transportation barriers, and a composite-type measure of these, using human-defined rule-based and machine learning (ML) classifier approaches. Materials and methods: We collected HRSN surveys as the reference standard and obtained EHR data from 1550 patients in 3 health systems from 2 states. We followed a Delphi-like approach to develop the human-defined rule-based CP. For the ML classifier approach, we trained supervised ML (XGBoost) models using 78 features. Using surveys as the reference standard, we calculated sensitivity, specificity, positive predictive values, and area under the curve (AUC). We compared AUCs using the Delong test and other performance measures using McNemar's test, and checked for differential performance. Results: Most patients (63%) reported at least one HRSN on the reference standard survey. Human-defined rule-based CPs exhibited poor performance (AUCs=.52 to .68). ML classifier CPs performed significantly better, but still poor-to-fair (AUCs = .68 to .75). Significant differences for race/ethnicity were found for ML classifier CPs (higher AUCs for White non-Hispanic patients). Important features included number of encounters and Medicaid insurance. Discussion: Using a supervised ML classifier approach, HRSN CPs approached thresholds of fair performance, but exhibited differential performance by race/ethnicity. Conclusion: CPs may help to identify patients who may benefit from additional social needs screening. Future work should explore the use of area-level features via geospatial data and natural language processing to improve model performance.Item Enhancing an enterprise data warehouse for research with data extracted using natural language processing(Cambridge University Press, 2023-06-13) Magoc, Tanja; Everson, Russell; Harle, Christopher A.; Health Policy and Management, School of Public HealthObjective: This study aims to develop a generalizable architecture for enhancing an enterprise data warehouse for research (EDW4R) with results from a natural language processing (NLP) model, which allows discrete data derived from clinical notes to be made broadly available for research use without need for NLP expertise. The study also quantifies the additional value that information extracted from clinical narratives brings to EDW4R. Materials and methods: Clinical notes written during one month at an academic health center were used to evaluate the performance of an existing NLP model and to quantify its value added to the structured data. Manual review was utilized for performance analysis. The architecture for enhancing the EDW4R is described in detail to enable reproducibility. Results: Two weeks were needed to enhance EDW4R with data from 250 million clinical notes. NLP generated 16 and 39% increase in data availability for two variables. Discussion: Our architecture is highly generalizable to a new NLP model. The positive predictive value obtained by an independent team showed only slightly lower NLP performance than the values reported by the NLP developers. The NLP showed significant value added to data already available in structured format. Conclusion: Given the value added by data extracted using NLP, it is important to enhance EDW4R with these data to enable research teams without NLP expertise to benefit from value added by NLP models.Item Evaluation of federated learning variations for COVID-19 diagnosis using chest radiographs from 42 US and European hospitals(Oxford University Press, 2022) Peng, Le; Luo, Gaoxiang; Walker, Andrew; Zaiman, Zachary; Jones, Emma K.; Gupta, Hemant; Kersten, Kristopher; Burns, John L.; Harle, Christopher A.; Magoc, Tanja; Shickel, Benjamin; Steenburg, Scott D.; Loftus, Tyler; Melton, Genevieve B.; Wawira Gichoya, Judy; Sun, Ju; Tignanelli, Christopher J.; Radiology and Imaging Sciences, School of MedicineObjective: Federated learning (FL) allows multiple distributed data holders to collaboratively learn a shared model without data sharing. However, individual health system data are heterogeneous. "Personalized" FL variations have been developed to counter data heterogeneity, but few have been evaluated using real-world healthcare data. The purpose of this study is to investigate the performance of a single-site versus a 3-client federated model using a previously described Coronavirus Disease 19 (COVID-19) diagnostic model. Additionally, to investigate the effect of system heterogeneity, we evaluate the performance of 4 FL variations. Materials and methods: We leverage a FL healthcare collaborative including data from 5 international healthcare systems (US and Europe) encompassing 42 hospitals. We implemented a COVID-19 computer vision diagnosis system using the Federated Averaging (FedAvg) algorithm implemented on Clara Train SDK 4.0. To study the effect of data heterogeneity, training data was pooled from 3 systems locally and federation was simulated. We compared a centralized/pooled model, versus FedAvg, and 3 personalized FL variations (FedProx, FedBN, and FedAMP). Results: We observed comparable model performance with respect to internal validation (local model: AUROC 0.94 vs FedAvg: 0.95, P = .5) and improved model generalizability with the FedAvg model (P < .05). When investigating the effects of model heterogeneity, we observed poor performance with FedAvg on internal validation as compared to personalized FL algorithms. FedAvg did have improved generalizability compared to personalized FL algorithms. On average, FedBN had the best rank performance on internal and external validation. Conclusion: FedAvg can significantly improve the generalization of the model compared to other personalization FL algorithms; however, at the cost of poor internal validity. Personalized FL may offer an opportunity to develop both internal and externally validated algorithms.Item Evolving availability and standardization of patient attributes for matching(Oxford University Press, 2023-10-12) Deng, Yu; Gleason, Lacey P.; Culbertson, Adam; Chen, Xiaotian; Bernstam, Elmer V.; Cullen, Theresa; Gouripeddi, Ramkiran; Harle, Christopher; Hesse, David F.; Kean, Jacob; Lee, John; Magoc, Tanja; Meeker, Daniella; Ong, Toan; Pathak, Jyotishman; Rosenman, Marc; Rusie, Laura K.; Shah, Akash J.; Shi, Lizheng; Thomas, Aaron; Trick, William E.; Grannis, Shaun; Kho, Abel; Health Policy and Management, Richard M. Fairbanks School of Public HealthVariation in availability, format, and standardization of patient attributes across health care organizations impacts patient-matching performance. We report on the changing nature of patient-matching features available from 2010-2020 across diverse care settings. We asked 38 health care provider organizations about their current patient attribute data-collection practices. All sites collected name, date of birth (DOB), address, and phone number. Name, DOB, current address, social security number (SSN), sex, and phone number were most commonly used for cross-provider patient matching. Electronic health record queries for a subset of 20 participating sites revealed that DOB, first name, last name, city, and postal codes were highly available (>90%) across health care organizations and time. SSN declined slightly in the last years of the study period. Birth sex, gender identity, language, country full name, country abbreviation, health insurance number, ethnicity, cell phone number, email address, and weight increased over 50% from 2010 to 2020. Understanding the wide variation in available patient attributes across care settings in the United States can guide selection and standardization efforts for improved patient matching in the United States.Item Generalizability and portability of natural language processing system to extract individual social risk factors(Elsevier, 2023) Magoc, Tanja; Allen, Katie S.; McDonnell, Cara; Russo, Jean-Paul; Cummins, Jonathan; Vest, Joshua R.; Harle, Christopher A.; Emergency Medicine, School of MedicineObjective: The objective of this study is to validate and report on portability and generalizability of a Natural Language Processing (NLP) method to extract individual social factors from clinical notes, which was originally developed at a different institution. Materials and methods: A rule-based deterministic state machine NLP model was developed to extract financial insecurity and housing instability using notes from one institution and was applied on all notes written during 6 months at another institution. 10% of positively-classified notes by NLP and the same number of negatively-classified notes were manually annotated. The NLP model was adjusted to accommodate notes at the new site. Accuracy, positive predictive value, sensitivity, and specificity were calculated. Results: More than 6 million notes were processed at the receiving site by the NLP model, which resulted in about 13,000 and 19,000 classified as positive for financial insecurity and housing instability, respectively. The NLP model showed excellent performance on the validation dataset with all measures over 0.87 for both social factors. Discussion: Our study illustrated the need to accommodate institution-specific note-writing templates as well as clinical terminology of emergent diseases when applying NLP model for social factors. A state machine is relatively simple to port effectively across institutions. Our study. showed superior performance to similar generalizability studies for extracting social factors. Conclusion: Rule-based NLP model to extract social factors from clinical notes showed strong portability and generalizability across organizationally and geographically distinct institutions. With only relatively simple modifications, we obtained promising performance from an NLP-based model.Item Primary care physicians’ electronic health record proficiency and efficiency behaviors and time interacting with electronic health records: a quantile regression analysis(Oxford University Press, 2022) Nguyen, Oliver T.; Turner, Kea; Apathy, Nate C.; Magoc, Tanja; Hanna, Karim; Merlo, Lisa J.; Harle, Christopher A.; Thompson, Lindsay A.; Berner, Eta S.; Feldman, Sue S.; Health Policy and Management, Richard M. Fairbanks School of Public HealthObjective: This study aimed to understand the association between primary care physician (PCP) proficiency with the electronic health record (EHR) system and time spent interacting with the EHR. Materials and methods: We examined the use of EHR proficiency tools among PCPs at one large academic health system using EHR-derived measures of clinician EHR proficiency and efficiency. Our main predictors were the use of EHR proficiency tools and our outcomes focused on 4 measures assessing time spent in the EHR: (1) total time spent interacting with the EHR, (2) time spent outside scheduled clinical hours, (3) time spent documenting, and (4) time spent on inbox management. We conducted multivariable quantile regression models with fixed effects for physician-level factors and time in order to identify factors that were independently associated with time spent in the EHR. Results: Across 441 primary care physicians, we found mixed associations between certain EHR proficiency behaviors and time spent in the EHR. Across EHR activities studied, QuickActions, SmartPhrases, and documentation length were positively associated with increased time spent in the EHR. Models also showed a greater amount of help from team members in note writing was associated with less time spent in the EHR and documenting. Discussion: Examining the prevalence of EHR proficiency behaviors may suggest targeted areas for initial and ongoing EHR training. Although documentation behaviors are key areas for training, team-based models for documentation and inbox management require further study. Conclusions: A nuanced association exists between physician EHR proficiency and time spent in the EHR.Item The OneFlorida Data Trust: a centralized, translational research data infrastructure of statewide scope(Oxford University Press, 2022) Hogan, William R.; Shenkman, Elizabeth A.; Robinson, Temple; Carasquillo, Olveen; Robinson, Patricia S.; Essner, Rebecca Z.; Bian, Jiang; Lipori, Gigi; Harle, Christopher; Magoc, Tanja; Manini, Lizabeth; Mendoza, Tona; White, Sonya; Loiacono, Alex; Hall, Jackie; Nelson, Dave; Biostatistics and Health Data Science, Richard M. Fairbanks School of Public HealthThe OneFlorida Data Trust is a centralized research patient data repository created and managed by the OneFlorida Clinical Research Consortium (“OneFlorida”). It comprises structured electronic health record (EHR), administrative claims, tumor registry, death, and other data on 17.2 million individuals who received healthcare in Florida between January 2012 and the present. Ten healthcare systems in Miami, Orlando, Tampa, Jacksonville, Tallahassee, Gainesville, and rural areas of Florida contribute EHR data, covering the major metropolitan regions in Florida. Deduplication of patients is accomplished via privacy-preserving entity resolution (precision 0.97–0.99, recall 0.75), thereby linking patients’ EHR, claims, and death data. Another unique feature is the establishment of mother-baby relationships via Florida vital statistics data. Research usage has been significant, including major studies launched in the National Patient-Centered Clinical Research Network (“PCORnet”), where OneFlorida is 1 of 9 clinical research networks. The Data Trust’s robust, centralized, statewide data are a valuable and relatively unique research resource.Item Understanding Health Care Administrators' Data and Information Needs for Decision Making during the COVID-19 Pandemic: A Qualitative Study at an Academic Health System(Sage, 2022-03-29) Guerrier, Christina; McDonnell, Cara; Magoc, Tanja; Fishe, Jennifer N.; Harle, Christopher A.; Health Policy and Management, Richard M. Fairbanks School of Public HealthObjective: The COVID-19 pandemic created an unprecedented strain on the health care system, and administrators had to make many critical decisions to respond appropriately. This study sought to understand how health care administrators used data and information for decision making during the first 6 mo of the COVID-19 pandemic. Materials and Methods: We conducted semistructured interviews with administrators across University of Florida (UF) Health. We performed an inductive thematic analysis of the transcripts. Results: Four themes emerged from the interviews: 1) common types of health systems or hospital operations data; 2) public health and other external data sources; 3) data interaction, integration, and exchange; and 4) novelty and evolution in data, information, or tools used over time. Participants illustrated the organizational, public health, and regional information they considered essential (e.g., hospital census, community positivity rate, etc.). Participants named specific challenges they faced due to data quality and timeliness. Participants elaborated on the necessity of data integration, validation, and coordination across different boundaries (e.g., different hospital systems in the same metro areas, public health agencies at the local, state, and federal level, etc.). Participants indicated that even within the first 6 mo of the COVID-19 pandemic, the data and tools used for making critical decisions changed. Discussion: While existing medical informatics infrastructure can facilitate decision making in pandemic response, data may not always be readily available in a usable format. Interoperable infrastructure and data standardization across multiple health systems would help provide more reliable and timely information for decision making. Conclusion: Our findings contribute to future discussions of improving data infrastructure and developing harmonized data standards needed to facilitate critical decisions at multiple health care system levels. Highlights: The study revealed common health systems or hospital operations data and information used in decision making during the first 6 mo of the COVID-19 pandemic. Participants described commonly used internal data sources, such as resource and financial reports and dashboards, and external data sources, such as federal, state, and local public health data. Participants described challenges including poor timeliness and limited local relevance of external data as well as poor integration of data sources within and across organizational boundaries. Results suggest the need for continued integration and standardization of health data to support health care administrative decision making during pandemics or other emergencies.