Browsing by Author "Cummins, Jonathan"

Now showing 1 - 4 of 4
  • Thumbnail Image
    Item
    Generalizability and portability of natural language processing system to extract individual social risk factors
    (Elsevier, 2023) Magoc, Tanja; Allen, Katie S.; McDonnell, Cara; Russo, Jean-Paul; Cummins, Jonathan; Vest, Joshua R.; Harle, Christopher A.; Emergency Medicine, School of Medicine
    Objective: The objective of this study is to validate and report on portability and generalizability of a Natural Language Processing (NLP) method to extract individual social factors from clinical notes, which was originally developed at a different institution. Materials and methods: A rule-based deterministic state machine NLP model was developed to extract financial insecurity and housing instability using notes from one institution and was applied on all notes written during 6 months at another institution. 10% of positively-classified notes by NLP and the same number of negatively-classified notes were manually annotated. The NLP model was adjusted to accommodate notes at the new site. Accuracy, positive predictive value, sensitivity, and specificity were calculated. Results: More than 6 million notes were processed at the receiving site by the NLP model, which resulted in about 13,000 and 19,000 classified as positive for financial insecurity and housing instability, respectively. The NLP model showed excellent performance on the validation dataset with all measures over 0.87 for both social factors. Discussion: Our study illustrated the need to accommodate institution-specific note-writing templates as well as clinical terminology of emergent diseases when applying NLP model for social factors. A state machine is relatively simple to port effectively across institutions. Our study. showed superior performance to similar generalizability studies for extracting social factors. Conclusion: Rule-based NLP model to extract social factors from clinical notes showed strong portability and generalizability across organizationally and geographically distinct institutions. With only relatively simple modifications, we obtained promising performance from an NLP-based model.
  • Thumbnail Image
    Item
    Natural language processing-driven state machines to extract social factors from unstructured clinical documentation
    (Oxford University Press, 2023-04-18) Allen, Katie S.; Hood, Dan R.; Cummins, Jonathan; Kasturi, Suranga; Mendonca, Eneida A.; Vest, Joshua R.; Health Policy and Management, School of Public Health
    Objective: This study sought to create natural language processing algorithms to extract the presence of social factors from clinical text in 3 areas: (1) housing, (2) financial, and (3) unemployment. For generalizability, finalized models were validated on data from a separate health system for generalizability. Materials and methods: Notes from 2 healthcare systems, representing a variety of note types, were utilized. To train models, the study utilized n-grams to identify keywords and implemented natural language processing (NLP) state machines across all note types. Manual review was conducted to determine performance. Sampling was based on a set percentage of notes, based on the prevalence of social need. Models were optimized over multiple training and evaluation cycles. Performance metrics were calculated using positive predictive value (PPV), negative predictive value, sensitivity, and specificity. Results: PPV for housing rose from 0.71 to 0.95 over 3 training runs. PPV for financial rose from 0.83 to 0.89 over 2 training iterations, while PPV for unemployment rose from 0.78 to 0.88 over 3 iterations. The test data resulted in PPVs of 0.94, 0.97, and 0.95 for housing, financial, and unemployment, respectively. Final specificity scores were 0.95, 0.97, and 0.95 for housing, financial, and unemployment, respectively. Discussion: We developed 3 rule-based NLP algorithms, trained across health systems. While this is a less sophisticated approach, the algorithms demonstrated a high degree of generalizability, maintaining >0.85 across all predictive performance metrics. Conclusion: The rule-based NLP algorithms demonstrated consistent performance in identifying 3 social factors within clinical text. These methods may be a part of a strategy to measure social factors within an institution.
  • Thumbnail Image
    Item
    Predictive Modeling of Hypoglycemia for Clinical Decision Support in Evaluating Outpatients with Diabetes Mellitus
    (Taylor & Francis, 2019) Li, Xiaochun; Yu, Shengsheng; Zhang, Zuoyi; Radican, Larry; Cummins, Jonathan; Engel, Samuel S.; Iglay, Kristy; Duke, Jon; Baker, Jarod; Brodovicz, Kimberly G.; Naik, Ramachandra G.; Leventhal, Jeremy; Chatterjee, Arnaub K.; Rajpathak, Swapnil; Weiner, Michael; Biostatistics, School of Public Health
    Objective: Hypoglycemia occurs in 20–60% of patients with diabetes mellitus. Identifying at-risk patients can facilitate interventions to lower risk. We sought to develop a hypoglycemia prediction model. Methods: In this retrospective cohort study, urban adults prescribed a diabetes drug between 2004 and 2013 were identified. Demographic and clinical data were extracted from an electronic medical record (EMR). Laboratory tests, diagnostic codes and natural language processing (NLP) identified hypoglycemia. We compared multiple logistic regression, classification and regression trees (CART), and random forest. Models were evaluated on an independent test set or through cross-validation. Results: The 38,780 patients had mean age 57 years; 56% were female, 40% African-American and 39% uninsured. Hypoglycemia occurred in 8128 (539 identified only by NLP). In logistic regression, factors positively associated with hypoglycemia included infection, non-long-acting insulin, dementia and recent hypoglycemia. Negatively associated factors included long-acting insulin plus sulfonylurea, and age 75 or older. The models’ area under curve was similar (logistic regression, 89%; CART, 88%; random forest, 90%, with ten-fold cross-validation). Conclusions: NLP improved identification of hypoglycemia. Non-long-acting insulin was an important risk factor. Decreased risk with age may reflect treatment or diminished awareness of hypoglycemia. More complex models did not improve prediction.
  • Thumbnail Image
    Item
    A randomized study on the usefulness of an electronic outpatient hypoglycemia risk calculator for clinicians of patients with diabetes in a safety-net institution
    (Taylor & Francis, 2020) Weiner, Michael; Cummins, Jonathan; Raji, Annaswamy; Ofner, Susan; Iglay, Kristy; Teal, Evgenia; Li, Xiaochun; Engel, Samuel S.; Knapp, Kristina; Rajpathak, Swapnil; Baker, Jarod; Chatterjee, Arnaub K.; Radican, Larry; Medicine, School of Medicine
    Objective: Hypoglycemia (HG) occurs in up to 60% of patients with diabetes mellitus (DM) each year. We assessed a HG alert tool in an electronic health record system, and determined its effect on clinical practice and outcomes. Methods: The tool applied a statistical model, yielding patient-specific information about HG risk. We randomized outpatient primary-care providers (PCPs) to see or not see the alerts. Patients were assigned to study group according to the first PCP seen during four months. We assessed prescriptions, testing, and HG. Variables were compared by multinomial, logistic, or linear model. ClinicalTrials.gov ID: NCT04177147 (registered on 22 November 2019). Results: Patients (N = 3350) visited 123 intervention PCPs; 3395 patients visited 220 control PCPs. Intervention PCPs were shown 18,645 alerts (mean of 152 per PCP). Patients’ mean age was 55 years, with 61% female, 49% black, and 49% Medicaid recipients. Mean baseline A1c and body mass index were similar between groups. During follow-up, the number of A1c and glucose tests, and number of new, refilled, changed, or discontinued insulin prescriptions, were highest for patients with highest risk. Per 100 patients on average, the intervention group had fewer sulfonylurea refills (6 vs. 8; p < .05) and outpatient encounters (470 vs. 502; p < .05), though the change in encounters was not significant. Frequency of HG events was unchanged. Conclusions: Informing PCPs about risk of HG led to fewer sulfonylurea refills and visits. Longer-term studies are needed to assess potential for long-term benefits.