Browsing by Subject "Generalizability"

Now showing 1 - 5 of 5
  • Thumbnail Image
    Item
    Generalizability and portability of natural language processing system to extract individual social risk factors
    (Elsevier, 2023) Magoc, Tanja; Allen, Katie S.; McDonnell, Cara; Russo, Jean-Paul; Cummins, Jonathan; Vest, Joshua R.; Harle, Christopher A.; Emergency Medicine, School of Medicine
    Objective: The objective of this study is to validate and report on portability and generalizability of a Natural Language Processing (NLP) method to extract individual social factors from clinical notes, which was originally developed at a different institution. Materials and methods: A rule-based deterministic state machine NLP model was developed to extract financial insecurity and housing instability using notes from one institution and was applied on all notes written during 6 months at another institution. 10% of positively-classified notes by NLP and the same number of negatively-classified notes were manually annotated. The NLP model was adjusted to accommodate notes at the new site. Accuracy, positive predictive value, sensitivity, and specificity were calculated. Results: More than 6 million notes were processed at the receiving site by the NLP model, which resulted in about 13,000 and 19,000 classified as positive for financial insecurity and housing instability, respectively. The NLP model showed excellent performance on the validation dataset with all measures over 0.87 for both social factors. Discussion: Our study illustrated the need to accommodate institution-specific note-writing templates as well as clinical terminology of emergent diseases when applying NLP model for social factors. A state machine is relatively simple to port effectively across institutions. Our study. showed superior performance to similar generalizability studies for extracting social factors. Conclusion: Rule-based NLP model to extract social factors from clinical notes showed strong portability and generalizability across organizationally and geographically distinct institutions. With only relatively simple modifications, we obtained promising performance from an NLP-based model.
  • Thumbnail Image
    Item
    Increasing participant diversity in AD research: Plans for digital screening, blood testing, and a community-engaged approach in the Alzheimer's Disease Neuroimaging Initiative 4
    (Wiley, 2023) Weiner, Michael W.; Veitch, Dallas P.; Miller, Melanie J.; Aisen, Paul S.; Albala, Bruce; Beckett, Laurel A.; Green, Robert C.; Harvey, Danielle; Jack, Clifford R., Jr.; Jagust, William; Landau, Susan M.; Morris, John C.; Nosheny, Rachel; Okonkwo, Ozioma C.; Perrin, Richard J.; Petersen, Ronald C.; Rivera-Mindt, Monica; Saykin, Andrew J.; Shaw, Leslie M.; Toga, Arthur W.; Tosun, Duygu; Trojanowski, John Q.; Alzheimer's Disease Neuroimaging Initiative; Radiology and Imaging Sciences, School of Medicine
    Introduction: The Alzheimer's Disease Neuroimaging Initiative (ADNI) aims to validate biomarkers for Alzheimer's disease (AD) clinical trials. To improve generalizability, ADNI4 aims to enroll 50-60% of its new participants from underrepresented populations (URPs) using new biofluid and digital technologies. ADNI4 has received funding from the National Institute on Aging beginning September 2022. Methods: ADNI4 will recruit URPs using community-engaged approaches. An online portal will screen 20,000 participants, 4000 of whom (50-60% URPs) will be tested for plasma biomarkers and APOE. From this, 500 new participants will undergo in-clinic assessment joining 500 ADNI3 rollover participants. Remaining participants (∼3500) will undergo longitudinal plasma and digital cognitive testing. ADNI4 will add MRI sequences and new PET tracers. Project 1 will optimize biomarkers in AD clinical trials. Results and discussion: ADNI4 will improve generalizability of results, use remote digital and blood screening, and continue providing longitudinal clinical, biomarker, and autopsy data to investigators.
  • Thumbnail Image
    Item
    Stress testing deep learning models for prostate cancer detection on biopsies and surgical specimens
    (Wiley, 2025) Flannery, Brennan T.; Sandler, Howard M.; Lal, Priti; Feldman, Michael D.; Santa-Rosario, Juan C.; Pathak, Tilak; Mirtti, Tuomas; Farre, Xavier; Correa, Rohann; Chafe, Susan; Shah, Amit; Efstathiou, Jason A.; Hoffman, Karen; Hallman, Mark A.; Straza, Michael; Jordan, Richard; Pugh, Stephanie L.; Feng, Felix; Madabhushi, Anant; Pathology and Laboratory Medicine, School of Medicine
    The presence, location, and extent of prostate cancer is assessed by pathologists using H&E-stained tissue slides. Machine learning approaches can accomplish these tasks for both biopsies and radical prostatectomies. Deep learning approaches using convolutional neural networks (CNNs) have been shown to identify cancer in pathologic slides, some securing regulatory approval for clinical use. However, differences in sample processing can subtly alter the morphology between sample types, making it unclear whether deep learning algorithms will consistently work on both types of slide images. Our goal was to investigate whether morphological differences between sample types affected the performance of biopsy-trained cancer detection CNN models when applied to radical prostatectomies and vice versa using multiple cohorts (N = 1,000). Radical prostatectomies (N = 100) and biopsies (N = 50) were acquired from The University of Pennsylvania to train (80%) and validate (20%) a DenseNet CNN for biopsies (MB), radical prostatectomies (MR), and a combined dataset (MB+R). On a tile level, MB and MR achieved F1 scores greater than 0.88 when applied to their own sample type but less than 0.65 when applied across sample types. On a whole-slide level, models achieved significantly better performance on their own sample type compared to the alternative model (p < 0.05) for all metrics. This was confirmed by external validation using digitized biopsy slide images from a clinical trial [NRG Radiation Therapy Oncology Group (RTOG)] (NRG/RTOG 0521, N = 750) via both qualitative and quantitative analyses (p < 0.05). A comprehensive review of model outputs revealed morphologically driven decision making that adversely affected model performance. MB appeared to be challenged with the analysis of open gland structures, whereas MR appeared to be challenged with closed gland structures, indicating potential morphological variation between the training sets. These findings suggest that differences in morphology and heterogeneity necessitate the need for more tailored, sample-specific (i.e. biopsy and surgical) machine learning models.
  • Thumbnail Image
    Item
    The Alzheimer's Disease Neuroimaging Initiative in the era of Alzheimer's disease treatment: A review of ADNI studies from 2021 to 2022
    (Wiley, 2024) Veitch, Dallas P.; Weiner, Michael W.; Miller, Melanie; Aisen, Paul S.; Ashford, Miriam A.; Beckett, Laurel A.; Green, Robert C.; Harvey, Danielle; Jack, Clifford R., Jr.; Jagust, William; Landau, Susan M.; Morris, John C.; Nho, Kwangsik T.; Nosheny, Rachel; Okonkwo, Ozioma; Perrin, Richard J.; Petersen, Ronald C.; Rivera Mindt, Monica; Saykin, Andrew; Shaw, Leslie M.; Toga, Arthur W.; Tosun, Duygu; Alzheimer’s Disease Neuroimaging Initiative; Radiology and Imaging Sciences, School of Medicine
    The Alzheimer's Disease Neuroimaging Initiative (ADNI) aims to improve Alzheimer's disease (AD) clinical trials. Since 2006, ADNI has shared clinical, neuroimaging, and cognitive data, and biofluid samples. We used conventional search methods to identify 1459 publications from 2021 to 2022 using ADNI data/samples and reviewed 291 impactful studies. This review details how ADNI studies improved disease progression understanding and clinical trial efficiency. Advances in subject selection, detection of treatment effects, harmonization, and modeling improved clinical trials and plasma biomarkers like phosphorylated tau showed promise for clinical use. Biomarkers of amyloid beta, tau, neurodegeneration, inflammation, and others were prognostic with individualized prediction algorithms available online. Studies supported the amyloid cascade, emphasized the importance of neuroinflammation, and detailed widespread heterogeneity in disease, linked to genetic and vascular risk, co-pathologies, sex, and resilience. Biological subtypes were consistently observed. Generalizability of ADNI results is limited by lack of cohort diversity, an issue ADNI-4 aims to address by enrolling a diverse cohort.
  • Thumbnail Image
    Item
    The Alzheimer's Disease Neuroimaging Initiative-4 (ADNI-4) Engagement Core: A culturally informed, community-engaged research (CI-CER) model to advance brain health equity
    (Wiley, 2024) Rivera Mindt, Mónica; Arentoft, Alyssa; Calcetas, Amanda T.; Guzman, Vanessa A.; Amaza, Hannatu; Ajayi, Adeyinka; Ashford, Miriam T.; Ayo, Omobolanle; Barnes, Lisa L.; Camuy, Alicia; Conti, Catherine; Diaz, Adam; Easter, Bashir; Gonzalez, David J.; Graham Dotson, Yolanda; Hoang, Isabella; Germano, Kaori Kubo; Maestre, Gladys E.; Magaña, Fabiola; Meyer, Oanh L.; Miller, Melanie J.; Nosheny, Rachel; Ta Park, Van M.; Parkins, Shaniya; Renier Thomas, Lisa; Strong, Joe; Talavera, Sandra; Verney, Steven P.; Weisensel, Trinity; Weiner, Michael W.; Okonkwo, Ozioma C.; Alzheimer's Disease Neuroimaging Initiative; Medicine, School of Medicine
    Introduction: The Alzheimer's Disease Neuroimaging Initiative-4 (ADNI-4) Engagement Core was launched to advance Alzheimer's disease (AD) and AD-related dementia (ADRD) health equity research in underrepresented populations (URPs). We describe our evidence-based, scalable culturally informed, community-engaged research (CI-CER) model and demonstrate its preliminary success in increasing URP enrollment. Methods: URPs include ethnoculturally minoritized, lower education (≤ 12 years), and rural populations. The CI-CER model includes: (1) culturally informed methodology (e.g., less restrictive inclusion/exclusion criteria, sociocultural measures, financial compensation, results disclosure, Spanish Language Capacity Workgroup) and (2) inclusive engagement methods (e.g., the Engagement Core team; Hub Sites; Community-Science Partnership Board). Results: As of April 2024, 60% of ADNI-4 new in-clinic enrollees were from ethnoculturally or educationally URPs. This exceeds ADNI-4's ≥ 50% URP representation goal for new enrollees but may not represent final enrollment. Discussion: Findings show a CI-CER model increases URP enrollment in AD/ADRD clinical research and has important implications for clinical trials to advance health equity. Highlights: The Alzheimer's Disease Neuroimaging Initiative-4 (ADNI-4) uses a culturally informed, community-engaged research (CI-CER) approach. The CI-CER approach is scalable and sustainable for broad, multisite implementation. ADNI-4 is currently exceeding its inclusion goals for underrepresented populations.