Browsing by Subject "Benchmarking"

Now showing 1 - 5 of 5
  • Thumbnail Image
    Item
    Artificial Intelligence Methods and Artificial Intelligence-Enabled Metrics for Surgical Education: A Multidisciplinary Consensus
    (Wolters Kluwer, 2022) Vedula, S. Swaroop; Ghazi, Ahmed; Collins, Justin W.; Pugh, Carla; Stefanidis, Dimitrios; Meireles, Ozanan; Hung, Andrew J.; Schwaitzberg, Steven; Levy, Jeffrey S.; Sachdeva, Ajit K.; Collaborative for Advanced Assessment of Robotic Surgical Skills; Surgery, School of Medicine
    Background: Artificial intelligence (AI) methods and AI-enabled metrics hold tremendous potential to advance surgical education. Our objective was to generate consensus guidance on specific needs for AI methods and AI-enabled metrics for surgical education. Study design: The study included a systematic literature search, a virtual conference, and a 3-round Delphi survey of 40 representative multidisciplinary stakeholders with domain expertise selected through purposeful sampling. The accelerated Delphi process was completed within 10 days. The survey covered overall utility, anticipated future (10-year time horizon), and applications for surgical training, assessment, and feedback. Consensus was agreement among 80% or more respondents. We coded survey questions into 11 themes and descriptively analyzed the responses. Results: The respondents included surgeons (40%), engineers (15%), affiliates of industry (27.5%), professional societies (7.5%), regulatory agencies (7.5%), and a lawyer (2.5%). The survey included 155 questions; consensus was achieved on 136 (87.7%). The panel listed 6 deliverables each for AI-enhanced learning curve analytics and surgical skill assessment. For feedback, the panel identified 10 priority deliverables spanning 2-year (n = 2), 5-year (n = 4), and 10-year (n = 4) timeframes. Within 2 years, the panel expects development of methods to recognize anatomy in images of the surgical field and to provide surgeons with performance feedback immediately after an operation. The panel also identified 5 essential that should be included in operative performance reports for surgeons. Conclusions: The Delphi panel consensus provides a specific, bold, and forward-looking roadmap for AI methods and AI-enabled metrics for surgical education.
  • Thumbnail Image
    Item
    Brain-age prediction: Systematic evaluation of site effects, and sample age range and size
    (Wiley, 2024) Yu, Yuetong; Cui, Hao-Qi; Haas, Shalaila S.; New, Faye; Sanford, Nicole; Yu, Kevin; Zhan, Denghuang; Yang, Guoyuan; Gao, Jia-Hong; Wei, Dongtao; Qiu, Jiang; Banaj, Nerisa; Boomsma, Dorret I.; Breier, Alan; Brodaty, Henry; Buckner, Randy L.; Buitelaar, Jan K.; Cannon, Dara M.; Caseras, Xavier; Clark, Vincent P.; Conrod, Patricia J.; Crivello, Fabrice; Crone, Eveline A.; Dannlowski, Udo; Davey, Christopher G.; de Haan, Lieuwe; de Zubicaray, Greig I.; Di Giorgio, Annabella; Fisch, Lukas; Fisher, Simon E.; Franke, Barbara; Glahn, David C.; Grotegerd, Dominik; Gruber, Oliver; Gur, Raquel E.; Gur, Ruben C.; Hahn, Tim; Harrison, Ben J.; Hatton, Sean; Hickie, Ian B.; Hulshoff Pol, Hilleke E.; Jamieson, Alec J.; Jernigan, Terry L.; Jiang, Jiyang; Kalnin, Andrew J.; Kang, Sim; Kochan, Nicole A.; Kraus, Anna; Lagopoulos, Jim; Lazaro, Luisa; McDonald, Brenna C.; McDonald, Colm; McMahon, Katie L.; Mwangi, Benson; Piras, Fabrizio; Rodriguez-Cruces, Raul; Royer, Jessica; Sachdev, Perminder S.; Satterthwaite, Theodore D.; Saykin, Andrew J.; Schumann, Gunter; Sevaggi, Pierluigi; Smoller, Jordan W.; Soares, Jair C.; Spalletta, Gianfranco; Tamnes, Christian K.; Trollor, Julian N.; Van't Ent, Dennis; Vecchio, Daniela; Walter, Henrik; Wang, Yang; Weber, Bernd; Wen, Wei; Wierenga, Lara M.; Williams, Steven C. R.; Wu, Mon-Ju; Zunta-Soares, Giovana B.; Bernhardt, Boris; Thompson, Paul; Frangou, Sophia; Ge, Ruiyang; ENIGMA-Lifespan Working Group; Psychiatry, School of Medicine
    Structural neuroimaging data have been used to compute an estimate of the biological age of the brain (brain-age) which has been associated with other biologically and behaviorally meaningful measures of brain development and aging. The ongoing research interest in brain-age has highlighted the need for robust and publicly available brain-age models pre-trained on data from large samples of healthy individuals. To address this need we have previously released a developmental brain-age model. Here we expand this work to develop, empirically validate, and disseminate a pre-trained brain-age model to cover most of the human lifespan. To achieve this, we selected the best-performing model after systematically examining the impact of seven site harmonization strategies, age range, and sample size on brain-age prediction in a discovery sample of brain morphometric measures from 35,683 healthy individuals (age range: 5-90 years; 53.59% female). The pre-trained models were tested for cross-dataset generalizability in an independent sample comprising 2101 healthy individuals (age range: 8-80 years; 55.35% female) and for longitudinal consistency in a further sample comprising 377 healthy individuals (age range: 9-25 years; 49.87% female). This empirical examination yielded the following findings: (1) the accuracy of age prediction from morphometry data was higher when no site harmonization was applied; (2) dividing the discovery sample into two age-bins (5-40 and 40-90 years) provided a better balance between model accuracy and explained age variance than other alternatives; (3) model accuracy for brain-age prediction plateaued at a sample size exceeding 1600 participants. These findings have been incorporated into CentileBrain (https://centilebrain.org/#/brainAGE2), an open-science, web-based platform for individualized neuroimaging metrics.
  • Thumbnail Image
    Item
    Development of the Illinois Surgical Quality Improvement Collaborative (ISQIC): Implementing 21 Components to Catalyze Statewide Improvement in Surgical Care
    (Wolters Kluwer, 2023) Bilimoria, Karl Y.; McGee, Michael F.; Williams, Mark V.; Johnson, Julie K.; Halverson, Amy L.; O'Leary, Kevin J.; Farrell, Paula; Thomas, Juliana; Love, Remi; Kreutzer, Lindsey; Dahlke, Allison R.; D'Orazio, Brianna; Reinhart, Steven; Dienes, Katelyn; Schumacher, Mark; Shan, Ying; Quinn, Christopher; Prachand, Vivek N.; Sullivan, Susan; Cradock, Kimberly A.; Boyd, Kelsi; Hopkinson, William; Fairman, Colleen; Odell, David; Stulberg, Jonah J.; Barnard, Cindy; Holl, Jane; Merkow, Ryan P.; Yang, Anthony D.; Surgery, School of Medicine
    Introduction: In 2014, 56 Illinois hospitals came together to form a unique learning collaborative, the Illinois Surgical Quality Improvement Collaborative (ISQIC). Our objectives are to provide an overview of the first three years of ISQIC focused on (1) how the collaborative was formed and funded, (2) the 21 strategies implemented to support quality improvement (QI), (3) collaborative sustainment, and (4) how the collaborative acts as a platform for innovative QI research. Methods: ISQIC includes 21 components to facilitate QI that target the hospital, the surgical QI team, and the peri-operative microsystem. The components were developed from available evidence, a detailed needs assessment of the hospitals, reviewing experiences from prior surgical and non-surgical QI Collaboratives, and interviews with QI experts. The components comprise 5 domains: guided implementation (e.g., mentors, coaches, statewide QI projects), education (e.g., process improvement (PI) curriculum), hospital- and surgeon-level comparative performance reports (e.g., process, outcomes, costs), networking (e.g., forums to share QI experiences and best practices), and funding (e.g., for the overall program, pilot grants, and bonus payments for improvement). Results: Through implementation of the 21 novel ISQIC components, hospitals were equipped to use their data to successfully implement QI initiatives and improve care. Formal (QI/PI) training, mentoring, and coaching were undertaken by the hospitals as they worked to implement solutions. Hospitals received funding for the program and were able to work together on statewide quality initiatives. Lessons learned at one hospital were shared with all participating hospitals through conferences, webinars, and toolkits to facilitate learning from each other with a common goal of making care better and safer for the surgical patient in Illinois. Over the first three years, surgical outcomes improved in Illinois. Discussion: The first three years of ISQIC improved care for surgical patients across Illinois and allowed hospitals to see the value of participating in a surgical QI learning collaborative without having to make the initial financial investment themselves. Given the strong support and buy-in from the hospitals, ISQIC has continued beyond the initial three years and continues to support QI across Illinois hospitals.
  • Thumbnail Image
    Item
    Genome-wide circadian rhythm detection methods: systematic evaluations and practical guidelines
    (Oxford University Press, 2021-05-20) Mei, Wenwen; Jiang, Zhiwen; Chen, Yang; Chen, Li; Sancar, Aziz; Jiang, Yuchao; Medicine, School of Medicine
    Circadian rhythms are oscillations of behavior, physiology and metabolism in many organisms. Recent advancements in omics technology make it possible for genome-wide profiling of circadian rhythms. Here, we conducted a comprehensive analysis of seven existing algorithms commonly used for circadian rhythm detection. Using gold-standard circadian and non-circadian genes, we systematically evaluated the accuracy and reproducibility of the algorithms on empirical datasets generated from various omics platforms under different experimental designs. We also carried out extensive simulation studies to test each algorithm’s robustness to key variables, including sampling patterns, replicates, waveforms, signal-to-noise ratios, uneven samplings and missing values. Furthermore, we examined the distributions of the nominal equation M1-values under the null and raised issues with multiple testing corrections using traditional approaches. With our assessment, we provide method selection guidelines for circadian rhythm detection, which are applicable to different types of high-throughput omics data.
  • Thumbnail Image
    Item
    Image harmonization: A review of statistical and deep learning methods for removing batch effects and evaluation metrics for effective harmonization
    (Elsevier, 2023) Hu, Fengling; Chen, Andrew A.; Horng, Hannah; Bashyam, Vishnu; Davatzikos, Christos; Alexander-Bloch, Aaron; Li, Mingyao; Shou, Haochang; Satterthwaite, Theodore D.; Yu, Meichen; Shinohara, Russell T.; Radiology and Imaging Sciences, School of Medicine
    Magnetic resonance imaging and computed tomography from multiple batches (e.g. sites, scanners, datasets, etc.) are increasingly used alongside complex downstream analyses to obtain new insights into the human brain. However, significant confounding due to batch-related technical variation, called batch effects, is present in this data; direct application of downstream analyses to the data may lead to biased results. Image harmonization methods seek to remove these batch effects and enable increased generalizability and reproducibility of downstream results. In this review, we describe and categorize current approaches in statistical and deep learning harmonization methods. We also describe current evaluation metrics used to assess harmonization methods and provide a standardized framework to evaluate newly-proposed methods for effective harmonization and preservation of biological information. Finally, we provide recommendations to end-users to advocate for more effective use of current methods and to methodologists to direct future efforts and accelerate development of the field.