Browsing by Author "Cardoso, M. Jorge"

Now showing 1 - 3 of 3
  • Thumbnail Image
    Item
    Metrics reloaded: recommendations for image analysis validation
    (Springer Nature, 2024) Maier-Hein, Lena; Reinke, Annika; Godau, Patrick; Tizabi, Minu D.; Buettner, Florian; Christodoulou, Evangelia; Glocker, Ben; Isensee, Fabian; Kleesiek, Jens; Kozubek, Michal; Reyes, Mauricio; Riegler, Michael A.; Wiesenfarth, Manuel; Kavur, A. Emre; Sudre, Carole H.; Baumgartner, Michael; Eisenmann, Matthias; Heckmann-Nötzel, Doreen; Rädsch, Tim; Acion, Laura; Antonelli, Michela; Arbel, Tal; Bakas, Spyridon; Benis, Arriel; Blaschko, Matthew B.; Cardoso, M. Jorge; Cheplygina, Veronika; Cimini, Beth A.; Collins, Gary S.; Farahani, Keyvan; Ferrer, Luciana; Galdran, Adrian; van Ginneken, Bram; Haase, Robert; Hashimoto, Daniel A.; Hoffman, Michael M.; Huisman, Merel; Jannin, Pierre; Kahn, Charles E.; Kainmueller, Dagmar; Kainz, Bernhard; Karargyris, Alexandros; Karthikesalingam, Alan; Kofler, Florian; Kopp-Schneider, Annette; Kreshuk, Anna; Kurc, Tahsin; Landman, Bennett A.; Litjens, Geert; Madani, Amin; Maier-Hein, Klaus; Martel, Anne L.; Mattson, Peter; Meijering, Erik; Menze, Bjoern; Moons, Karel G. M.; Müller, Henning; Nichyporuk, Brennan; Nickel, Felix; Petersen, Jens; Rajpoot, Nasir; Rieke, Nicola; Saez-Rodriguez, Julio; Sánchez, Clara I.; Shetty, Shravya; van Smeden, Maarten; Summers, Ronald M.; Taha, Abdel A.; Tiulpin, Aleksei; Tsaftaris, Sotirios A.; Van Calster, Ben; Varoquaux, Gaël; Jäger, Paul F.; Pathology and Laboratory Medicine, School of Medicine
    Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.
  • Thumbnail Image
    Item
    Presymptomatic atrophy in autosomal dominant Alzheimer's disease: A serial magnetic resonance imaging study
    (Elsevier, 2018-01) Kinnunen, Kirsi M.; Cash, David M.; Poole, Teresa; Frost, Chris; Benzinger, Tammie L. S.; Ahsan, R. Laila; Leung, Kelvin K.; Cardoso, M. Jorge; Modat, Marc; Malone, Ian B.; Morris, John C.; Bateman, Randall J.; Marcus, Daniel S.; Goate, Alison; Salloway, Stephen P.; Correia, Stephen; Sperling, Reisa A.; Chhatwal, Jasmeer P.; Mayeux, Richard P.; Brickman, Adam M.; Martins, Ralph N.; Farlow, Martin R.; Ghetti, Bernardino; Saykin, Andrew J.; Jack, Clifford R.; Schofield, Peter R.; McDade, Eric; Weiner, Michael W.; Ringman, John M.; Thompson, Paul M.; Masters, Colin L.; Rowe, Christopher C.; Rossor, Martin N.; Ourselin, Sebastien; Fox, Nick C.; Neurology, School of Medicine
    INTRODUCTION: Identifying at what point atrophy rates first change in Alzheimer's disease is important for informing design of presymptomatic trials. METHODS: Serial T1-weighted magnetic resonance imaging scans of 94 participants (28 noncarriers, 66 carriers) from the Dominantly Inherited Alzheimer Network were used to measure brain, ventricular, and hippocampal atrophy rates. For each structure, nonlinear mixed-effects models estimated the change-points when atrophy rates deviate from normal and the rates of change before and after this point. RESULTS: Atrophy increased after the change-point, which occurred 1-1.5 years (assuming a single step change in atrophy rate) or 3-8 years (assuming gradual acceleration of atrophy) before expected symptom onset. At expected symptom onset, estimated atrophy rates were at least 3.6 times than those before the change-point. DISCUSSION: Atrophy rates are pathologically increased up to seven years before "expected onset". During this period, atrophy rates may be useful for inclusion and tracking of disease progression.
  • Thumbnail Image
    Item
    Understanding metric-related pitfalls in image analysis validation
    (ArXiv, 2023-09-25) Reinke, Annika; Tizabi, Minu D.; Baumgartner, Michael; Eisenmann, Matthias; Heckmann-Nötzel, Doreen; Kavur, A. Emre; Rädsch, Tim; Sudre, Carole H.; Acion, Laura; Antonelli, Michela; Arbel, Tal; Bakas, Spyridon; Benis, Arriel; Blaschko, Matthew B.; Buettner, Florian; Cardoso, M. Jorge; Cheplygina, Veronika; Chen, Jianxu; Christodoulou, Evangelia; Cimini, Beth A.; Collins, Gary S.; Farahani, Keyvan; Ferrer, Luciana; Galdran, Adrian; Van Ginneken, Bram; Glocker, Ben; Godau, Patrick; Haase, Robert; Hashimoto, Daniel A.; Hoffman, Michael M.; Huisman, Merel; Isensee, Fabian; Jannin, Pierre; Kahn, Charles E.; Kainmueller, Dagmar; Kainz, Bernhard; Karargyris, Alexandros; Karthikesalingam, Alan; Kenngott, Hannes; Kleesiek, Jens; Kofler, Florian; Kooi, Thijs; Kopp-Schneider, Annette; Kozubek, Michal; Kreshuk, Anna; Kurc, Tahsin; Landman, Bennett A.; Litjens, Geert; Madani, Amin; Maier-Hein, Klaus; Martel, Anne L.; Mattson, Peter; Meijering, Erik; Menze, Bjoern; Moons, Karel G. M.; Müller, Henning; Nichyporuk, Brennan; Nickel, Felix; Petersen, Jens; Rafelski, Susanne M.; Rajpoot, Nasir; Reyes, Mauricio; Riegler, Michael A.; Rieke, Nicola; Saez-Rodriguez, Julio; Sánchez, Clara I.; Shetty, Shravya; Summers, Ronald M.; Taha, Abdel A.; Tiulpin, Aleksei; Tsaftaris, Sotirios A.; Van Calster, Ben; Varoquaux, Gaël; Yaniv, Ziv R.; Jäger, Paul F.; Maier-Hein, Lena; Pathology and Laboratory Medicine, School of Medicine
    Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.