- Browse by Author
Browsing by Author "Haas, Brian"
Now showing 1 - 2 of 2
Results Per Page
Sort Options
Item Best practices to evaluate the impact of biomedical research software-metric collection beyond citations(Oxford University Press, 2024) Afiaz, Awan; Ivanov, Andrey A.; Chamberlin, John; Hanauer, David; Savonen, Candace L.; Goldman, Mary J.; Morgan, Martin; Reich, Michael; Getka, Alexander; Holmes, Aaron; Pati, Sarthak; Knight, Dan; Boutros, Paul C.; Bakas, Spyridon; Caporaso, J. Gregory; Del Fiol, Guilherme; Hochheiser, Harry; Haas, Brian; Schloss, Patrick D.; Eddy, James A.; Albrecht, Jake; Fedorov, Andrey; Waldron, Levi; Hoffman, Ava M.; Bradshaw, Richard L.; Leek, Jeffrey T.; Wright, Carrie; Pathology and Laboratory Medicine, School of MedicineMotivation: Software is vital for the advancement of biology and medicine. Impact evaluations of scientific software have primarily emphasized traditional citation metrics of associated papers, despite these metrics inadequately capturing the dynamic picture of impact and despite challenges with improper citation. Results: To understand how software developers evaluate their tools, we conducted a survey of participants in the Informatics Technology for Cancer Research (ITCR) program funded by the National Cancer Institute (NCI). We found that although developers realize the value of more extensive metric collection, they find a lack of funding and time hindering. We also investigated software among this community for how often infrastructure that supports more nontraditional metrics were implemented and how this impacted rates of papers describing usage of the software. We found that infrastructure such as social media presence, more in-depth documentation, the presence of software health metrics, and clear information on how to contact developers seemed to be associated with increased mention rates. Analysing more diverse metrics can enable developers to better understand user engagement, justify continued funding, identify novel use cases, pinpoint improvement areas, and ultimately amplify their software's impact. Challenges are associated, including distorted or misleading metrics, as well as ethical and security concerns. More attention to nuances involved in capturing impact across the spectrum of biomedical software is needed. For funders and developers, we outline guidance based on experience from our community. By considering how we evaluate software, we can empower developers to create tools that more effectively accelerate biological and medical research progress. Availability and implementation: More information about the analysis, as well as access to data and code is available at https://github.com/fhdsl/ITCR_Metrics_manuscript_website.Item Enhancing Patient Communication With Chat-GPT in Radiology: Evaluating the Efficacy and Readability of Answers to Common Imaging-Related Questions(Elsevier, 2023) Gordon, Emile B.; Towbin, Alexander J.; Wingrove, Peter; Shafique, Umber; Haas, Brian; Kitts, Andrea B.; Feldman, Jill; Furlan, Alessandro; Radiology and Imaging Sciences,School of MedicinePurpose To assess ChatGPT's accuracy, relevance, and readability in answering patients' common imaging-related questions and examine the effect of a simple prompt. Methods 22 imaging-related questions were developed from categories previously described as important to patients: safety, the radiology report, the procedure, preparation before imaging, meaning of terms, and medical staff. These questions were posed to ChatGPT with and without a short prompt instructing the model to provide an accurate and easy-to-understand response for the average person. Four board-certified radiologists evaluated the answers for accuracy, consistency, and relevance. Two patient advocates also reviewed responses for their utility for patients. Readability was assessed by Flesch Kincaid Grade Level (FKGL). Statistical comparisons were performed using chi-square and paired t-tests. Results 264 answers were assessed for both unprompted and prompted questions. Unprompted responses were accurate 83% (218/264) of the time, which did not significantly change for prompted responses (87% [229/264]; P=0.2). The consistency of the responses increased from 72%f (63/88) to 86% (76/88) when prompted (P=0.02). Nearly all responses (99% [261/264]) were at least partially relevant for both question types. Fewer unprompted responses were considered fully relevant at 67% (176/264), though this increased significantly to 80% when prompted (210/264) (P=0.001). The average FKGL was high at 13.6 [12.9-14.2], unchanged with the prompt (13.0 [12.41-13.60], P=0.2). None of the responses reached the eighth-grade readability recommended for patient-facing materials. Conclusions ChatGPT demonstrates the potential to respond accurately, consistently, and relevantly to patients' imaging-related questions. However, imperfect accuracy and high complexity necessitate oversight before implementation. Prompts reduced response variability and yielded more targeted information but did not improve readability. Relevance and Application ChatGPT has the potential to increase accessibility to health information and to streamline the production of patient-facing educational materials, though its current limitations require cautious implementation and further research.