Browsing by Subject "Reproducibility"
Now showing 1 - 9 of 9
Results Per Page
Sort Options
Item Clearinghouse Standards of Evidence on the Transparency, Openness, and Reproducibility of Intervention Evaluations(Springer, 2021) Mayo-Wilson, Evan; Grant, Sean; Supplee, Lauren H.; Epidemiology, Richard M. Fairbanks School of Public HealthClearinghouses are influential repositories of information on the effectiveness of social interventions. To identify which interventions are “evidence-based,” clearinghouses review intervention evaluations using published standards of evidence that focus primarily on internal validity and causal inferences. Open science practices can improve trust in evidence from evaluations on the effectiveness of social interventions. Including open science practices in clearinghouse standards of evidence is one of many efforts that could increase confidence in designations of interventions as “evidence-based.” In this study, we examined the policies, procedures, and practices of 10 federal evidence clearinghouses that review preventive interventions—an important and influential subset of all evidence clearinghouses. We found that seven consider at least one open science practice when evaluating interventions: replication (6 of 10 clearinghouses), public availability of results (6), investigator conflicts of interest (3), design and analysis transparency (3), study registration (2), and protocol sharing (1). We did not identify any policies, procedures, or practices related to analysis plan registration, data sharing, code sharing, material sharing, and citation standards. We provide a framework with specific recommendations to help federal and other evidence clearinghouses implement the Transparency and Openness Promotion (TOP) Guidelines. Our proposed “TOP Guidelines for Clearinghouses” includes reporting whether evaluations used open science practices, incorporating open science practices in their standards for receiving “evidence-based” designations, and verifying that evaluations used open science practices. Doing so could increase the trustworthiness of evidence used for policy making and support improvements throughout the evidence ecosystem.Item From Model Organisms to Humans, the Opportunity for More Rigor in Methodologic and Statistical Analysis, Design, and Interpretation of Aging and Senescence Research(Oxford University Press, 2022) Chusyd, Daniella E.; Austad, Steven N.; Brown, Andrew W.; Chen, Xiwei; Dickinson, Stephanie L.; Ejima, Keisuke; Fluharty, David; Golzarri-Arroyo, Lilian; Holden, Richard; Jamshidi-Naeini, Yasaman; Landsittel, Doug; Lartey, Stella; Mannix, Edward; Vorland, Colby J.; Allison, David B.; Anatomy, Cell Biology and Physiology, School of MedicineThis review identifies frequent design and analysis errors in aging and senescence research and discusses best practices in study design, statistical methods, analyses, and interpretation. Recommendations are offered for how to avoid these problems. The following issues are addressed: (a) errors in randomization, (b) errors related to testing within-group instead of between-group differences, (c) failing to account for clustering, (d) failing to consider interference effects, (e) standardizing metrics of effect size, (f) maximum life-span testing, (g) testing for effects beyond the mean, (h) tests for power and sample size, (i) compression of morbidity versus survival curve squaring, and (j) other hot topics, including modeling high-dimensional data and complex relationships and assessing model assumptions and biases. We hope that bringing increased awareness of these topics to the scientific community will emphasize the importance of employing sound statistical practices in all aspects of aging and senescence research.Item Genome-wide circadian rhythm detection methods: systematic evaluations and practical guidelines(Oxford University Press, 2021-05-20) Mei, Wenwen; Jiang, Zhiwen; Chen, Yang; Chen, Li; Sancar, Aziz; Jiang, Yuchao; Medicine, School of MedicineCircadian rhythms are oscillations of behavior, physiology and metabolism in many organisms. Recent advancements in omics technology make it possible for genome-wide profiling of circadian rhythms. Here, we conducted a comprehensive analysis of seven existing algorithms commonly used for circadian rhythm detection. Using gold-standard circadian and non-circadian genes, we systematically evaluated the accuracy and reproducibility of the algorithms on empirical datasets generated from various omics platforms under different experimental designs. We also carried out extensive simulation studies to test each algorithm’s robustness to key variables, including sampling patterns, replicates, waveforms, signal-to-noise ratios, uneven samplings and missing values. Furthermore, we examined the distributions of the nominal equation M1-values under the null and raised issues with multiple testing corrections using traditional approaches. With our assessment, we provide method selection guidelines for circadian rhythm detection, which are applicable to different types of high-throughput omics data.Item Opinion: Why institutional review boards should have a role in the open science movement(National Academy of Sciences, 2019-10-22) Grant, Sean; Bouskill, Kathryn E.; Social and Behavioral Sciences, School of Public HealthItem Pilot Evaluation of the Long-Term Reproducibility of Capillary Zone Electrophoresis-Tandem Mass Spectrometry for Top-Down Proteomics of a Complex Proteome Sample(American Chemical Society, 2024) Sadeghi, Seyed Amirhossein; Chen, Wenrong; Wang, Qianyi; Wang, Qianjie; Fang, Fei; Liu, Xiaowen; Sun, Liangliang; Biomedical Engineering and Informatics, Luddy School of Informatics, Computing, and EngineeringMass spectrometry (MS)-based top-down proteomics (TDP) has revolutionized biological research by measuring intact proteoforms in cells, tissues, and biofluids. Capillary zone electrophoresis-tandem MS (CZE-MS/MS) is a valuable technique for TDP, offering a high peak capacity and sensitivity for proteoform separation and detection. However, the long-term reproducibility of CZE-MS/MS in TDP remains unstudied, which is a crucial aspect for large-scale studies. This work investigated the long-term qualitative and quantitative reproducibility of CZE-MS/MS for TDP for the first time, focusing on a yeast cell lysate. Over 1000 proteoforms were identified per run across 62 runs using one linear polyacrylamide (LPA)-coated separation capillary, highlighting the robustness of the CZE-MS/MS technique. However, substantial decreases in proteoform intensity and identification were observed after some initial runs due to proteoform adsorption onto the capillary inner wall. To address this issue, we developed an efficient capillary cleanup procedure using diluted ammonium hydroxide, achieving high qualitative and quantitative reproducibility for the yeast sample across at least 23 runs. The data underscore the capability of CZE-MS/MS for large-scale quantitative TDP of complex samples, signaling its readiness for deployment in broad biological applications. The MS RAW files were deposited in ProteomeXchange Consortium with the data set identifier of PXD046651.Item Reproducibility of Glycemic Measures Among Dysglycemic Youth and Adults in the RISE Study(The Endocrine Society, 2023) Tjaden, Ashley H.; Edelstein, Sharon L.; Arslanian, Silva; Barengolts, Elena; Caprio, Sonia; Cree-Green, Melanie; Lteif, Amale; Mather, Kieren J.; Savoye, Mary; Xiang, Anny H.; Kahn, Steven E.; Medicine, School of MedicineAims: Previous work found poor reproducibility for measures of glycemia in individuals at risk for dysglycemia. Differences between youth and adults have not been assessed. Using youth and adults in the Restoring Insulin Secretion Study, we tested variability and classification concordance for hemoglobin A1C (HbA1c), fasting and 2-hour glucose from oral glucose tolerance tests (OGTTs). Methods: HbA1c and glucose on repeated samples obtained ∼6 weeks apart were compared in 66 youth (mean age 14.2 years) and 354 adults (52.7 years). Changes, coefficient of variation (CV), and concordance of diagnostic categories between the 2 visits were compared. Results: Mean difference between the 2 visits in HbA1c was higher in youth than adults (P < .001), while fasting glucose was similar and 2-hour glucose was lower in youth (P = .051). CV was smallest for HbA1c compared to fasting and 2-hour glucose. For HbA1c, youth had higher CV (P < .001); whereas CV for 2-hour glucose was lower for youth (P = .041). Classification concordance by HbA1c was lower in youth (P = .004). Using OGTT or HbA1c for classification, intervisit variability produced discordant classification in 20% of youth and 28% of adults. Using both fasting glucose and HbA1c, intervisit variability reduced discordant classification to 16% of adults while not improving classification in youth. Conclusions: Poor reproducibility and lack of classification concordance highlight the limitations of one-time testing, with important implications for assessing eligibility in clinical trials. Consideration should be given to using more than a single parameter for screening and diagnosis, especially when classification category is important.Item Stability of MRI metrics in the advanced research core of the NCAA-DoD concussion assessment, research and education (CARE) consortium(Springer Nature, 2018-08) Nencka, Andrew S.; Meier, Timothy B.; Wang, Yang; Muftuler, L. Tugan; Wu, Yu-Chien; Saykin, Andrew J.; Harezlak, Jaroslaw; Brooks, M. Alison; Giza, Christopher C.; Difiori, John; Guskiewicz, Kevin M.; Mihalik, Jason P.; LaConte, Stephen M.; Duma, Stefan M.; Broglio, Steven; McAllister, Thomas; McCrea, Michael A.; Koch, Kevin M.; Radiology and Imaging Sciences, School of MedicineThe NCAA-DoD Concussion Assessment, Research, and Education (CARE) consortium is performing a large-scale, comprehensive study of sport related concussions in college student-athletes and military service academy cadets. The CARE "Advanced Research Core" (ARC), is focused on executing a cutting-edge investigative protocol on a subset of the overall CARE athlete population. Here, we present the details of the CARE ARC MRI acquisition and processing protocol along with preliminary analyzes of within-subject, between-site, and between-subject stability across a variety of MRI biomarkers. Two experimental datasets were utilized for this analysis. First, two "human phantom" subjects were imaged multiple times at each of the four CARE ARC imaging sites, which utilize equipment from two imaging vendors. Additionally, a control cohort of healthy athletes participating in non-contact sports were enrolled in the study at each CARE ARC site and imaged at four time points. Multiple morphological image contrasts were acquired in each MRI exam; along with quantitative diffusion, functional, perfusion, and relaxometry imaging metrics. As expected, the imaging markers were found to have varying levels of stability throughout the brain. Importantly, between-subject variance was generally found to be greater than within-subject and between-site variance. These results lend support to the expectation that cross-site and cross-vendor advanced quantitative MRI metrics can be utilized to improve analytic power in assessing sensitive neurological variations; such as those effects hypothesized to occur in sports-related-concussion. This stability analysis provides a crucial foundation for further work utilizing this expansive dataset, which will ultimately be freely available through the Federal Interagency Traumatic Brain Injury Research Informatics System.Item The case for open research in entomology: reducing harm, refining reproducibility and advancing insect science(Agricultural and Forest Entomology, 2024) Cuff , Jordan P.; Barrett , Meghan; Gray , Helen; Fox , Charles; Watt , Allan; Aimé , Emilie1. Open research is an increasingly developed and crucial framework for the advancement of science and has seen successful adoption across a broad range of disciplines. Entomology has, however, been slow to adopt these practices compared to many adjacent fields despite ethical and practical imperatives to do so. 2. The grand challenges facing entomology in the 21st century require the synthesis of evidence at global scales, necessitating open sharing of data and research at a pace and scale incompatible with the slow adoption of open research practices. Open science also plays a vital role in fostering trust in research and maximizing use of research outputs, which is ethically crucial for reducing harms to insects. 3. We outline these imperatives and how open research practices can enhance entomological research across a range of contexts. We also highlight the holistic nature of open science across the full research lifecycle through several specific examples of open research practices, which can be adopted easily by individual entomologists. 4. We do, however, argue that the responsibility of promoting, integrating and encouraging open research is most crucially held by publishers, including scholarly societies, which have leveraged widespread adoption in adjacent fields. Entomology must advance quickly to become a leading discipline in the open research transition.Item Transparent, Open, and Reproducible Prevention Science(Springer, 2022) Grant, Sean; Wendt, Kathleen E.; Leadbeater, Bonnie J.; Supplee, Lauren H.; Mayo-Wilson, Evan; Gardner, Frances; Bradshaw, Catherine P.; Pediatrics, School of MedicineThe field of prevention science aims to understand societal problems, identify effective interventions, and translate scientific evidence into policy and practice. There is growing interest among prevention scientists in the potential for transparency, openness, and reproducibility to facilitate this mission by providing opportunities to align scientific practice with scientific ideals, accelerate scientific discovery, and broaden access to scientific knowledge. The overarching goal of this manuscript is to serve as a primer introducing and providing an overview of open science for prevention researchers. In this paper, we discuss factors motivating interest in transparency and reproducibility, research practices associated with open science, and stakeholders engaged in and impacted by open science reform efforts. In addition, we discuss how and why different types of prevention research could incorporate open science practices, as well as ways that prevention science tools and methods could be leveraged to advance the wider open science movement. To promote further discussion, we conclude with potential reservations and challenges for the field of prevention science to address as it transitions to greater transparency, openness, and reproducibility. Throughout, we identify activities that aim to strengthen the reliability and efficiency of prevention science, facilitate access to its products and outputs, and promote collaborative and inclusive participation in research activities. By embracing principles of transparency, openness, and reproducibility, prevention science can better achieve its mission to advance evidence-based solutions to promote individual and collective well-being.