Metrics for Evaluating the Impact of Data Sets

Abstract

Research is a social activity, involving a complex array of resources, actors, activities, attitudes, and traditions (Sugimoto & Larivière 2018). There are many norms, including the sharing of new work in the form of books and journal articles and the use of citations and acknowledgments to recognize the influence of earlier work, but what it means to produce impactful scholarship is difficult to define and measure. The goals, methods, metrics, and utility of evaluating the impact of data sets are situated within this broader context of scholarly communication and evaluation. An understanding of the dynamic history, current practices, concepts, and critiques of measuring impact for and beyond research data sets can help researchers navigate the scholarly dissemination landscape more strategically and gain agency in regard to how they and their work are evaluated and described.

What is research impact? As Roemer and Borchardt (2015) describe, the concept involves two important ideas: the change a work influences and the strength of this effect. These effects can include, but are not limited to, advances in understanding and decision making, policy creation and change, economic development, and societal benefits. For example, rich documentation of an endangered language might lead to and support community and governmental revitalization efforts. However, the linkages between a specific scholarly product and its effects are rarely direct, there are disciplinary differences between how research is communicated and endorsed, and some outcomes take a very long time to manifest (Greenhalgh et al. 2016). This makes the assessment of research impact very labor intensive, even at a small scale, so researchers and decision makers often rely on data and metrics that are regarded as indicative of certain kinds of impact.