Techniques for Improving the Robustness of Visual Analytics

Abstract

Interactive visualization systems, such as Tableau, are integral parts of the data analysis workflow. While such tools were built to help analysts perform exploratory data analysis with minimal effort, analysts have also been using them to make statistical inferences (e.g., predicting future trends) based on patterns revealed by the dataset. However, in addition to revealing true patterns, visualizations can also surface noise and other random fluctuations in data, which could lead to spurious discoveries. The latter poses a threat to the trustworthiness of analyses, especially given the increased reliance on visualizations across various domains. My central thesis is that it is possible to reduce the incidence of false discovery by introducing lightweight user interface elements in visualization tools. In particular, I propose eliciting and incorporating analyst beliefs into visualizations as an approach for guarding against spurious patterns and reducing the risk of analysts “overfitting” the data. To study how analysts would respond to such intervention, I first designed an interactive tool that combined visual belief elicitation with traditional visualization functionalities. In a qualitative study with data analysts, the tool appeared to allow users to operationalize their working knowledge into analyses, nudging them to adopt normative analysis practices (e.g., specifying hypotheses before peeking at data). I then conducted a crowdsourced experiment to investigate if this design could indeed help reduce the incidence of false discovery. Compared to a control condition, participants who used our intervention made significantly more accurate inferences and reported fewer false discoveries. Lastly, I investigated the capability of human intuition by comparing inferences from participants against those generated by statistical machines to understand the advantages and limitations of each. Overall, my thesis paves the way toward the development of a robust visual analytics system that facilitates collaborative decision-making processes, leveraging the complementary abilities of humans and machines.