Visual analytics evaluation process: Practice guidelines for complex domain

Abstract

Visual analytics fundamentally influences the analytical process. Without proper guidance, the use of visual analytics in complex domain can become an obstacle that hides the usefulness of analytical data. Since evaluation is the only way to identify the effectiveness of visual analytics to represent analytical outcomes, misconception of the evaluation process will bury the relevancy of visualization to support valuable decision. Recently, the nature domain of data has changed and we are now dealing with data that is massive, ambiguous, and dynamic, is often processed in real time. Hence, the data is more complex and cognitive activities that visual analytics facilitates are also getting more complex. Thus, this research revisits the way to evaluate visual analytics in complex cognitive conditions that are natural, uncertain and context dependent. Governed by the Design Science Research Methodology (DSRM), there are three phases involve during the evaluation process; i). demonstration, ii). evaluation and iii). communication. The design process is embedded with the human-activity centered design approach to gain better understanding on the visual analytics users and the complex cognitive activities involved. Thus, the research proposes Focus Group Observation method in conducting the evaluation in authentic setting. By offering a set of evaluation recommendations, this research aims to enhance visual analytics among users. It also recommends the evaluation criteria, sampling strategy and participation, focus group tasks and settings and data management and analysis that are suitable for complex domain in visual analytics.