Advancing the theory and practice of system evaluation: a case study in geovisual analytics of social media

ABSTRACT

This paper advances the state-of-the-art in methodology design for empirical evaluation of (geo)visual analytics software. Specifically, we describe the process of design, development and application of a prototypical user study tailored to the evaluation of complex geovisual analytics tools that focus on social media analysis. We fist perform a synthesis of existing theory and best practices for software evaluation of comparable systems. We then demonstrate how the product of said synthesis – a methodological ‘check list’ – can be used to inform a proof-of-concept user study of an actual geovisual analytics software system. The resulting user study design accommodates for the use of real geographic social media datasets, the complexity of the intended analytical process, and for the learning challenges faced by the participants working with a fully-functional and mature geovisual analytics application, and is likely representative of a wide range of evaluation scenarios in (geo)visual analytics. A complete summary of all the study instruments is included to encourage their scrutiny, reuse and modification by others. Finally, we have discovered that participants’ curiosity and desire for autonomy played a noticeable role in the evaluation process – something not previously reported.

RÉSUMÉ

Ce papier présente un état de l’art sur la conception méthologique pour l’évaluation empirique de logiciels de (géo)visualisation analytiques. En particulier nous décrivons le processus de conception, de développement et d’application pour une étude utilisateur standard centrée sur l’évaluation d’un outil de geovisualisation analytique complexe axé sur l’analyse des médias sociaux. Nous faisons d’abord une synthèse des théories existantes et des bonnes pratiques sur l’évaluation de logiciels de systèmes comparables. Puis nous montrons comment le produit de cette synthèse – une méthode basé sur une liste de vérification – peut être utilisé pour décrire une étude utilisateur de type validation pour un logiciel d’analyse géovisuelle. La conception de l’étude utilisateur résultante s’adapte à l’utilisation de vrais jeux de données géographiques de médias sociaux ; à la complexité du processus analytique envisagé et aux difficultés d’apprentissage que connaissent les participants, confrontés à une application analytique géovisuelle fonctionnelle et éprouvée. Elle est représentative d’un large éventail de scénario d’évaluation en analyse géovisuelle. Un résumé complet de tous les éléments de l’étude est inclus pour encourager leur analyse détaillée, leur réutilisation et leur modification par d’autres. Enfin nous avons découvert que la curiosité et l’envie d’autonomie des participants jouent un rôle non négligeable dans le processus d’évaluation, point qui n’avait pas été identifié auparavant.

KEYWORDS:

• Evaluation
• user study
• geovisual analytics
• social media

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes on contributors

Alexander Savelyev has a B.S. degree from Saint Petersburg State University, Saint Petersburg, Russia (2006), a M.A. degree from the University of Northern Iowa (2008) and a Ph.D. degree from the Pennsylvania State University (2015), all in Geography. He joined the Pennsylvania State University as a postdoctoral scholar in 2015 and the Texas State University in 2016, where he is currently an Assistant Professor of Geography. He was Vice-Chair, Chair and Past Chair of the Cartography Specialty Group of the American Association of Geographers (2016–2019). His main areas of research interest are information- and geovisualization, geo-visual analytics, and cartography.

Alan M. MacEachren has a BA from Ohio University (1974) and MS (1976) and PhD (1979) from the University of Kansas, all in Geography. He has held the following academic appointments: Virginia Tech, Assistant Professor of Geography (1979–1983); University of Colorado, Assistant Professor of Geography (1983–1985); Penn State University, Associate Professor of Geography (1985–1992), Professor of Geography (1992-present), Affiliate Professor of Information Sciences and Technology (2007-present), Visiting Professor, Department of Computer Science, Stanford University, Director, GeoVISTA Center, 1998-present, faculty Associate, Institute for CyberScience (2017-present). He was Chair of the Visualization Commission of the International Cartographic Association (1995–2005), Associate Editor for Information Visualization (2001-present), Associate Editor of IEEE Transactions on Visualization and Computer Graphics, 2007–2011, a member of multiple National Research Council Committees (including the Committee for the Workshop on Intersections between Geospatial Information and Information Technology). He was also a member of the National Visualization and Analytics Center R&D Agenda panel (2004–2005). MacEachren was named an Honorary Fellow of the International Cartographic Association in 2005 and was elected as a Fellow of the American Association for Advancement of Science in 2013.