Introduction to this Special Issue on Ubiquitous Multi-Display Environments

Introduction

Until recently, most of our computing was carried out in environments with a single display. In contrast, modern computing environments are evolving into multi-display environments. Simple dual-monitor setups in personal offices are by now quite common. However, computing environments are increasingly becoming even more sophisticated, with individual offices containing several heterogeneous displays in a variety of layouts. Although multi-display environments have been shown to offer benefits for individuals (Robertson, Czerwinski, Baudisch, Meyers, Robbins, Smith, & Tan, 2005), they are arguably even more suited to collocated collaboration, as they can better facilitate multiple people working and coordinating synchronously. Collocated collaborative computing differs in many ways from individual computing, including the need to switch back and forth between private and shared interaction. Such collaboration is arguably better supported with an appropriately configured multi-display environment.

The first multi-display environments were installed in military war rooms, stock exchanges, and airports. Within the human-computer interaction (HCI) research community, the lineage of multi-display environments can be traced back to the 1980's where the use of large projection displays was explored within collaboration scenarios. For example, the CoLab project (Stefik, Foster, Bobrow, Kahn, Lanning, & Suchman, 1987) explored the use of a large display in concert with several networked desktop computers clustered around it. The advent of large interactive displays using pens and touch, such as the Xerox Liveboard made the interaction between users and shared display devices more immediate (Elrod, Bruce, Gold, Goldberg, Halasz, Janssen, Lee, McCall, Pedersen, Pier, Tang, & Welch, 1992). Early Liveboard applications, such as Tivoli, demonstrated how such displays can be used to support collaborative processes and also hinted at a long series of new user interface challenges resulting from the new input devices and from shared use (Pedersen, McCall, Moran, & Halasz, 1993).

The cost of displays, as well as their physical footprint, limited the use and integration of large displays into multi-display environments for many years. More recently, the commoditization of projection and flat screens (LCD, plasma, etc.) has resulted in lower cost and in smaller footprints, enabling such displays to be more viable for an increasing number of environments. At the same time, new types of display and input technologies have led to a diversity of new sizes and shapes. In particular, the emergence of small mobile devices significantly altered the ecology of computing, requiring systems that could support intermittent connectivity and reconfiguration of displays on the fly as users brought mobile devices in and out of their computing environment.

Today, display units differ not only in size but also in form factor, and orientation. These include vertical displays (such as traditional desktop screens and large wall mounted screens), horizontal displays (such as table tops), tilted displays (such as drafting tables), [Immersadesk, clearboard2], anandand variable configurations (such as hand-held PDAs or tablet computers). Input capabilities of modern multi-display environments also range widely, from traditional mouse-and-keyboard to pen-and-single-touch interaction, to more recent multi-touch and vision based systems that can sense a much richer set of human gestural actions.

Over the past two decades, researchers have explored connecting multiple display units into heterogeneous multi-display environments, such as a combination of an interactive table and a wall display (Johanson, Fox, & Winograd, 2002), plus multiple mobile devices (Streitz, Tandler, Müller-Tomfelde, & Konomi, 2001). Such multi-display environments can facilitate group interaction in interesting ways. For example, horizontal surfaces allow users sitting around them to see each other, with some systems like DiamondTouch enabling user identification which can facilitate unique interface designs (e.g., Forlines, Esenther, Shen, Wigdor, & Ryall, 2006). In contrast, wall displays evoke very different interpersonal interactions amongst participants. The combination of displays, of course, provides a broader range of support for computer mediated human-human interactions.

Multi-display environments differ not only in the nature and configuration of their display units and input capabilities, but also in the backend infrastructure. For example, all display units may be connected to the same processing unit (Shupp, Andrews, Dickey-Kurdziolek, Yost, & North, 2009) or to different processing units that are interconnected by a network (Robertson et al, 2005). The latter allows for the opportunistic creation and breaking apart of display components, providing users with flexibility as required by the task.

RESEARCH ISSUES

While the heterogeneity of display and input technologies enable an increasingly wide range of usage scenarios, they also pose significant new challenges. Two major themes are:

1. How do different types of displays impact users and human collaboration? Unlike other types of collocated collaboration, collaboration in a multi-display environment is particularly multi-faceted, and the range of effects substantially more complex. On the one hand, there are effects resulting from multiple users (temporarily) collaborating around a single display unit. On the other hand, different display units may be used, requiring that access to display resources be managed — typically using social protocols. In addition, users can switch between private and different public roles, e.g., between being a listener and a moderator. All of these variables are impacted by the available displays. The result is a high-dimensional space of display configurations, users, and tasks that makes social studies in this space a challenge.

2. How to make multi-display environments work more seamlessly? Multi-display environments are complex not only because of the diversity of the display units, but also because of the multitude of possible arrangements: Not only do the displays themselves matter, but also the spaces in between. To add to the complexity, units can be added, moved, and removed, often dynamically. This results in a whole range of new challenges. How can we reduce the cost of switching between display units or help users interacting across distances (Robertson et al., 2005) How can we represent the seams between the displays in a way that matches the users' cognitive model? How can we help users transfer contents between display units in an intuitive way? How do we keep screen space and contents consistent, when additional display units are added or removed?

The articles in this Special Issue, by exploring these two themes, present state-of-the-art of research in multi-display environments with design-oriented and empirically-grounded explorations of various issues surrounding multi-display environments.

USERS AND COLLABORATION

The first three articles focus on the human-human collaboration aspects of multi-display environments. The first article kicks off the discussion by giving an overview of a range of collaborative interactions that can take place in a multi-display environment, using as a motivating example the collaborative design process used at automotive and product design studios. The next two articles take a closer look at the impact of specific display configurations on group participation and social behavior.

Khan, Matejka, Fitzmaurice, Kurtenbach, Burtnyk, and Buxton describe a decade long effort in their organization studying the collaboration process in automotive and product design studios, and building new interfaces to better support that process in a variety of multi-display environments. Their exploration ranges across a wide gamut of display styles and configurations, including very large “powerwalls,” display walls augmented with multiple auxiliary displays akin to a modern instantiation of CoLab (Stefik et al., 1987), plus more specialized custom technologies such as boom mounted displays engineered to support viewing of 3D models. They describe the design and usage experience of several unique techniques designed to facilitate efficient interaction with and navigation of the digital assets used in the design process. In addition to proposing unique solutions, this article thoughtfully reveals the many tradeoffs and challenges that future research must tackle.

Robles, Nass, and Kahn study the effect of display configuration and usage context on people's behavior in group situations. In their first laboratory experiment, they explore how people respond when their answers and feedback evaluation are shared with a group via either a large central display or via personal displays. They found that display configuration can impact various aspects of participant behavior, including social anxiety and willingness to change answers. By introducing two additional factors – common vs. personal context, and common vs. interpersonal content presentation styles – into a second experiment, they explore additional behavioral elements including participant's level of engagement with the content and sense of social distance from one another. Their findings illuminate the human psychological relationships between display configurations and social perceptions along several dimensions. Such an understanding of how and when displays can significantly shape social behavior will help us make more informed decisions when designing new multi-display environments for different application domains.

Rogers, Lim, Hazlewood, and Marshall investigate whether shareable interfaces designed for multiple users to interact with can facilitate more equitable participation in co-located group settings compared with single user displays. They situate their investigation within a framework that conceptualizes how the configuration of displays and other interaction artifacts provide a range of “entry points” that allow a user to engage with the interaction environment. Their study compared sharable interfaces along the dimension of device constraint: a highly constrained laptop with a mouse shared by three users, a moderately constrained tabletop display where three sides were available for users to stand at, and a lightly constrained physical-digital setup where the tabletop display was augmented with a room full of tagged objects. They found that the most equitable participation amongst group members took place in the least constrained conditions. Their work provides empirical data to support the value of a dynamic and reconfigurable multi-display environment for collaborative activity.

SEAMLESSNESS

The last three articles in this Special Issue take more of a technology perspective. The first two focus on novel styles of interaction techniques specific to multi-display environments. These techniques enable users to combine multiple heterogeneous display units into a single coherent display system, and to configure the display space such that it is understandable and manageable for them. The final article rounds out the special issue with a look back at another key reason for combining display units in the first place, namely to increase display space.

Ramos, Hinckley, Wilson, and Sarin present a novel approach to creating multi-display environments opportunistically. Their interactive systems allow users to join multiple mobile devices and tablet computers into a single logically integrated display system or to establish a link between mobile devices and tabletop computers. They identify configuring devices as a major problem in this domain. They address it with synchronous gestures across two or more devices that not only select contents and transfer it, but also identify the involved devices, create a connection, and break down the connection when the transfer is complete. They frame the discussion within the construct of proxemics – how people handle the space surrounding an individual – and discuss social issues resulting from collocated use, such as a user's hesitancy to reach into other user's private space. Through a nuanced discussion of the various design tradeoffs, this work provides a solid foundation for future explorations into the creation and management of ad hoc conglomerations of multi-display environments.

Nacenta, Gutwin Aliakseyeu, and Subramanian further our understanding of the geometry of multi-display environments and its implications on interaction. Traditional computer screens are flat and as a result the metaphors we use today to convey the behavior of pointers and windows are derived from a two-dimensional world. In a multi-display environment, however, display units are embedded in a three-dimensional space. The authors analyze different metaphors and then explore the idea of using perspective from the user's current position as a means for transferring the traditional 2-D metaphors to the new 3-D world. They describe techniques for cross-display object movement based on perspective and discuss the resulting challenges, which include the fact that perspective varies between users and when users move around and the risk of losing track of a pointer traveling between screen units. Their work adds a novel twist to the literature on this topic in that it is perhaps the first to systematically consider taking a user's current viewpoint into account when designing interactions for use in a multi-display environment.

Shupp, Andrews, Dickey-Kurdziolek, Yost, and North study the effect of display size and configuration for very large high-resolution displays. While their user studies focused on single-person use, it is clear that the findings have direct implications for collaborative use of multi-display environments, such as how much resolution is useful and what types of task certain display configurations afford. The authors present two user studies, both of which involves up to 24 display units each with 17” diagonal size and 1280x1024px resolution. They study user performance and find key benefits for large physical size, primarily because it allows users to navigate physically to a target rather than having to revert to virtual navigation, such as panning the screen. In addition, they find that a curved display surrounding the user simplifies the inspection of details, while a flat display provides better overview. While further investigation is certainly necessary to validate their findings in collaborative multi-person usage scenarios and to see if their results hold when larger seamless displays become a reality, their results clearly point to the need for designers to take size, resolution, and curvature into account when building such environments.

The broad range of approaches, techniques, and results described in the articles in this Special Issue is a clear indicator of both the importance and maturity of this topic. In describing a decade long effort in this area, the Khan et al. article exemplifies the longevity and continuing challenges presented by multi-display environments. The empirical studies described in the other articles have approached various issues from very different angles — new display types, new configurations both static and reconfigurable, new applications scenarios, as well as analyses grounded in psychology interpreting human behavior in the context of collocated collaboration.

Multi-display environments have evolved rapidly in the past two decades, and we hope the knowledge and insights in this Special Issue will assist in building a deep foundation for the design of the next generation of multi-display environments.