Why mess with users’ autonomy? On ideals and dualism in design

KEYWORDS:

• User autonomy
• design ideal
• esthetic axioms
• soma design

1. Introduction

The MRL Studio in Nottingham has a longstanding commitment to designing for the uncomfortable, the thrilling, the unexpected, the unusual and the artistic. In this article, they revisit three of their designs that in different ways mess with users’ sense of control over the interaction – or as they frame it, users’ autonomy in deciding the course of action.

With Bronco, they explore how breath could be used to control a bucking mechanical horse. Breathing is interesting as it is for the most part controlled by our autonomous nervous system, but we can also choose to consciously control it. Bronco challenges this control by increasing its bucking for each breath riders takes, in effect asking riders to hold their breath until they can no longer do so.

The second design is an interactive film called The Disadvantages of Time Travel, a disturbing story about childhood bullying. The unfoldings of the story depend on the viewer’s brain activity and blinking (based on EEG). Each time the viewer blinks, the film cuts between four different portrayals of the same story. By blinking, the viewer can shift from disturbing scenes, or by not blinking, they can stay with less disturbing unfoldings of the story. But at some point, their eyes become dry and they must blink.

The third design exploration is slightly different. It describes a technology that lets musicians control interactive performance through their playing. If they, for example, play a particular sequence of pitches and/or durations, it triggers actions, such as audio- or visual effects. A musician can create their own codes, and thereby fully control when the effects are triggered. Or they can play a piece of music with hidden codes that someone else created. The latter means they have to react to events unfolding in the moment.

After discussing their three example designs, the authors unpack the underlying dimensions what it means to challenge users’ autonomy. They arrive at three experiential dimensions that provide different facets of control that are challenged in these interactions:

1. users may have to surrender control – to the system or to their own autonomic system (as at some point they have to breathe or blink their eyes)

2. users may have to become self-aware of how and when they can control the interaction – sometimes attending to the mechanics of their own movements, sometimes attending to how those movements in turn render various effects such as music or visual effects

3. finally, looseness in control can arise from imprecise sensors or technology glitches, but it can also arise from inexactness of users’ own movements or behaviors

A designer can orchestrate the unfoldings of an experience along these dimensions, shifting in-between them or increasing or decreasing control. This is where the most important value of this paper lies: the understanding that users’ autonomy is a material of sorts, a material that designers can shape into interesting interactions that travel from forcing users to surrender control to making users more aware of their interactions. While the classical, early HCI-research emphasized that users should be given total control over the tools they use (Shneiderman, 1997), as we turn to richer user experiences, control becomes a malleable material of sorts.

2. Critique I: why surrender control?

But why would we want users to surrender their control? What novel experiences are enabled and work really well and which ones fail to engage? Here the authors leave us with some question marks.

The problem of answering the why-question lies partly in that an infinite space of possible interactions and experiences is opened. Depending on domain and aims, playing with users’ autonomy can be the basis of all sorts of design aims: it can be anything from the core of a thrilling, playful, comfortable or uncomfortable experience to the basis in a utilitarian function aiming to make interaction simpler by suggesting or even acting on behalf of users. That is, prescribing why or when we should meddle with users’ control over the interaction will depend on which application we are designing, its aims and purpose, and the domain at hand. It is hard to provide any generic answer to the why-question and exactly how to do it to create a successful interaction.

Still, I find myself longing for some stronger conclusions on what engagements they have uncovered that they find most interesting or unexpected? Within the domains they have been working, they could have told us some of what worked and what did not work? In-between the lines there are, in my view, some really interesting results.

For example, in The Disadvantages of Time Travel, there is an interesting uncomfortable interaction loop going back and forth between bodily discomfort and narrative discomfort: not blinking makes users’ eyes go uncomfortably dry, forcing them to blink, which in turn spurs a negative turn in the interactive narrative toward darker content. The user experience becomes one of traveling between personal, bodily discomfort to darker accounts of childhood bullying. This back and forth engagement between different forms of discomfort is to me touching on a deeper lesson learnt. It speaks of the strong back and forth linkage between movement and experience.

A long time ago, I tried framing somewhat similar interactions as affective loops (Höök, 2008, 2009): by deliberately designing for certain physical movements that in turn touch on or somehow are related to certain emotional or visceral experiences, we can build strong user experiences. In one of our design studies (Paiva et al., 2002), we noted how, for example, upwards, energetic, loose, arm movements versus tensed, inwards, arm movements with an interactive plush toy, helped build very strong affective identification with the character in a game. With the upwards movements with the plush doll the character in the game would become ‘happy’, while with the slower, tensed inwards movements, the character would become ‘sad’. Especially interesting to me was how this engagement in turn resulted in a stronger, bodily mirroring of what the character in the game was portraying. For example, when a battle was won, the character would raise both arms to celebrate the victory. Users often mirrored the movement, celebrating together with their character. The emotional experience and identification flowed back and forth between character behavior and user behavior.

While the interactive plush toy was designed to make players be in total control of the interaction, the design of the Influencing Machine (Höök et al., 2003) played with the scale from controlling to influencing to complete randomness in how user and machine interacted. From that work, we learnt the importance of letting users (in some way) experience their influence on the interaction – even in situations where the aim is to make users surrender their control. Later, we designed a machine creating fine arts drawings based on the viewers biodata – the Metaphone (Šimbelis et al., 2014). Viewers would often start by assuming that they could control the Metaphone through controlling their heart rate or arousal levels (as measured by a GSR-sensor). But then, step by step, many experienced this sense of surrendering control, letting the monotony of the machines soundscape and behavior calm them down – but in a creepy manner. The Metaphone works with media speaking against one-another: for example, there is a monotonous sound based on users’ heart rate that can feel calming, but there were also creepy low-frequency sounds based on the viewer’s arousal level (GSR). These seemingly contradictory experiences come together into an experience transporting the user into the strange and somewhat scary land of machines.

Thus, if we want to design for really strong user experiences, beyond engaging solely through the glass screen using symbols and language, we can choose to involve more of our senses – playing with, for example, users’ control over semi-autonomous processes such as breathing or blinking our eyes. But the design challenge then becomes figuring out how this interaction should unfold over time to achieve interesting and relevant effects on our whole selves. We might argue that much can be learnt from other media, such as fine arts, movies, story-telling, sports experiences and so on. But those fail to tell us how to best make use of our interactive technological materials – changing their behavior in response to user behaviors. An infinite landscape of possible designs opens to us, a territory where we do not yet know where to go. Benford and colleagues point to one interactive element that we can fiddle with, but they could have, more strongly, told us what specific design insights this led to. In what corners of the three-dimensional space did they achieve successful interactions? Why were those deemed successful? Why are some journeys through the three-dimensional space they outline more interesting than others?

3. Critique II: by what ideals?

An even more difficult way of asking the why-question, is to not only classify and explain those moments where we can align (or misalign) different sensorial and intellectual experiences in order to achieve some specific user experience, but on a more existential level ask ourselves: why would we do that in the first place? I realize that this is a bit like asking “why should we do art?” or “what is the meaning of life?”. But as designers, we all come to that question sooner or later. We have to articulate the underlying esthetic axioms, or manifestos, that drive our explorations in some specific direction (Redström, 2017). We have to, on an existential level, decide what we find worth designing for.

In HCI, the problem has recently been framed as the existential crisis of HCI (Janlert & Stolterman, 2015). When HCI was founded in the 1980s, it was concerned with shaping dialogue-based interactions through a glass interface aiming to let users work efficiently. The ideals were given by that work context: create tools that lets users work more efficiently. When the field later had to deal with how digital interactions had expanded to concern meaningful experiences in all walks of life, the ideals became concerned with a richer plethora of design challenges. In addition, technologies coming onto the market, such as the EEG and breathing sensors used in Bronco creating for an interactive narrative, pushes interaction design closer and closer to our bodies.

In work I did recently with Jonas Löwgren (under review), we were asked to define interaction design. We summarized our understandings (through the lens of design ideals) as:

“it is a field that seems to change and follow in the wake of novel materials and technologies, where the resulting designs penetrate one use domain after another, therefore restlessly asking for new ideals, new design frameworks, novel interdisciplinary know-how. But in the midst of all this change, we believe that interaction designers will continue to serve as catalysts of ongoing interplay involving human and nonhuman actors, driven by empathy and compassion in order to care for the human condition: our corporeality, experience, and how practice and culture completes us.”

My understanding of Benford and colleagues’ long-standing commitment to engage with artists; uncomfortable interactions; using mixed reality technologies; leads me to believe that they are addressing esthetics involving all our senses on a profound level. This could have been stated and elaborated on more clearly in the paper. By pushing on the boundaries of human control, be it over artistic performance as in the musicians playing with the muzicodes; over the autonomous processes in our corporeal bodies; or with the practices and cultures they reside in; they are engaging with the richness of human experience – with how to improve on and deepen esthetic experience through all our senses. As they play with and push those boundaries, they uncover, or perhaps discover or invent, interactive experiences that touches on the core of what it means to be human. Any design studio will have certain commitments, certain esthetic axioms, certain methods and ways of approaching design. While the MRL-studio often lets artists, such as Blast Theory, direct their design projects, their own competence and choices clearly color which projects they take on and how they are shaped.

In my design studio, we have during the last years framed our esthetic commitments as soma design (Höök, 2018). Building on the somaesthetics theory by Shusterman (Shusterman, 2008) and the ‘corporeal turn’ by Sheets-Johnstone (Maxine Maxine Sheets-Johnstone, 2011), we have argued that design with novel technologies on or around our bodies and movement may help deepen (our and our end-users’) esthetic appreciation through all our senses. We have argued, based on Shusterman’s theories, that such deepened somesthetic appreciation lets us live a better, richer life. George Khut framed it as allowing us to, as designers, examine and improve on connections among sensation, feeling, emotion, and subjective understanding and values (Khut, 2006) and how this in turn lets us orchestrate user experiences that lets our end-users have richer experiences.

4. Critique III: from dualism and back?

Finally, the core of the work presented relates to bridging the gaps introduced by different dualistic stances, a position I strongly agree with. But here I find myself troubled by one statement: “Thus, while we agree with calls for more holistic approaches to mind-body experiences in embodied interaction, we suggest that more attention be paid to the inherent tension between mind and body”. Here, I believe that a non-dualistic/holistic stance should not be confused with the idea that all the different processes that makes up a human being are necessarily aligned as one non-divisible unit or cannot be subdivided into different parts.

There are many processes going on in our corporeal bodies in response to being in the world that may well speak against one-another or where it is unclear where the motivation to act or think starts or ends. What a non-dualistic position entails instead the recognition that thought does not reign over matter, that corporeal hormonal, muscular, nervous system responses are all connected – even if sometimes in confusing ways. It is a position that maintains that thought without a body is a non-entity.

Furthermore, as noted already by William James (1884), emotion without an embodied experience as an impossibility: “A purely disembodied human emotion is a nonentity.” James cites an example from Henle to describe the close connection between feeling and bodily movement or experiences:

“So small a thing as a bubble of air rising from the stomach through the oesophagus, and loitering on its way a few minutes and exerting pressure on the heart, is able during sleep to occasion a nightmare, and during waking to produce a vague anxiety. On the other hand, we see that joyous thoughts dilate our blood-vessels, and that a suitable quantity of wine, because it dilates the vessels, also disposes us to joyous thoughts” (James [1890] 1981, 462).

This does not mean that we can create a user experience by solely forcing our end-user to perform some particular movement or that we will always be “joyous” when we drink wine. Mimicry alone, as in mirroring facial expressions or movements, will not necessarily be enough to create an emotional experience. All the different parts need to be there for emotion to arise, and we cannot deterministically spur changes in our internal organs or hormone levels through portraying certain movements or facial expressions. But as Sheets-Johnstone says, there is a dynamic congruency between movement and emotion (M. Sheets-Johnstone, 1999) – and thereby by emotion and thought and action. By studying emotion as if it was a still picture, we miss its temporal aspects and, more importantly, its outcome: to motivate action.

By characterizing a non-dualistic/holistic stance as ignoring the tension between mind and body, the authors are, in a sense, going back to a dualistic stance. What is the “mind” that they are speaking of? If we see the mind as consisting of movements in the nervous system, interconnected with other movements, as those introduced by emotion, hormones, muscular or autonomous system reactions, then the whole idea of a mind as being in an oppositional or tensed relationship to a body becomes a non-entity. A non-dualistic design stance is not the same as refuting that there are different body parts, different processes, different interactions with our surrounding world. A non-dualistic stance simply says that they are interrelated in intricate and interesting ways.

Furthermore, what I think is also lacking from the theoretical starting points here is the notion of learning. A “body” is not a given, pre-defined state. Our bodies change with the experiences we are exposed to. We train our muscles in certain ways when holding our breath; the looseness in our musical performance can be changed and improved by attending to and challenging different aspects of the execution; or we can improve on our compassion for those who have been bullied. The material we design with is not only users’ autonomy but how users’ autonomy changes in and through the interactions we expose users to. The body is malleable. As are the digital, interactive materials.

5. Summary

The work described in this paper is an important design exploration of users’ autonomy, challenging the old HCI-ideals of complete end-user control. By designing systems that take as input semi-autonomous bodily processes or introducing elements of randomness in the loss of control over musical performances, the authors are opening a rich space of possible design work. The authors state that: “We therefore recognise that multiple states of mind-body connection and separation are possible, often inevitable and even desirable as part of embodied experiences.” What we might want to add is the explicit articulation of the esthetic ideals that underlie their explorations. While those ideals can be understood from reading in-between the lines, the risk is that the work comes across almost as a mechanistic manipulation of users’ autonomy along three dimensions without telling us why such manipulations are, as they say, desirable?

Additional information

Funding

This work was supported by the Marianne and Marcus Wallenberg Foundation [MMW 2019.0228]; Stiftelsen för Strategisk Forskning [RIT15-0046]; Vetenskapsrådet [2016-04709].

Notes on contributors

Kristina Höök

Kristina Höök is a professor in interaction design at KTH. Her research focuses on designing with the body, in particular the design stance soma design.