Counterfactual Thinking: What Theories Do in Design

ABSTRACT

This essay addresses a foundational topic in applied sciences with interest in design: how do theories inform design? Previous work has attributed theory-use to abduction and deduction. However, design is about creating an intervention, a possible state that does not exist presently, and these accounts fail to explain how theories permit taking this leap. We argue that the practical value of a theory lies in counterfactual thinking. Theories are like “speculation pumps”: they produce (pump) counterfactual thought experiments of the type: If design was <like this>, then interaction would be <like that>. The more valid these thought experiments are and the better they direct the solution of design problems toward desirable and reliable outcomes, the more useful the theory. Counterfactual thinking sheds new light to design methods and, importantly, can reconcile an underlying tension between design sciences and applied sciences.

1. Introduction

The question of how theories inform design is foundational to research on disciplines interested in designing technology for human use, including ergonomics, design research, and human–computer interaction (Card et al., 1984; Carroll, 1997, 2003; Carroll & Campbell, 1986; Carroll & Kellogg, 1989; Dalsgaard & Dindler, 2014; Gaver, 2012; Höök & Jonas, 2012; Hornbæk & Oulasvirta, 2017; John Millar. Carroll, 1991; Kaptelinin & Nardi, 2006; Ling et al., 2005; Nardi, 1996a; Newell & Card, 1985; Redström, 2017; Reeves, 2015; Rogers, 2012; Stolterman, 2008; Velt et al., 2017; Zimmerman & Forlizzi, 2008; Zimmerman et al., 2007). A theory can inspire, foster reflection, and help take a new perspective. A theory can describe, explain, and predict interaction beyond intuition and observations (Shneiderman, 1986).

In this essay, we argue at length that in order to guide decisions in design, descriptions and explanations are insufficient. From a description or explanation of what is, one cannot deduce how things ought to be. Knowledge that is locked to the present state of affairs does not allow entertaining entirely new options. We claim that in order to be useful in design, a theory should help generate and refine design ideas, take new stances and, critically, advise decisions on “what ought”. One may think that this is about prediction. The regular way to conceive prediction is that it is the estimation of a future state. However, this characterization misses a central aspect of design: That we are changing that future state via our design. This is the core problem in design and what distinguishes it from natural sciences. While science aims at shedding light on the existing and the universal, design aims to create the yet-unexisting and the particular (Stolterman, 2008). The goal of this essay is to shed light on one under-studied aspect of theories that contribute to their usefulness.

The question of what theories do in design is controversial. Two camps exist on the matter. For the sake of the argument, we call them the scientific and the pragmatist camp. Early research in design was influenced by the notion of the sciences of the artificial (Simon, 1996), which elevated theory, models and simulation as the prime way to inform design. Design should be informed by deriving implications from first principles, in particular models of how the human mind works. Calculations based on models would allow estimating the properties of a solution (Card et al., 1984). Such models could also be used after design, for instance, in evaluation with (Hartson et al., 1999) or without users (Polson et al., 1992). Simulations would imitate reality to predict how it reacts to change. In engineering design this view is still strong (Cross & Roy, 1989; Pahl & Beitz, 2013), but in particular in human–computer interaction the scientific camp was marginalized for a while (Oulasvirta, 2019).

After the early successes of the scientific camp (Newell & Card, 1985; Olson & Olson, 1990), criticisms were raised for this type of research to be limited in scope, unscalable, and lagging behind technology development (Carroll, 1997; Carroll & Campbell, 1986; Green et al., 1996; Sutcliffe, 2000). Carroll (1997), for instance, argued that human-computer interaction is not “merely applied psychology” (p. 62) but a field that challenges and helps create basic science because of its focus on design. At first, it was insisted that the idea that design decisions are derived or simulated needs to be relaxed. It was proposed that documenting rationale behind design decisions is needed, even if theories were not available to back up each decision (Moran & Carroll, 1996). At the same time, calls to get “real” (Landauer, 1991), “situated” (Suchman, 1987), and “wild” were voiced (Rogers, 2012). These criticisms led to the formation of an alternative, pragmatist stance. It emphasizes designerly ways of knowing that are often tacit but manifest in the practice of design “in the thick of it” (Rogers, 2012). According to this view, theory has little or no value in vitro, only in vivo. This has also been called a craft view (Long & Dowell, 1989). In this view “craft knowledge is acquired by practice and example, and so is experiential; it is neither explicit nor formal” (p. 9). Some saw the traditional criteria held for a good theory – such as validity – as secondary (Gaver, 2012). Instead, it was argued, what matters is how theories inspire, reflect, and help a designer in practical choices. The designer is seen as an agent embodying theories, as opposed to being subjected to using them (Redström, 2017; Zimmerman et al., 2007). Consequently, it is practically impossible to trace the role that a theory had in a particular design decision (Gaver, 2012). The field needs a view that can reconcile these two camps.

This essay develops a particular perspective on what theories “do” in design. We propose a view on information that a theory contains, which can be used in design, as well as the reasoning offered to drive its use. To this end, we will be mostly looking at two elements of a theory in this paper: (1) propositions – that is, claims about the world – that theories contain and (2) methods for reasoning. We scope our analysis to the design of interactive artifacts for human use, and leave broader discussion of design thinking and design processes for future work. Although our main focus is on theories, we do touch on many forms of knowledge-formation from experiments to innovation. Note that our view is deliberately asituated. We want to understand the general potential of a theory to inform design. Although each project has its unique characteristics, exposing the logical potential of a theory to contribute to design can be illuminating.

1.1. Counterfactual thinking

Our argument centers on counterfactual thinking. Counterfactual thinking refers to “what if” thinking, or thoughts about the likely consequences of a hypothetical event (Lewis, 2013). Hypothetical events are alternatives to real events that could have happened. In the context of design, counterfactual thinking is a form of epistemic activity that bridges theories and actions, such as interventions and design choices. The essential idea is a prediction of a possible – but not necessarily previously observed – decision’s consequence to humans; what the decision implies in interaction and to people.

One can think of theories as metaphorical speculation pumps, a concept that is inspired by Daniel Dennett’s “intuition pumps” (Dennett, 2013). By reasoning with a theory, a designer “pumps” thought experiments (speculations) on the consequences of how a design could be organized (Figure 2). By pumping propositions, a designer can identify design choices with desirable consequents. Propositions also support reflection on the consequents of design and the design space. From this viewpoint, the counterfactual reasoning afforded by a theory determines its value in design. From any presented theory, one pumps counterfactual propositions by reasoning. How pumping happens differs? Mathematically represented models further allow dedicated reasoning methods, such as simulation and numerical derivation. As we discuss here, many other forms exist, some that one might not normally think of as design knowledge (see Figure 1)

Figure 1. Comparison of counterfactual reasoning with abduction and deduction

.

Figure 2. Theory as a metaphorical speculation pump: To “use” a theory in design, a designer engages in speculation, or counterfactual thinking: if design was a, then interaction would be c. We argue that it is through counterfactual propositions that theories help solve problems, elaborating objectives, and redefine assumptions

Counterfactual reasoning is a type of reasoning that goes from a theory to a possible – but not yet existing – world. This type has gained much less attention. By contrast, the logical basis of theory-use in design has been often attributed to abduction, which goes from observations to theory, and deduction, which goes from theory to world (see Figure 1).

A counterfactual proposition in design can be expressed as follows:

$\mathrm{I}\mathrm{f}\mathrm{d}\mathrm{e}\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}\mathrm{w}\mathrm{a}\mathrm{s}\mathbf{a}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{n}\mathrm{i}\mathrm{n}\mathrm{t}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{w}\mathrm{o}\mathrm{u}\mathrm{l}\mathrm{d}\mathrm{b}\mathrm{e}\mathbf{c}$

The argument has two parts: an antecedent (a) and a consequence (c), linked by the counterfactual step. In short, this can be written as $a>c$. When “a” concerns design and “c” concerns interaction, we have a counterfactual proposition that can be used to inform design. Note that the antecedent “a” has no necessary basis in current reality. While this formulation is simplistic – in particular in the sense that it does not expose the system of other propositions that one must account for when designing – it does make clear the possible world semantics of counterfactuals (Lewis, 2013), and exposes a unique problem in design. For a review of propositional reasoning, see Johnson-Laird et al. (1992).

This general type of reasoning is long-recognized in fields like engineering and political science, as well as in history and psychology. While formal treatment of counterfactuals is beyond the scope of this paper, it is important to note that there are formalisms in statistics for counterfactual reasoning, such as Judea Pearl’s do-operator for causal graphs (Pearl, 2000). Pearl defines a counterfactual as a proposition of the form: “The value that Y would have obtained, had X been x” (Pearl, 1999). Beyond offering a tool for counterfactuals, Pearl’s treatment of the topic points out that this type of reasoning is susceptible to confounding factors. Here, to permit wide discussion of design-informing theories, we refer to counterfactual thinking as a relaxed form of counterfactual reasoning that does not require a formal propositional system.

The rest of this essay studies counterfactual thinking to shed new light on what theories “do” in design. We develop a treatment of counterfactual thinking in design. Three broader arguments are made: (1) that counterfactuality, not abduction nor deduction, is the main type of reasoning that links design to theory; (2) that the constructive power of a theory, its value as a “pump” in design, boils down to properties of the counterfactual propositions it produces, which can be further assessed; and (3) that, in practice, designers combine very diverse counterfactual capacities and practices, and some combinations lead to fallacious propositions or preclude theory-use.

The paper will conclude that reconciliation between the scientific and the pragmatist stance is possible if antirealism is rejected with the idea that theoretical propositions do not refer to the world. By accepting that theoretical propositions have truthlikeness, we can make more systematic use out of theories in design, even when they are understood from the pragmatist perspective. Vice versa, the scientific camp benefits from the analysis of practices that foster and prevent proper theory use. If our claim about the centrality of counterfactual thinking in design is accepted, it will direct research to consider, formalisms such as causal graphs or simulation models, that allow rigorous treatment of counterfactuals.

2. Assumptions

This section lays the groundwork on which the rest of our arguments stand: We claim that, epistemically, design is counterfactual reasoning. Our reasoning goes:

• A design implies a possible world or worlds

• The choice of a design must thus be based on some belief that informs how good the choice will be

• Theories can produce those beliefs via counterfactual thinking

These claims entail two commitments: the possible worlds assumption and scientific realism.

First, we need to agree that design is a constructive activity: It aims at the construction of an artifact, conceptual or experimental prototypes, or plans or proposals for design (Gaver, 2012). We can call any activity where artifacts and plans for artifacts with desirable qualities are created for design. In this sense, there is nothing special in design; It is not limited to professional, research, artistic, or wicked settings. We encounter design when a gain function of a mouse is tuned or when new concepts for smart cities are proposed.

A stronger claim is the possible worlds assumption. It suggests that, epistemically, design is about choice, and moreover about possibilities that do not exist yet but that might. There are numerous possible future states of affairs, and the designer’s choices shape, which one will realize The artifact that is being designed does not exist in the present world but steers toward a new possibility of how we might live. The act of manufacturing or implementing an artifact, by contrast, does not center on speculation about possible worlds, although it results in a concrete object. The idea that design is about changing the future is broadly accepted in current views of design (e.g., Gaver, 2012; Zimmerman & Forlizzi, 2008). Research-through-Design is explicit about this. Design is about what might be, or about making the “right” thing, as opposed to an activity about making statements about what is (Zimmerman & Forlizzi, 2008). Note that a counter-argument can be made that design is not limited to possible worlds. Design can also be about representing the present or the past. However, we believe that the possible worlds assumption is central to human–computer interaction and other fields where constructive efforts aim at artifacts that improve the human condition.

There are many ways in which designers can approach possible worlds. One approach, albeit a hopeless one, is to pick a random design and see what happens. A designer throwing a dart would have $4\cdot {10}^{26}$ keyboard layouts to try to come up with the Qwerty layout. This thought experiment suggests that the designer needs some way to anticipate the right consequences of their choice and avoid the wrong ones. Counterfactual thinking lends this vital anticipatory capacity.

Counterfactual thinking, at the most general level, is about likely consequences of hypothetical events (Lewis, 2013): “If some conditions were (hypothetically) changed to this, these consequences will/may occur.” When applied to design, it engages the designer with a speculation on the possible consequences of a design choice. A counterfactual proposition is a conditional that links an antecedent (“If speech was used for typing”) with a consequent (“drivers could better keep their gaze on road”). We denote the counterfactual step with the operator $>$:

$\mathrm{A}\mathrm{n}\mathrm{t}\mathrm{e}\mathrm{c}\mathrm{e}\mathrm{d}\mathrm{e}\mathrm{n}\mathrm{t}>\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{s}\mathrm{e}\mathrm{q}\mathrm{u}\mathrm{e}\mathrm{n}\mathrm{t}$

or $a>c$ for short.

Theories can serve as the basis of counterfactuality. A theory provides some deterministic way to link the antecedent and the consequent (Hornbæk & Oulasvirta, 2017). Fitts’ law (Fitts, 1954), as a practical example that is well known, has four variables describing antecedent conditions: (1) $D$, or distance to target from present location, (2) $W$, or width of target from present angle-of-approach, (3) free parameter $a$, and (4) free parameter $b$. A consequent, or outcome, describes events that occur during or after interaction, such as user performance, errors, or satisfaction. In Fitts’ law, the consequent is $MT$, or movement time. Many other types of counterfactual reasoning are discussed in the next sections.

However, counterfactual thinking most often cannot fully determine a design. What makes design complex is that it “involves many different decisions, dealing with many different and potentially independent factors of an artefact, all situated within the specific circumstances of production and use” (Gaver, 2012, p. 940). However, to the extent that the ultimate decisions achieved with a theory are better than what would have been achieved without it, it has what we dub constructive power. Turning this upside down, the artifact to some degree embodies the speculative assumption of the theory. It becomes a “theory nexus” (Carroll & Kellogg, 1989). The underpinning counterfactual reasoning is thus reflected in the quality of the designed artifact.

Several writers distinguish different types of theory (Bederson et al., 2003; Rogers, 2012). Bederson and Shneiderman separate descriptive, explanatory, predictive, prescriptive, and generative theories. In our view, this is not different types of theories but different uses of theory. All these types of uses may occur in trying to do counterfactual reasoning about a possible design. For now, we therefore assume that any one theory may play multiple roles in design and that many theories may support counterfactual reasoning.

This takes us to the last, and most contentious, assumption of this section: The truthlikeness (Hilpinen, 1976; Niiniluoto, 2012) of a counterfactual tells how similar the proposition is to reality. Formally, one can think of truthlikeness as some function $\mathrm{\Phi }$ that assigns a counterfactual proposition a truth value $t$:

$\mathrm{\Phi }:(a>c)\to t$

The position that theoretical propositions are truthlike is developed in scientific realism (Niiniluoto, 1999; Psillos, 2005). Theories contain information about the world that is independent of the scientist’s opinion. In design, propositions that are truthlike will help make better predictions on what may happen when a choice is taken. Truthlikeness helps developing a design as a hypothesis on how “good” or “right” the possible world might be.

This assumption is contested in two ways. First, it may be claimed that theories do not need to be truthlike to be useful in design. This view is correct: The pursuit of truthlikeness may curb creative thinking where one needs to take distance from the present world. However, rejecting thruthlikeness also leads to a disconnect between design and science, because no verifiable connection remains between a theoretical proposition and a choice in design. Thus, we need truthlikeness to understand what theories do in design in an abstract way that is not contingent on the particular designer and the particular context.

Second, one can reject the notion of truth in the context of scientific theories. However, the notion of truthlikeness is softer: truthlikeness refers to there being some verifiable resemblance between observations and theoretical propositions (Niiniluoto, 1999). This view is not the same as positivism, a view that is generally dismissed in social sciences and design, which insists that only propositions that can be rigorously verified or formally proven are scientific. The notion of truthlikeness admits the imperfection of theoretical propositions, the biases, measurement errors, and flawed constructs. It calls for a critical stance. Science progresses where those imperfections are identified and addressed in theories and methods. It is thus entirely possible to entertain a view that theories contain truthlike propositions useful in design without subscribing to positivism.

3. Theories and counterfactual thinking in design

If the claim is accepted that theories inform design via thought experiments, the question follows: so what? What do these thought experiments “do” in design? We identify three principled uses:

1. Theories contain informative propositions that can direct design choices

2. Theories solve design problems by identifying (or biasing toward) the best choice among a set of options

3. Theories help rethink design problems by exposing novel design spaces and phenomena

We draw these together in a definition of a theory’s constructive power. While this section talks about theories, exemplifying the main points with a few known theories, in the next section we extend this discussion beyond theories to discuss other forms of inquiry. Our focus here is on theories that make explicit statements about design and interaction, for example, coming from psychology and engineering. Philosophical or sociological theories, or theories that do not aim to talk directly to design, are left out of scope here.

3.1. Theories inform choices

A theory can be seen as a collection of claims (Niiniluoto, 1999). There are two types of claims of interest here: counterfactual and others. The counterfactual propositions of a theory can be used to inform design. The non-counterfactual propositions are silent about design. However, as we argue below, a theory can be potentially design-informing even if it does not contain explicit counterfactual propositions. When combined with empirical observations, or some other way of enriching the set of propositions, one can deduce more counterfactual propositions.

There are many types of design-informing counterfactual propositions present in the theories of HCI and ergonomics. While the following section provides more examples, we here give three stereotypical examples.

Human performance models, cognitive models, and many others follow a general form that maps task conditions or design to some effect on users:

In specific variant, a feature of design is what affects the consequent:

Consider for example, Threaded Cognition (Salvucci & Taatgen, 2008). It is a theory of multitasking based on ACT-R, one of the most popular cognitive architecture models applied in HCI. Threaded Cognition explains conditions in which two or more tasks interfere with each other. To the extent that task conditions $t$ can be determined or shaped by design, this type of proposition is design-informing. Threaded cognition, for example, explains why different modality combinations lead to interference, a factor that designers can influence.

Theories of situated interaction, activity theory, distributed cognition, yield counterfactuals of this type:

$\mathrm{a}(\mathrm{S}\mathrm{i}\mathrm{t}\mathrm{u}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n})>\mathrm{c}(\mathrm{S}\mathrm{i}\mathrm{t}\mathrm{u}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n})$

The difference to human performance models is that the antecedent situation can contain aspects of the consequent situation. While this may sound tautological, it is not necessarily so. If even a single variable in $a$ is different from $c$, a theory can yield non-trivial consequences.

3.2. Theories can solve full or partial design problems

What does one do with a set of speculations? A single counterfactual proposition is potentially design-informing: it can help take a decision regarding some features of a design. It can fix one or more dimensions of a design problem. In the case of multiple such propositions, they contain information for fixing multiple dimensions. For example, to design a graphical layout, one needs to fix the locations, colors, and types of elements on the canvas.

Some theories have the form

$a>c\Rightarrow \mathrm{\forall }a:a\Rightarrow c$

In other words: if the antecedent is present (in some design), consequent c will follow. Consider design guidelines for example. Design guidelines are simple rules, commonly used by practitioners, that tell which kinds of designs to favor. For example, designs that provide feedback or help users recover from errors should be favored. By contrast, if the guideline is violated (antecedent), it is assumed that usability will be compromised, although guidelines do not say exactly how. Models and simulations can express the $c$ part in more detail. For example, cognitive models (e.g., ACT-R, bounded agent models) can predict consequents related to task completion time, errors, or cognitive load.

If one has not one but a set of counterfactual propositions, larger problems can be solved. Counterfactual propositions can drive design if it is seen as optimization: the identification of best (or acceptable) design among some set. Here, the design space $D$ is the Cartesian product of design variables $\{x_1,\cdots ,x_n\}$. Assuming design objectives $\{o_1,\cdots ,o_n\}$, to “solve a problem” is to find the set of variable values that maximize those objectives. Counterfactual propositions that map variable values to objective values can identify the best design; in other words, the O-maximizing X–vector in $D$. For a review of optimization methods for the design of user interfaces, see Oulasvirta et al. (2020).

3.3. Theories can help explore design spaces

So far, we have focused on problem-solving in design. Here, counterfactual thinking centers on possible consequences of a design choice. Theories can also help explore and catalog the options available in design. Here, however, we reach the boundaries of counterfactual thinking.

There are two ways in which theories can chart design ideas. In the weaker form, the connection between an antecedent and its consequent is broken. Consider a theory that generates consequents (c) with no reference to antecedents (a). For example, many taxonomies in HCI are like this (e.g., Card et al. (1991)). Such theory-use essentially enumerates consequents but without saying in which conditions each consequent should be preferred in design.

In the stronger form, an antecedent is mapped to a set or distribution of consequents. There is a connection between (a) and (c), but it remains plural and therefore ambiguous. For example, imagine a designer that reads about game theory and imagines several online auctioning interfaces for the prisoner’s dilemma. Is this counterfactual thinking? It depends. If the design ideas were produced by reasoning with the theory, yes. If, on the other hand, the theory directly declared ideas on how to build such designs, without specifying preceding conditions, then no.

3.4. Theories can redefine problems

A theory can also help a designer redefine a design problem and take a new view. This takes place in two general ways. First, as mentioned above, a theory can expose aspects of the design space that were not known to the designer. For this, no counterfactual reasoning is needed. Any proposition that refers to antecedents (a) that were previously unknown to the designer suffice.

Second, a theory can expose design objectives or interesting consequences of a design. Again, counterfactuality is not necessary. Any proposition that refers to consequences (c) that is novel to the designer suffices. Consider, for example, a theory of accidents in aviation, such as the domino theory, the normal accidents theory, or the Rasmussen’s SRK model (Wiegmann & Shappell, 2001). These theories can inspire a designer to place a higher weight on error-prevention mechanisms in the design of a control panel.

These arguments imply that scientific theories can be a basis for reflection and designerly ways of knowing. Designers shift between problem-definitions and problem-solutions. New perspectives on design can be as important as direct solutions. As noted, these sorts of reasoning may break the connection that a strict definition of counterfactual reasoning insists.

4. Constructive power

With these definitions, we can turn to assess the potential value that a theory has in design. Constructive power of a theory $T$ can be assessed in a design project, assuming that we know its (theoretical or practical) design space and design objectives: The more precisely the counterfactual propositions of theory $T$ steer toward decisions that yield desirable outcomes to end-users, the higher its constructive power. Constructive power tantamount to the increase in a designer’s ability to produce good designs: Given the theory, how much better designs can be produced? Thus, a useful theory speaks about those factors that are actually important for good outcomes – about user, design, context, task, and so on. If the propositions that link conditions to outcomes are ambiguous or flawed, the constructive power is low.

This reasoning suggests a few characteristics of a design-informing theory: It (1) contains speculations that talk to the most relevant design objectives; (2) contains speculations with truthlike consequents (c); (3) contains few if any speculations with (unexpectedly) poor outcomes in interaction. Note that whether and which propositions a designer actually chooses to use is not part of constructive power. It is a “best-case analysis”. In any real project, a designer may fail to achieve the full potential, as we discuss below.

4.1. Example: Fitts’ law

We provide a concrete and deliberately simple example to illustrate the above points. Fitts’ law (Fitts, 1954) is a candidate for the most widely used theory in HCI. It is a theoretical proposition and a quantitative model developed over decades of research on interaction techniques and input devices. But what kind of speculation pump is it?

Building on information theory, Fitts’ law assumes that the human motor system is a capacity-limited communication channel. The smaller ($W$) or more distant a target ($D$) is, the more information the motor system must express to select it. Speeding up movement compromises accuracy, as throughput stays constant. This trade-off is captured in a regression model of movement time: $MT=a+b{log}_2(D/W+1)$.

First, what are its speculations about? The equation makes visible the general structure of its propositions:

$\mathrm{a}(\mathrm{P}\mathrm{o}\mathrm{i}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{d}\mathrm{e}\mathrm{m}\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{s})>\mathrm{c}(\mathrm{S}\mathrm{p}\mathrm{e}\mathrm{e}\mathrm{d}\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{a}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y})$

The antecedent is determined by the spatial organization of an interface (sizes and distances), and the consequent yields speed and accuracy of aimed movements. The two are linked by the formula. Thus, the law’s counterfactual propositions are about spatial organizations of buttons and their consequence to a user’s performance.

Second, how does one “pump” useful speculations? Fitts’ law allows pumping by algebraic operations. By fixing a value of any of the variables, one can solve for any of the other: $D$, $W$, or $MT$. For example, spatial designs can be compared by plugging in corresponding $D$s and $W$s, and input devices can be compared for performance at different ranges of target dimensions.

How (constructively) powerful is Fitts’ law, then? It is relevant to a very low number of decisions: two spatial parameters of a selection task. However, if applied to the design of a selection region, it makes the trivial suggestion that it should be made as large and possible and close to the pointer. However, when a designer must decide on many selection regions, Fitts’ law can help deciding how to balance their sizes and distances on a layout. Although Fitts’ law talks about only two parameters, and is limited to motor performance, its ubiquitous application possibilities make it useful. Aimed movements are the prime way of expressing intention to computers, and efficiency is a central objective in design. Moreover, the law is trivial to apply, and it has relatively high truthlikeness; only a few exceptions to it are known. Fitts’ law, then, appears like a handy “mini-pump” for UI design.

5. A counterfactual way of understanding design practice

Equipped with these concepts, we now turn to diagnose some textbook-level branches of design-related methods, from design heuristics, over field studies, to modeling. While we do not claim our analysis to be comprehensive – we admittedly miss many types of knowledge-formation – it uncovers some principally different ways in which knowledge informs design. These differences are rooted in two main sources of counterfactual reasoning: the counterfactual propositions contained in the theory and the reasoning apparatus that allows pumping them.

Three general observations are made: (1) that popular speculation pumps manifest two recurring deficiencies: narrow scope and weak (and even outright flawed) reasoning; (2) that some forms of research counted as empirical, and not constructive, such as experimentation and field studies, can produce design-informing propositions; and (3) that counterfactual reasoning is a mine field to the unassuming. Going from theory and observations to speculations can be fraught with logical pitfalls. We do not yet understand this well beyond the most straightforward cases.

5.1. Design heuristics

Design heuristics are rules-of-thumb describing conventions and maxima for design. Numerous heuristics have been presented, arguably the most famous set being the heuristics of Nielsen (2005), derived from work by Nielsen and Molich (1990). In textbooks of interface and interaction design, heuristics are recognized as a key type of knowledge. But how do they achieve this and are they powerful as speculation pumps?

A heuristic is a counterfactual proposition that links some property H to some consequence in interaction:

$\mathrm{a}(\mathrm{D}\mathrm{e}\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}\mathrm{h}\mathrm{a}\mathrm{s}\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{t}\mathrm{y}\mathrm{H})>\mathrm{c}(\mathrm{S}\mathrm{o}\mathrm{m}\mathrm{e}\mathrm{t}\mathrm{h}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{g}\mathrm{o}\mathrm{o}\mathrm{d}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{n}\mathrm{t}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n})$

The famous visibility heuristic by Nielsen (2005) is a good example: “The system should always keep users informed about what is going on, through appropriate feedback within reasonable time.” Taken as a counterfactual:

$\mathrm{a}(\mathrm{D}\mathrm{e}\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{v}\mathrm{i}\mathrm{d}\mathrm{e}\mathrm{s}\mathrm{f}\mathrm{e}\mathrm{e}\mathrm{d}\mathrm{b}\mathrm{a}\mathrm{c}\mathrm{k})>\mathrm{c}(\mathrm{U}\mathrm{s}\mathrm{e}\mathrm{r}\mathrm{k}\mathrm{n}\mathrm{o}\mathrm{w}\mathrm{s}\mathrm{s}\mathrm{y}\mathrm{s}\mathrm{t}\mathrm{e}\mathrm{m}\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{t}\mathrm{u}\mathrm{s})$

These examples expose the inherent weakness of heuristics: as speculation pumps, they contain very little information. A heuristic is but a single proposition.

There are at least three principled ways to reason with heuristics in design. First, although logically false, heuristics may be used to reason for the contraposition; that is, if the antecedent is false, negative consequences follow:

$\mathrm{a}(\mathrm{D}\mathrm{e}\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}\mathrm{d}\mathrm{o}\mathrm{e}\mathrm{s}\mathrm{n}\mathrm{o}\mathrm{t}\mathrm{h}\mathrm{a}\mathrm{v}\mathrm{e}\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{t}\mathrm{y}\mathrm{H})>\mathrm{S}\mathrm{o}\mathrm{m}\mathrm{e}\mathrm{t}\mathrm{h}\mathrm{i}\mathrm{n}\mathrm{g}\mathrm{b}\mathrm{a}\mathrm{d}$

For example, in the case of the visibility heuristic: if appropriate feedback is not offered, the user may not know the system status. This is logically false, because there may be other ways in which the user is aware of the system status.

Second, heuristics can be used for evaluation of design ideas:

$\mathrm{H}\mathrm{i}\mathrm{s}\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{s}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}\Rightarrow \mathrm{T}\mathrm{h}\mathrm{e}\mathrm{d}\mathrm{e}\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}\mathrm{i}\mathrm{s}\mathrm{g}\mathrm{o}\mathrm{o}\mathrm{d}$

However, this is not a design-generating use of a heuristic. It is mute about how to implement H. For example, implementing the visibility heuristic in a complex information display requires careful understanding of human visual attention.

Third, although a full design is never specified by a heuristic, one can generate a partial solution with a heuristic. In this reasoning, the suggested property H is added to the design. For example, one could add “appropriate feedback within reasonable time” to a system. However, many of the heuristics that we know of are underspecified. They do not determine the exact way in which that property should be implemented. To design “appropriate feedback”, many design parameters must be decided, from modality to content and timing.

Because of the inherent weakness of heuristics, one may want to use multiple in concert. This leads to another potential fallacy. An important property of counterfactual reasoning is that from $a>c$ it does not follow that a $a\wedge b>c$. That is, the counterfactual may not hold when another antecedent enters. Nielsen’s consistency heuristic may not hold when combined with the visibility heuristic. While consistency asks for predictability, to make a critical status message attended, this predictability has to be disrupted. Without extraneous information, such issues cannot be resolved.

To sum up, although heuristics have low information content and questionable validity, they do allow speculation about which specific features that could be added to a design. When many heuristics are combined, increasingly many features can be added to the design. However, heuristics also easily lure designers to fallacious reasoning: heuristics are underspecific. They can be implemented in numerous ways, some of which may lead to unresolvable contradictions in the design.

Finally, there is scarcely any evidence for the truthlikeness of heuristics that could be considered rigorous, although there have been attempts at evidence-based heuristics. When combined with the problem of underspecification, a designer that relies on heuristics might create design instances that simply do not work.

5.2. Design standards and examples

Design standards include collections of heuristics and example designs. Throughout the history of operating systems, for example, numerous standards have been presented (e.g., ISO 9241, Apple’s style guides, Smith and Mosier’s collection of guidelines). Google’s Material Design¹ is a representative example. It consists of several “patterns” that address design issues like confirmation, data formats, errors, gestures, navigation, permissions, search, settings, etc. These patterns consist of goals (“try to ensure your design is like this”), heuristics (“Minimize errors by designing apps that make it easy for users to input information flexibly”), and concrete examples. Sometimes they are accompanied by counterexamples and contextualizing information. The part of a design standard that function as a rule of thumb faces the same shortcomings as heuristics in general. However, having a coherent set of heuristics and examples can dodge the issue of contradictory heuristics. A coherent set of heuristics pushes speculation toward a safer subset of the feature space.

More interesting is the case of design examples. Designers routinely collect, curate, and utilize examples in their practice. Examples like screenshots and videos are powerful, because they contain much more information than a heuristic. They pinpoint several properties. Examples, we argue, invite speculation by analogical reasoning. This implies that changing the shared design parameters in the exact way the example tells, makes the design conform to the standard:

$\mathrm{a}(\mathrm{D}\mathrm{e}\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}\mathrm{f}\mathrm{e}\mathrm{a}\mathrm{t}\mathrm{u}\mathrm{r}\mathrm{e}\mathrm{s}\mathrm{a}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{E}\mathrm{x}\mathrm{a}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{e})>\mathrm{c}(\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{s}\mathrm{t}\mathrm{o}\mathrm{S}\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{a}\mathrm{r}\mathrm{d})$

Design patterns (Borchers, 2008; Dearden & Finlay, 2006) use a similar reasoning, whereby abstracted patterns of essential elements of a design is used for inspiration and for mimicking. All of these examples raise the question: why conform to the standard? By implication of being part of the standard, the design is (fallaciously) expected to be good. However, standards rarely come with evidence for usability. The only reason we can come up it is usability-by-association: by merit of following a standard, users who use other designs in that standard, may find the design more predictable and therefore easier to use.

Whereas design heuristics were under-specifying, a design example is over-specifying. That is, unless the design constraints and objectives are almost exactly the same, this analogical step cannot be taken. Some features are so different that the example does not fully transfer. And if it does not, the designer must creatively come up with changes. This is where design examples and standards break down.

5.3. Mathematical models

A long tradition exists in mathematical modeling of elementary aspects of interaction such as aimed movement, choice, learning, and navigation. A mathematically expressed model in this space is often a statistical model, such as a regression model, or a computational model executed stepwise as a computer program. Fitts’ law and most other models of human performance are good examples of regression models. Mathematical models in HCI include models of decision-making, mental reasoning, motor performance, choice, and learning.

But how do such models “speculate” about design? A nonlinear function can represent much more information than a heuristic. In theory, a continuous real-valued function affords an infinite number of speculations, although in practice only a few of them are practically meaningful. To be useful for design, the variables and parameters of the model must refer to some design-related antecedents (a) and some interaction-related consequents (c). Models that do not fulfill this condition are not design-generating. To then solve a concrete design problem with a mathematical function, a design feature that best matches a desirable outcome must be identified. This can happen analytically, such as in the case of Fitts’ law, or by numerical methods in the case of more complex functions.

Although mathematical models offer a principled way to speculate, the models we have for the design of technology have a low number of design-related variables, limiting the speculations they allow (Hornbæk & Oulasvirta, 2017).

5.4. Simulation models

Simulation models are step-wise executed programs. What distinguishes simulators are their representational power and fidelity. Every step in the program is essentially a function. Examples include cognitive models like GOMS and ACT-R, neural network models, computational rationality, biomechanical models, and dynamic models. Like mathematical models, simulation models map model parameters to outcome and process predictions. If random variables are involved in the simulator, the outputs can fluctuate randomly even when parameters are fixed. Simulation models differ from mathematical models in one crucial respect.

While some models define the relation between antecedents and consequences through formula, simulator models define them through algorithms. They are essentially black boxes. Analytical solutions like in the case of mathematical models are mostly impossible. This has an unfortunate consequence for speculation. In principle, a simulation produces only one counterfactual proposition:

$S(\theta )\to a>c$

When simulation $S$ is run with parameters $\theta$, it produces the counterfactual proposition $a>c$.

Does this mean that simulations are inherently limited in their speculative capacity? Not necessarily. We can imagine a meta-simulator that goes through multiple designs as part of the simulation and outputs all antecedent–consequent propositions. This idea has been exploited in computational design where simulators are used in the objective function. An algorithm can speculate as many candidate designs in a predefined design space as its computational budget allows, and it can evaluate them by means of simuation. This is a very powerful way to generate design ideas, but also challenging to implement.

5.5. Experimental research

Our decision to include experimental research here as a speculation pump may be surprising at first. However, it becomes less so when remembering that experimental research is carried out because it allows counterfactual reasoning (Shadish et al., 2002). The prototypical form of a controlled study has two conditions, between which only one variable changes. It could be for example, the presence of some system or interface feature. Experiments rely on ceteris paribus reasoning: all other being equal, the cause of any difference observed between the two conditions can be attributed to that single variable.

While not all experiments produce speculations for design, some can. We are thinking here of experiments that produce insights about interaction, and not formative evaluations or quick-and-dirty think-aloud tests. An experiment that manipulates a design-related variable D produces a set of counterfactual propositions of the type:

$\mathrm{a}(\mathrm{D}\mathrm{e}\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}\mathrm{h}\mathrm{a}\mathrm{s}\mathrm{D}=\mathrm{d})>\mathrm{c}(\mathrm{I}\mathrm{n}\mathrm{t}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{i}\mathrm{s}\mathrm{O}=\mathrm{o})$

That is, when design variable is set to $d$, we expect interaction to follow empirical observations from that condition $o$. The larger the experimental design, the larger number of such propositions are produced.

This analysis exposes experimental research as design-informing activity in HCI. This is hardly surprising to anyone who carries out experimental research. However, it is surprising within the decades of debates on design knowledge that has not recognized this role, but kept empirical research separate from the epistemic activities that generate design ideas.

This analysis also exposes the inherent weakness of point studies, a common and berated type of evaluation. In a point study or single-shot study, a prototype is evaluated but no baseline condition is present. Importantly, the result cannot be attributed to any single factor or design variable. The only counterfactual reasoning made possible is

$\mathrm{a}(\mathrm{D}\mathrm{e}\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}\mathrm{i}\mathrm{s}\mathrm{e}\mathrm{x}\mathrm{a}\mathrm{c}\mathrm{t}\mathrm{l}\mathrm{y}\mathrm{D})>\mathrm{c}(\mathrm{I}\mathrm{n}\mathrm{t}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{i}\mathrm{s}\mathrm{e}\mathrm{x}\mathrm{a}\mathrm{c}\mathrm{t}\mathrm{l}\mathrm{y}\mathrm{O})$

The speculations that are produced are thus overly constrained. they insist that in order to apply this proposition, the whole design must be like in the study. Usability testing is often of this type.

5.6. Field and case studies

Field study is a form of empirical research where an existing system or a prototype is studied in the field. Case studies are empirical studies with a focus on learning from a particular group, design, or organization. Qualitative (e.g., observations, field notes, interviews) and quantitative data can be collected. In the field of human–computer interaction, field and case studies among the most common categories of empirical research (Kjeldskov & Paay, 2012). In which way can a field or case study produce design-informing knowledge? After all, the outcome of a field study is a description of contingent and circumstantial phenomena. It does not afford ceteris paribus logic like experiments.

A field study produces observations of the type:

$\mathrm{a}(\mathrm{o}\mathrm{b}\mathrm{s}\mathrm{e}\mathrm{r}\mathrm{v}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{s})\Rightarrow \mathrm{c}(\mathrm{o}\mathrm{b}\mathrm{s}\mathrm{e}\mathrm{r}\mathrm{v}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{s})$

“In these particular antecedents $a$, these particular consequents $c$ were observed.” This is not a form of counterfactual proposition, however. These propositions are not in themselves design-informing.

However, field studies of prototypes do allow counterfactual reasoning when an argument can be formed how some antecedent would have behaved if it was not as observed in that study. There are plenty of examples like this. A study of cashless payment in London buses (Pritchard et al., 2015), for example, made observations of this type:

a ((Cashless payment) Pc (Fear, compromise of freedoms)

However, by comparison to a reality where cashless payment is not in use, and fears and freedoms not caused, a counterfactual proposition can be reached:

$\mathrm{a}(\mathrm{C}\mathrm{a}\mathrm{s}\mathrm{h}\mathrm{l}\mathrm{e}\mathrm{s}\mathrm{s}\mathrm{p}\mathrm{a}\mathrm{y}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{t})>\mathrm{c}(\mathrm{F}\mathrm{e}\mathrm{a}\mathrm{r},\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{m}\mathrm{i}\mathrm{s}\mathrm{e}\mathrm{o}\mathrm{f}\mathrm{f}\mathrm{r}\mathrm{e}\mathrm{e}\mathrm{d}\mathrm{o}\mathrm{m}\mathrm{s})$

Therefore, even if not explicitly set as experimental research, field studies can support counterfactual reasoning.

However, there is another, perhaps even more surprising way in which case and field studies can be design-informing. This form has been pointed out in the field of future studies (Booth et al., 2009). Here, rich idiographic descriptions of a case study, of events $E$, can permit counterfactual reasoning of the kind of “what if some variant $E{ }^{\mathrm{\prime }}$ had happened instead. This is made possible by detailed knowledge of the $a$ and $c$ in $E$. By increasing the level of description of the various relationships that the two have, a historian can speculate about likely consequences of $E{ }^{\mathrm{\prime }}$. In a way, the thick description of the event becomes a weak simulator; weak in the sense that it does not include deterministic steps but every step is argued for. Many authors of field study reports would, however, be cautious about drawing generalizable conclusions from their studies, for instance, about the importance of specific design features through their contribution to positive outcomes (so-called “implications for design”, see Dourish (2006) for an argument against drawing such conclusions). In much case study methodology (Yin, 2013), however, drawing such conclusions are warranted but they seem to be used rarely.

5.7. Verbal theories

It is more a rule than an exception that a theory is expressed as verbally. Some theories of interaction have been recently summarized under topics like interaction-as-embodiment, interaction-as-tool-use, interaction-as-experience (Hornbæk & Oulasvirta, 2017), and include such theories as situated action, distributed cognition, activity theory, and embodied interaction. In which way can these theories be design-informing?

There are two answers to this question. First, the verbal medium is rich enough to allow the expression of counterfactual propositions, although it can at the same time be more susceptible to ill-informed conclusions. Given this limitation, they could be considered weakly design-informing.

Second, even theories that no counterfactual propositions can support such reasoning when combined with empirical data. We assume that a theory is a set of claims. When combined with a set of claims from an empirical study, one may be able to form counterfactual propositions that were not supported by either. For example, if theory $T$ claims that $a\Rightarrow c$ and study $S$ find that $\sim a\Rightarrow \sim c$, a counterfactual claim may be drawn: $a>c$.

6. Obstacles to counterfactual thinking

To what extent is counterfactual reasoning present in ongoing research, rather than in the idealized forms just discussed? If it happens rarely, analyzing why seems important. And further, if counterfactual reasoning happens rarely, how can design nevertheless be successful?

In this section, we consider this issue and claim that:

1. systematic reasoning using theory is disappointingly uncommon – albeit not inexistent – even in the best papers in a top conference of a relevant field

2. theory use is discouraged by beliefs about counterfactual thinking and about the epistemology of theories

3. technology design has been successful despite these problems thanks to “ratcheting practices”.

6.1. Lack of counterfactual reasoning in CHI papers

To obtain a snapshot of the state of counterfactual reasoning in human–computer (HCI) research, we next turn to the 25 best papers from CHI proceedings from year 2017. We expect that best papers would be treating the link from theory to design similarly to other papers; indeed, those papers likely write about this link with more clarity than the average HCI paper.

Seventeen of the papers report on constructing systems (spanning the planning, design, and implementation of the systems); the remainder are empirical (5) or conceptual (3). In most cases, the systems are key to the paper; in others, the systems are of minor importance. For instance, Williamson and Williamson (2017) studied experimenter roles in the evaluations of public displays, but only use one paragraph to describe the system evaluated. This makes it hard to assess the role of theory in the design. Nevertheless, most papers offer sufficient details to analyze their writing about theory in the design process.

In eight of the papers dealing with construction (47%), we find some use of theory about interaction beyond the discussion of related work (Barry et al., 2017; Jacobs et al., 2017; Kim et al., 2017; Lazar et al., 2017; Liu et al., 2017; Steinberger et al., 2017; Valentine et al., 2017; Webber et al., 2017). Kim et al. (2017) used theories from psychology and education on self-explanation and “devise and evaluate several elicitation techniques for information visualization with associated hypotheses” (p. 1376). In this case, the designs are relatively simple but their benefits clearly underpinned through reasoning with the theory. In a paper on creating procedural art through direct manipulation, Jacobs et al. (2017) noted that “the design of Para was partially based on the theory that a direct-manipulation procedural tool would be compatible with manual practice and enable manual artists to leverage existing skills” (p. 6338). These two papers show the clearest and most explicit link we could find between theory and design in this sample.

In other cases, the theory provides mainly high-level ideas for design, with many details to be fleshed out (Barry et al., 2017; Lazar et al., 2017; Liu et al., 2017; Valentine et al., 2017; Webber et al., 2017). Liu et al. (2017) used Bayesian Experimental design, “an approach where the system ‘runs experiments’ on the user to maximize an expected utility”, to design a new way of navigating information spaces. Lazar et al. (2017) discussed how certain views of dementia suggest stigma and only emphasize the biomedical aspects of dementia. They presented a critique of those views and develop three technology exhibits “aimed at providing a critical lens on dementia” (p. 2179).

In a few cases, theory is used to generate empirically testable hypothesis. For instance, Steinberger et al. (2017) reviewed a theory by Kurzban on subjective effort and task performance, develop prototypes, and conclude that “our prototype evaluations have shown promising results in terms of increasing task engagement. These results upheld the predictions of the opportunity-cost-model by Kurzban et al. Our work indicates that this model holds true in the driving context.” (p. 2835).

In nine papers constructing technology (53%), there is little or no discernable use of theory about interaction to drive design (Gugenheimer et al., 2017; Kan et al., 2017; Kazemitabaar et al., 2017; Schorr & Okamura, 2017; Tolmie et al., 2017; Ur et al., 2017; Williamson & Williamson, 2017; Yi et al., 2017). This does not mean that there is no background information or related work or supporting theories from other fields; just that theories contained in these parts are not used explicitly used to drive design or engage in counterfactual thinking about designs. For instance, Gugenheimer et al. (2017) noted that “prior work showed that this physical interaction can potentially increase enjoyment, social interaction and has cognitive benefits” (p. 4022) but proceed to inform their design process in other, non-theoretical ways. This is similar to other drivers of key decisions about design such as iterative design (Kazemitabaar et al., 2017; Tolmie et al., 2017; Ur et al., 2017), ethnography (Erete & Burrell, 2017; Tolmie et al., 2017), observations and interviews (Webber et al., 2017), and contextual design (Webber et al., 2017), and surveying users (Gugenheimer et al., 2017).

Across all best papers, a few other connections between theory and design may be observed. In some papers, design intermediaries make that connection include formulating design lenses from theory and developing personas (Steinberger et al., 2017). Design lenses are “analytical tools, which have been shown in HCI research to be useful for recognizing design opportunities, formalizing design criteria, and critiquing design work” (p. 2827). The design lenses do “not represent a complete account, but rather a first articulation based on our engagement with the literature [e.g.,…] and our own research experience in the domain [e.g.,…]. They are a result of many cycles of concept development, design work, and user testing, and therefore the synthesis of an iterative process”. Thus, theory in this case is mixed with other sources for speculating about the future design. Some are “proto-theories” that may be poorly articulated and in the process of formation. In some cases, papers are explicit about the tenuous link between the theory and the construction, talking about “taking inspiration” (Valentine et al., 2017) or they apply theory after construction as expressed in “we reflect on the design process for an app to support psychological wellbeing in pregnancy” (Barry et al., 2017). Sometimes the intermediaries are simply formulated as goals of the design.

In addition to design intermediaries, other mechanisms are used to link from designs back to theories. This appears to help papers relate to their theoretical backdrop or, alternatively, to aid future work in speculations about design. These include (a) design implications (Epstein et al., 2017; Lazar et al., 2017; Tolmie et al., 2017), for instance, “Given how crucial prediction accuracy is to menstrual cycle tracking, designers should evaluate additional techniques for modeling and communicating predictions about a woman’s cycle” (Epstein et al., 2017, p. 6882); (b) formulating a design space (e.g., Gugenheimer et al. (2017)), and (c) formulating models (e.g., Jokinen et al. (2017)).

In summary, the use of theory in design/construction is present in about half of the papers: those papers, however, rarely engage in explicit counterfactual reasoning and theory therefore plays a minor role in CHI’s best papers. Note, that this is not claiming that all of HCI is not using theory; merely that the sample of papers does not contain much counterfactual reasoning. Also note that papers may require new technology or new domains to become best papers; again we do not make claims about all of HCI, just about the present sample.

6.2. Beliefs that hinder reasoning

The discussion of CHI 2017 papers – and a browsing of the HCI at large – suggests that systematic and logical engagement with theories is not common. Next, we outline first explanations to this general observation.

First, we see the belief in the power of iterative design as one impediment. Iterative design is a hallmark of user-centered design and of the field of HCI more generally. It prescribes that there “must be a cycle of design, test and measure, and redesign, repeated as often as necessary” (Gould & Lewis, 1985, p. 300). Whereas iterative design has much use in practical development, it might not be equally well suited for research. To give an example from CHI 2017, Tolmie et al. (2017) presented an ethnography and subsequent iterative design of an application for journals to work with user-generated content. They conclude that “The work we have presented in this paper provides some scope for scaffolding this problem by enabling tools to be sensitive to the different ways in which variability in news production may be encountered” (p. 3641). Whereas the insights about the variability in news production are valuable and important, there is a sense in which the iterative design prevents a discussion of what we already knew about the work in news organizations. Counterfactual reasoning is likely also – to the extent we engage in it and report on it – limited because we can retort to iterative design. So while there is reason to be impressed with iterative design, it can effectively inhibit theory use.

Second, research often focuses on if user interfaces work, not why they work or fail. The discussions in papers that use theories to design systems are mainly about whether the systems failed or succeeded. They do not return to or argue about the theory or the choices directly inspired by theory. Generalizations and theory-building and other conceptual work do not happen this way. It shift the focus from why a user interface or system works to if it works. In the sample of CHI best papers that employ iterative design, conclusions concern the system build (say, if it works as desired or if it can be bbuilt. If speculation were properly used, one might expect to see theory used in design as well as some discussion about the implications of the design and any empirical findings about the design back on theories: how well did they fit, where assumptions in theories met, was boundary conditions challenged, and so on. Interestingly, and to us somewhat sad, is that CHI 2017 best papers, and few papers more broadly in HCI, contain such “implications for theorizing”.

Third, theory-use is distrusted in constructive activity. Such distrust could be based on a belief that constructive activity is a craft: that choices in design are based on designers’ intuitions and ways of knowing and that any use of theory in that craft is irrelevant, or possibly dangerous. This belief might lead to not reporting counterfactual reasoning in papers, and possibly to not engaging in it at all. A further belief possibly behind this distrust is that theory do not generalize and any use of theory in design is based on an untenable rationalist assumption.

Fourth, perhaps researchers focus too little on intended outcomes. Speculations about antecedents occur occasionally, both about principled possibilities and about specific design decisions. However, speculations about why those would be good are almost absent. So, systematic thinking about why desirable outcomes would ensure is not easy to do or not well supported by our theories. In an odd contrast, Epstein et al. (2017, p. 6877) talk about speculating about unintended outcomes: “Our work makes what Bardzell defines as a critique-based contribution […] by ‘analyz[ing] designs … to expose their unintended consequences’, such as the downsides to normative design choices”. Although this is a refreshing example, why does unintended consequences seem to draw more interest that a systematic discussion about intended ones? The latter could be established upfront and made subject to discussion and reflection.

Fifth, one might believe that theory is for positioning and post hoc analysis, not for design. In the sample of CHI best papers, several mention drawing inspiration from theories; in some cases, theoretical work stimulates thinking about new antecedents (e.g., design for new situations). The role of theory here is as backdrop or sensitizing concepts, and only indirectly as tools of thinking about actual design decisions and why such decisions may lead to desirable outcomes. This role of theory is echoed in books like Rogers (2012), which manages to cover more than 20 theories in 100 pages. It is also evident in, for instance, how activity theory has struggled to move from a frame of reference toward strong influence on design (Clemmensen et al., 2016; Nardi, 1996b). Another practice consist in using theories only after design. This, of course, is hardly counterfactual reasoning. At best it is abductive reasoning, at worst it is just a justification for preconceived ideas. A case in point is the theory building surrounding embodied interaction. Whereas the field has developed several accounts of theories of embodied interaction such as Klemmer et al. (2006); Dourish (2004), we are struggling to show examples where those theories have proven generative (let alone predictive) about specific designs that would tap those theories and lead to desirable outcomes. A second concern with theories as lenses applied after the fact is well expressed by Dubin (1978) in his work on theory building. He argues that there is limited scientific value in applying a theory after the fact because that exercise is unlikely to produce advances in theory building unless it is based on a explicit goal to challenge or extend the theory.

6.3. Ratcheting to success

Given the state of theory use in the paper discussed, and the ideas of why counterfactual reasoning is rarely done, it seems reasonable to discuss why it is that excellent designs can nevertheless be produced? For instance, in the CHI 2017 sample, there are many excellent and inspiring systems with no explicit theory behind the design (e.g., Ur et al., 2017; Yi et al., 2017). And outside of this sample, excellent systems are similarly being created. Thus, success in constructive efforts is somewhat of a paradox: Why, despite the claimed shortcomings of the involved design knowledge, can design nevertheless succeed?

Designing any system for human use insists on a large number complex decisions that must work together to meet design objectives. Our argument is that by triangulating different types of knowledge, and by iterating, a designer compensates for the low constructive power of available design knowledge. When triangulating activities are organized as they often are, “ratcheting” can happen.² That is, success in previous steps creates new knowledge or boundary objects that allow the team to avoid previous pitfalls and more directly find good solutions. However, at its extreme, this approach is wasteful and prone to errors. It incurs a high cost in terms of time and effort, depends on the prior knowledge and supporting practices of the team, is susceptible to mimicking existing solutions and ending up in a local optimum, and yet it offers no guarantee for the quality of the outcome.

Considering a counter-example to ratcheting: a point study. As discussed, a point study is an empirical study measuring outcomes in a fixed condition of interest. The inherent weakness of this form of design is that only a single design is assessed. Without some prior (e.g., theory, previous experiences), the results (c) do not inform decisions on other conditions. For every design candidate, even if a single feature changed, a new study must be run, they cannot be accepted or rejected with the information that a point study generates. If the point study in question happens to draw a blank, the designer is left in her own vehicles to generate a new idea. When considered in isolation, point studies are an incredibly wasteful way of progressing in design.

In a real project, epistemic endeavors like this are not taken in isolation. Even if error prone, patterns and guidelines and standards can help narrowing decisions from multiple choices to few. However, lacking constructive power, emphasis is put more on ideating many design candidates. Alas, the number of candidate designs often explodes as a function of design decisions. This means that the set of heuristics and examples will never be sufficient alone, and the designer must engage in iterative development. An important enabler of ratcheting is the capability to quickly generate designs. Rapid sketching techniques, for example, increase the rate with which partial and complete ideas can be generated, and encourages exploration of diverse ideas by suspending critique. This can compensate for some of the lack of counterfactual capacity in design.

Here, we find the explanation to ratheting’s success. By externalizing, sharing, standardizing, educating, and publishing results, more and more triangulators becomes available. Consider the popularity of design repositories and searching for design solutions online. Or the development of guidelines and standards and portfolios. By engaging in user research the problem statement is refined. This aids counterfactual reasoning by specifying the objective and limiting design space. By engaging in evaluation, one reduces uncertainty in estimates of goodness that might otherwise stand on opinion and experience. By using heuristics, patterns, standards, and examples in design, one generates partial solutions with higher probability of being good. Via this kind of process, a designer can slowly “ratchet” toward better designs.

Finally, we have portrayed an extreme, theory-poor version of ratcheting to illustrate a point. Practically all projects, however, involve iteration in some form or the other. Simply because there is no unified theory that could inform the totality of choices in design. Iteration can also serve counterfactual thinking. It can help refine theories, parametrize models, and define their constraints. Such iterative engagements with theoretical knowledge, however, are rarely described in a research paper.

7. Discussion

This essay has shed new light to the question of what makes theory informative in the design of interactive technology. In particular, we have aimed to integrate positions emphasizing science and views emphasizing design research. Our main answer is a new analytic view that emphasizes counterfactual reasoning as the link between theory and design; metaphorically, we see theories as tools that help designers speculate about the future, to allow steering solutions of design problems toward desirable outcomes.

In our view, counterfactual reasoning is what steers crafting, design, and engineering toward some future. Theories do not design. Models do not design. Data do not design. But they can be made to design via counterfactual reasoning. For a theory to be called a “theory for design”, it must – like a ship in heavy weather – take design forward and reach good futures. The more precise and valid speculations it allows for important design choices, the higher its problem-solving capacity.

Although many have remarked about design being speculative by nature, few have proposed how design is informed by knowledge and by empirical data. We have reviewed here how heuristics can inform design, although their low information content easily leads to issues like conflict resolution. Mathematical models can inform design by allowing numerical derivation of design consequences. Computational models can inform design by simulating what happens when a design is fixed in a particular way. Verbal theories can contain counterfactual information, albeit with the caveat that these can be misinterpreted without appropriate disclaimers. Even experiments can inform design by producing counterfactual propositions that stem from their ceteris paribus logic. While the logic of counterfactual reasoning does not recognize iteration, iteration can help evolve better and better counterfactual thoughts via these mechanisms.

Taking human–computer interaction as an example, previous work has lamented its “black holes” (Kostakos, 2015) or called for strong concepts (Höök & Jonas, 2012), theory reuse (Sutcliffe, 2000), or increased problem-solving capacity (Oulasvirta & Hornbaek, 2016). However, we now see that they have fallen short in important two ways: First, by failing to propose how exactly knowledge and theory informs design; and, second, by failing to point out systemic obstacles inhibiting theory-use. Let us discuss these two points in turn.

The first main benefit of our account of how theory informs design is that it allows an assessment of how well methods and practices support this activity. Simply, this concerns the counterfactual propositions that they help create which yields desirable outcomes to end-users. Activities like empirical evaluation and case studies, which have been kept separate from design, can be assessed in terms of the counterfactual propositions they propose. The limits of design heuristics, which have been realized in practice but not theoretically explained, can now be exposed. While simulation models were for long heralded as the desire data of design, their inherent weaknesses as black boxes has become clear. We have seen how “in the wild” approaches can drive themselves into a corner where speculation becomes arduous. But, we have argued how empirical observations can salvage supposedly “weak” theories. These insights call for serious discussion on methodology and theories that can uniquely inform the design of technology for human use.

Two important caveats need mentioning. We do not claim that the only use of theory is for thinking about design. We already explained the role of theory to redefine problems and sensitize designers to the design problem; they further play role outside of design in integrating empirical phenomena and factors that shape those phenomena, in addition to supporting understanding and predicting same. Note that we do also not claim that theory-use is all there is to design: idea-generation techniques, methods, and exemplary practices are non-theory examples that come to mind. Theories can provide ethical and philosophical perspectives that help frame and evaluate design choices.

The other main benefit of our account is to point out the systemic obstacles inhibiting theory-use. The incentive system is perversely biased for quick wins, asking for iteration and trial and error. The field begs for novelty, innovation, and flash. This renders us to a “ratcheting” mode. We also believe that theorizing in applied areas has been held captive by beliefs and concepts adopted from its parental disciplines. These fields are concerned about description, explanation, and prediction, not with speculation. However, while prediction is the hallmark of science, speculation is the hallmark of design.

In this paper, we have identified some obstacles to theory-use. They include relying exclusively on iterative design while ignoring theory, using theories to position work and explain it after the fact, and focusing on outcomes of design rather than on the mechanisms and the reasoning that shape those outcomes. We argue that all of these prevent researchers from engaging in and reporting on what in our view is a very important part of their work: Accounting for how theory has helped speculate about how their solutions work toward desirable outcomes. This proposal is more radical than the ideas of claims that have been much discussed in particular in the HCI literature (Carroll & Kellogg, 1989; Sutcliffe, 2000). Claims are design rationales motivated in psychological theory; they try to capture the upsides and downsides of a particular decision. However, previous treatment has not addressed counterfactuality as a necessary capacity. Further, we speculate that the view of theory-use as counterfactual reasoning might also lead to a different way of developing and framing theories, so as to adapt them better for supporting counterfactual reasoning. In our view, not engaging in counterfactual reasoning (and reporting on it!) may be one reason why some assert that fields like HCI are not cumulative (Kostakos, 2015).

A final obstacle is the undercurrent against scientific realism. Scientific realists hold that scientific claims have truthlikeness, which is independent of the researcher’s opinion. Reflective practitioner, design thinking, and designerly ways of knowing have been used as slapsticks, wrongly, to reject “scientific” or “positivist” theorizing across the board. We hold that in order to keep empirical data, theory, and design together, scientific realism must be embraced. This does not mean rejecting the reality and value of designerly ways of knowing; It means that in order to build a bridge to theory-formation and empirical research, design cannot treat theories as mere opinions and inspirations. Counterfactuality is one bridge between these viewpoints.

What, then, would we like to see changed based on these insights? Improving our writing culture is an obvious step. This means explicating the logic that ties assumptions to design decisions, and the speculations permitted. This is possible for all types of papers; Carroll and Kellogg (1989) discussed in particular how to make explicit the psychological claims made when constructing artifacts. That would make discussing the mechanisms by which design works easier, and might allow for more implications to be drawn back to the theory that informed design. This is largely missing from the papers we reviewed.

A more ambitious aspiration is to make the discussion of design, and how theory and knowledge inform design, less mysterious and more clear about underpinning types of reasoning. As part of that, we would like to see also more explicit discussions about when design assumptions fail and result in non-desirable outcomes. There is a need for developing theories in HCI explicitly with constructive power in mind. We need to compare theories for their constructive power so that we can evolve them. We often face a sparsity of theories but, when we do have theories, they can be contradictory or incommensurable. Consider a design project where you have two theories with partially non-overlapping antecedents and consequents. Which one should one favor? What HCI could do as a field is to compare theories not only for their empirical validity but for their constructive power. For example, how well would a team do that uses theory A versus theory B? A critical related challenge is to understand the scope within which a theory can produce valid constructive propositions. Fitts’ law is a notorious example: it is well-known that it breaks down when target sizes decrease. Thus the counterfactual predictions it makes beyond its training data are increasingly worse. In machine learning, algorithms are trained with one part of a dataset and tested with another. Should the constructive power of a theory be tested similarly? HCI theories could be developed in the same spirit: asking to ask how their constructive power transfers to novel domains and contexts.

Finally, two decades ago Barnard et al. (2000) discussed what they call macrotheory. Their point was that a host of local theories would not solve the problems of design; deeper theory is needed that connects different levels of analysis. We agree and although we find the notion of counterfactual thinking valuable, a coherent account of how to do that across different levels of analysis remains to be fully developed.

8. Conclusion

Counterfactual reasoning offers an actionable view on useful theories. The most critical question we should ask ourselves is how our research improves our ability to speculate with design. Instead of asking if some piece of research contributes design ideas, or empirical findings, or theoretical insights, one should ask in what way it improves reasoning from antecedents to consequents. As we have argued, it has its role in problem-solving type design where the conditions and goals of design are well-articulated. In particular,

• What are the counterfactual propositions – here noted in the form $a>c$ – that a theory entails, and how truthlike they are?

• What is the reasoning apparatus that produces those propositions?

• How easy it is to pump, or use this reasoning apparatus, to come up with informative propositions for a problem at hand?

• How can we put these propositions into practice?

This is a new transdisciplinary mind-set for disciplines centered on the design of technology for human use.

Additional information

Funding

A.O. has received funding from the Academy of Finland project BAD

Notes on contributors

Antti Oulasvirta

Antti Oulasvirta leads the User Interfaces research group at Aalto University and the Interactive AI research program at FCAI (Finnish Center for AI). He received his doctorate in Cognitive Science from the University of Helsinki in 2006.

Kasper Hornbæk

Kasper Hornbæk received his and M.Sc. and Ph.D. in Computer Science from the University of Copenhagen, in 1998 and 2002, respectively. Since 2014 he has been a professor at University of Copenhagen. In 2020, he was elected to the CHI Academy.

Notes

1. material.io

2. We refer to “ratcheting” in reference to Michael Tomasello’s explanation of how culture and cognition evolve together.