Causal Inference in Generalizable Environments: Systematic Representative Design

ABSTRACT

Causal inference and generalizability both matter. Historically, systematic designs emphasize causal inference, while representative designs focus on generalizability. Here, we suggest a transformative synthesis – Systematic Representative Design (SRD) – concurrently enhancing both causal inference and “built-in” generalizability by leveraging today’s intelligent agent, virtual environments, and other technologies. In SRD, a “default control group” (DCG) can be created in a virtual environment by representatively sampling from real-world situations. Experimental groups can be built with systematic manipulations onto the DCG base. Applying systematic design features (e.g., random assignment to DCG versus experimental groups) in SRD affords valid causal inferences. After explicating the proposed SRD synthesis, we delineate how the approach concurrently advances generalizability and robustness, cause-effect inference and precision science, a computationally-enabled cumulative psychological science supporting both “bigger theory” and concrete implementations grappling with tough questions (e.g., what is context?) and affording rapidly-scalable interventions for real-world problems.

Keywords:

• Brunswik
• cause and effect
• experimental design
• games
• generalizability
• representative design
• systematic design
• systematic representative design
• virtual environments
• virtual reality

Notes

1 Context matters; but, rarely is it defined. Van Overwalle (1997) draws on Rescorla and Wagner’s (1972, p. 88) definition from the animal learning literature in which context refers to the relatively constant “situational stimuli arising from the …environment” (see Van Overwalle & Van Rooy, 1998 for further treatment of this issue). For example, some aspects of context may provide a more or less constant background across individuals (e.g., a bar has different affordances than a church or a school). But, even there, other aspects of the “context” can be changing (e.g., whether we are in a “pick-up” situation or not and if so what is our role (given goals, beliefs, etc.) and where in the potential “pick-up” scenario are we).

2 Rarely is a single factor, ”X”, deterministic (i.e., always necessary and sufficient) in causing “Y”.  Instead, causal relationships tend to be probabilistic (Eells, 1991), such that “many factors are usually required for an effect to occur, but we rarely know all of them and how they relate to each other.” (Shadish et al., 2002, p. 5). Thus, across experiments “a given causal relationship will occur under some conditions but not universally across time, space, human populations, or other kinds of treatments and outcomes that are more or less related to those studied.” (Shadish et al., 2002, p. 5). Furthermore, modern causal analysis has broad implications, including for the transportability of experimental results to new populations (e.g., in observational studies), or external validity (Pearl & Bareinboim, 2014; Pearl & Mackenzie, 2018).  But, the generalizability of interest in these analyses (i.e., external validity) does not insure generalizability to everyday life (GEL).

3 Historically, the capacity for causal inference within experiments is defined in terms of a series of procedures that eliminate alternative explanations for differences in the dependent variable between, for example, the experimental and control groups, other than as due to the prior manipulation of the independent variable (see below in the section on systematic design and the promise of causality; see also, Brewer & Crano, 2014).

4 Generalizability to everyday life (GEL) is a new concept that is defined as the “built in” capacity to generalize the results of a study. We do so by first identifying – using formative research – what for the persons of interest (POI) are the situations and sequences of interest (SOI) leading up to the behaviors of interest (BOI). Using sampling theory we representatively sample from these SOI for BOI to which we wish to generalize, implementing these SOI and BOI in the default control group (DCG), for example in a digital game (see Miller, Jeong, & Christensen, in press; Appendix). External validity has historically been used as a measure of generalizability in experiments: It is defined in terms of the capacity to generalize the cause-effect relationship found in one study to the effects found in another with a different sample (e.g., of participants; or stimuli, etc.). External validity, however, does not insure GEL (see below for further discussion).

5 But, because they were stripped down, they might produce smaller effects (i.e., such stimuli might be less involving and impactful) (Aronson, Ellsworth, Carlsmith, & Gonzales, 1990).

6 A manipulation’s experimental realism and social impactfulness may enhance effects but may add extraneous variables: But, reducing extraneous variables (adding control) may weaken (potentially reducing replicability of) experimental effects (Aronson et al., 1990). Field research, with greater naturalism, is one compromise in this tension, but it neither necessarily affords the representative situational/setting and samples to which we would like to generalize nor the experimental control typically found in a laboratory study (Paluck & Cialdini, 2014). Furthermore, an event in the lab may be similar to an event in real life (exhibiting mundane realism) (Aronson et al., 1990) without it having GEL. To insure that the effects and relationships (e.g., among variables) in the control are generalizable to everyday life, the virtual environment (described later) by design needs to be representative of the cues and settings/situations of interest (SOI) likely to lead to the behavior of interest (BOI) for the population of interest (POI) as in everyday life. And, a condition with high experimental realism (e.g., some conformity studies (Aronson et al., 1990)) may include situations that aren’t representative of challenging conformity situations, given the BOI, that individuals in a POI may confront in everyday life.

7 As some experiments, within a condition, “self-selection” occurs, especially in long interventions. An advantage of the proposed SRD is that these self-selection opportunities within conditions can be kept constant across conditions or systematically manipulated.

8 In correlational designs, there can be a predictor or measured independent variable “X” that precedes a dependent variable “Y," where there is sufficient variability on both variables and where X and Y significantly covary. Such a measured independent variable typically involves naturally occurring behavioral variations that are then correlated with the dependent variable.  This provides a necessary, albeit not sufficient basis, for causal inference (Brewer & Crano, 2014). Causal analysis (e.g., Pearl & Mackenzie, 2018) suggests, however, greater causal inference potential from correlational data, especially for "big data" over time.

9 Brunswik's (1956) ecological validity (actually Brunswik refers to the validities of the cues) and representative design are often used incorrectly and interchangeably (Araújo et al., 2007). Here, we use them as Brunswik intended.

10 Trying to learn the ecological validities of these cues, and their intercorrelations, via experience was presumed, with the organism ideally learning the equivalence and substitutability of different cues (e.g., if one set of cues were unavailable or unreliable).

11 Dhami and Belton (2017) argue that their Brunswikian idiographic assessments and their heuristics either match or outperform alternatives relevant to cue utilization in making decisions.

12 Nonprobability sampling techniques might require a validity check since they might not cover the ecology to which the researcher wished to generalize (Brunswik, 1955b; Dhami et al., 2004).

13 Gigerenzer et al. (1991) showed that representative sampling of cities (vs. not) could dramatically impact inferences about bias, consistent with claims that participants’ use probabilistic mental models in making these inferences and judgments. 

14 We cannot even assume that across studies with faces, the same cause-effect relationships will result. For example, the orientation of the face can affect social perception (Witkower & Tracy, 2019).

15 This situation may be analogous to the situation in chemistry before the development of The Periodic Table that is used as the basis for predictions, hypothesis testing, and theory in chemistry that some have argued moved chemistry from a pre-paradigmatic science to a science as physicists would think of it (see Scerri, 2007).

16 Procedurally, in classic experimental designs, participants do not self-select themselves into condition. Instead, they are randomly assigned to conditions, to eliminate the possibility that differences between conditions are due to participants’ pre-existing differences instead of the manipulation. Procedurally, in SRD, participants are randomly assigned to the DCG or a given experimental condition; participants then make choices (e.g., in a virtual environment game) just like in classic designs. Others (e.g., virtual intelligent agents) subsequently respond to the participants’ responses, and those responses are adjusted given the participant’s behavior.  Different agent responses can then affect the participant’s subsequent options and choices, creating interactional sequences into which participants “self-select” that are narratively designed to be more generalizable to everyday life (see below; Appendix; Miller, Wang, Jeong, & Gillig, in press).  However, there are not between group differences in self-selection opportunities in SRD, unless systematically manipulated by design with all other variables held constant or controlled as a source of alternative explanations. Thus, this type of “self-selection” during the course of a game does not present a threat to internal validity.

17 VR can consume the user’s audio and visual senses, activating haptic responses.

18 Related technologies include augmented reality (AR) in which virtual objects are overlaid onto one’s real-life world and mixed reality (MR) that augments as above but also anchors those virtual objects, making it possible to interact with them in the real-world. (Garon, Boulet, Doironz, Beaulieu, & Lalonde, 2016; Tepper, Rudy, Lefkowitz, Weimer, Marks, Stern, & Garfein, 2017; Tokareva, 2018). Haptic interfaces may be especially interesting for applications with mixed reality and smartphones with digital/VR games “in the wild” for example (Lee, Sinclair, Gonzalez-Franco, Ofek, & Holz, 2019).

19 The broader category of games includes games that are not video games, such as card games. There are other things described as games (e.g., economic games, such as “prisoner’s dilemma”) that are associated with game theory, involving the modeling of strategic interactions between rational players (see for example, Myerson, 1991). These literatures, however, are well outside the scope of the current work.

20 The video game industry, including the much smaller VR video game industry, is a major industry, rapidly overtaking the 125 billion dollar annual mark (Gaudiosi, 2016).

21 A commercial version is the well-known video game simulation (and its many variants) called the “Sims.”

22 As Adams (2013) notes there are also subgenres within each of these genre types (e.g., shooter games are a subgenre of action video games).

23 Simulations are sometimes classified within the category of digital games and sometimes not, depending upon one’s specific definition of “video games” and the specific nature of the simulation in question. For our current purposes – we are interested for research and behavior change purposes in the capabilities of such software and hardware for implementing the “gist” of representative SOI and BOI for a POI within a virtual world – when it comes to language pertaining to such distinctions (e.g., when is a simulation not a game) we find ourselves in agreement philosophically with Wittgenstein (2009).

24 Simulators, for example flight simulators, have been used extensively for decades and found to be highly effective (Hays, Jacobs, Prince, & Salas, 1992).

25 In studies with commercial games as well as randomized trials with serious games, whether VR or virtual games, the control group is most often either a wait-list control group or it’s another game. Each has weaknesses, in that effects in the former may be due to using a game without the intervention components critical; The effects in the later are hard to compare because so many variables differ across games (Granic et al., 2014). Furthermore, for meta-analyses the control comparison can be many things. It could be a comparison game what differs considerably across studies. It could be a wait-list control group where that control condition is more “constant” but one isn’t controlling for participants’ experiencing a game. SRD addresses this issue by first developing the DCG that can be used as the appropriate control in a randomized controlled trial.

26 Price and Anderson (2007) argued that although VR could facilitate a sense of presence, it did not cause a positive treatment outcome.

27 This externally generated RHI paradigm it is argued involves mostly top-down processes while alternative similar paradigms (where the user self-initiates movement) involving body ownership (e.g., moving rubber hand (mRHI), virtual hand illusion (VHI), are more likely to be “actively shaped by processes which allow for continuous comparison between the expected and the actual sensory consequences of the actions…[These additional illusions provide the basis with a motor task using VR to test hypotheses about] whether during goal-oriented tasks body ownership may result from the consistency of forward models” (brackets added for clarity) involving both self-generated and distal multisensory cue integration (Grechuta, Ulysse, Ballester, & Verschure, 2019, p. 1).

28 Researchers have suggested that we need to be careful in designing representative environments, avoiding and testing for problems like the uncanny valley (e.g., where the agents are so similar to the target person that they activate disbelief) or VR sickness (Cobb, Nichols, Ramsey, & Wilson, 1999). However, although one should test for this in a given population (e.g., Benoit et al., 2015), that as participants are more immersed and experience greater presence in more representative environments, and better leverage emerging technologies for their target populations and behaviors of interest (e.g., García-Betances et al., 2015), that virtual environments may provide closer and closer approximations to everyday behavior in similar everyday affording situations without detrimental effects.

29 Here, the underlying person parameters (goals, beliefs) that might drive behavior of those other actors should be considered, bearing in mind the technology used to implement agents in the game and leveraging its capabilities. For example, we manipulated the parameters of intelligent agents (Marsella et al., 2004), including various goal weights and belief parameters to guide automatic scenarios affording different sexual risk challenges (SOI) leading to our BOI, risky sex, for our POI, as in the real-world.

30 In theory, VR could involve intelligent “other” agents. However, this is currently not the case.

31 Unity games can be built once and then used across over 20 different platforms including on smartphones, iPads/tablets, computers of every sort; it is used in about half of all games developed since 2005, and is relatively easy to learn and use, and has a real-time game engine – software development environment (https://unity3d.com/unity/features/multiplatform).

32 Of course the VR headset is also modified so that it does not contain ferromagnetic metals.

33 See Emmert et al. (2016) for a meta-analysis of fMRI self-regulation neurofeedback.

34 Rutherford, who won a Nobel Prize in Chemistry was overheard noting that science is either physics or stamp collecting.

35 The physicist Hoffman more recently noted, “When I was in high school, I loved science and mathematics, but I could never get too excited about biology. It seemed like a lot of tedious memorization and ad hoc theories and appeared to lack the coherence, clarity, and universality of physics. This remained my opinion for many years” (Hoffman, 2012, p. 2).

36 There is a Nobel prize in physics, chemistry, and medicine/physiology, but not biology per se. Nor is there a Nobel Prize (officially) in the social sciences.

37 Note that the level of scale here for this precision and dynamic examination of molecules in their context is at the level of molecules and cells, not whole organisms.

38 The 1998 Nobel Prize in Chemistry (Nobel Media AB, 1998) was won for computational tools: Pople for developing computational methods for quantum chemistry and Kohn’s involved a density functional approach. The 1999 Nobel Prize in Physics (Nobel Media AB, 1999) involving computational modeling was awarded to Veltman and Hooft in quantum field theory.

39 Providing the basis for refuting “bigger theory” or strong systematic evidence that undermines major theoretical assumptions of established theory could also provide a basis for an award in science at this level.

40 Furthermore, additional researchers, building on this work, investigating the combination of multisensory self-motion and place/landmark information in virtual environments with mice, developed a network model, further testing those principles. This is moving scientists towards a theoretical framework for understanding how environment and self-cues produce the spatial representations guiding goal-directed behavior (Campbell et al., 2018).

41 Indeed, the hippocampus plays a significant role across rodents and humans in decision-making involving approach-avoidance conflict: it is key, however, to study these motives concurrently (Bach et al., 2014; Ito & Lee, 2016; Oehrn et al., 2015; O'Neil, Newsome, Li, Thavabalasingam, Ito, & Lee 2015, 277) reminding us of N. E. Miller’s (1944) classic approach-avoid conflict research (and the importance of measurements in the rat’s movement in space as it negotiated this conflict). This suggests the need to revisit this work on movement to assess this conflict (Boyd, Robinson, & Fetterman, 2011) using today’s technologies (e.g., Oculus Rift/VR; animated characters interacting with humans) similar to what has and is currently being done, involving fine-grained head movements in both approach and avoid motivations in conflict situations Jeong, Feng, Krämer, Miller, & Marsella (2017).

42 In building initial DCG, for example, we are concurrently testing assumptions about key features in it (e.g., settings and their affordances; structures in it (e.g., scripts); beginning and ending points, etc.). These evidence-based assumptions, for example, can be challenged (e.g., with comparisons with alternative models; by experimentally eliminating/altering aspects of the model in experimental comparisons to judge their altered virtual validity) to enhance cumulative science precision, accuracy, and insight into when, why, and how they differ in terms of impact.

43 Furthermore, virtual validity checks in real-time (e.g., using smartphone and sensor technologies, including ecological momentary assessments (Shiffman, Stone, & Hufford, 2008) afford continued feedback and opportunities for cumulative measurement and prediction improvement over time.

44 In building their computational models, Read and his colleagues argued that humans have universal approach and avoid systems and nested in them, universal goals: But the relative levels of chronic goals differ between individuals. Situations have different goal affordances as well. As individuals move into different situations (e.g., a friend appears; an alarm goes off), the situational affordances change: These combine with chronic goal activations to affect current competing goal activations, with the most activated goal driving behavior. Computational models of virtual personalities (VP) – where VP chronic activations were systematically manipulated – indicated that there was tremendous within-person variability in behavior across situations, but at the same time entering each VP’s data (as we would for real subjects), and performing factor analyses produced across persons, the “Big 5”.

45 In a way a given SRD DCG could be our “best guess” instantiation of the probability distributions of cues and sequences that constitute a specific context and sequential options and consequences in the real-world. As suggested earlier, this seems analogous to the “model system” concept so critical in modern biology.

46 In addition, because computational models can be used across scale (e.g., the interpersonal level, the individual level, and the neural level) to address personality and social psychological dynamics in producing emergent behavior (e.g., Read, Brown, Wang, & Miller, 2018), they can also suggest (across scale, for example in fMRI studies) what to measure and afford in building SRD.

47 In short, computational modeling, since it requires the math and precision to build, provides psychologists with new methods in our toolkit for illuminating hidden assumptions and theoretical gaps, while also affording ways to iteratively build and improve SRD as well as testable theories (Farrell & Lewandowsky, 2018; Marsella, Gratch, & Petta, 2010; Marsella & Gratch, 2016; Vallacher, Read, & Nowak, 2017).

48 It can take a decade or more for basic science to produce useful applications (Morris, Wooding, & Grant, 2011).

49 Technological advances here are rapid, including in exquisite capabilities for voice recognition and emotion differentiation (see for example, Huang & Narayanan, 2017; Somandepalli et al., 2016) and the capacity to “pick up” complex contextual cue reactivity in craving (Traylor, Parrish, Copp, & Bordnick, 2011).

50 In this era of “big data” (Cai & Zhu, 2015; Kitchin, 2014; Provost & Fawcett, 2013), one question is what will we do with so much rich and complex data? Machine learning may provide one set of answers, but the cues, and the relationships among them that go into these algorithms can often be a “black box.” Furthermore, these cues may or may not be the cues that humans use in the same way (Cai & Zhu, 2015). SRD is a methodology through which big data can be leveraged to better create systematic control and experimental groups and to more systematically test how to “structure” data contextually to build better predictive models of human smart-phone and sensor data patterns over time.

51 Although Clark (2013) mostly presents one predictive coding algorithm, different predictive coding models using alternative algorithms still vye for which better capture the data and which is the most neurobiologically plausible (Spratling, 2017).

52 Friston (2013, p. 212) notes that “predictive coding is a consequence of surprise minimisation, not its cause.”

53 Read and Miller (1998) also closely examined the developmental literature (Read & Miller, 1995). For example, young children have a readiness to communicate wants (Gelman, 1990).

54 The frontal cortex plays a domain general cognitive control function, selecting among competing representations and shifting and weighting algorithms between dorsal and ventral multimodal streams and numerous points of integration across ventral and dorsal streams (Bornkessel-Schlesewky et al., 2015a, 2015b).

55 A critical feature in experiments is timing, the independent variable for example must precede the dependent variable as one important criterion for causal inference. Of course, in everyday life, individuals use many of the criteria we use in experiments to make their own everyday causal inferences about the meaning of sequences of behavior.

56 Face processing (and the anticipation of face processing) is especially associated with the fusiform face area (FFA) of the brain (Furl, Garrido, Dolan, Driver, & Duchaine, 2011). That is, specific “who” or “whom” assessments in understanding sequences of actions may be based on connections in one’s representation there.

57 Roseman (2011) provides a hierarchical motive-based model pertaining to emotional “effects” that may serve to also motivate (e-motion) action. Might this theoretical model suggest possible neural (perhaps narratively based) slot unit linked underpinnings?

58 Exciting work (i.e., Chang, Gianaros, Manuck, Krishnan & Wager, 2015) in multivoxel pattern analysis (MVPA) that appears to afford sensitive and specific neural signatures for affect induced stimuli (e.g., aversive images) could benefit from fMRI recording during participant SRD representative scenarios engagement. It is an intriguing possibility that we could examine if such stable neural specific signature activations and their links recapitulate ongoing narratives (and conceptual plot units) in social interaction.

59 These social concepts themselves are apt to be based on underlying learning histories with respect to various combinations of plot units.

60 There are extensive literatures focused on “person”/“object” perception as well as action perception– and the links among these to social judgments. For example, there is considerable work on features underpinning face perception and the relationship of these features to social judgements such as dominance and competence (e.g., Todorov, Dotsch, Porter, Oosterhof, & Falvello, 2013) or attractiveness (e.g., Bronstad, Langlois, & Russell, 2008; Todorov et al., 2013).

61 As Clark notes, Helmholtz (1860) had “the key idea that sensory systems are in the tricky business of inferring sensory causes from their bodily effects. This in turn involves computing multiple probability distributions, since a single such effect will be consistent with many different sets of causes distinguished only by their relative (and context dependent) probability of occurrence” (Clark, 2013, p. 182, italics added).

62 For other discussions pertaining to contextualization and its importance, see Kveraga et al. (2007), Bar (2007), Barrett and Bar (2009), and Fabre-Thorpe (2011).

63 Computational models can take priors (prior constructions; top down concepts) into account in guiding subsequent causal inferences in ongoing social interactions. This could help model and predict users’ causal meaning inferences in interacting in virtual (and to the extent possible real-life corresponding) situations. Those computational models could also generate testable hypotheses for participants causal inferences within and about both virtual and real-life ongoing situations over time.

64 Given a predictive coding approach to developing SRD, there is also the possibility of building in representative environments that afford opportunities for examining variability in how individuals make causal inferences within, for example, a given DCG and in assessing what can alter those patterns (e.g., with experimental groups built on the DCG base).

Additional information

Funding

Research reported in this article was supported by the National Institute on Drug Abuse under R01DA031626 awarded to Stephen Read (PI), by the National Institute of Mental Health under R01MH082671, awarded to Lynn Miller (PI), and the National Institute of General Medical Sciences under R01GM10996 awarded to Stephen Read and Lynn Miller (PIs). The content is solely the responsibility of the authors and does not necessarily represent the official views of NIDA, NIMH, or NIGMS.