Measuring in-situ engagement during structured experiences: Validation & reliability study using immersion neuroscience software

Abstract

Effective measurement of participant engagement is vital to the advancement of experience design research. This paper addresses the feasibility, validity, and reliability of one possible measure, Immersion Neuroscience Software (INS). INS uses wearable cardiac sensors and a proprietary algorithm to process heart rate data to provide a passive real-time score of in-situ engagement—i.e., a second-by-second neurophysiological score that is interpreted as participant attention to the experience (dopamine) and emotional resonance during the experience (oxytocin). For this study, we test INS in the context of a structured experience of theatrical performance. This study included a sample of 72 individuals across four performances of the same dramatic play. Our findings provide initial evidence of INS as a promising approach to measuring in-situ engagement during structured experiences.

Keywords:

• Engagement
• experience
• measurement
• neurophysiological
• quantitative research

Engagement during structured experiences is paramount to the field of leisure and is commonly evaluated as an indicator of the value or outcomes of an experience. In their in-depth review, Ellis et al. (2022) noted that the “complex and elusive concept” (p. 1147) of engagement has often suffered from a jingle-jangle fallacy where engagement is conceptualized as both a transitory or momentary phenomenon and a stable one in which changes occur gradually. Thus, advancing empirical knowledge about engagement (as either momentary or stable) “hinges on formal definition and quality measurement” (Ellis et al., 2022, p. 1147). In this paper, we seek to contribute to the quality of measurement when conceptualizing engagement as momentary (i.e., in-situ) by introducing a new physiological measurement that provides a second-by-second neurophysiological score for engagement during structured experiences. We provide initial evidence of Immersion Neuroscience Software (INS) as a promising approach to measuring in-situ engagement during structured experiences (i.e., theatrical performance).

Background

Conceptualizing engagement

Engagement has been conceptualized in many ways across disciplines. Some scholars define it as a cognitive operation like attention, effort, or agency when completing a task (e.g., Beymer et al., 2018; Ellis et al., 2022); others refer to it as participation in an activity (Finn & Zimmer, 2012; Fredricks et al., 2014). In this paper, we conceptualize engagement as defined in the theory of structured experience (TSE) as “a transitory condition of heightened attention, emotion, and motivation that varies by degree and is characterized by (a) extraordinarily high focus of attention on an unfolding narrative or story told in words, actions, and music; (b) heightened emotions; and (c) agentic inclinations.” (Ellis et al., 2019, p. 103). Later, Ellis and colleagues (2022) dive into the concept of engagement within the TSE and describe engagement as a momentary or in-situ condition, and position engagement as a determinant of deep structured experience. They further specify that engagement within the TSE has three determinants: provocation, coherence, and personalization—all of which can be deliberately affected through intentional experience design.

Operationalizing engagement

Ellis and colleagues (2022) provide an 11-item scale of engagement to measure the different domains of in-situ engagement (α = 0.97). The scale leads with the prompt: “During approximately what percentage of your time did you feel each of the following?” Items include: Fully engaged? Fully focused? Strong emotions? etc. In their instrument, engagement is measured as a ratio-level, continuous variable. The measurement is administered after completing a structured experience and therefore utilizes post-hoc self-report recall-based measurement. It effectively contributes to the resolution of the jingle-jangle problem of measuring engagement because the items ask participants to estimate the percentage of time they experienced each of the domains of engagement during the structured experience (i.e., in-situ engagement).

Other approaches used to measure in-situ engagement include the experience sampling method (ESM). Also known as ambulatory assessment or ecological momentary assessment (EMA), these approaches facilitate the collection of self-report engagement data as an experience occurs within a naturalistic environment (Sather, 2014). Data can be collected multiple times during a structured experience by soliciting responses from participants via surveys delivered at either pre-specified or random times or intervals (Hodge et al., 2022). Although ESM approaches have advantages like reducing recall bias by collecting data temporally and physically closer to a specified event, they are not without their limitations (Csikszentmihalyi & Larson, 2014). One limitation, especially in the context of structured experiences, is that ESM disrupts the flow of the structured experience because the participant needs to stop the experience activity to engage in the research activity. This disruption is detrimental to measuring concepts like in-situ engagement with fidelity.

With the advancement of mobile and wearable data collection tools, there are new opportunities for non-disruptive, non-self-report ESM-based approaches to in-situ measurement of engagement. Technology like personal devices (e.g., smartphones) and sensors (e.g., smart watches) can facilitate ambient data collection during structured experiences that do not disrupt the experiences as they unfold. For example, when a participant dons a sensor (e.g., smartwatch) and participates in a structured experience, the researcher can collect data throughout without disrupting the participant’s experience. Depending on device capabilities and corresponding software, data collected via wearable sensors offer opportunities for scholars to passively collect participants’ physiological data during a structured experience at varying resolutions (e.g., once per second or even hundreds of times per second), which can offer additional insights into in-situ engagement.

Considering these new technological advancements, we caution scholars when considering the use of passive physiological data collection: such measurements are not infallible, nor are they a panacea (Csikszentmihalyi & Larson, 2014). When used appropriately and in conjunction with other types of data, technology like wearable sensors may be well-positioned to contribute to the call for quality measurement of in-situ engagement (Ellis et al., 2022). Indeed, other disciplines have begun including physiological data to garner data and insights that complement more ‘traditional’ research methods (e.g., Martin et al., 2023; Mastromatteo et al., 2023; Törmänen et al., 2023; Zaccoletti et al., 2023).

In general, physiological data provides indicators of individual-level responses during a structured experience. Examples of physiological indicators include electrodermal activity, heart rate and heart rate variability, breathing rate, blood pressure, and hormones. With some (but not all) physiological indicators like heart rate and brain activity, scholars are able to zoom in the scope of a research study to examine one specific activity and observe the moment-by-moment responses to specific stimuli throughout the structured experiences. In some cases, studies use heart rate or heart rate variability to make inferences about an individual’s engagement (e.g., Mulaffer et al., 2019; Richardson et al., 2020). In other cases, however, physiological indicators like neurohormones (e.g., cortisol, dopamine, oxytocin) that are collected via urine or saliva can only be measured in aggregate after the experience has been completed. For example, to measure oxytocin, researchers may collect spinal fluid, blood, saliva, or urine at the end of a structured experience, as noted in scholarship on couples’ recreational activities (e.g., Melton et al., 2019; 2022). In these cases, the physiological indicators cannot be used to determine moment-to-moment constructs like in situ engagement. Thus, some types of physiological indicators—and more specifically, the ways they are measured—affect their utility in understanding in-situ engagement.

Regarding measuring in-situ engagement, neuroscientists have linked novelty (also a factor linked to engagement; Ellis et al., 2022) to the release of neurotransmitters (e.g., dopamine, serotonin, acetylcholine, and noradrenaline) (Rangel-Gomez & Meeter, 2016). Yet, observing the release of neurotransmitters on a moment-to-moment scale presents a challenge, given that measuring them typically requires complex procedures that would disrupt the continuity of a structured experience, and sufficient expertise and resources for the proper collection, management, and analysis of bodily fluids (Melton et al., 2022). Thus, newer data collection platforms may circumvent such complications and increase the accessibility of measuring neurotransmitters (i.e., dopamine, oxytocin) in structured experiences.

Immersion Neuroscience Software

In this study, we explore the use of one such data collection platform, known as Immersion Neuroscience Software (INS; Immersion Neuroscience, Henderson, NV). INS provides a neurophysiological score that “quantifies the extent to which your brain is connected to and getting value from an experience” (Results from Participating in an Immersion Experience | Immersion Knowledge Center, n.d.). INS was developed to measure the dynamic nature of experience that cannot be captured in static post-experience surveys. More specifically, INS aims to capture two key elements of an experience: attention to the experience and emotional engagement during the experience (Zak & Barraza, 2018). In field studies of consumer behaviors, INS has predicted top-rated reality TV shows and music, as well as consumer spending (Zak, 2022a).

INS is based on two decades of research that links the peripheral nervous system (the nerves of the heart) to the central nervous system (the nerves of the brain). The INS platform links second-by-second changes in cardiac rhythm to infer neural states associated with the downward streams of dopamine and oxytocin. From this inference, the algorithm creates the INS scores (1 to 100) which have “accurately predicted behaviors after a stimulus, especially those that elicit emotional responses (Lin et al., 2013; Barraza et al., 2015)” (Merritt et al., 2023, p. 3). Thus, using a proprietary algorithm, INS software transforms heart rate data into a continuous, interval-level score. This score is based on capturing attention—i.e., heart rate changes correlated with dopamine release—and emotional resonance—i.e., heart rate changes correlated with oxytocin release (Gutkowska & Jankowski, 2012; Zak, 2020; 2022a; 2022b; Zak & Barraza, 2018). Previous research has demonstrated that oxytocin is a neurohormone associated with emotional empathy and emotional resonance (Domes et al., 2007: Hurlemann et al., 2010; Preckel et al., 2014). Similarly, previous research has linked dopamine to attention and focus (D'Ardenne et al., 2008; Schultz, 2002; & Cools, 2016). In the context of engagement, oxytocin can enhance empathy, a component of social engagement (Shamay-Tsoory & Abu-Akel, 2016), and enhance the salience of social stimuli, thereby making social experiences more engaging (Bartz et al., 2010). Dopamine influences engagement-related behaviors (Cools et al., 2011) and positive motivation (Bromberg-Martin et al., 2010).

Ellis et al. (2019) differentiate between three similar but distinct TSE constructs based on context: immersion, absorption, and engagement. Of most salience to this study, we focus on Ellis and colleagues’ constructs of immersion and engagement, because although INS has been previously conceptualized as neurophysiological “immersion” (e.g., Lin et al., 2022; Merritt et al., 2023), we argue it more closely aligns with the TSE construct of engagement. A close read of the definition of immersion as articulated by Ellis et al. indicates that although both immersion and engagement are “transitory condition[s] of action, attention, and motivation that [vary] by degree”, immersion is characterized by

(a) high focus of attention on a limited stimulus field, (b) environmental demand for immediate action, and (c) automatic responses (i.e., action without conscious, calculated decisions), (d) immediate feedback on the efficacy of those actions, and (e) perception of control (p. 11).

In contrast, engagement is characterized by “(a) extraordinarily high focus of attention on an unfolding narrative or story told in words, actions, and/or music; (b) heightened emotions; and (c) agentic inclinations” (Ellis et al., 2019, p. 11).

In this study, which focused on the attentional and emotional responses to a theatrical narrative-driven story written and produced with the intention of eliciting a heightened emotional response, INS more closely aligns with the TSE construct of engagement. Moreover, the physiological links between dopamine and heightened attention, and oxytocin and emotional resonance further suggest INS is likely to operate as a proxy measure for engagement. However, in other contexts, those using the TSE framework may suggest that INS scores provides a proxy for immersion.

Experts familiar with data collection and analysis of physiological data will readily acknowledge that “healthy biological systems exhibit complex patterns of variability that can be described by mathematical chaos” (Shaffer & Ginsberg, 2017, p. 1). In this paper, we suggest that INS may provide a user-friendly and cost-efficient way of collecting unobtrusive indicator of in-situ engagement data in naturalistic leisure settings and structured experiences for social scientists not familiar with the mathematical chaos of physiological data. In seeking to advance the use of INS as a proxy indicator for in-situ engagement, we present (1) the logistical requirements for using INS, and (2) findings on the reliability and validity of INS data in the context of a structured experience.

Software requirements

INS is a software as a service (SaaS) and thus provides a software licensing subscription. The software provides a user-friendly platform for collecting data and a digital dashboard that visually tracks second-by-second neurophysiological scores (i.e., INS scores).

Hardware requirements

In addition to a computer to run the software, additional hardware is required: heart rate sensors and a transmitting device.

Heart rate sensors

Participant data is collected via a heart rate sensor. INS supports a wide variety of smartwatches and fitness sensors, such as Scosche (Scosche Industries Inc., Oxnard, CA), Apple Watch (Apple Inc., Cupertino, CA, USA), Fitbit (Fitbit Inc., San Francisco, CA, USA), Google Watch, and Whoop fitness devices. The use of HR trackers can provide a variety of useful information, although each device has potential limitations, especially in comparison to standard electrocardiographs (Benedetto et al., 2018). Currently, INS scientists recommend Scosche devices or other heart rate sensors that have been previously validated for research use (e.g., Gillinov et al., 2017). Scosche Rhythm sensors use the Valencell® biometric chip, which has been benchmarked for providing above 99% reliability (Milstein and Gordon, 2020; Stahl et al., 2016).

Transmitting device

Researchers can use one of two options for transmitting the data from heart rate sensors: (1) the INS App and smartphone or (2) the INS hub. The INS App is freely available and can be downloaded onto most smartphones. Using the app requires each participant to have or be provided with a compatible smartphone. The app requires participants to complete additional setup before data collection can be launched. Heart rate sensors transmit data to the on-phone app via Bluetooth. The INS hub is the second option for transmitting data. The hub can be purchased through INS and is configured to the researcher’s INS account. One-time setup for researchers is minimal, and participants do not have to complete additional setup INS. Data collection with the hub is optimal with a hard-wired internet connection; however, wireless internet can also be used.

Data management & interpretation

The INS algorithm produces a second-by-second score. Thus, a one-minute experience will produce 60 INS scores per participant, and a 60-minute experience will produce 3,600 INS scores per participant. This is a form of non-linear time series data. Scores range from 1 to 100 and can be interpreted using the pre-established categories of engagement: captivated (91–100), absorbed (66–90), interested (36–65), indifferent (11–35), and tuned out (1–10) (Overview of Immersion Metric | Immersion Knowledge Center, n.d.). The INS score can be interpreted as “participant engagement with the content” (Overview of Immersion Metric | Immersion Knowledge Center, n.d.). In general, a score of 50 is considered “average” engagement. Thus, INS scores greater than 50 can be interpreted as heightened emotion and attention within a structured experience. However, it would be best for scholars to establish their own interpretation guidelines after conducting multiple studies in their particular context (e.g., what is an “average” score in theater, amusement parks, education sessions, etc.?).

Limited peer-review evidence

As stated, INS may be a promising tool for experience designers and researchers in understanding engagement during structured experiences and, relatedly, identifying the mechanisms of optimal or highly engaging experiences. However, much of the evidence that would support accepting INS as a reliable and valid measurement approach is not publicly available. In part, this is because the software is based on a proprietary algorithm that is protected for commercialization (personal communication with Paul Zak on November 11, 2021, and December 8, 2022). Many of the details regarding the creation of the algorithm and software have been chronicled in Zak’s book Immersion; however, the book is meant for a lay audience and does not provide detailed data that can be scientifically examined. Furthermore, although INS is increasingly used for market research, peer-reviewed published research is limited (Lin et al., 2022; Merritt et al., 2022; 2023; Rancati & Maggioni, 2023; Zak & Barraza, 2018). Of these studies, no article has reported the validity or reliability of using INS. Therefore, this study contributes to the field by providing an initial examination of the validity and reliability of INS data.

Current study: Theater

Given the limited peer-reviewed scholarship using INS, the aim of the present study was to assess the validity and reliability of INS in the context of structured experiences. Some scholars have suggested that our research team should reverse engineer the proprietary algorithm to fully assess its utility; however, such efforts would require both expertise and financial support beyond our current abilities and resources. Therefore, we adopt more accessible research methods to examine the reliability and validity of INS as a tool for understanding in-situ engagement during structured experiences.

This study is contextualized to theatrical performance as a structured experience. Indeed, Ellis has frequently used theater as a context for the study of structured experiences as well as the concept of in-situ engagement (Ellis et al., 2022). Our research questions are as follows:

RQ1

Is INS a feasible tool for measuring in-situ engagement in real-world settings? We hypothesize INS is a feasible tool for measuring engagement in real-world settings. In this study, feasibility will be tested by reporting the amount of missing data. Missing data are common in quantitative studies and physiological research (e.g., Strijbosch et al., 2021); however, thresholds for missing data, when exceeded, undermine the quality of data. Using the INS system requires a 2-step transmission of data from the heart rate monitor to the hub or smart device, and then to the INS platform, potentially increasing the possibility of losing data to technical issues. Therefore, we test feasibility by examining missing data.

RQ2

Is INS a valid tool for measuring in-situ engagement? The intent of this question is to address validity—the extent to which INS measures what it purports to measure: engagement. Literature has established that there is not always a correlation between psychological and physiological measures of the same construct (e.g., Hoareau et al., 2021; Morrow & Labrum, 1978), thus it could be inappropriate to use psychological or survey-based measures to evaluate validity of INS. Therefore, in this paper, we use predictive validity—i.e., whether an instrument can predict a future outcome (i.e., engagement). If INS measures engagement, then we should be able to predict when the neurophysiological score will be high in a theatrical performance. Similar to previous studies on narrative engagement and physiological measures (e.g., Richardson et al., 2020; Strijbosch et al., 2021; Sukalla et al., 2016), we expect that emotionally relevant scenes for the characters to elicit high engagement responses. To address this question, the playwright/director identified two scenes that were intended to be high-engagement for the audience. If INS is a valid measure of in-situ engagement, then we expect INS scores during those two scenes will be high.

RQ3

Is INS a reliable tool for measuring in-situ engagement? We evaluate the reliability by comparing INS data across four performances of the same play. If the INS scores indicating audience engagement are accurate, then we can anticipate that the engagement profiles of the four performances will display comparable patterns. This is due to the expectation that the audience will react similarly to each performance. However, it is not expected that engagement profiles will be an exact match due to variations in audience members and performances.

Methods

Participants

Participants were recruited from a community in Hawaii, in collaboration with a theater department from {INSTIUTION} that was showing the play. To enroll for the study, individuals scanned a QR code provided on recruitment materials (e.g., fliers, email announcements), which directed them to an online survey that provided study information and screened for eligibility criteria. Inclusion criteria limited participants to adults 18 years old and older, English-speakers with access to a smartphone or smart device with either iOS or Android operating systems. Because of the nature of the biological data being collected, individuals with chronic pain were excluded from the study. Chronic pain introduces systematic differences into heart rate that cannot be corrected for statistically via the INS. Thus, INS recommends excluding from research studies.

Seventy-four individuals were invited to attend one of four evening performances (k = 4) of the same play in June 2022. We note that because the thematic content of the play could have been triggering to some participants, all participants were informed in advance of their participation of the play’s nature and its themes. No participants withdrew from the study upon learning of the play’s themes. Of the 74 invited to participate, n = 72 were included in the study. INS data was not recorded for two participants, and both instances occurred in the first data collection session (no total data loss occurred in any subsequent performances). These missing data are considered a procedural failure of the research team and not of INS technology; therefore, these participants were excluded from the analysis. Participants chose a performance based on their availability. Our sample size of 72 is larger than most in traditional psychophysiological studies (n = 20 to 25; Bastiaansen et al., 2022). Tickets to the play were free, and participants were offered food at the end of the study as an incentive for participation. On average, participants were 26.72 years old (SD = 8.89) and majority female (n = 44, 61.1%). Slightly less than half of participants self-identified as White (n = 33, 45.8%); two self-identified as Black (2.8%); one self-identified as American Indian/Native American or Alaska Native (1.4%); 20 self-identified as Asian (27.8%); 10 self-identified as Native Hawaiian or Other Pacific Islander (13.9%); two self-identified as Other (2.8); and four did not report. Eight participants self-identified as Spanish, Hispanic, or Latino (11.1%). Most participants self-identified as never been married (n = 53, 73.6%), 15 were married (20.8%), one self-identified as divorced/separated (1.4%), and three did not report. Most participants (n = 50, 69.4%) reported a household income of $10,000 - $24,999.

Structured experience

The theatrical performance “Double Cupcakes” is a one-act play (approximately 30-minutes) that was performed for four consecutive nights at a university theater. The stage was situated in a “black box” or “studio theater” configuration (i.e., a minimalist performance space with the floor of the stage at the same level as the first audience row; seating on three sides). Each performance was sold out. The play depicts the relationship between five female friends. In scene 1, the friends meet for the first time at their university orientation week. They bond and share snacks, including cupcakes. Scene two occurs several years later, with the five friends reuniting to learn that one friend, the main character, has been diagnosed with breast cancer. The scene concludes with more cupcakes being shared. The third and final scene depicts the four friends visiting the main character who is now in home hospice. They reminisce, and the four eventually leave their friend to rest on the couch while they go to the kitchen to prepare snacks. When they return to the living room, they realize their friend has passed away. The doorbell rings and cupcakes are delivered with a note from the main character. The now-group-of-four commemorates the passing of their friend with cupcakes, and the play ends.

Intended experience: In-situ engagement

According to the TSE (Ellis et al., 2019), engagement can be amplified in a structured experience using narrative-driven storytelling that elicits heightened emotion. We asked the playwright/director to provide a temporal profile outlining the dramatic structure based on Freytag’s pyramid—exposition, inciting incident, rising action, climax, falling action, resolution, and denouement. Based on these elements, the playwright was then asked to identify the key corresponding moments in the performance. Of these key moments, the research team selected two scenes that should result in in-situ engagement—specifically, we chose scenes that were expected to elicit both heightened attention and emotion (Rühlemann, 2022): the revelation of the cancer diagnosis and the death of the main character. The research team created and used field note observation documents to record the specific time stamp at which the scenes occurred during each performance.

Study design

Data were collected over four days (Tuesday, Wednesday, Thursday, and Friday), with a performance occurring at 7:30 p.m. each day. Participants arrived 30 – 45 minutes before curtain time and met in a classroom near the auditorium where the play would be performed. After completing the informed consent process, participants were provided with a heart rate (HR) sensor and instructed how to wear it. Participants then downloaded the INS app onto their smartphone and connected their HR sensor via Bluetooth. Research assistants and the lead field researcher recorded extensive observational field notes. Observational data included recording the hour:minute:second of pre-specified events (e.g., when the play started) as well as unanticipated events (e.g., if a participant fell asleep). The observing researcher would record the participant ID number (printed on adhesive stickers that participants wore facing outward on their shoulders, chest, and back), the event, and the timestamp of the event. Research assistants used their smartphones and a time-zone clock app to record the hour:minute:second of events. Other field notes included non-time-specific observations of audience size (i.e., crowding), audience reactivity, and the environment (e.g., temperature, lighting, aroma, etc.). These observations were used to determine whether any phenomena across the four performances would have introduced systematic differences into the INS data (caused by extreme variation in the unfolding of the structured experience). No such variation occurred.

After completing informed consent and setup, participants completed an online survey, watched a five-minute nature video as a baseline INS measure, and were then escorted to the theater. Participants were seated with other nonparticipating audience members to watch the performance. After the performance, participants returned to the designated classroom and completed a post-performance survey. After completing the study, participants were provided food (pizza, fruit, veggies, cookies, and drinks) as compensation for their involvement. We note that none of the survey data were used for the analysis of the reliability and validity of INS. Online supplementary materials are provided with description of pre- and post-survey measurement.

Measures

Lived experience: In-situ engagement

In-situ engagement was collected via cardiac sensors and the INS. As described above, heart rate data were collected via cardiac sensor at 1 Hz (Scosche Rhythm+ 2.0 or a Scosche Rhythm+). This HR sensor was worn on the forearm two-finger-widths below the crook of the elbow. Participants wore the sensor on the inside of the arm, with the sensor touching the skin. The sensor was firmly secured to the arm via an adjustable elastic armband so that inadvertent movement of the sensor was minimized. As detailed earlier in this paper, heart rate data collected by the sensor is transmitted to a piece of hardware (in this case, participants’ smartphones with the INS app installed). The INS app then transfers data to the online INS software, and data are transformed using the proprietary algorithm to generate second-by-second INS scores. Data were exported as CSV data file from the INS online software. Using field notes, data were synchronized with the start and end of each performance. Time stamps of key events were recorded in observation data and used to create experience segments, including the two high engagement scenes. Data were then analyzed.

Data analysis

Data were analyzed using STATA 13 (StataCorp. 2013. College Station, TX) and R (R Core Team. 2021. Vienna Austria). Data used in this study are INS scores collected from start of the performance to the end of the performance.

Feasibility

To address feasibility of INS (RQ1), researchers examine INS raw scores for missing data per individual. Although some controversy over margins for missing data exists (e.g., Madley-Dowd et al., 2019), we determined a priori that INS would be considered a successful tool for data collection if the following thresholds were achieved: No more than 20% of observations would be missing for 90% of all participants. On average, each participant netted 2,126 observations. Thus, for this criterion of feasibility, at least 67 participants (90%) would need to achieve 1,701 (80%) observations.

Validity

To address the validity of INS (RQ2), we examine the composite score of audience INS (i.e., group mean score per second) during two scenes anticipated to be high in engagement. We operationalize high engagement as an INS score of 66 or higher, which is identified by INS as a state of “absorbed,” and aligns with the conceptual definition of in-situ engagement. Because the structured experience was a live theater performance, some variation in the timing of each scene across the four performances occurred. Therefore, we use timestamped field observations to identify the beginning and ending of each scene for each performance. At a minimum, we examine the 15 seconds before and after the recorded time stamp noting the beginning of each scene. This was to evaluate if an instance of high engagement occurred as part of the rising action to the key moment (i.e., 15 seconds before time stamp) or reaction to key moment (i.e., 15 seconds after time stamp). If, however, engagement levels remained high beyond the initial 30-second window, then additional time points were included to provide a full picture of how long the high-engagement response lasted. Data selection for this analysis therefore continued until five consecutive seconds of engagement scores dipped below the INS threshold of “absorbed.” For this analysis, a total of eight comparisons are made (two scenes across four performances). Thus, for this criterion of validity, if an instance of INS engagement scores during the 30 second window meet or exceed the “absorbed” level (i.e., 66 or greater) in at least six of the intended high-engagement scenes (75%), then we would consider INS a valid measurement of in-situ engagement.

Reliability

Physiological-based time series data are often classified as nonlinear and nonstationary (Staffini et al., 2021). Therefore, to address the reliability of INS (RQ3), we examine synchrony among the four performances relying on a windowed time-lagged extension of cross-correlation, called the windowed cross-correlation (WCC; Boker, Xu, Rotondo, & King, 2002) using the composite score of audience INS (i.e., group mean score per second). WCC obtains cross-correlations between two time series when these time series are nonstationary—as is the case with live performances where the timing of scenes may differ slightly across performances. Using WCC, a pair of time series (i.e., two performances) are cross-correlated at differing lags. This is completed by lagging one short-window of observations positively or negatively in time allowing WCC to account for time delays between performances.

Following best practices identified in Boker et al. (2002) and Behrens et al. (2020), we specified four parameters: window size, window increment, maximum lag, and lag increment. The window size (wSize) determines how long each window is; for this study, the final parameter for analysis was set at 60 (which is a full minute of the performance). The window increment (wInc) indicates the size of the steps between two adjacent (overlapping) windows; in the final analysis, wInc = 5. The maximum lag (tMax) regulates how far the segments of the two time series are shifted away from each other; in the final analysis, tMax = 30. And the lag increment (tInc) determines the size of the steps with which the segments are shifted; in the final analysis, tInc = 1.

The score used for strength of synchrony between performances is the average correlation across the windows between two performances. For this study, this means that for every 60-second window (one minute), the analysis provided a correlation between the performances. There were at least 397 windows per performance. These correlations are then averaged to provide a single correlation of the performance. In some cases, the windows will not be correlated or even negatively correlated as the INS scores did not correspond together in real-time. However, in using the WCC analysis, the expectation is that if there is a similarity in performance patterns, then the average correlation will be positive.

Results

Table 1 provides summary statistics of the measurements for the four performances. Across 72 individuals who attended one of four performances, 150,919 observations were collected at the individual-level; and 8,503 observations were collected at group-level across four performances. The average length of performance was 36 minutes. The average audience engagement score across the four performances was 54.59 (SD = 13.48), which according to the thresholds established by INS, indicates the audience was “interested.” Across the four performances, the INS scores ranged from a minimum of 24 (indifferent) to a maximum of 95 (captivated), demonstrating INS is sensitive to variation in engagement scores.

Table 1. Summary statistics of INS observations of Engagement.

		In-Situ Engagement (INS score)
Performance	N	Observations	Mean	SD	Min	Max
1	13	2,071	49.57	14.79	19	94
2	20	2,201	54.48	13.27	21	96
3	20	2,106	54.36	13.55	26	96
4	19	2,125	59.94	12.29	28	94
Average	18	2,126	54.59	13.48	24	95

Note. Data is based on composite engagement-per-second scores (group-level). Interpretation example: in Performance 1, there were 13 individuals; the performance lasted for 2,071 seconds (i.e., observations). The composite mean engagement-per-second score across the 2,071 observations was 49.57 (i.e., interested), with a standard deviation of 14.79. The range of composite scores for Performance 1 was from 19 (i.e., indifferent) to (94 (i.e., captivated).

Figure 1 provides a descriptive summary of audience engagement across the four performances using INS-established categories. On average, most observations (68.2%) were categorized as “interested” in the performance, with an additional 24.3% of observations achieving either “absorbed” or “captivated” (i.e., high engagement). Only 7.53% of observations fell within the “indifferent” category, and no observations fell in the “tuned out” category” (i.e., disengaged).

Figure 1. INS Observation at each engagement level per performance. Note. Data is based on composite engagement-per-second (group). Interpretation of INS Scores: captivated (91–100), absorbed (66–90), interested (36–65), indifferent (11–35), and tuned out (1–10).

Feasibility

To address our first research question, we examined occurrences of missing data using raw scores at the individual level. Observations across the four performances ranged from 2,071 to 2,201 per person with 26,923 observations occurring in performance 1 (2,071 observations per 12 participants); 44,020 observations (2,201 observations per 20 participants) in performance 2; 42,120 observations (2,106 observations per 20 participants) in performance 3; and 4 a total of 40,375 observations (2,125 observations per 19 participants) in performance 4.

Table 2 provides an overview of missing data. Of the 72 participants, 97.2% of participants (n = 70) were missing less than 20% of observations. Some 63 participants (87.5%) had no missing data at all. Two participants (2.8%) were missing more than 20% of observations. One participant was missing 21% of observations, and the other participant was missing 66% of observations. Thus, results meet the criterion set for feasibility.

Table 2. Missing observations.

% of Missing Observations	n	% of Sample
0%	63	87.50%
1 to 5%	5	6.94%
5 to 10%	1	1.39%
10 to 15%	1	1.39%
15 to 25%	1	1.39%
25% to 50%	0	0.00%
50% to 99%	1	1.39%
100%	0	0.00%

Note. Data is based on individual engagement-per-second scores.

Validity

To address research question 2, we examined INS engagement-per-second scores during the performance of two scenes. The 30-second window allotted for the scenes in each performance rendered a total of 8,503 observations for analysis. Table 3 provides a summary of INS scores that met the criteria of 66 or higher per each scene and performance. It is also worth noting that Table 3 demonstrates the expected variability that naturally occurs across different nights of the performance. Findings provide evidence that all eight scenes elicited a high-engagement response from the audience. On average, high engagement during the first scene lasted 8.5 seconds with an average INS score of 68.49, with the highest score (77) reached in performance 1. High engagement in the second scene lasted almost twice as long, 15 seconds, with an average score of 72.15. The highest INS score during this scene was 87 in performance 1. Thus, results meet the criterion set for validity.

Table 3. High-engagement for diagnosis and death scenes.

Scene	Counts	Min	Max	M	SD
Cancer Diagnosis	8.5	66	72	68.49	2.09
Performance 1	13	66	77	70.54	3.71
Performance 2	3	66	67	66.67	0.58
Performance 3^a	8	66	71	68.25	1.91
Performance 4^a	10	66	72	68.50	2.17
Death	15	67	77	72.15	3.70
Performance 1	18	67	87	77.67	7.12
Performance 2	12	66	68	67.00	0.43
Performance 3	13	67	76	71.46	2.88
Performance 4	17	67	78	72.47	4.37

Note. Composite engagement-per-second (group-level) scores were used for this analysis. Counts is the number of INS scores at 66 or above in the designated timeframe. Counts can be interpreted as seconds.

^aNot consecutive seconds; INS score dipped below 66 and then increased to 66 and above within the designated timeframe.

Reliability

To address research question 3, we use engagement-per-second (INS) at the group level to examine synchrony in engagement across the four performances. A visual portrayal of similarities in data patterns can be found in Figure 2. Visually, the performances follow a similar pattern of engagement.

Figure 2. Profile of audience engagement for each performance. P1: Performance 1; P2: Performance 2; P3: Performance 3; P4: Performance 4; C: Cancer Diagnosis scene; D: Death scene.

Beyond visual comparison, we provide a quantitative expression of the similarities in the performance patterns. Table 4 provides an overview of the correlations between each performance; more details about the WCC analysis can be found in the supplemental materials. Table 4 provides a summary of the WCC results. The average WCC for all four performances was 0.31, and the median is 0.35. Performances 3 and 4 have the strongest synchrony—an average WCC of 0.43; and performance 2 and 3 have the weakest synchrony—an average WCC of 0.21. These results are interpreted as correlations but should be interpreted in the context of nonlinear and nonstationary data. Thus, results suggest that there was positive synchrony among the performances and therefore meet the criterion set for reliability.

Table 4. Cross-correlation and windowed cross-correlation between performances.

	Cross-Correlation	Windowed Cross-Correlation (WCC)
Performance	r	# of 60 sec Windows	r Mean	r Median	r Min	r Max
P1 & P2	0.12	397	0.26	0.30	−0.83	0.94
P1 & P3	0.08	397	0.31	0.36	−0.72	0.92
P1 & P4	0.06	397	0.38	0.42	−0.74	0.96
P2 & P3	0.05	404	0.21	0.22	−0.67	0.88
P2 & P4	0.12	408	0.30	0.36	−0.67	0.96
P3 & P4	0.30	404	0.43	0.46	−0.78	0.95
Average	0.12	401.2	0.31	0.35	−0.74	0.93

Note. Mean is the average correlation across the # of windows. Median is the middle of the set of numbers. Whereas, Min is the lowest windowed correlation and Max is the highest window correlation.

Discussion

For leisure scholars, in-situ engagement is an important component of structured experiences because it is a determinant of participants’ delight, perceived value, and positive affect (Ellis et al., 2022). Differences in definitions and measurement in previous research has led some scholars (e.g., Ellis et al., 2022) to call for increased specificity in conceptualization and operationalization of the construct engagement. Such specificity has the potential to advance empirical knowledge of engagement and enhance scholars’ ability to inform and shape practice. Therefore, the purpose of this study was to assess the feasibility, validity, and reliability of INS scores as a proxy indicator for in-situ engagement. Our findings provide initial support for the use of INS to measure in-situ engagement during structured experiences. Furthermore, our findings provide opportunities for conceptual connections and practical application for experience design.

Initial support for using INS: Feasibility, validity, and reliability

Findings from this study provide support for the feasibility of using INS to collect in-situ engagement data. Two participants’ data were lost due to researcher error. Minimal missing data occurred such that data were lost at rates of more than 20% for only two participants (2.8% of the sample). For 87.5% of the sample (n = 63), no missing data occurred. In comparison, it can be common for researchers to lose 10 or more participants due to missing data, equipment failure, or excessive artifacts when using electrodermal measurement of emotional arousal (Li et al., 2022).

Feasibility for individual research teams or contexts may be affected in other ways. As noted, INS requires additional software and hardware, the acquisition of which necessitates additional resources (e.g., financial resources for purchases; temporal resources to learn new processes and data measures). Indeed, physiological measurement procedures can increase researcher burden because data collection is a time-consuming endeavor that results in small sample sizes (e.g., Bastiaansen et al., 2022; Bolls et al., 2001; Guo et al., 2015; Ohme et al., 2009). In this study, the addition of physiological data collection added some time to the data collection procedures in the field (e.g., 15 – 20 minutes for participant setup and training). Subsequent data management and preparation for analyses add additional time requirements. Researchers seeking to add INS to their data collection procedures should consider these and other factors when determining feasibility. Overall, however, findings from this study offer initial support for the general feasibility of INS.

Current findings also provide initial support for the validity of INS. Given that in-situ engagement is described as a transitory state, a physiological measure of in-situ engagement would be expected to vary moment-to-moment as a structured experience unfolds (Ellis et al., 2022). Findings indicate that INS can detect variations in in-situ engagement during structured experiences, thus demonstrating a sensitivity to the conceptually expected transitory nature of in-situ engagement (see Figure 2). Indeed, data demonstrate the ability of INS to differentiate between distinct types of engagement levels as defined by INS (e.g., “indifferent”, “interested”, “absorbed”, and “captivated”; see Figure 1). Additionally, INS detected high audience engagement during 8 a priori scenes across 4 distinct performances (see Table 3).

Finally, our findings provide initial support for INS as a reliable data collection tool. Similar patterns of audience engagement emerged across all four performances (see Figure 2). Because live performances are nonstationary, it is difficult to identify similarities in patterns using quantitative analysis; however, using WCC, we also demonstrated statistical synchrony across the performances.

Altogether, based on the evidence for feasibility, validity, and reliability, we conclude that INS is a promising approach to collect data on in-situ engagement during structured experiences. Our cautious optimism stems from lessons learned in collecting the data presented here as well as those we have learned when collecting INS data in more than 20 other pilot and research studies. The lessons learned and nuances of using INS will be further detailed for interested scholars in a forthcoming methodological note. Though INS represents a promising approach to the study of in-situ engagement, it is not a panacea. As always, the utility of any given research method should be determined by theory and targeted research questions.

It is our viewpoint that different types of measurements and methods (e.g., survey, physiological, qualitative, quantitative) provide a unique value-add to each study. Thus, combining measurements and methods rather than relying on a single data stream or type of data in a study is best when time and financial resources allow. Doing so can enhance interpretability of findings, particularly when physiological data are present. Biological or physiological data can be challenging to interpret—and although INS is based on a straightforward measurement scale of 1 – 100, the meaning of an INS score at a given moment in each structured experience may be dependent upon participants’ self-report descriptions of the target of their attention, their individual antecedent characteristics, and/or group dynamics in the case of a shared experience. Thus, it is not the view of the author nor the purpose of this paper to suggest that one form of measurement is better than the other. Rather, we suggest social scientists need different tools to examine different facets of the same concept. Neurophysiological-based tools provide one additional method for understanding the human experience that can be used in combination with other tools to address research questions more fully. As these tools become increasingly affordable and interpretable, their accessibility and applicability to social science is amplified.

Conceptual connections and practical application

In Duerden’s (2022) address to the leisure field, he called for scholars and practitioners to become the leaders in designing experiences that promote human flourishing. To date, several books and journal articles have been written on the topic of experience design (e.g., Ellis et al., 2019; Pine & Gilmore, 2011; Rossman & Duerden, 2019; Szóstek, 2021). From these texts, one feature of successful experiences is that they are engaging. Other scholars have distinguished between (1) subjective experiences of psychological engagement characterized by immersion and complete concentration and (2) behavioral participation—a type of engagement indicated through “meaningful and sustained involvement in an activity outside of the self” (Ramey et al., 2015, p. 237). Conceptually, engagement as behavior may serve as the gateway to psychological or in-situ engagement, such that psychological or in-situ engagement is contingent upon participation or behavior (i.e., a person must engage in an experience behaviorally if any degree of psychological engagement is to be achieved). However, in the context of the theory of structured experiences (TSE) (Ellis et al., 2019), it is psychological or in-situ engagement that is of most value to the design and staging of structured experiences. Likewise, psychological engagement rather than behavioral engagement has been recognized as the driver of developmental outcomes when studying youth and adolescent leisure programs (Ramey et al., 2015). Finally, experiences have been typified by the degree to which attention and emotion are at play. In their experience type framework, Duerden and colleagues (2018) argued that conscious experience depends first upon attention (a component of psychological or in-situ engagement) being directed toward stimuli within the context of the experience. Then, heightened emotion elevates conscious experiences from ordinary to extraordinary (Rossman & Duerden, 2019).

It is in the realm of in-situ engagement, then, where perhaps the most fruitful avenues of scholarship and practice may be built. As scholars across multiple disciplines have noted, the construct of in-situ engagement is one that can be influenced. For example, engagement can be influenced by novelty (Ellis et al., 2022; Rossman & Duerden, 2019) and incongruity (Melton, 2017) as those elements draw attention and can elicit emotional responses. Within the theory of structured experience, Ellis and colleagues (2022) identify three additional prioritized pathways to or determinants of engagement: provocation, coherence, and personalization (Ellis et al., 2022). These three determinants can be effectively designed within the context of narrative-driven structured experiences. Specifically, Ellis and colleagues (2022) state that, within their theory of structured experience, “engagement…is a subjective experience associated with focusing attention on an unfolding story.” Therefore, theater performances like those examined in this study may provide an especially rich context for continued research into engagement.

Integrative models of human development emphasize the joint operations of biological, psychological, and social processes during leisure experiences (Kleiber & McGuire, 2016). New technologies provide opportunities and affordances to consider the physiological factors during structured experiences. Physiological measures can provide new insight. Thus, the advancement of tools that allow for passive collection of physiological data that represent in-situ engagement during leisure experiences may offer increased understanding. Scholars and practitioners alike may be better equipped to understand who is and is not engaged during a structured experience. Moreover, scholars and practitioners may be better positioned to assess the continuity of the experiences they provide. Specifically, both scholars and practitioners could use measurement tools such as INS to answer questions about engagement with increased precision. For example, moment-to-moment data on in-situ engagement provided through means like INS can help scholars and practitioners identify moments or events within structured experiences that lead to disengagement. They can then, through prototyping and testing, identify experience elements that may be effective in reducing or reversing such disengagement. Such lines of inquiry address and reiterate work underway by many researchers (e.g., Ellis et al., 2022) who focus on optimizing key determinants of engagement, and the outcomes of increased engagement.

Limitations

This paper is among the first to report on a systematic evaluation of the reliability and validity of the INS software. Although data and analyses support the feasibility, validity, and reliability of INS, those conclusions are based data gathered using the INS proprietary algorithm. Our study did not directly test the INS algorithm via direct assessment of the neurotransmitters upon which the platform is built (i.e., oxytocin and dopamine). Nor did our study compare the scores generated by the INS platform to raw heart rate data. Future studies may consider these options; however, doing so would represent a substantial investment of resources and a need for additional expertise (e.g., neuroscience). To continue to test the validity of INS, future research may consider examining not only expected peak moments of performances but also any expected trough or low engagement moments. We did not test for any additional confounding variables that may influence INS scores beyond chronic pain, as directed by the creators of INS; however, future research may want to test the influence of factors such as caffeine intake, disability, or other chronic conditions. Additionally, this study did not seek to determine the relationship between physiological and survey data on in-situ engagement. Future studies may consider adding survey measures of in-situ engagement to determine the correlation between psychological and physiological measures. However, scholars in other fields have established that physiological and psychological measures will not necessarily be correlated (e.g., Hoareau et al., 2021; Morrow & Labrum, 1978). We used commercial grade of heart rate monitors to collect data. These heart rate monitors are deemed appropriate for research studies (Gillinov et al., 2017). Different sensors, however, could create variability in data collection, particularly if used in a physically active experience (see Gillinov et al., 2017). It remains a question whether systematic differences would emerge across different sensors, and if those different scores could be compared. Future research may also seek to examine this.

Conclusion

In this study we tested the feasibility, validity, and reliability of a passive, non-self-report approach to measuring in-situ or psychological engagement. Our findings, derived from the assessment of four live theater performances (a narrative-based structured experience), provide initial support for the feasibility, validity, and reliability of this approach. Advancing research on engagement in the context of experience design depends upon effective and high-quality measurement. Thus, the findings from this study may spur additional progress that can be translated from scholarship to practice and inform the design of effective structured experiences in leisure and other contexts.

Acknowledgments

Data collection for this project was supported by undergraduate researchers from BYU-Hawaii and BYU, including Bryan Larson, Morgan Mickelsen, Abri Dela Cruz, Joey Fung, and McKena Lowther. Funding for this project was supported by internal funds from the authors’ universities. Funding for Open Access was made possible by BYU.

Ethical approval

The research study (#22-46) was approved by the Institutional Review Board of Brigham Young University-Hawaii on May 26, 2022.

Disclosure statement

No potential conflict of interest was reported by the author(s).