Exercisable Learning-Theory and Evidence-Based Andragogy for Training Effectiveness using XR (ELEVATE-XR): Elevating the ROI of Immersive Technologies

Abstract

eXtended Reality (XR) promises to provide a powerful capability to disseminate a vast array of instructional content appropriate for all types and levels of learners, ultimately revolutionizing education and training. Yet, while XR presents an opportunity for a powerful paradigm shift in education, a scientifically grounded andragogy for harnessing the didactic power of XR has yet to emerge. Herein, a unified framework of research-based instructional strategies for guiding the design of proficiency-based XR educational and training content is proposed, along with an associated naval maintenance use case. While this work draws on relevant pedagogical and andragogical theories, as well as fundamental learning theories applicable to learners of all ages, the framework specifically targets adult learners and training of complex skills applicable to various professional domains. Within the framework, learning theories have been aligned with learner proficiency level to specify competency-based learning objectives and activities, XR form factors, and learning loops that are anticipated to boost XR training efficacy. The proposed framework—the Exercisable Learning-theory and EVidence-based Andragogy for Training Effectiveness using XR (ELEVATE-XR)—is meant to serve as a guide to drive effective and efficient design of XR applications that enhance adult learning.

Keywords:

• Augmented reality
• extended reality
• adaptation
• education
• training
• simulation
• andragogy
• learning theory

1. Introduction

As interactive technologies advance so do their application to education and training. For example, e-learning, gamification, serious games, and interactive learning environments have all seen significant growth in utilization in recent years (Bennani et al., 2022; Squires, 2019; Wang et al., 2022). In this vein, eXtended Reality (XR) technologies (e.g., the umbrella term that encompasses all real-and-virtual combined environments and human-system interactions generated by computer technology and wearables, including representative forms such as augmented reality (AR), mixed reality (MR) and virtual reality (VR); Blumberg, 2018; Fast-Berglund et al., 2018) and their application to learning have been increasingly recognized by researchers and practitioners alike (Sirakaya & Alsancak-Sirakaya, 2018; Ziker et al., 2021). So much so, that by 2027 it is anticipated that the XR education market will conservatively reach $22.4B (IndustryArc, 2022). XR technologies’ value to education and training derive from their potential to provide authentic, appropriately realistic learning experiences that increase skill acquisition and knowledge retention. Meta-analyses have quantified these gains from XR learning experiences.

• In VR training, learning gains (pre- to post- test versus traditional training) average 2.5% more per 1⁄4 hour of training for cognitive skills and 2.95% per hour gain in technical skills (Angel-Urdinola et al., 2021). From a performance perspective, as compared to traditional training VR yields, on average, ∼30% more performance efficiency and equivalent fewer errors, with ∼30% higher confidence and self-efficacy, though standard deviations tend to be high. Immersive VR benefits over more traditional approaches have mostly been demonstrated with regard to cognitive learning outcomes, especially in areas requiring high degrees of visualization and experiential understanding. In terms of benefits, VR has been found to support development of cognitive (e.g., remembering and understanding spatial relationships or concepts; knowledge acquisition), procedural (e.g., ability to understand and apply a technique; however, see refuting evidence from Chen et al., 2020), gross psychomotor (e.g., visual scanning and observational skills), and affective (e.g., building resilience to performance under stress; emergency response procedures) skills (Abich et al., 2021; Davids et al., 2021; Jensen & Konradsen, 2018). However, some studies have found VR to be equally or even less effective than more traditional training approaches (Hamilton et al., 2021; Howard et al., 2021). As discussed later, some VR features may result in increased cognitive load, and therefore may not be appropriate for less experienced learners (Fiorella & Mayer, 2021; Parong & Mayer, 2108, 2021). Thus, it is essential to consider task-technology fit when specifying VR training programs, as no XR form factor is “best” across all contexts (Hamilton et al., 2021).

• Learning gains from AR/MR training versus traditional strategies are also notable, with overall average effect sizes in the medium range of 0.36–0.68 (Garzón et al., 2020; see Cohen [1988] for effect size ranges). Drilling down deeper, it has been reported that average effect sizes tend to fall in the large range when using AR/MR to improve performance (0.74) (Chang et al., 2022) and in informal (non-classroom) settings (0.8), as well as in the very large range when using AR/MR in manufacturing settings (1.24) (Garzón & Acevedo, 2019). Further, effect sizes may increase (on average by 0.03) with each additional hour of AR/MR training, up to 18 hours before effects taper off (Chang et al., 2022). Thus, to realize learning gains training time matters. In this regard, meta-analyses have found that optimal learning gains from AR/MR-based interventions occurred with programs lasting one to four weeks and may require as much as twelve weeks to realize (Chang et al., 2022; Garzón et al., 2020; Li et al., 2021). From a performance perspective, AR/MR tends to increase retention, improve physical task performance, improve collaboration, increase student motivation, interest, and engagement, encourage student exploration and creativity, decrease performance time and reduce errors, while potentially increasing physical and mental workload (Di Pasquale et al., 2022; Garzón et al., 2020; Kaplan et al., 2021; Perifanou et al., 2022). In terms of detrimental learning effects, AR/MR may lead to attention tunneling, perceptual issues associated with the headset, which could both hinder learning, and, in addition, may not benefit high-achieving learners (Radu, 2014). Thus, AR/MR training solutions must be carefully designed and matched to learner capabilities.

In general, XR technologies have the capability to disseminate a vast array of instructional content ranging in fidelity, complexity, affordances, and limitless combinations of virtual and real components. Yet, relatively little is known regarding how best to apply this medium for training purposes (Howard et al., 2021). Further, while there are many XR use cases, perhaps only about one third have been developed according to learning theories (Doran, 2021; Matovu et al., 2022; Radianti et al., 2020; Rousell, 2019; Zahabi & Razak, 2020; Zhang & Wang, 2021). Those that are grounded in educational theory have been found to be more effective than unstructured curricula (Khan et al., 2019). Within the subset considering theories, constructivism may be most frequently considered, with cognitive load theory also often cited (Kavanagh, Luxton-Reilly, Wuensche, & Plimmer. 2017; Putnam et al., 2020). Even when considered, however, these theories were often not directly applied to the design of XR learning activities (i.e., lacked a direct link between theory and design). Thus, there has yet to be a defining andragogy (Galustyan et al., 2019) that establishes unity between theory and practice nor has a repertoire of scientifically grounded strategies that direct and optimize adult learning in XR applications been developed (Guilbaud, Guilbaud, & Jennings, 2021; Perez et al., 2016). This is important because it is the underlying andragogy of a learning environment, regardless of the technology or process it is built upon, that is typically responsible for ensuing learning gains (Clark, 1983). While taking a learning theory approach cannot provide a complete and definitive compendium that will address all XR andragogical questions, it can provide a tool to organize and translate theoretical principles into design recommendations that aim to realize substantial learning gains. Further, it is vital to examine how XR technologies can be aligned with both theoretical and practical andragogical principles, as poor design and use of XR training has been shown to lead to negative transfer of training and reduced learning gains (Barsom et al., 2016; Parong & Mayer, 2018, 2021). The purpose of this article is to present an andragogy, grounded in both theory and practice, that provides an understanding of the underlying principles that need to be upheld to realize reliable learning gains from XR applications.

2. ELEVATE-XR framework

This article proposes the Exercisable Learning-theory and EVidence-based Andragogy for Training Effectiveness using XR (ELEVATE-XR framework), which unifies aspects of learning theories and educational frameworks to provide both theoretical (i.e., why) and practical (i.e., how) foundations to guide the design of XR applications that enhance adult learning across the continuum from novice to expert.

2.1. Theoretical underpinnings for the application of XR technologies in learning

While there is no definitive understanding of how individuals learn, there are a plethora of learning theories that can provide a theoretical basis for the application of XR technologies within education and training. These theories generally fall under three main umbrellas (Goel, 2011), including behaviorism (i.e., learning involves changes in behavior due to linking stimuli and responses), cognitivism (i.e., learning is a result of processing and organizing information in a meaningful way), and constructivism (i.e., learning involves constructing and adapting new knowledge by experiencing authentic tasks anchored in meaningful contexts). To ensure a rich set of practices were developed, learning theories across these categories were incorporated into the ELEVATE-XR framework. Foundational principles from each theory and how they apply to the XR andragogy are described next (see Table 1).

Table 1. Implications of learning theories to XR Andragogy.

Learning theory	Implications to XR andragogy
Behaviorism Theory	Structure scenarios to support observing, imitating, and embodying modeled correct behavioral responses, as well as discovery and invention of new meaningful behaviors that enhance performance; arrange scenarios such that difficulty level primarily elicits positive reinforcement, and provide feedback to motivate desired performance outcomes
Cognitive Load Theory	Consider level of stress, complexity, assistance, feedback, and fidelity (SCAFFolding) that is appropriate given proficiency of the learner
Component Display Theory	Involve all performance forms (remembering, using, generalizing) for all content types (facts, concepts, procedures, principles)
Conditions of Learning Theory	Ensure conditions of learning (gain attention, state objective, stimulate recall, trigger selective perception, foster semantic encoding, trigger a meaningful response, provide feedback, assess performance, foster generalization) are met across the hierarchy of learning
Embodied Learning Theory	Involve learners in realistically embodied behaviors that are highly integrated into task performance and avoid incidental forms of embodiment
Constructivist Theory	Involve learners in making connections based on authentic inquiry and active engagement with the environment to extend basic declarative knowledge and build schemas; provide feedback that results in revision and restructuring of previous knowledge constructs
Experiential Learning Theory	Engage learners in concrete, self-directed, hands-on learning experiences that support reflection, abstract conceptualization, and active experimentation to construct knowledge based on experience
Situated Learning Theory	Connect content and context through situated experiences in authentic domain settings and situations

• Behaviorist Learning Theory: This theory suggests that learning is a behavioral response to environmental stimuli, which requires repetition (i.e., learning by doing) and extrinsic (i.e., negative versus positive reinforcement) motivation (Skinner, 1989). To enhance learning, early learners can observe modeled correct behavioral responses. As understanding grows, learners can begin to imitate and embody target behaviors. Eventually, learners can discover or invent new behaviors that enhance performance (Murtonen et al., 2017). Throughout learning, responses that are followed by positive reinforcement are more likely to be retained. Thus, according to this theory, XR scenarios should support observing, imitating, and embodying modeled correct behavioral responses, as well as discovery and invention of new meaningful behaviors that enhance performance, be arranged such that the difficulty level elicits positive versus negative reinforcement, and provide feedback to motivate desired performance outcomes (Moreno, 2004; Shute, 2008).

• Cognitive Load Theory: This theory, which falls under cognitivism, posits that learning experiences should be designed to reduce working memory “load” in order to promote acquisition of schemas (i.e., knowledge structures and associated retrieval cues that support recall of learned information; Jalani & Sern, 2015; Sweller et al., 2019; Paas & van Merriënboer, 2020). Specifically, as only a few elements (or chunks) of information can be held in working memory and processed at any given time without overloading a learner’s cognitive capacity, cognitive overload decreases the effectiveness of schema development. Further, there are three different factors that contribute to total cognitive load, including intrinsic (i.e., load inherent to the content to be learned), extraneous (i.e., load evoked by the instructional experience), and germane (i.e., load imposed by the learning processes) load (de Jong, 2010). Extraneous load is typically high early in learning before schemas are formed that can guide attention during learning activities. Germane load is typically high once competency has been achieved, as it involves more complex reasoning during learning, such as through interpreting, exemplifying, classifying, inferring, differentiating, and organizing new knowledge. Thus, according to the Cognitive Load Theory, XR learning scenarios should start with simple (as opposed to complex), primarily passive, observational tasks for which extraneous cognitive load is managed. XR learning scenarios should provide assistance to orient learners to relevant cues but fade the assistance as proficiency advances from dependence to relative dependence to relative autonomy to autonomy (Hu & Zhang, 2017). Additionally, these scenarios should provide feedback—progressing from immediate outcome-based (e.g., knowledge of correct response) to immediate elaborative (e.g., explanations) to immediate/delayed strategy-based (e.g., strategic hints that help learners reject erroneous and consider alternative hypotheses) (Fyfe et al., 2015; Taxipulati & Lu, 2021), gradually increase complexity (e.g., by advancing from worked examples to conventional tasks), and distribute cognitive load across modalities. Fidelity of XR-based training can contribute to cognitive load. For example, presenting too much vivid detail and visual information can overload a learner’s cognitive capacity during VR training and thus hinder learning gains (Albus et al., 2021). Similarly, multimedia learning has been shown to result in extraneous overload (Fiorella & Mayer, 2021). The free-play nature of VR could, in particular, exacerbate extraneous processing, as VR training is generally designed without a specified sequence and thus it permits learners to follow any learning path that captures their interest regardless of their level of readiness. This may affect novices, who are prone to high levels of extraneous processing, most profoundly (Fiorella & Mayer, 2021; Parong & Mayer, 2018, 2021), and thus a guided VR experience may be preferred for less experienced learners (Simeone et al., 2019). Further, in some contexts there is mixed evidence for skill acquisition and transfer efficacy via VR, particularly with regard to psychomotor skills training at the various stages of skill acquisition (Nassar et al., 2021). It may be that in some cases the internal model for psychomotor tasks created during VR training cannot readily generalize to real world tasks because in VR perceptual-motor couplings may be altered due to a lack of or altered haptic feedback (Levac et al., 2019), which may hinder training transfer from VR training solutions focused on development of psychomotor skills. Conversely, it has been shown that experts do not necessarily require haptic feedback to perform well in immersive VR environments (e.g., estimating the thickness of custom-made membranes during robotic surgery), and it is suggested that their expertise may enable them to rely on visual, rather than tactile, cues, thereby making VR an appropriate form factor for more proficient learners (Meccariello et al., 2016). As proficiency advances, cognitive load can be increased within XR scenarios via cognitive (e.g., uncertainty, time pressure, fidelity), physiological (e.g., pain, fatigue, environmental stress), and affective (e.g., emotional scenarios) stressors, which may foster more reasoning-based activities by advanced learners.

• Component Display Theory. This theory, which falls under cognitivism, classifies learning along two dimensions: content (facts, concepts, procedures, and principles) and performance (remembering, using, generalizing) (Merrill, 1994). To facilitate learning, according to the Component Display Theory, XR scenarios should involve all performance forms for all content types.

• Conditions of Learning Theory: This theory, which falls under cognitivism, suggests that learning is hierarchical based on its complexity, advancing from stimulus recognition to response generation to procedure following to use of terminology to discriminations to concept formation to rule application to problem solving (Gagne et al., 2004). Further, to foster learning several events must take place including gaining attention, informing learners of the objective, stimulating recall of prior learning, presenting stimuli to trigger selective perception, providing learning guidance to foster semantic encoding, eliciting performance by triggering a response, providing feedback to reinforce learning, assessing performance by eliciting knowledge retrieval, and enhancing retention and transfer through fostering generalization. Thus, XR scenarios can meet the conditions of learning by incorporating these events across the hierarchy of learning.

• Embodied Learning Theory: This theory, which falls under cognitivism, suggests that cognitive learning processes are inextricably linked with sensory and motor functions within the environment, including gestures and other human movements (Skulmowski & Rey, 2018). Specifically, a combination of bodily engagement (i.e., degree to which bodily activity is involved in learning from observing gestures to using peripheral devices to making bodily gestures to physical locomotion) and task integration (i.e., whether or not bodily activities are integral—rather than incidental—and meaningfully related to learning tasks) leads to richer encoding and therefore richer cognitive representations and greater learning gains. The latter can lead to an intuitive feel and eventually automaticity (Soylu, 2016). However, while involvement of more basic motor systems seems to reduce load on working memory during learning (Goldin-Meadow et al., 2001), high levels of bodily embodiment can increase cognitive load; therefore, prudence is necessary. Thus, to facilitate learning, XR scenarios should allow for realistically embodied behaviors that are highly integrated into task performance and avoid incidental forms of embodiment.

• Constructivist Learning Theory. This theory suggests that learning is an active process in which the learner must discover, construct and transform knowledge into cognitive structures called schemas that provide meaning and organization to information (Bruner, 1990). To construct knowledge, early learners should be provided the opportunity to observe, discover, and recognize relevant information and then choose meaningful hands-on activities that help them interpret and make sense of domain-specific information. Once developed, domain-specific, hierarchically organized schemas allow more competent learners to categorize different problem states and select the most appropriate actions. After having been sufficiently developed and applied, proficient and expert learners develop schemas that provide an intuitive sense of situations given the goal, and ultimately operate under automatic rather than controlled processing (Kotovsky et al., 1985). Thus, based on constructivism, XR scenarios should involve learners in actively making connections and extending basic declarative knowledge (facts and concepts) to build schemas as their proficiency advances.

• Experiential Learning Theory: This theory, which falls under constructivism, posits that learning occurs in four stages—concrete learning (i.e., self-directed, hands-on experience), reflective observation (i.e., intentional analysis of learning experience from multiple perspectives), abstract conceptualization (i.e., set goals to connect or form new ideas and concepts based on experience), and active experimentation (i.e., form and test hypotheses based on newly created ideas and concepts in new situations; try out actions involving decision making and problem solving; Kolb, 2015). Thus, XR scenarios should support experiential learning through providing concrete experiences learners can be physically immersed in to reflect upon and decode what is taking place, compare to previous knowledge to deepen understanding of underlying concepts, and engage in active decision making and problem solving, informed by learned concepts, to advance their knowledge.

• Situated Learning Theory: This theory, which falls under constructivism, suggests that learning activities must be situated in an unpredictable and authentic context and involve changes in knowledge and action so that learned principles can be generalized (Lave & Wenger, 1991). Contextualization is considered key, which refers to the degree to which to be learned content is connected, through experience, to a real-world context and community of practice (Giamellaro, 2017). The more contextualization that is used, the greater the quality of learning, as learners can be required to attend to relevant areas of a task context and demand authentic levels of attentiveness and effort as if they were engaged in real-world activities. As situated learning is typically ill-structured, transfer and generalization can be fostered through incorporation of activities that promote reflection within the design of the learning environment (Stoner & Cennamo, 2018). Specifically, during situated learning experiences, learners can be provided with metacognitive strategies to enhance reflection-in-action (i.e., reflection in situ to understand performance outcomes). Such metacognitive strategies can take on multiple forms and evolve as the learner’s proficiency advances, such as from identifying and summarizing the delta between desired and actual performance, to explaining and exemplifying desired performance, and from relating and transferring knowledge to new situations, to focusing learning on situational dissonance—differences, contradictions, discrepancies, and fostering consideration of alternative hypotheses based on situational factors. Thus, based on the Situated Learning Theory, the degree to which content and context are connected through situated experience and foster reflection should be central to the application of XR in adult learning applications.

Beyond learning theories, to effectively inform the design of XR-based adult learning solutions, an XR andragogy needs a framework within which to wrap principles derived from learning theories. In particular, a competency-based framework can support bridging the gap between the theoretical and the practical aspects of andragogy (Ermenc et al., 2015). Thus, ELEVATE-XR incorporates such a framework based on the Dreyfus and Dreyfus (1980) model of skill acquisition and Bloom’s Revised Taxonomy (Anderson et al., 2001).

• Skill Acquisition Model: This model can be used to assess the competency level of learners. Competence is the ability to successfully meet complex demands in a particular context through mobilization and integration of knowledge, action, and reflection (Ermenc et al., 2015). According to the Dreyfus and Dreyfus (1980) Skill Acquisition Model, to achieve competence, learners progress through a series of five proficiency levels: novice, advanced beginner, competent, proficient, and expert. According to this model (Dreyfus, 2004; Persky & Robinson, 2017; Rousse & Dreyfus, 2021; van de Pol et al., 2015):

  • Novices are focused on absorbing declarative knowledge, but they lack schemas and thus are inclined to extraneous processing. As a result, they tend to find it difficult to determine the essential parts of learning material. Novices thus learn best with focused, guided (e.g., step-by-step instruction) scenarios that support following rules, discriminating relevant situational features, prioritizing important information, and organizing new knowledge.

  • Advanced beginners start to understand the context of a situation, can quickly access the particular rules that are relevant to a specific task and/or context, and formulate organizing principles into maxims. However, they still lack an understanding of how to filter incoming information based on relevance, and thus can become easily overwhelmed. Those at the advanced beginner stage thus learn best with increasingly complex scenarios that require application of rules and choosing a course of action, connecting new knowledge with existing knowledge, integration of extraneous information, provision of specific and targeted feedback, and techniques to manage stress.

  • At the third stage, the competent individual no longer struggles with basic rules, is able to develop intuition to guide their decision making, and has the ability to devise personalized rules to formulate plans. Learning at the competent stage can be facilitated by provision of authentic, complex scenarios and inverse problems (i.e., those that require calculating from a set of observations, causal factors that produced them) that foster self-reflection, consideration of “why” decisions are made, and autonomy but with supportive feedback when needed.

  • At the fourth level, the proficient individual can see the big picture, know which cues and information to focus on, and have an intuitive sense of the goal given the situation. Learning can be facilitated at this stage through provision of complex, unique scenarios that foster goal-setting, solving problems in novel and imaginative ways, managing multiple distractions and emotional stimuli, self-reflection, and use of intuition.

  • The expert no longer needs rules and can work intuitively, knowing what’s important and what’s not, what to do, and what is the expected outcome in any given situation. Learning at the expert stage can be facilitated by providing scenarios that deal with uncertainty during decision making and challenge one’s understanding, reflecting on a situation while engaging metacognitive knowledge and regulatory control processes that foster discovery of new knowledge, and sharing knowledge with others.

To define learning objectives and associated activities appropriate for each proficiency level, these five-stages of mastery can be mapped on to concepts of cognitive development characterized by Bloom’s Revised Taxonomy.

• Bloom’s Revised Taxonomy: This two-dimensional taxonomy (Anderson et al., 2001) provides learning objectives that trainees should be able to accomplish in terms of cognitive process dimensions and knowledge dimensions. The cognitive process dimensions represent a continuum of increasing cognitive complexity- from lower order to higher order thinking skills (i.e., remember, understand, apply, analyze, evaluate, create). Within each cognitive process dimension, the knowledge dimensions also evolve from concrete factual to conceptual to procedural and finally to more abstract metacognitive knowledge. XR scenarios can be structured around the intersection of the cognitive processes and knowledge dimensions through learning activities.

  •  ○ Novice Learning Objective and Activities: At the novice level, the primary objective is to learn and remember facts, concepts, and procedures (i.e., declarative knowledge). This objective can be met by designing activities that involve defining, describing, discovering, identifying, arranging, labeling, listing, matching, naming, observing, and recalling new knowledge, among other related activities.

  •  ○ Advanced Beginner Learning Objective and Activities: At the advanced beginner level, the primary learning objective is to demonstrate understanding of newly learned facts, concepts, and procedures. This objective can be met through activities that foster organizing knowledge into schemas and using them to assess, compare, classify, differentiate, semantically encode, imitate, explain, recognize, respond, translate, and interpret situations, among other related activities.

  •  ○ Competent Learning Objective and Activities: At the competent level, the primary learning objective is to problem solve by applying acquired facts, concepts, and procedures (i.e., procedural knowledge). This objective can be met through activities that require demonstrating, employing, examining, experimenting, illustrating, implementing, integrating, and using acquired knowledge in a new situation, among other related activities.

  •  ○ Proficient Learning Objective and Activities: At the proficient level, the primary learning objective is to analyze acquired knowledge to break it up into constituent parts, determine how the parts are interrelated, and identify generalities. This objective can be met through activities that require deconstructing, deducing, differentiating, explaining, integrating, linking, organizing, predicting, relating, and structuring acquired knowledge, among other related activities.

  •  ○ Expert Learning Objective and Activities: At the expert level, the primary learning objective is to use acquired knowledge to evaluate situations to create new patterns or propose new, alternative solutions. This objective can be met through activities that require analyzing, contrasting, critiquing, estimating, inventing, hypothesizing, synthesizing, and validating new knowledge structures, among other related activities.

The XR implications derived from learning theories (see Table 1) can be brought together with the competency-based framework outlined above to provide a theoretical foundation for designing XR learning scenarios (see Table 2). By building on this theoretical foundation, effective means of linking theory to practice can be devised from which to derive practical guidelines.

Table 2. XR Competency-based Learning Theory Framework.

	Proficiency Level
Learning Theory	Novice	Advanced Beginner	Competent	Proficient	Expert
Behaviorism Theory	Observe modeled correct behavioral responses; Provide repetition & in situ reinforcement	Guide & imitate modeled correct behavioral responses; Provide repetition & in situ reinforcement	Embody target behavioral responses; Provide repetition & in situ reinforcement	Discover new meaningful behaviors; Provide repetition & in situ reinforcement	Evaluate & create new meaningful behaviors; Provide repetition & post hoc reinforcement
Cognitive Load Theory	Provide simple, passive observation-based scenarios, minimal stress, orienting assistance, immediate outcome-based feedback	Provide guided yet challenging hands-on scenarios, minimal contextually relevant stressors, relevance assistance, immediate elaborative feedback	Provide relative autonomy, authentically complex scenarios, contextually-relevant stressors that don’t overtax learner, facilitative feedback	Provide autonomy with limited assistance, high complexity real-life scenarios, high frequency / high fidelity stressors, strategic feedback	Provide autonomy with no assistance, high complexity real-life scenarios, novel, uncertain, high tempo stressors, strategic feedback
Component Display Theory	Focus on absorbing & remembering facts	Focus on understanding (e.g., assess situation, recognize consequence) concepts & procedures	Focus on applying concepts & procedures in self-guided scenarios	Focus on analyzing & generalizing facts, concepts, procedures, principles	Focus on evaluating & extending facts, concepts, procedures, principles
Conditions of Learning Theory	Orient & gain attention via “absorb” activities (e.g., state objective, discover, observe, recall), use Q&A appraisals	Trigger selective perception via “do” activities (e.g., assess, recognize, respond, encode), use Q&A appraisals	Trigger intuition via “elaboration” activities (e.g., examine, experiment, integrate), use practical skills appraisals	Foster generalization via “connect” activities (e.g., deconstruct, link, deduce, explain, predict), use peer rating appraisals	Foster generalization via “transfer” activities (e.g., synthesize, analyze, estimate, invent), use self-assessment appraisals
Embodied Learning Theory	Involve learners in observing realistically embodied behaviors that are associated with task performance	Involve learners in imitation of realistically embodied behaviors that are associated with task performance	Involve learners in executing embodied behaviors that are highly integrated into task performance	Involves learners in situation discrimination to foster discovery & selection of intuitive situational responses	Involves learners in situations that foster creation & generation of automaticity in situational responses
Constructivist Theory	Provide activities that help learners discover & recognize relevant information	Allow learners to choose meaningful hands-on activities to help them interpret & make sense of information	Engage learners in self-initiated activities to differentiate problem states & select appropriate actions	Immerse learners in real-life experiences & provide time & opportunity to goal-set & generalize knowledge	Immerse learners in real-life experiences that foster critiquing existing & generating new knowledge
Experiential Learning Theory	Support observation of concrete learning activities (i.e., recognizing & retrieving relevant information)	Support concrete learning activities (i.e., hands-on experiences that require applying declarative knowledge in context)	Foster reflective observation (i.e., intentional analysis of learning experience from multiple perspectives)	Foster abstract conceptualization (i.e., set goals to connect or form new ideas & concepts based on experience)	Foster active experimentation (i.e., form/test hypotheses); try out actions involving decision making & problem solving
Situated Learning Theory	Situate within conceptually focused scenarios; Reflect in situ by summarizing delta between desired & actual performance in a given situation	Situate within procedurally focused scenarios; Reflect in situ by explaining & exemplifying desired performance in a given situation	Situate within embodied, integrative learning experiences; Reflect in situ by relating & transferring knowledge to new situations	Situate within complex, discovery scenarios; Reflect in situ by focusing on situational dissonance (differences, contradictions, discrepancies)	Situate within complex, generative scenarios; Reflect in situ by fostering consideration of alternative hypotheses based on self-critique

2.2. Practical application of theory in XR technologies in learning

By linking the theoretical framework (see Table 2) to competency, and aligning learner needs to XR affordances, the ELEVATE-XR framework (see Table 3) provides a means to devise practical XR learning principles that can be used to tailor training to an individual’s level of proficiency. This andragogy defines the learning objectives, learning activities, and learning loop (i.e., a series of activities focused on fostering learning), and aligns these to appropriate XR form factors (i.e., VR, AR, MR) most suitable for a given proficiency level. Additionally, this framework introduces theoretically- and evidence- based XR design guidelines for appropriate use of critical scenario elements (i.e., Stressors, Complexity, Assistance, Feedback, and Fidelity) across the continuum from novice to expert, referred to by the authors as “XR SCAFFolding”. If effectively implemented, these practical principles are anticipated to lead to highly effective XR adult learning applications.

Table 3. ELEVATE-XR Framework.

	Proficiency Level
	Novice	Advanced Beginner	Competent	Proficient	Expert
Over-Arching Learning Objective	Remember – Facts, Concepts, Procedures	Understand – Facts, Concepts, Procedures	Apply – Facts, Concepts, Procedures	Analyze – Facts, Concepts, Procedures	Evaluate/Create – Facts, Concepts, Procedures
Learning Activities	Define, Describe, Identify, Arrange, Label, List, Match, Name, Recall	Compare, Classify, Differentiate, Imitate, Explain, Translate, Interpret	Demonstrate, Employ, Examine, Execute, Illustrate, Manipulate, Use	Deconstruct, Differentiate, Deduce, Organize, Relate, Structure	Assess, Contrast, Critique, Invent, Hypothesize, Validate
Appropriate XR Form Factors	AR, Guided VR	AR, MR, Guided VR with Force Feedback	AR, MR, Free-Form VR with Haptics	AR, MR, Free-Form, Multimodal VR	AR, MR, Free-Form, Multimodal VR
Learning Loop	Situate > Orient > Observe > Reflect by Summarizing > Appraise via Remembering > Repeat	Situate > Guide > Organize > Reflect by Exemplifying > Appraise via Understanding > Repeat	Situate > Give Autonomy > Embody > Reflect by Relating > Appraise via Applying > Repeat	Situate > Goal-Orient > Generalize > Reflect by Contradicting > Appraise via Analyzing > Repeat	Situate > Challenge > Create > Reflect by Self Critiquing > Appraise via Evaluating > Repeat
SCAFFolding Scenario Elements	Stress: Low Complexity: Low Assistance: High Feedback: Immediate outcome-based (e.g., knowledge of correct response) Fidelity: Low	Stress: Low Complexity: Medium Assistance: Medium Feedback: Immediate elaborative (e.g., explanations) Fidelity: Low (Visual) with High (Force Feedback)	Stress: Medium Complexity: Medium Assistance: Medium Feedback: Immediate facilitative (e.g., advice, suggestion) Fidelity: Medium (Visual) with High (Haptics)	Stress: High Complexity: High Assistance: Low Feedback: Immediate strategy-based (e.g., in situ hints) Fidelity: High	Stress: High Complexity: High Assistance: None Feedback: Delayed strategy-based (e.g., point out inconsistencies) Fidelity: High

2.2.1. Novice XR learning practices

The theoretical framework in Table 2 can be used to define novice XR learning practices within the ELEVATE-XR framework (see Table 3).

2.2.1.1. Novice XR learning objective

According to Bloom’s Revised Taxonomy (Anderson et al., 2001) and Component Display Theory, the over-arching novice level learning objective should be to remember facts, concepts, and procedures.

2.2.1.2. Novice XR learning activities

According to Bloom’s Revised Taxonomy (Anderson et al., 2001), novice learning activities should include defining, describing, identifying, arranging, labeling, listing, matching, naming, recalling new knowledge, among other related activities.

2.2.1.3. Novice XR learning form factor

Appropriate XR form factors for novice learners include AR and guided VR. As novices have no discerning schemas and contend with copious extraneous processing, AR is proposed as the most effective XR form factor. According to Behaviorist, Cognitive Load, Conditions of Learning, Embodied Learning, Constructivist, Experiential Learning, and Situated Learning Theories, at the novice level an XR form factor is needed that can support focused, discovery-based, guided observational learning activities, while orienting learners, controlling extraneous processing, and avoiding cognitive overload. Scenario elements should not include stressors, should include minimal complexity and visual fidelity, and should include maximal assistance and feedback. AR overlays can be used to both focus attention on relevant knowledge sources via spatialized cues at the point-of-interest and convey declarative knowledge via instructional overlays (Alzahrani, 2020). The concomitance of the real environment overlaid with augmented virtual objects afforded by AR capabilities also allows for visualization of concepts and communication of spatial relationships, all situated within an authentic context (Tobisková et al., 2022). AR-based training solutions designed for novices should, however, carefully consider potential distractors present in the real-world task environment that may increase extraneous processing. For example, if using AR for a medical task, such as training a surgical procedure, the real-world task environment includes blood and complex anatomical features that may present a challenge when attempting to focus the novice learner on basic concepts and skills, such as suturing (see Figure 1(left)). Similarly, many maintenance task environments and associated equipment are visually complex, and may include distractors when attempting to teach basic skills with AR overlays, such as the order in which to perform steps or identification of relevant equipment parts and features when troubleshooting an engine (see Figure 1(right)). Thus, while AR overlays are effective at focusing attention and conveying declarative knowledge to guide novice learning, the real world context onto which these overlays are projected must be considered and controlled, if necessary, to reduce extraneous processing.

Figure 1. Example real-world robotic surgery task environment (Left) and example real-world maintenance task environment (Right).

VR has also been used extensively in recent years for training at the novice level (c.f., Mendes et al., 2022; Xie et al., 2021; Yu et al., 2022). For example, the da Vinci^® Skills Simulator™ provides immersive VR exercises that are performed on the da Vinci^® surgical robot surgeon’s console, which enables novices to become familiar with the actual console hardware (see Figure 2). In general, VR can be used to present visual imagery and support novice training with guided scenarios that provide assistance and immediate feedback in the form of simulated visual cues and prompts, while also minimizing distractors in order to reduce cognitive processing demands; this is critical for novice learners and must be considered if using VR for novice-level training.

Figure 2. Example guided VR task environment for Novice robotic surgery skills training.

If VR scenarios are free-form and lack appropriate guidance to direct attention, novice learners may experience high levels of extraneous processing and thus target learning gains main not be realized (Fiorella & Mayer, 2021; Parong & Mayer, 2108; 2021). This may be why, at times, VR has not been found to be an effective form factor to convey declarative knowledge (Meyer et al., 2019; Wang et al., 2021). Thus, when using VR at the novice level, to reduce cognitive load it is essential to manage complexity and guide interaction in a manner that is linked to learning goals. Additionally, it may also be best to wait until the advanced beginner proficiency level to introduce embodied MR solutions, because according to Behaviorist, Cognitive Load, Embodied Learning, Constructivist, and Experiential Theories, novice learning should involve primarily passive observation. As novice learners have less spare attentional resources to attend to embodiment cues (e.g., haptics, force feedback), benefits of incorporating such cues into XR training solutions for novices is questionable (Overtoom et al., 2019).

2.2.1.4. Novice XR learning loop

Based on the theoretical framework, the novice learning loop should be structured as: Situate > Orient > Observe > Reflect by Summarizing > Appraise via Remembering > Repeat.

• Situate: According to the alignment of the competency-based framework and Situated Learning Theory (see Table 2), novices should be situated within conceptually focused scenarios that assist in developing a foundation of declarative knowledge. For example, in situ spatialized 3D AR step-by-step instructional overlays or VR scenarios concentrated on conveying a particular concept could be used to focus the attention of a novice at the point-of-interest, while conveying semantic, temporal, and spatial elements that clarify facts, concepts, and procedures (Blattgerste et al., 2018).

• Orient: According to the alignment of the Competency-Based Framework and Cognitive Load and Conditions of Learning Theories (see Table 2), novices are dependent on assistance (Hu & Zhang, 2017) to orient them to relevant cues, which guides them in development of appropriate schemas. Specifically, novice scenarios could use AR overlays to orient novices and direct their attention only to those aspects of a problem or procedure essential to schema acquisition (i.e., problem states, goal states, and their associated solution spaces; Hynes et al., 2022; Persky & Robinson, 2017). VR solutions would similarly need to provide means of orienting novices to relevant aspects of a scenario to gain attention and reduce extraneous processing.

• Observe: According to the alignment of the competency-based framework and Behaviorist, Cognitive Load, Embodied Learning, Constructivist, and Experiential Theories (see Table 2), novice XR scenarios should mostly involve passive “learning on the move” (Zimmerman et al., 2019) activities that foster observation of basic declarative knowledge to support discovery and recognition of relevant information (e.g., understanding facts, identifying concepts, listing steps).

• Reflection via Summarization: According to the alignment of the competency-based framework and Situated Learning Theory (see Table 2), novice XR scenarios should incorporate reflection-in-action via prompts that foster identification and summarization of the delta between desired and actual performance.

• Appraisal via Remembering: According to the alignment of the competency-based framework and Behaviorist and Cognitive Load Theories (see Table 2), novice XR scenarios should involve learners with appraisals that provide immediate outcome-based feedback that reinforces desired outcomes and supports remembering acquired declarative knowledge. This can be achieved via formative assessments (i.e., in situ assessments during learning; Yan et al., 2021). Formative assessments provide learners with the opportunity to engage and construct deeper learning by interactively assessing and working with their newly acquired knowledge in situ. These assessments can take the form of short answer questions, completion questions, multiple choice questions, extended matching items, imitated worked examples, and other such appraisals that test factual recall objectively and efficiently (Wass et al., 2001). Worked examples, which illustrate an expert’s problem solution, have particular value for novices because they reduce extraneous processing (Sweller, 2006). To support such appraisal activities, novice XR scenarios can make use of spatial capabilities within the augmented world to situate and contextualize traditional formative evaluations. For example, AR overlays on a system could present novice learners with a completion question (e.g., where is a system subcomponent located?), which would require the learner to recognize, locate, and point (through hand gestures tracked by the AR headset) to the item of interest. Such interactive appraisal activities are anticipated to increase learning speed (Barana et al., 2021; Kummar, 2020). Similar techniques could be used in VR scenarios for novices.

• Repeat: It is anticipated that 20 hours of training focused on select novice principles may be sufficient to progress to the advanced beginner stage (Ericsson & Harwell, 2019; Kaufman, 2014). This is an estimate based on the alignment of the competency-based framework and Behaviorist Theory, which suggest learning requires repetition (see Table 2), and the need for purposeful/deliberate practice to realize gains in proficiency.

2.2.1.5. Novice XR SCAFFolding scenario elements

According to the alignment of the competency-based framework and Behaviorist, Cognitive Load, and Conditions of Learning Theories (see Table 2), novice XR scenarios should have a low level of stress (e.g., no time constraints, no consequences for errors, etc.), a low level of complexity (e.g., isolated topics with limited steps, involve recognition rather than recall, observation of worked examples, etc.) and stimulus fidelity (e.g., only visual and auditory cues to minimize extraneous processing), a high level of assistance (e.g., positive/negative reinforcement, guided scenarios, avoid open exploration, etc.), and ample feedback that is immediate, error-sensitive, elaborative, outcome-based, and allows the learner to try again without consequence.

2.2.2. Advanced beginner XR learning practices

The theoretical framework in Table 2 can be used to define advanced beginner XR learning practices within the ELEVATE-XR framework (see Table 3).

2.2.2.1. Advanced beginner XR learning objectives

According to Bloom’s Revised Taxonomy (Anderson et al., 2001) and Component Display Theory, the over-arching advanced beginner level learning objective should be to understand newly acquired facts, concepts, and procedures.

2.2.2.2. Advanced beginner XR learning activities

According to Bloom’s Revised Taxonomy (Anderson et al., 2001), advanced beginner learning activities should foster organizing knowledge into schemas and using them to compare, classify, differentiate, imitate, explain, translate, and interpret situations, among other related activities.

2.2.2.3. Advanced beginner XR learning form factor

Appropriate XR form factors for advanced beginner learners include AR, MR, and guided VR with force feedback. As advanced beginners learn best with increasingly complex scenarios that require application of rules and choosing a course of action, connecting new knowledge with existing knowledge, integration of extraneous information, provision of specific and targeted feedback, and techniques to manage stress, MR is proposed as the most effective XR form factor. According to Behaviorist, Cognitive Load, Conditions of Learning, Embodied Learning, Constructivist, Experiential Learning, and Situated Learning Theories, at the advanced beginner level an XR form factor is needed that can support guided, concrete, hands-on learning activities that trigger selective perception, involve learners in imitation of realistically embodied behaviors that are associated with task performance (e.g., procedure following), and foster concept formation in situ. Such embodied activities can be best supported by MR platforms. In particular, MR can be used to promote multisensory processing with tangible objects and manipulatives (Pellas et al., 2020). In addition, AR overlays can be used to direct attention while conveying instructional information within these scenarios. Specifically, by using AR in conjunction with real-world objects, physical interaction can be enabled during learning. However, as AR augmented overlays alone do not engage physical interaction and in some cases the internal psychomotor task models formed during VR training may not readily generalize to real world tasks (Levac et al., 2019), AR overlays alone and VR may not be the most appropriate form factors for advanced beginners to acquire psychomotor skills. The beneficial effect of haptic feedback increases at this stage of learning, thus, the lack of authentic haptic feedback in VR is one of the most important characteristics that limits its efficacy for the advanced beginner. Such limitations have been demonstrated in the literature, as results are variable regarding the efficacy of adding haptic feedback to VR simulators for psychomotor skills training (Overtoom et al., 2019). Further, this type of feedback has been shown to be most important for complex psychomotor skills, which may not even be introduced at the advanced beginner stage. It is important to note, however, that VR has been found to be effective at providing force feedback to guide force application and gross motor behaviors when learning psychomotor skills, resulting in benefits to task performance, force regulation, and task completion time (Weber & Eichberger, 2015). Further, for more complex and delicate tasks, force feedback has been found to be critical in guiding adjustment of input forces and avoidance of exaggerated forces (e.g., damaging tissue, breaking threads). Thus, VR trainers that are supplied with force sensing and force feedback mechanisms may improve psychomotor skills training at the advanced beginner level. Figure 3 provides such an example of a guided, end-to-end VR-based simulated surgical trainer with force feedback. When choosing a form factor for advanced beginners, MR is thus considered most suitable, and while AR and VR may be suitable, careful consideration must be given to haptic fidelity and task complexity to support effective psychomotor skills training.

Figure 3. Example guided VR task environment for Advanced Beginner robotic surgery skills training with force feedback.

Figure 4. Central Control Station (Left), Input/Output Drop Box (Center), and field device (Right).

2.2.2.4. Advanced beginner XR learning loop

Based on the theoretical framework (see Table 2), the advanced beginner learning loop should be structured as: Situate > Guide > Organize > Reflect by Exemplifying > Appraise via Understanding > Repeat.

• Situate: According to the alignment of the competency-based framework and Situated Learning Theory (see Table 2), advanced beginners should be situated within hands-on scenarios that allow for exemplifying and reflecting upon desired performance in a given situation, thereby supporting development of procedural knowledge. Thus, situated XR scenarios can support advanced beginners in proceduralizing their newly acquired declarative knowledge through hands-on guided practice (Saks et al., 2021).

• Guide: According to the alignment of the Competency-Based Framework and Behaviorist, Cognitive Load, and Conditions of Learning Theories (see Table 2), advanced beginners are relatively dependent on assistance (Hu & Zhang, 2017) and should be provided with guidance that models correct behavior and supports imitation. Specifically, as advanced beginners can become easily overwhelmed as they work to translate newly acquired knowledge into maxims, XR scenarios should provide guidance that supports filtering incoming contextual information to differentiate between relevant versus irrelevant cues while following hands-on procedures (Hoffmann et al., 2022).

• Organize: According to the alignment of the Competency-Based Framework and Cognitive Load and Conditions of Learning Theories (see Table 2), as advanced beginners start to understand the context of a situation and make context-aware decisions, they should be guided in organizing knowledge into situationally based maxims (i.e., situational heuristics that guide appropriate application of knowledge; Rousse & Dreyfus, 2021). The latter involves identification of aspects of a situation that are discriminable, such as recurrent phenomena that can be recognized by the learner (e.g., sound of a system fault alert). XR scenarios can gradually increase in complexity and element interactivity, while guiding learners to assess situations (Sankaran et al., 2019) to ensure learners test learned principles and recognize consequences of actions and decisions (Dreyfus, 2004; Rousse & Dreyfus, 2021). Advanced beginner XR scenarios should also guide learners to incorporate both situational elements (e.g., the characteristic “look” of a trouble state within a system), as well as non-situational elements (e.g., purpose of specific components within a system) into their maxims.

• Reflection via Exemplification: According to the alignment of the competency-based framework and Situated Learning Theory (see Table 2), advanced beginner XR scenarios should incorporate reflection-in-action via prompts that foster explaining and exemplifying desired performance in a given situation (e.g., reflecting on active decision-making in choosing a course of action and connecting previously mastered declarative knowledge to newly acquired procedural knowledge).

• Appraisal via Understanding: According to the alignment of the competency-based framework and Behaviorist and Cognitive Load Theories (see Table 2), advanced beginner XR scenarios should involve learners with appraisals that provide immediate, error-sensitive, elaborative, response-contingent feedback that reinforces understanding regarding how to strategically reason about relevance and accurately code perceived information into appropriate problem states and associated schemas (Dalinger et al., 2020; Mayer, 2010). As with novices, this can be achieved via formative assessments.

• Repeat: According to the alignment of the competency-based framework and Behaviorist Theory that suggests learning requires repetition (see Table 2), it is estimated that 40 hours of deliberate practice focused on select advanced beginner principles and skills may be sufficient to progress to the competent stage (Ericsson & Harwell, 2019; Kaufman, 2014).

2.2.2.5. Advanced beginner XR SCAFFolding scenario elements

According to the alignment of the competency-based framework and Behaviorist, Cognitive Load, Component Display, Conditions of Learning, Embodied Learning, Constructivist, and Experiential Learning Theories (see Table 2), advanced beginner XR scenarios should incorporate minimal, contextually relevant stressors only during instruction but not during formative evaluation (e.g., time awareness, no consequences for errors, etc.), an increase in complexity over novice scenarios specifically with regard to difficulty but not with regard to abstractness (e.g., self-directed, hands-on scenarios that require assessing situation, discriminating relevance, recognizing consequences, concept formation, rule application, etc.), remain with a low level of fidelity to minimize extraneous processing but add in high fidelity force feedback for psychomotor skills training, a high level of assistance (e.g., positive/negative reinforcement, guide and imitate modeled correct behavioral responses), and directive feedback that is immediate, error-sensitive, elaborative, response-contingent, allows the learner to try again, includes time to completion, and provides insight on progress.

2.2.3. Competent XR learning practices

The theoretical framework in Table 2 can be used to define competent XR learning practices within the ELEVATE-XR framework (see Table 3).

2.2.3.1. Competent XR learning objectives

According to the Bloom’s Revised Taxonomy (Anderson et al., 2001), the over-arching competent level learning objective should be to apply newly acquired facts, concepts, and procedures.

2.2.3.2. Competent XR learning activities

According to the Bloom’s Revised Taxonomy (Anderson et al., 2001) competent learning activities should require demonstrating, employing, examining, executing, illustrating, implementing, and using acquired knowledge in a new situation, among other related activities.

2.2.3.3. Competent XR learning form factor

Appropriate XR form factors for competent learners include AR, MR, and free-form VR with high fidelity haptics. As competent learners should display a mastery of procedural knowledge while acquiring integrative knowledge (i.e., connects, relates and unifies concepts in various situations; Anderson et al., 2001) via hands-on experiences, MR continues to be an especially fitting form factor. According to Behaviorist, Cognitive Load, Conditions of Learning, Embodied Learning, Constructivist, Experiential Learning, and Situated Learning Theories, at the competent level an XR form factor is needed that can provide embodied learning activities that support connecting, relating, and unifying facts, concepts, and procedures in various situations. Such integrative, embodied activities can be supported by XR platforms that engage learners in “elaboration” activities (e.g., examine, experiment, integrate; Anderson et al., 2001) that foster intuition during self-directed problem solving and decision making within relevant context (Sankaran et al., 2019). For the same reasons as those for advanced beginners (i.e., lack of physical interaction), AR (augmented overlays alone) may not be an appropriate form factor to support psychomotor skill training in competent level learners. However, AR when used in conjunction with physical objects to enable physical interaction may be appropriate for competent level learners to acquire psychomotor skills. The ability to allow for free-form, self-directed exploration in VR (i.e., using scenarios designed without a specified sequence, where learners can experience any kind of interaction that captures their interests) could prove particularly effective at training at the competent level. Such self-guided scenarios can allow the competent learner to apply newly acquired knowledge (e.g., facts, concepts, and procedures) to increasingly more complex scenarios, thereby supporting development of more generalizable and integrated knowledge that can be robustly applied in differing context. Further, VR scenarios that include force and motion parameters (e.g., pathway and position cues) may be particularly valuable in supporting competent learners in better interpreting use of force (e.g., peak force, force volume) and motion (e.g., path length, motion volume) parameters during more dynamic, psychomotor tasks, as they have been found to exceed the maximum force values applied by both novices and experts (Horeman et al., 2014). This is particularly important for tasks, such as complex craniofacial surgical procedures, that rely on high fidelity visual and haptic cues for high accuracy performance (Arikatla et al., 2018). Thus, as previously noted, careful consideration must be given to haptic fidelity when using VR to train psychomotor skills at the competent level (Overtoom et al., 2019).

2.2.3.4. Competent XR learning loop

Based on the theoretical framework (see Table 2), the competent learning loop should be structured as: Situate > Give Autonomy > Embody > Reflect by Relating > Appraise via Applying > Repeat.

• Situate: According to the alignment of the competency-based framework and Situated Learning Theory (see Table 2), competent learners should be situated within embodied learning experiences that allow for relating and unifying concepts in various situations and transferring knowledge to new situations, thereby supporting development of integrative knowledge. Thus, situated XR scenarios can support competent learners in the process of integrating their newly acquired declarative and procedural knowledge via contextually relevant physical interactions that foster an internal “knowing” or intuition (Lindgren & Johnson-Glenberg, 2013).

• Give Autonomy: According to the alignment of the Competency-Based Framework and Behaviorist, Cognitive Load, Conditions of Learning, and Experiential Learning Theories (see Table 2), competent learners should be provided with autonomy via engagement in self-directed, self-guided scenarios. While relative autonomy (Hu & Zhang, 2017) is important at this level of proficiency, XR scenarios should provide subtle coaching when mistakes are made that facilitates elaboration and reflective observation (i.e., intentional analysis of learning experience from multiple perspectives).

• Embody: According to the alignment of the Competency-Based Framework and Behavioristic, Component Display, Embodied Learning and Constructivist Theories (see Table 2), as competent learners must acquire the ability to apply appropriate concepts and procedures given the situation, they should be immersed in authentic learning experiences for which bodily activities are integral and meaningfully related to learning tasks, thereby fostering rich encoding that can lead to an intuitive feel. This can be achieved in XR through scenarios that allow for physically applying concepts and procedures in self-guided scenarios requiring differentiating problem states and selecting appropriate actions (Anderson et al., 2001; Rousse & Dreyfus, 2021). If embodied XR scenarios support perception of hands-on actions in relation to goals, deliberate planning, and formulation of routines in context, integrative knowledge will be gained.

• Reflection via Relation Building: According to the alignment of the competency-based framework and Situated Learning Theory (see Table 2), competent XR scenarios should incorporate reflection-in-action via prompts that foster relating and transferring knowledge to new situations (e.g., reflecting on why decisions were made and what the resulting consequences were in a given situation and how that relates to other situations, as well as awareness of what knowledge has been acquired and what is yet to be mastered). Further, to foster development of tacit knowledge, competent XR scenarios can immerse learners in solving inverse, ill-defined problems (i.e., determining from a set of observations the causal factors that produced them) that cannot be solved following rules, maxims, or guidelines, but rather require synthesis, or regressive reasoning, from conclusions to premises or from effects to causes (Peña, 2010).

• Appraisal via Applying: According to the alignment of the competency-based framework and Behaviorist and Cognitive Load Theories (see Table 2), competent XR scenarios should involve learners with appraisals that provide in situ opportunities for practical skills assessment during application of knowledge to new situations. As with novices and advanced beginners, this can be achieved via formative assessments, such as by asking embodied questions to reveal subtle moments of learning, and asking probing questions that allow hidden knowledge and insights to emerge (Kotchoubey, 2018), such as by requiring learners to recognize key information depicted in an augmented 3D space (e.g., presence or absence of a critical component, a missing step, or an error in a step/procedure).

• Repeat: According to the alignment of the competency-based framework and Behaviorist Theory that suggests learning requires repetition (see Table 2) when focused on select skills, 100 hours of deliberate practice focused on competent principles may be sufficient to progress to the proficient stage (Ericsson & Harwell, 2019; Simmons, 2022).

2.2.3.5. Competent XR SCAFFolding scenario elements

According to the alignment of the competency-based framework and Behaviorist, Cognitive Load, Component Display, Conditions of Learning, Embodied Learning, Constructivist, and Experiential Learning Theories (see Table 2), competent XR scenarios should incorporate contextually relevant stressors (e.g., time pressure, consequences for errors, etc.)—while avoiding use of highly stressful elements (e.g., errors lead to significant consequences, high levels of distraction, etc.), an increase in complexity with regard to both difficulty and abstractness (e.g., provide a complex problem that does not have an immediately obvious solution, etc.), a medium level of visual fidelity with high haptic fidelity, a medium level of assistance (e.g., positive/negative reinforcement during self-directed learning), and facilitative feedback (e.g., advice, suggestions; Black & William, 1998) that is immediate, provides opportunities for self-reflection and self-explanation, allows the learner to try again, includes time pressure, and provides insight on progress.

2.2.4. Proficient XR learning practices

The theoretical framework in Table 2 can be used to define proficient XR learning practices within the ELEVATE-XR framework (see Table 3).

2.2.4.1. Proficient XR learning objectives

According to the Bloom’s Revised Taxonomy (Anderson et al., 2001), the over-arching proficient level learning objective should be to analyze acquired knowledge to break it up into constituent parts, determine how the parts are interrelated, and identify generalities.

2.2.4.2. Proficient XR learning activities

According to the Bloom’s Revised Taxonomy (Anderson et al., 2001) proficient learning activities should require deconstructing, differentiating, integrating, organizing, relating, and structuring acquired knowledge, among other related activities.

2.2.4.3. Proficient XR learning form factor

Appropriate XR form factors for proficient learners include free-form, multimodal VR, and potentially AR and MR. As proficiency advances, and learners can adeptly complete scenarios with increasing levels of abstractness and active decision-making, VR becomes potentially the most appropriate form factor for learners at the proficient level. According to Behaviorist, Cognitive Load, Conditions of Learning, Constructivist, Experiential Learning, and Situated Learning Theories, at the proficient level an XR form factor is needed that can provide a diverse set of scenarios that support intuitively discovering, prioritizing, relating, and generalizing relevant aspects of a problem space without the need for overt physically interactive assessment. Such discovery-based activities can be supported by free-form VR platforms that engage multiple learner modalities (visual, auditory, haptic) in “connect” activities (e.g., deconstruct, link, deduce, explain, predict; Anderson et al., 2001) that foster intuitive situational responses in a wide variety of contextually rich, high tempo, and operationally relevant simulated scenarios that require adapting to the situation at hand. As AR/MR may be limited in the conditions they can simulate in the real world to foster robust generalization, and there is not a need for overt embodiment, these form factors may not be appropriate for proficient level learners. However, AR/MR may prove valuable at the proficient level for on-the-job upskilling (see Figure 4), as these form factors can provide point-of-need support to experienced workers to amplify their skills (Steel, 2019).

2.2.4.4. Proficient XR learning loop

Based on the theoretical framework (see Table 2), the proficient learning loop should be structured as: Situate > Goal-Orient > Generalize > Reflect by Contradicting > Appraise via Analyzing > Repeat.

• Situate: According to the alignment of the competency-based framework and Situated Learning Theory (see Table 2), proficient learners should be situated within contextually relevant discovery-based experiences that allow for application of integrative knowledge and skills to novel and varied scenarios, thereby supporting generalization of knowledge. Thus, situated XR scenarios should support proficient learners in the process of generalizing their integrative knowledge in diverse, contextually realistic situations (Rousse & Dreyfus, 2021). To build such flexible, generalizable skills, proficient XR scenarios can foster reliance on intuition, unconscious pattern recognition that quickly filters information as pertinent versus non-pertinent, and context linking, while displaying more confidence and accountability (Persky & Robinson, 2017).

• Goal-Orient: According to the alignment of the Competency-Based Framework and Behaviorist, Constructivist Learning, and Experiential Learning Theories (see Table 2), proficient learners should be engaged in scenarios focused on seeing the big picture, connecting existing knowledge, and discovering new meaningful behaviors to intuitively sense the goal given the situation at hand (Schunk, 1991). XR scenarios can facilitate such purpose-driven learning via provision of complex, unique scenarios that foster goal-setting, solving problems in novel and imaginative ways, managing multiple distractions and emotional stimuli, and self-reflection.

• Generalize: According to the alignment of the Competency-Based Framework and Behavioristic, Component Display, Embodied Learning and Constructivist Theories (see Table 2), as proficient learners must acquire the ability to generalize previously developed schemas and maxims and adapt them to the situation at hand, they should be immersed in complex, highly varied scenarios that require situational discrimination to foster discovery and selection of intuitive situational responses. This can be achieved in XR through scenarios that foster analyzing and generalizing facts, concepts, procedures, and principles (Anderson et al., 2001; Rousse & Dreyfus, 2021). If embodied XR scenarios support the learner in perceiving deviations from the normal pattern and determining what to do without assistance, the ability to generalize knowledge should be achieved.

• Reflection via Contradiction: According to the alignment of the competency-based framework and Situated Learning Theory (see Table 2), proficient XR scenarios should incorporate reflection-in-action via contradictory prompts that foster identifying consistencies and contradictions to support generalizing knowledge across diverse and varied situations (i.e., focusing on situational dissonance, such as differences, discrepancies) via self-reflection.

• Appraisal via Analyzing: According to the alignment of the competency-based framework and Behaviorist and Cognitive Load Theories (see Table 2), proficient XR scenarios should involve learners with appraisals that require analysis and in situ self-explanation, as well as peer assessments. Self-reflection can be achieved via formative assessments, while peer assessment can be derived from existing assessment instruments, such as the Multisource Assessment (360°) Methodology (Williams et al., 2017).

• Repeat: According to the alignment of the competency-based framework and Behaviorist Theory that suggests learning requires repetition (see Table 2) and expertise requires 10,000 hours of deliberate practice (Ericsson & Harwell, 2019), using the 80/20 rule (i.e., 80% of the results are usually achieved by 20% of the effort) 2000 hours of deliberate practice focused on proficient principles may be sufficient to progress to the expert stage (Medium, 2018).

2.2.4.5. Proficient XR SCAFFolding scenario elements

According to the alignment of the competency-based framework and Behaviorist, Cognitive Load, Component Display, Conditions of Learning, Constructivist, and Experiential Learning Theories (see Table 2), proficient XR scenarios should incorporate high frequency/high fidelity multimodal cues and stressors (e.g., significant time pressure, significant consequences for errors and longer performance times, high levels of distraction, etc.), high complexity with regard to both difficulty and abstractness (e.g., differentiate among conflicting critical cues, differentiate relative efficacy between alternative approaches, deconstruct approach taken to generalize concepts, apply complex skills in challenging real-world contexts to overcome plateaus and limits on performance improvement and promote transfer [Ericsson, 2009], etc.), provide limited assistance (e.g., positive/negative reinforcement during self-reflection; in situ support only as needed—allow multiple errors to occur prior to providing a prompt; autonomy is important at this stage [Hu & Zhang, 2017] and too much assistance can induce the expertise reversal effect; Kalyuga & Renkl, 2010), and strategy-based (e.g., in situ hints that guide learner without explicitly presenting correct answer) feedback that is immediate, provides opportunities for self-reflection and self-explanation, includes time pressure, and provides insight on progress.

2.2.5. Expert XR learning practices

The theoretical framework in Table 2 can be used to define expert XR learning practices within the ELEVATE-XR framework (see Table 3).

2.2.5.1. Expert XR learning objectives

According to the Bloom’s Revised Taxonomy (Anderson et al., 2001), the over-arching expert level learning objective should be to evaluate situations to create new patterns or propose new, alternative solutions.

2.2.5.2. Expert XR learning activities

According to the Bloom’s Revised Taxonomy (Anderson et al., 2001) expert learning activities should require assessing, contrasting, critiquing, inventing, hypothesizing, and validating new knowledge structures, among other related activities.

2.2.5.3. Expert XR learning form factor

Appropriate XR form factors for expert learners include free-form, multimodal VR, and potentially AR/MR. As the focus at the expert level is on developing robust skills that close the gap between intuition and generation of new perspectives even under stressful and uncertain conditions, VR continues to be perhaps the most appropriate form factor. According to Behaviorist, Cognitive Load, Conditions of Learning, Constructivist, Experiential Learning, and Situated Learning Theories, at the expert level an XR form factor is needed that can foster development of higher-order skills, analytic abilities, self-regulation strategies, extensive domain knowledge, and the ability to transfer tacit and declarative knowledge to new scenarios and generate the ideal course of action within a novel context. Such evaluative, creation-based activities can be supported by VR platforms that engage learners in “transfer” activities (e.g., synthesize, analyze, estimate, invent; Anderson et al., 2001) that foster creation and generation of automaticity in situational responses under novel, uncertain, high tempo context. For the same reason as those at the proficient level (i.e., limited capacity to simulate multiple real world scenarios to foster robust generalization and transfer), AR and MR may not be the most appropriate form factors for expert level learners; however, these form factors may prove effective for upskilling experts on-the-job.

2.2.5.4. Expert XR learning loop

Based on the theoretical framework (see Table 2), the expert learning loop should be structured as: Situate > Challenge > Create > Reflect by Self Critiquing > Appraise via Evaluating > Repeat:

• Situate: According to the alignment of the competency-based framework and Situated Learning Theory (see Table 2), expert learners should be situated within contextually relevant evaluative, creation-based experiences that allow for generation of new meaningful facts, concepts, procedures, and principles, thereby supporting transfer and extension of knowledge. Thus, situated XR scenarios should support expert learners in the process of generating new knowledge in high tempo, novel situations (Rousse & Dreyfus, 2021). To build such resilient, generalizable skills, expert XR scenarios can provide high situational fidelity, such as through dependence on deep understanding, incorporation of complex decision-making, involvement in information sharing, and being challenged by others (Kalyuga et al., 2003; Persky & Robinson, 2017).

• Challenge: According to the alignment of the Competency-Based Framework and Behaviorist, Constructivist Learning, and Experiential Learning Theories (see Table 2), expert learners should be engaged in scenarios focused on challenging the accuracy of hypotheses, the relevance of critical cues, the criticality and root cause of errors, and the relative efficacy of alternative approaches. XR scenarios can facilitate such generative learning via provision of novel, uncertain, high tempo scenarios that foster synthesis, evaluation, and recognition of complex patterns, are time sensitive and highly dependent on conditional knowledge, increase environmental stressors, and introduce significant consequences if actions are not taken quickly or accurately enough.

• Create: According to the alignment of the Competency-Based Framework and Behavioristic, Component Display, Embodied Learning and Constructivist Theories (see Table 2), as expert learners are focused on evaluating and creating new meaningful behaviors and knowledge, they should be immersed in novel, uncertain, high tempo scenarios that remove all scaffolds and use complex, ambiguous scenarios that require metacognition, such as monitoring their own thinking and problem solving or predicting outcomes of their performance, to foster reflective thinking (Ertmer & Newby, 1996). This can be achieved in XR through scenarios that foster evaluating and extending facts, concepts, procedures, and principles (Anderson et al., 2001; Rousse & Dreyfus, 2021). If XR scenarios support the learner in monitoring and self-regulatory skills that enable experts to know not only what (declarative knowledge) but also how (procedural knowledge), where/why/when (conditional knowledge) to apply specific knowledge and actions, the ability to create new knowledge should be achieved (Ertmer & Newby, 1996).

• Reflection via Self Critique: According to the alignment of the competency-based framework and Situated Learning Theory (see Table 2), expert XR scenarios should incorporate reflection-in-action via self-critique prompts that foster consideration of alternative hypotheses by questioning, for example, the accuracy of hypotheses, the relevance of critical cues, the criticality and root cause of errors, and the relative efficacy of alternative approaches.

• Appraisal via Evaluating: According to the alignment of the competency-based framework and Behaviorist and Cognitive Load Theories (see Table 2), expert XR scenarios should involve learners with appraisals that require in situ self-assessments that are high in complexity and abstractness. Such evaluations should allow the learner to practice conditional knowledge, pattern-recognition, and dynamic decision-making (e.g., dual task performance with non-technical skills such as intraoperative communication; White, McMahon, Walsh, Coffey, & Leonard, 2018).

• Repeat: According to the alignment of the competency-based framework and Behaviorist Theory that suggests learning requires repetition (see Table 2), 10,000 hours of deliberate practice focused on expert principles may be sufficient to achieve mastery (Ericsson & Harwell, 2019).

2.2.5.5. Expert XR SCAFFolding scenario elements

According to the alignment of the competency-based framework and Behaviorist, Cognitive Load, Component Display, Conditions of Learning, Constructivist, and Experiential Learning Theories (see Table 2), expert XR scenarios should incorporate novel, uncertain, high tempo, high fidelity multimodal cues and stressors (e.g., extreme time pressure, extreme consequences for errors and longer performance times, high levels of multimodal distractions, etc.), high complexity with regard to both difficulty and abstractness (e.g., require synthesis, evaluation, and recognition of complex patterns, invention of new unique solution to complex, novel, ambiguous problem), provide no assistance (e.g., post hoc performance feedback, allow learner to proceed despite unrecoverable errors; autonomy is important at this stage [Hu & Zhang, 2017] and expertise reversal must be monitored; Kalyuga & Renkl, 2010), and strategy-based (e.g., point out inconsistencies) feedback that is delayed, provides opportunities for self-reflection and self-explanation, and provides insight on progress.

Based on this ELEVATE-XR framework, better informed decisions regarding the learning practices to incorporate into an XR solution given an individual’s level of proficiency can be made. Such personalized solutions are anticipated to lead to higher learning gains, increased transfer, and higher engagement than non-personalized training solutions (Claypoole et al., 2020; Parong & Mayer, 2018, 2021).

3. Application of ELEVATE-XR: a naval maintenance use case

In order to provide an illustrative example of how ELEVATE-XR can be applied to a specific training domain, a naval maintenance use case is provided here. The example maps the andragogical guidelines summarized in Table 3 to associated XR-based training design guidelines, which are summarized in Table 4 for each proficiency level. The selected use case is focused on Network Casualty Response Team (NCRT) Information Technology (IT) Technician corrective maintenance training for the Machinery Control Monitoring System (MCMS) on board the USS Gerald R. Ford (CVN 78) aircraft carrier. The CVN 78 is designed to operate effectively with almost 700 fewer crew members than a CVN 68-class ship. Reduced crew ships such as the CVN 78 provide inherently fewer opportunities for On-the-Job-Training (OJT) due to the limited number of maintainers aboard each ship and the limited time available for existing maintainers to dedicate to training on ship. Furthermore, due to frequent turn-over of senior maintainers to new assignments and deployments, training solutions are needed to assist maintainers at all proficiency levels. As such, this section presents a use case that can greatly benefit from XR-based training across the continuum of novice to expert learners. Leveraging the ELEVATE-XR framework and XR technologies for this use case on board the CVN 78 has led to significant increases (∼30%) in learning gains over the course of a single training scenario (Claypoole et al., 2020). This indicates that XR technologies can be an effective medium for training and operational support when coupled with andragogical approaches.

Table 4. ELEVATE-XR machinery control monitoring system use case.

	Proficiency Level
	Novice	Advanced Beginner	Competent	Proficient	Expert
Over-Arching Learning Objective	Remember System/ subsystem component names and locations, and how subsystems are connected. Order of steps in procedure.	Understand What each system/ subsystem does and how they work together. Why procedural steps are performed in the specified order.	ApplyTroubleshoot simple/ common system faults involving individual subsystems.	Analyze Troubleshoot complex/ uncommon system faults involving multiple subsystems.	Evaluate/CreateTroubleshoot rarely encountered and novel system faults. Develop novel problem-solving solutions.
Learning Activities	Name and locate system/ subsystem components. Correctly list order of procedural steps.	Explain the functions of system/ subsystem components and how they work together. Explain why procedural steps are performed in specified order.	Demonstrate ability to identify simple/ common system faults involving individual subsystems, and determine causes of faults.	Demonstrate ability to diagnose complex/ uncommon system faults involving multiple subsystems, and determine causes of faults.	Demonstrate ability to diagnose rarely encountered and novel system faults, and determine causes of faults.
Example XR-Based Training Scenario	Use AR overlays to isolate/ identify individual IODB components and walk through procedural steps.	Use AR overlays to demonstrate relationships between IODB components and provide explanations for the ordering of procedures.	Use MR overlays to simulate IODB faults and support both visual and physical inspection of subsystem components and faults.	Use VR to simulate a wide range of complex and uncommon IODB system-level faults, that may require navigation to various parts of the ship and support realistic training and testing scenarios.	Use VR to simulate highly realistic, novel or rarely-encountered faults that may require additional information or navigation to various parts of the ship in “serious games”-like scenarios with exceptions to rules.
Example XR Scenario Elements	AR overlays focus user attention and provide assistance/ feedback in gaining declarative knowledge (e.g., multiple choice questions and answers) with no added stressors.	AR overlays to simulate network and power flow through components and provide assistance/ feedback in answering “why” questions regarding procedures with modest time pressure.	MR overlays simulate simple, common IODB faults and provide on-demand assistance and feedback in identifying and determining causes of faults with minimal consequences for errors.	End-to-end VR scenarios provide variations on faults involving multiple subsystems and failure types with minimal assistance and feedback, and significant consequences for errors.	Highly realistic simulations provide exposure to rarely encountered, complex scenarios requiring simulated communication and navigation features, and potentially catastrophic consequences for errors.

The MCMS spans equipment throughout the entirety of the ship. It includes a Central Control Station (CCS), a computer-based system that is used to track all system faults and failures, Input/Output Drop Boxes (IODBs) that provide power and networking capabilities to specified areas of the ship, and a wide variety of field devices, such as valves and pumps, that are each connected to an IODB. These subsystems are shown in Figure 4.

This use case will focus specifically on troubleshooting faults from an IODB, as this requires an understanding of the entire MCMS and can require all levels of expertise, depending on the fault. The primary tasks involved in troubleshooting faults from an IODB include visual inspection of IODB components such as wires, connections, and indicator lights, physically checking network and power connections, and testing network and power connectivity by cycling system components on and off. IODB repairs may include replacing individual components (e.g., switches, power sources, terminal blocks, circuit breakers, and fans) or replacing entire modules (e.g., logic controllers and communication modules). Additionally, IODB troubleshooting may identify the need for repairs to connected field devices.

3.1. Use case training guidelines for novice maintainers

Based on ELEVATE-XR, overarching learning objectives for novice IODB maintainers should focus on declarative knowledge such as the names and basic functions of MCMS and IODB subsystems and components, how the subsystems and components are connected to one another, and their physical locations on the ship, as well as the location of subsystem components within the IODB and their defining features to enable visual recognition. Additionally, novice maintainer learning goals include remembering the order of steps in completing basic IODB troubleshooting procedures. As indicated in Table 4, learning activities at this level of proficiency thus include naming and locating system/subsystem components and correctly listing the order of IODB maintenance procedural steps. An example of an appropriate XR-based training scenario to support this level of learning is to use AR overlays to isolate/highlight and identify individual IODB components, and to walk through procedural steps. Scenario elements could include AR overlays to focus user attention and provide assistance/feedback in gaining declarative knowledge (e.g., multiple choice questions and answers) with no added stressors.

3.2. Use case training guidelines for advanced beginner maintainers

Overarching learning objectives for advanced beginner IODB maintainers include understanding what each subsystem and component does, and how the various subsystems and components work together. Additionally, advanced beginner maintainer learning goals include understanding why steps are performed in the specified order when completing basic IODB troubleshooting procedures. As indicated in Table 4, learning activities at this level of proficiency thus include explaining the functions of subsystems and their components, how they work together, and why procedural steps are performed in specified orders. An example of an appropriate XR-based training scenario to support this level of learning is to use AR overlays to demonstrate relationships between IODB components and provide explanations for the ordering of procedures. For example, AR overlays could be used to simulate network and power flow through components and provide assistance/feedback in answering “why” questions regarding procedures. Minimal stressors such as modest time pressure could also be introduced at this stage of training.

3.3. Use case training guidelines for competent maintainers

Overarching learning objectives for competent IODB maintainers should focus on applying knowledge gained, such as troubleshooting system faults involving individual subsystems within the IODB. As indicated in Table 4, learning activities at this level of proficiency thus include demonstrating the ability to identify simple and/or common system faults involving individual subsystems, and determine causes of those faults. An example of an appropriate XR-based training scenario to support this level of learning is to use MR overlays to simulate IODB faults and support both visual and physical inspection of subsystem components and faults. For example, MR overlays could be used to simulate simple, common IODB faults such as an intermittent network signal within the IODB. Competent level scenarios could use MR to provide on-demand assistance and feedback in identifying and determining the cause of the fault such as a loose network cable connection within the IODB or a faulty module. Additional stressors can be included at this stage such as minimal consequences for errors.

3.4. Use case training guidelines for proficient maintainers

Overarching learning objectives for proficient IODB maintainers include analyzing information provided in a training scenario such as troubleshooting more complex faults involving multiple subsystems and/or less common system faults. As indicated in Table 4, learning activities at this level of proficiency thus include demonstrating the ability to diagnose complex system faults, and determine causes of those faults. An example of an appropriate XR-based training scenario to support this level of learning is to use free-form VR to simulate a wide range of complex and uncommon IODB faults involving multiple subsystems that may require navigation to various parts of the ship (e.g., other IODBs within the network), as well as additional features that support more realistic training and testing scenarios such as simulated communication with other maintainers to obtain required information. For example, end-to-end VR scenarios could be used to provide variations on faults involving multiple subsystems and failure types with minimal assistance and feedback. Stressors can be increased at this stage, such as the introduction of significant consequences for errors.

3.5. Use case training guidelines for expert maintainers

Overarching learning objectives for expert IODB maintainers should focus on evaluating complex scenarios and creating solutions, such as troubleshooting rarely encountered and novel system faults, and developing novel problem-solving approaches. As indicated in Table 4, learning activities at this level of proficiency thus include demonstrating the ability to diagnose rarely encountered and novel system faults, and determining causes of those faults using innovative diagnostic techniques. An example of an appropriate XR-based training scenario to support this level of learning is to use free-form, high-fidelity VR to simulate highly realistic, novel, or rarely-encountered faults in “serious games”-like scenarios with exceptions to rules and no assistance. For example, highly realistic simulations could provide exposure to rarely encountered, complex scenarios requiring simulated communication and navigation throughout the ship to assess various subsystems and/or multiple system faults. Stressors can be maximized at this stage, including potentially catastrophic consequences for errors.

4. Conclusion and future directions

As XR technologies become used more frequently in education and training domains, it is imperative that researchers and practitioners identify best practices for the development and presentation of XR instructional content. This article presents the ELEVATE-XR framework to guide design of XR content for educational and training purposes by considering proficiency level, learning objectives and activities, XR form factors, learning loops, and scenario elements. Though the focus within the ELEVATE-XR framework has been placed on the XR medium, importantly, this andragogy can also be instantiated in other personalized learning environments, such as computer-based training, where appropriate. As the guidelines within the ELEVATE-XR framework were developed from previous research in related areas (Claypoole et al., 2020), evaluations are needed to explore these recommendations in varied empirical contexts to ensure they consistently lead to intended learning gains. Ultimately, researchers, practitioners, and designers of XR learning environments have a responsibility to ensure this emerging technology is being used properly with respect to the content presented, educational area for which it is developed, environments in which it is being used, and individuals for which it is being designed. Such an approach has the potential to elevate the ROI of XR technology in the context of education and training.

Acknowledgements

The authors would like to dedicate this work to the late Dr. Ray S. Perez, who inspired this research with his benevolent leadership and strategic vision; his commitment to the advancement of mission readiness through provision of technology-based education and training solutions has led to profound advancements for our servicemembers. The authors would also like to thank the Sailors who participated in the aforementioned evaluations; their support and availability are crucial for developing next-generation training and operational support using emerging technology that is fleet-centric and optimized for the Sailor. Additionally, the authors would like to thank former MP&T Lead for PMS 378L (PEO Carriers) for support in the execution of this effort, Cape Henry Associates and CACI for their logistical support in the execution of this evaluation, as well as the Naval Education and Training Command, who partnered with Center for Naval Analysis to provide an independent review of the training concepts presented in this article. The authors also appreciate the insights of Cali M. Fidopiastis during early conceptualization of the XR andragogy, as well as Catherine Hodges’ support with Sailor engagement.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

The work described in this article was funded by the Office of Naval Research [Contract #: N68335-18-C-0535] and PEO Carriers [Contract #: W911NF-17-D-0021].

Notes on contributors

Kay M. Stanney

Kay M. Stanney is CEO/Founder of Design Interactive, Inc. She was inducted into the National Academy of Engineering for contributions to human factors engineering through XR technology and strategic leadership. She received her Masters and Ph.D. in Industrial Engineering, with a focus on Human Factors Engineering, from Purdue University.

Anna Skinner

Anna Skinner is the President of Black Moon, LLC. Dr. Skinner's technical expertise blends her educational background in biomedical engineering and human factors psychology. She has over 20 years of professional experience spanning the domains of education and training, human-machine interaction, and operational neuroscience, with a focus on military application.

Claire Hughes

Claire Hughes is a Senior Research Associate and Project Lead at Design Interactive, Inc. with ten years of Human Factors Engineering experience focusing on project management, end-user evaluations, and research analysis. Her knowledge and expertise support human factors end-user evaluations, literature reviews, needs analysis, task analysis, and leading projects.