In-Person or Virtual Training?: Comparing the Effectiveness of Community-Based Training

ABSTRACT

The purpose of this study was to examine differences in knowledge gain by curriculum delivery platform for participants receiving a community-based healthy relationships curriculum for a vulnerable population with limited income and education. Using data gathered from 613 participants gathered in 2019–2020, those who received in-person training (n = 440) were compared to those who received synchronous virtual training (n = 173) on curriculum knowledge. Results indicate that in-person participants had statistically significantly higher gains in knowledge at posttest compared to the participants who received synchronous virtual training. The differences in knowledge gain were not accounted for by the demographics of the two groups. Implications include consideration of adaptations in virtual delivery that may close the gap between in-person and synchronous virtual training such as instruction techniques, presentation style, content and materials, and participant preparation.

Introduction

Background

During the global COVID-19 pandemic, virtual meetings became a necessity. Businesses and educational institutions adopted virtual communication platforms to continue daily classes, meetings, health appointments, and community events (AlAteeq, Aljhani, & AlEesa, 2020; Andrews, Berghofer, Long, Prescott, & Caboral-Stevens, 2020; Chatziralli et al., 2020; Rajab, Gazal, & Alkattan, 2020). Community-based in-person trainings were also challenged to transfer the on-site training into a web-based delivery.

Online education – asynchronous and synchronous – has been offered in two formats as an alternative to in-person instruction for decades. Asynchronous distance learning was the most used and studied training delivery before the pandemic, where participants are often offered recorded videos for use anytime (Stöhr, Demazière, & Adawi, 2020). Studies comparing in-person and asynchronous distance community-based trainings have been reported to have no significant difference in outcomes (Benjamin et al., 2008; Neuenschwander, Abbott, & Mobley, 2013).

Synchronous virtual distance learning, on the other hand, provides participants with real-time interactive communication. The pandemic has drawn more attention to synchronous virtual online training (AlAteeq et al., 2020; Chatziralli et al., 2020; Porpiglia et al., 2020). Chatziralli et al. (2020) reported a significant increase in the use of a range of alternatives to in-person communication during the pandemic for ophthalmic education, and Zoom was the most frequently used platform for instruction (Chatziralli et al., 2020). Some studies have suggested real-time virtual meetings may have advantages such as overcoming the obstacles of the long-established format of face-to-face meetings: the opportunity to exchange or disseminate information without the restriction of physical location, cost-effectiveness, and being environmentally friendly (Porpiglia et al., 2020; Rubinger et al., 2020). Challenges include technical difficulties and pandemic-related anxiety or stress induced by the reliance on virtual meetings themselves (Rajab et al., 2020). Studies have shown no difference between synchronous distance and in-person training for school (AlAteeq et al., 2020; He et al., 2021). A meta-analysis shows that there was no significant difference between synchronous distance education and traditional education in effectiveness for health science students (He et al., 2021). The effectiveness of synchronous virtual training and instruction in the community compared to in-person, particularly during the COVID19 era, has not received scientific attention.

Natural experiment

Disaster and hazards research has provided considerable scientific knowledge on the impact of rare events. Appraisal-focused, problem-focused, and emotion-focused coping under conditions of susceptibility to an event have been linked to increased stress, and even more so for those who experienced psychological distress from factors such as unemployment, social isolation, and fear due to uncertainty (Chatziralli et al., Mohler et al., 2020; Richardson, Sorensen, & Soderstrom, 2006). However, unlike disasters and hazards, the pandemic also limited social support due to various requirements and restrictions (e.g., social distancing). Some research suggests that domestic violence increased during 2020 due to increased stress and conflict from the extended time family members were required to share space during stay-at-home orders and social distancing (Mohler et al., 2020). To ease the tension happening during the pandemic, the Healthy Relationships Iowa (HRI) program, an intervention designed to help build a healthier relationship both at home and at work continued to serve the community virtually. The program had been operating since 2015, but beginning April 1, 2020, it was converted to virtual delivery to serve individuals in the community. An unintended result was that the circumstances also provided an opportunity for a natural experiment to be conducted comparing the effectiveness of virtual delivery of the training curriculum with in-person delivery.

Limited research is available on the effectiveness of the synchronous community-based program training compared to in-person. To fill this gap, we examined differences in participants’ knowledge gains for real-time virtual training instruction compared to in-person instruction to deliver the HRI curriculum. Our specific aim is to understand what the difference in the effectiveness is between the synchronous delivery of Healthy Relationships Education and the in-person delivery of the same curriculum for this study.

Curriculum

The Healthy Relationships curriculum is based on much research indicating that a healthy, committed relationship can positively influence mental and physical health for both parents and children (Butler, 1999; Harold, Aitken, & Shelton, 2007; Whisman, Uebelacker, & Settles, 2010). Conflicts and violence are prevalent among families in the United States, especially those with children or limited resources (Cox, Kotch, & Everson, 2003; Niolon et al., 2009). Interventions to support healthy relationships have been identified as necessary, especially for families experiencing interpersonal conflicts, difficulties with employment and access to adequate resources, and the stress of parenting (Bradley, Friend, & Gottman, 2011; Niolon et al., 2009).

Healthy Marriage and Relationship Education (HMRE) program sponsored nearly 100 community-based programs throughout the U.S. such as the Healthy Relationships Iowa (HRI) program. These programs were funded by the U.S. Department of Health and Human Services (DHHS), Administration for Children and Families (ACF), Office of Family Assistance (OFA). HRI received an award through the HMRE program to implement the delivery of a curriculum for relationship education and communication skills through instruction based on the evidence-based World Class Relationships for Work and Home (Howell & Jones, 2013). The curriculum was designed to help improve relationships at home for lower-income level participants by instructing “relational fitness skills for life,” including communication and conflict management skills. The curriculum is described as six pillars, or critical content areas, including setting specific goals, avoiding blame, using power listening, changing behaviors instead of trying to change your partner, resolving conflicts and disagreements, and giving apologies and forgiveness.

Training of instructors began in 2016 with in-person delivery of the curriculum starting in the summer of 2016. With widespread shutdowns in March 2020, the program converted from in-person to synchronous courses and began delivering classes using the Zoom platform in April 2020. To prepare for transitioning to the virtual format, instructors completed a 90-min training of Zoom features such as scheduling, use of breakout rooms, polling, and chat features before beginning their first online curriculum delivery.

The content for the in-person and synchronous classes was the same, and both were delivered in a one-day, 8-h format with a one-hour break for lunch and a short morning and afternoon break. The instructor would remind the participants of the location and time within 24 hours of the registration. Activities and roleplays that are part of the curriculum were conducted in breakout rooms in Zoom to simulate the in-person delivery format in which participants physically moved to create separation for breakouts in a training facility.

Method

Participants and recruitment

Participants were recruited through various outreach approaches, including social media, radio advertising, community agencies, word of mouth from those who already participated in a workshop or instructors, and the program website. The participants completed the initial registration for each workshop through the program website. To be eligible, participants had to be at least 18 years of age.

Data

Data used were downloaded from the project database and did not include any identifiable information. To compare equivalent and recent periods, we used data gathered from October 2019 through March 2020 for the in-person condition and from April 2020 through September 2020 for the synchronous virtual training condition. Data were collected from participants using electronic tablets (iPads) with secure links to electronic instruments in the federally sponsored data system (nFORM) with knowledge tests located in a secure local electronic system. Informed consent was completed as part of the federal enrollment process. Instructors informed participants about information protection and the voluntary nature of the program and answering questions in the electronic systems. The instruments were administered before the class began (pretests) and after (posttest) the class ended. The pretest surveys gathered demographics, pretest knowledge of the content of the curriculum, and measurements on attitudes, values, and beliefs. The posttest included key indicators of knowledge of the curriculum along with measures of attitudes, values, and beliefs gathered in nFORM (“exit survey”) one month after class completion. For this study, the focus is on the key measures of knowledge of the curriculum content though demographics were also used in the analysis to determine if there were any potential exogenous factors that could account for differences. None were found.

Measures

Demographics

Demographic information included sex, age, race, ethnicity, primary language, income, and education levels. These data were gathered as part of the nFORM client characteristics survey. Given the increased psychological distress reported following other hazards and disasters (Chatziralli et al., 2020; Mohler et al., 2020; Richardson et al., 2006), we also analyzed the 6-item Kessler Psychological Distress Scale gathered in the nFORM data system collected at pretest and posttest (K6; https://bit.ly/3nHgShb) (Kessler et al., 2003) as a clinical characteristic. The K6 has been widely used as a screener for measuring the level of psychological distress in the general population (Kessler et al., 2003; Mitchell & Beals, 2011).

Curriculum knowledge

An 8-item knowledge test was designed with the curriculum developer who contributed to content identification and item construction including foils. The instrument was tested for content validity and reliability prior to use, revised based on the results obtained with the curriculum developer, and re-tested. The process included testing in the training of trainers’ instruction followed by testing in pilots of instruction with participants in the training. Knowledge test results were gathered from more than 4,000 participants who enrolled in the program. The items assessed participant learning and understanding of the curriculum (view the knowledge test items at https://bit.ly/3nHgShb). Participants answered the questions developed as indicators of critical content knowledge by selecting one of four response options for each question. The questions covered the core elements, including steps to solving conflicts and disagreements, the XYZ formula, formula for Power Listening Lite, types of relationship goals, ways to deal with resistance, and ways to get rid of blame. For example, one item asked, “what is the first step in solving conflicts or disagreements?.” The response options included: a.) Selecting a solution or solutions, b.) Defining the problem in terms of needs, c.) Brainstorming possible solutions, and d.) Figuring out how to start using the solution. The knowledge test at posttest showed good internal consistency (α = .82). In contrast, little association among test item scores was found prior to training with pretest internal consistency of α = .39, which supports the intended design of the instrument.

Analysis

Data were analyzed using SPSS 27.0 to test the hypothesis that no differences exist between in-person and virtual curriculum delivery platforms in knowledge gains. Knowledge total scores were derived by summing the number of correct on the eight key knowledge items at pre and posttest. Difference scores were calculated by subtracting the number of pretest items correct from the posttest items correct. Paired samples t-tests were used to compare difference scores between pretest and posttest for in-person and real-time virtual platforms. Spearman’s rho correlation and chi-square analyses were used to examine associations between participant demographic variables, instructional platform, and knowledge percent difference scores. For psychological distress, an independent samples t-test was used to compare in-person to real-time virtual groups on mean K6 score differences between pretest and posttest.

Results

A total of 613 participated in the program between October 2019 and September 2020; 440 participated in the in-person curriculum delivery from October 2019 through March 2020 while 173 completed the curriculum remotely from April 2020 through September 2020. The average attendance per workshop was 7–8 participants for in-person and 4–5 for synchronous workshops.

Participant characteristics

Table 1 provides the demographic and clinical characteristics of the participants. Of the 613 participants, 444 (72.4%) were male. Most participants were between the age of 25 and 44 (67.3%). Most (79.4%) participants identified themselves as white and 41 (6.7%) identified as Hispanic or Latino. Most participants were low-income with 452 (81.1%) earning less than $1,000 a month. About 50% of participants reported earning a high-school diploma (n = 277); 73 (11.9%) of the participants did not have a degree or diploma. Sex, age, race, ethnicity, and education were not significantly different between groups. The in-person group had slightly lower income (χ² (2, N = 557) = 8.415, p = .015). Most participants reported low or moderate levels of psychological distress and psychological distress scores were not statistically significantly different for virtual and in-person groups, (t(472) = .989, p = .323).

Table 1. Demographic characteristics of participants (N = 613)

Characteristic	Full sample		Virtual		In-person
	n	%	n	%	n	%
Sex
Male	444	72.4	119	68.8	325	73.9
Female	169	27.6	54	31.2	115	26.1
Age
18–24 years	88	14.4	20	11.8	68	15.5
25–34 years	208	33.9	61	35.3	147	33.4
35–44 years	205	33.4	57	32.9	148	33.6
45–54 years	72	11.7	19	11.0	53	12.0
55 or older	40	6.5	16	9.2	24	5.5
Race^a
American Indian or Alaska Native	22	3.6	4	2.3	18	4.1
Asian	2	0.3	0	0	2	0.5
Black or African-American	102	16.6	35	20.2	67	15.2
Native Hawaiian or other Pacific Islander	3	0.5	1	0.6	2	0.5
White	487	79.4	139	80.3	348	79.1
Other	33	5.4	7	4.0	26	5.9
Ethnicity
Hispanic or Latino	41	6.7	10	5.8	31	7.1
Not Hispanic or Latino	571	93.1	163	94.2	408	92.9
Monthly Income^b,d
less than $1,000	452	81.1	112	73.7	340	84.0
$1,001-$2,000	63	11.3	22	14.5	41	10.1
$2,000 or more	42	7.5	18	11.8	24	5.9
Education^c
No degree or diploma earned	73	11.9	16	10.3	57	13.9
HS Diploma or GED	277	48.9	81	52.3	196	47.7
Vocational/technical certification	58	9.5	12	7.7	46	11.2
Some college but no degree	103	16.8	26	16.8	77	18.7
Associate’s or higher	55	9.0	20	15.1	35	8.5
Psychological Distress^e
Low	211	35.3	57	34.3	154	35.7
Moderate	304	50.9	90	54.2	214	49.7
Severe	82	13.7	19	11.4	63	14.6

^aN = 649; this is a multiple response item to allow participants to self-identify as more than one race; participants could report more than one race

^bN = 557; 60 participants did not report income

^cN = 566; 51 participants did not report education

^dparticipant income was different by condition among respondents to the question, p < .05

^eN = 597

Program outcomes

The results of the paired samples t-tests showed that both in-person and synchronous virtual groups had statistically significant improvements in the number of curriculum knowledge items correct from pretest to posttest. For the in-person group, ($\bar{X}$ =2.82, SD =1.60) at pretest and ($\bar{X}$ =5.64, SD =2.46) at posttest, t(439) = 19.53, p <.001. For the virtual group, ($\bar{X}$ =3.23, SD =1.32) at pretest and ($\bar{X}$ =4.66, SD =2.20) at posttest, t(172) = 7.68, p <.001. Difference scores were compared for the in-person and virtual groups using independent samples t-tests. The virtual group ($\bar{X}$ =1.43) had significantly less change in items correct from pretest to posttest, t(385) =5.87, p <.001, compared to the in-person group ($\bar{X}$ =2.82). Cohen’s d effect size value was d =0.48.

Discussion

This study examined results using a reliable and valid measure of curriculum knowledge to compare in-person and synchronous virtual training during the coronavirus pandemic. Using data from 613 program participants in HRI collected from October 2019 to September 2020, the results showed that the curriculum was effective in increasing knowledge in virtual and in-person groups with significantly higher knowledge scores at posttest than pretest in both groups. However, the improvement from pretest to posttest for the in-person group was significantly greater than for the virtual group. The virtual group started with a higher average score at pretest and showed less improvement at posttest relative to the in-person group, resulting in a change in scores for the virtual group that was approximately half the size of the in-person group. These results provide support for the virtual curriculum delivery as an effective method, however, not as effective as in-person instruction.

Demographic differences did not account for differences in knowledge gain. There was no significant difference in sex, age, race, ethnicity, and education between groups. The in-person group had a slightly lower mean income; however, 92.5% of the overall sample reported income of less than $2,000 per month, suggesting that most individuals in both groups were living near or below poverty. Additionally, psychological distress scores on the K6 did not differ between in-person and virtual groups. Therefore, no study variables could account for the difference in knowledge gain between the in-person and virtual groups supporting the validity of the finding that in-person instruction had a greater effect on knowledge gain.

While extant literature demonstrated that online learning can be as effective as in-person (Benjamin et al., 2008; Berland et al., 2019; Neuenschwander et al., 2013; Wantland, Portillo, Holzemer, Slaughter, & McGhee, 2004), our data suggest that real-time, interactive virtual delivery is not as effective as in-person delivery for the HRI program. Reasons related to instruction quality participant preparation, presentation style, or content modifications and materials due to the sudden changes during the pandemic, may account for the differences in knowledge gain between groups. The study and discussion could contribute to the disaster preparation for community-based programs like HRI. First, instructors received extensive instructional training to be prepared for in-person curriculum delivery and not adequate training for virtual curriculum given the sudden change to respond to the pandemic. The 90-min training provided only focused on Zoom platform utilization, which may not have been the only challenge in the rapid transition. There was no time to practice teaching virtually and little information about adapting in-person instruction to virtual instruction was available.

Secondly, as new and challenging as adapting to a virtual instructional environment is, virtual learning is also new and challenging to participants as well. Technical difficulties, internet stability, and a less-controlled environment (e.g., the physical environment of the participants usually at home, online environment such as e-mail) make it more difficult for people to concentrate on the classes (Mahmood, 2021; Rajab et al., 2020). Cognitive load theory (Sweller, van Merrienboer, & Paas, 1998) and multimedia theory (Mayer, 2009) suggest that extraneous cognitive load and the burden on working memory are important considerations for online learning. “Zoom fatigue” is another challenge that emerged during the era of virtual learning, caused by close-distanced eye gaze, continually seeking non-verbal cues, reduced mobility, and lack of adequate breaks interfering with learning (Bailenson, 2021). Distractions from seeing our image could increase stress because of constant self-awareness (Daigle, 2020, May 27).

Finally, the instruction material was designed for in-person delivery and was delivered in the virtual environment without adaptation (i.e., changes in content and format). It is conceivable that in a different learning environment, changes in the material may be needed to aid participants in better understanding the content. For example, using more relatable real-world case studies, developing new interaction capacity in the virtual classroom, or adapting the presentation could help participants engaged in the class (Daigle, 2020, May 27; Mahmood, 2021). Changes in instruction to bridge the gap between how instructors communicate virtually, how participants learn remotely, and content adaptation represents a critical area for research to achieve greater effectiveness of instruction in a virtual environment.

Limitations

Posttest data were gathered at the end of the curriculum, which is a common issue for studies of curriculum delivery. Without a subsequent follow-up, learning retention and its effect on behavioral outcomes are not known. This study included only quantitative analysis to compare content knowledge between two groups. More information about the curriculum delivery from interviews, focus groups or observation with instructors would be beneficial to understand the dynamics during the classes and to identify opportunities to improve instruction (Harn, Parisi, & Stoolmiller, 2013).

Although income and education were measured based on approved measures, these measures were gathered in ordinal groups and were not normally distributed. Therefore, it could be due to errors in measures that more individual-level differences could not be explained by the data.

The findings suggest that further research is needed to answer questions about how the differences in knowledge gain occur. Although this was a natural experiment without robust controls, the putative difference was simply the difference in the instructional platform: synchronous training compared to in-person. Participants in the virtual group may have been under increased external stressors due to the COVID pandemic and associated economic recession (Salari et al., 2020), but these stressors, while could lead to a reduction in learning capacity (Sandi & Pinelo-Nava, 2007), were not found to be significantly different between the two groups.

Implications

Given that virtual delivery has proved to be an effective alternative approach and the advantages of utilizing virtual platforms cannot be overlooked, especially when responding to disaster and hazard, the central question is what can be done to improve virtual curriculum delivery to close the gap between virtual and in-person delivery. Instructors may need training specifically on delivering content virtually. Another factor may be practice. Instructors had been providing instruction for several years in an in-person environment and were suddenly cast into the role of virtual instructor. Virtual teaching strategies on how to help participants focus on the content of classes and maximizing the teaching quality may bring virtual instruction closer to traditional teaching. Voice and vocal functions may have more of an effect on participants to bring attention to critical content when non-verbal or eye contact could not be detected (Mahmood, 2021). It is also important to discover new approaches to increase virtual class interaction. For example, purposeful interpersonal interaction to rebuild a sense of community has been suggested by some (Chatterjee & Correia, 2020; Mahmood, 2021; Mehall, 2020). The framework used to leverage participants’ attention in the traditional classroom and promote active learning may be able to be adapted for virtual settings (Keller, Davidesco, & Tanner, 2020). Professional training to help instructors be aware of the difference, grasp the skills to do remote teaching, and feel confident about online teaching could increase instructors’ readiness for the transition and teaching quality (Scherer, Howard, Tondeur, & Siddiq, 2021). Small but meaningful changes to the presentation of course materials in a virtual format may be needed to facilitate learning, such as omitting non-essential words from PowerPoint slides that accompany figures (Garrison & Cleveland-Innes, 2005; Khalil & Elkhider, 2016; Swan, 2001).

To minimize distractions and fatigue, instructors and participants need to be more understanding and collaborative (Chatterjee & Correia, 2020). On one hand, instructors could help with technical and hardware issues (Mahmood, 2021). Allowing participants to turn off cameras, checking in regularly on participants’ emotions and feelings, and adding breaks to help regain focus and energy could help participants be relaxed and engaged (Daigle, 2020, May 27). On the other hand, participants could contribute by increasing their readiness for virtual learning, like being flexible to be able to “self-manage their learning,” setting up a private and quiet place for virtual classes, and being willing to “interact and communicate with others online (Joosten & Cusatis, 2020).

Conclusion

This study suggested that synchronous training delivery of healthy relationships education is effective for knowledge acquisition but may not be as effective as in-person curriculum delivery. The findings are an initial step forward in identifying and closing gaps to improve the quality of real-time, synchronous virtual learning to achieve better instructional outcomes. Further adaptation of the curriculum including changes in delivery may be needed to improve effectiveness. Research on the effectiveness of adaptations is important to determine if changes to virtual instruction are capable of approximating in-person instruction or if intractable differences exist. In-person instruction has existed since antiquity. Relatively speaking, virtual instruction is in its infancy, and refinements and improvements are underway. A potential positive impact is that through a better understanding of techniques that improve synchronous virtual training, these developments may also have implications for improving in-person curriculum delivery.

Abbreviations

HRI: The Healthy Relationships Iowa programK6: The 6-item Kessler Psychological Distress ScaleOFA: Office of Family Assistance

Data availability

Data were generated at The University of Iowa School of Social Work, National Resource Center for Family Centered Practice and nFORM maintained by Mathematica Policy Research, Inc. for the Administration for Children and Families. Data supporting the findings of this study are available from the corresponding author on request.

Acknowledgments

The study was supported in part by Healthy Relationships Iowa (HRI) with funding by HMRE program by the U.S. Department of Health and Human Services, Administration for Children and Families, Grant Number 90FM0068. These services are available to all eligible persons, regardless of race, gender, age, disability, or religion. Daryl Vanderwilt served as the principal investigator of the HRI project.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the Administration for Children and Families [90FM0068].