System providing automated feedback improves task learning outcomes during child restraint system (CRS) installations

Abstract

Objectives

The objective was to build and test an automated, interactive educational system to teach adults how to install a child restraint system (CRS) into a vehicle seat.

Methods

The automated feedback system (AFS) consisted of a mockup vehicle fixture, convertible CRS, and doll. Sensors were implemented into the equipment so that forward-facing (FF) CRS installation errors could be detected. An interactive display monitor guided users through the CRS installation process and alerted them when steps were done incorrectly. Sixty adult volunteers were recruited and randomized into either the treatment group or the control group. The treatment group used the AFS to guide them through a practice installation. The control group also completed a practice installation using the same equipment fixture without the feedback feature turned on; they only had standard printed instruction manuals to guide their tasks. Then, participants from both groups completed a second CRS installation in a real vehicle with standard instruction manuals only. The frequencies and types of errors in all the installations were evaluated by a Child Passenger Safety Technician (CPST). Error rates were compared between the treatment and control groups using lower-tailed t-tests and Pearson’s chi-square tests. Error rates were evaluated considering minor and serious errors together and also considering serious errors alone.

Results

Compared to the control group, participants who trained with the AFS exhibited fewer overall errors (minor and serious) in their fixture installations (p < 0.0001) as well as their follow-up vehicle installations (p < 0.0001). Specifically, participants in the treatment group had fewer errors in choosing an installation method, locking the seat belt (SB), tightening the SB or lower anchors (LA), and tightening the harness (p = 0.0002, p = 0.0003, p = 0.0084, and p = 0.0098, respectively, compared to control group during follow-up vehicle installations). The treatment group also performed significantly better than the control group when only serious errors were considered.

Conclusions

An automated feedback system is an effective way to teach basic CRS installation skills to users.

Key Words:

• Child restraints
• child safety
• child seat
• misuse
• CRS
• installation errors

Introduction

Studies estimate that 46% to 95% of child restraint systems (CRS) are misused in family vehicles (Greenwell 2015; Hoffman et al. 2016). The efficacy of various educational methods have been investigated over the past several decades. Improvements on passive educational materials such as written instruction manuals, CRS/vehicle labeling, and online resources have been only moderately successful in reducing CRS misuse in laboratory studies (Klinich et al. 2010; Mansfield et al. 2019). Some studies suggest that a lack of feedback to the caregiver might be a barrier to proper CRS use (Hall et al. 2018). They report that caregivers tend to repeat mistakes because they fail to recognize their own errors (Klinich et al. 2014).

Hands-on training with a Child Passenger Safety Technician (CPST) is an effective way to educate caregivers and reduce CRS misuse. Caregivers who reported any type of prior interactive, hands-on instruction were more likely to have properly installed CRS in their vehicles compared to caregivers who reported none (Lane et al. 2000). In a controlled study, adults assigned to a hands-on training group were four times more likely to be able to install a CRS correctly than adults in the control group, even though both groups watched an educational video and were allowed to use instruction manuals and other written guides (Tessier 2010). Duchossois et al. (2008) confirmed the long-term benefits of hands-on instruction, observing decreased rates of installation errors between families’ baseline inspections and a follow-up appointment 6–12 months later. Emerging evidence suggests that working with a CPST via interactive virtual presence (IVP) is also effective at reducing CRS misuse (Swanson et al. 2020).

Installing and using a CRS is considered a psychomotor skill (Tessier 2010). There are several critical steps for learning a new psychomotor skill: identification of task(s) and learner skill level, pre-conceptualization, verbal and visual demonstration, immediate error correction followed by self-guided error correction, practice, and post-skill feedback (Nicholls et al. 2016). Hands-on CRS training with a CPST typically includes many of these critical learning steps. These factors likely contribute to the documented success of such training. Unfortunately, there can be several obstacles to administering hands-on, interactive CRS training to caregivers. Caregivers must seek out locations offering the service and bring their vehicle, CRS, and child to the appointment. Appointments require a time commitment of 20 minutes or more. Some caregivers might be intimidated by the one-on-one session or may not be aware that the service exists. As a result of these barriers, fewer than 20% of caregivers seek out hands-on instruction from a CPST (Greenwell 2015).

This proof-of-concept study investigates a potential tool to make hands-on CRS education more accessible to the 80% of caregivers who choose not attend an in-person seat check. An automated feedback system (AFS) consisting of a vehicle seat, CRS, and doll was constructed to teach CRS installation skills to caregivers. The system provided computerized, real-time feedback about the user’s CRS installation via sensors in the equipment and a large touchscreen display. The hypothesis was that users who practice a CRS installation using the AFS will produce follow-up installations with fewer errors compared to participants who practice without constructive feedback. The long-term goal is to develop the AFS for use by the general public in locations such as retail stores, waiting rooms, and airports to reach caregivers who would otherwise be deterred by the barriers of attending a one-on-one CRS check.

Methods

Equipment

A frame was constructed to support late-model year, second row sedan seat cushions according to the CAD model of the original vehicle. A popular convertible CRS was selected for instrumentation along with a size-appropriate doll. The AFS was designed to teach only forward-facing (FF) installation at this time. Sensors were implemented in the following locations. Photos are provided in Supplementary material Appendix A.

• A six-axis load cell (6A40B 6-Axis Force Torque Load Cell, Interface Inc., Scottsdale, AZ) was incorporated into the lower anchor (LA) assembly to detect the tension produced on the LA strap (Mansfield et al. 2018). A single-axis load cell (LPU-250-C, Transducer Techniques Inc., Temecula, CA) was attached to the seat belt (SB) at the outboard lower anchor point on the vehicle seat frame to detect the tension produced on the SB.

• A magnetic Hall effect sensor (SR3F-A1, Honeywell International Inc., Morris Plains, NJ) was incorporated into the back of the CRS near the storage location of the tether. The actuator magnet was attached to the top tether connector hardware. When the tether was in its storage location on the back of the CRS, the sensor recognized its location.

• A second magnetic Hall effect sensor was incorporated near the top tether anchor on the vehicle seat fixture. This sensor recognized when the top tether was attached to its proper anchor via the same actuator magnet in the top tether connector hardware.

• A magnetic Hall effect sensor and actuator magnet were positioned in the bottom of the CRS such that the sensor was activated when the rear-facing (RF) recline foot was extended and not activated when the recline foot was retracted for FF mode.

• A miniature single-axis tension load cell (MLP-50, Transducer Techniques, Temecula, CA) was incorporated in the 5-point harness webbing system underneath the seating surface (Mansfield et al. 2019). This load cell detected the tension in the harness.

• A magnetic Hall effect sensor was embedded in the doll’s thorax at the proper height for the chest clip (retainer clip). An actuator magnet was attached to the back of the chest clip to activate the thorax sensor when the chest clip was buckled and positioned at the proper height (Figure 1).

  Figure 1. The fully-instrumented AFS is shown above. The monitor on the desk served as the interactive interface.

  

All sensors were wired into a data acquisition system (NI-9237, C-series, RJ50; and NI-9201, C-series; National Instruments, Austin, TX). LabVIEW NXG 2.1 (National Instruments, Austin, TX) was used to program the user interface (Figure 2). The sensor outputs were developed into six installation steps: 1. Set recline angle, 2. Remove tether from storage; 3. Install snugly using SB or LA; 4. Attach top tether; 5. Tighten harness; 6. Adjust chest clip. The steps were listed down the left side of the user's screen. Each step included a visual indicator which was red when the step was not completed and turned green when the system sensed adequate completion. "Help" buttons located next to each step brought up detailed photos and instructions about how to complete that particular step (see Supplementary material Appendix B). The user could check his/her work within each help screen before returning to the main screen. The steps could be completed in any order, but the order given on the screen represented the easiest path to completion. The main screen also displayed a photo of the goal: a properly installed FF CRS. When all six steps were completed correctly, a congratulatory message appeared.

Figure 2. Main screen of the AFS user interface. In the example above, the user has correctly completed steps #1–3 (in green) and still needs to work on steps #4–6 (in red).

Study design

The study protocol was approved by the Institutional Review Board at The Ohio State University (OSU; Protocol Number 2018H0318). Sixty adult volunteers were recruited using OSU employee newsletters and departmental mailing lists. Participants were allowed to have previous CRS experience but it was not required. Participants were randomized into either the treatment group (n = 30) or control group (n = 30). The treatment group completed a "practice" FF CRS installation on the AFS, using the interface to guide them through the process and check their work. The control group completed a practice installation on the same seat fixture using the same equipment, but the user interface was turned off. Thus, they received no feedback on their performance. Both groups had the manufacturer-provided CRS and vehicle instruction manuals available. An instructional script was read to each participant (see Supplementary material Appendix C). The instructions did not specify to use the SB or LA; this decision was left to the participants. The recline setting of the CRS was presented in FF mode. When finished installing, all participants were given a survey to respond to each of these statements on a Likert scale of −3 (strongly disagree) to +3 (strongly agree), with 0 being neutral: 1. I am confident that I performed the tasks correctly; 2. The tasks were easy to complete; 3. The instructions were helpful; 4. I learned something new about child seats.

Next, each participant was taken outside to a recent model year minivan (see Supplementary material Appendix D). The vehicle environment was slightly different than the mockup sedan seat used in the practice fixture: the vehicle had bucket seats in the second row rather than bench seats. The top tether anchor was located on the back of the seat rather than the back shelf. The third row minivan seats were stowed away and the hatch back was open improve accessibility to the top tether anchor. Small differences existed in the seat belt geometry, lower anchor locations, and seat contours compared to the fixture’s sedan seat. Participants were presented with another convertible CRS which was a different brand but otherwise similar in design to the one they had just practiced with. They were asked to install it in FF mode and secure the doll occupant. Both groups had access to the manufacturer-provided instruction manuals but no other performance feedback was available. Thus, this portion of the study aimed to evaluate whether users could retain the information they learned during their practice installations when presented with a more realistic scenario.

Data analysis

The same CPST checked each practice installation in the fixture and each follow-up installation in the vehicle. All observed errors were recorded on a checklist. The errors identified by the AFS were recorded for both the treatment group (who could see the errors on the screen) and the control group (who did not have access to this information). A validation analysis was completed by comparing the errors observed by the CPST vs. the errors identified by the AFS for each participant.

Errors were categorized as "serious" or "minor." Serious errors were defined as those that would likely result in harm to the occupant during a crash: not locking the seat belt, very loose harness, forgetting the top tether, etc. (Brown et al. 1995; Hummel et al. 1997; Legault et al. 1997; Lalande et al. 2003; Arbogast et al. 2005; Arbogast and Jermakian 2007; Lesire et al. 2007; Menon and Ghati 2007; Bulger et al. 2008; Skjerven-Martinsen et al. 2011; Rudin-Brown et al. 2017; National Highway Traffic Safety Administration (NHTSA) 2014; Manary et al. 2019). Minor errors were considered to be less consequential, such as twisted but otherwise tight LA or harness webbing, using lower anchors and seat belt together, etc. A full list of minor and serious errors are provided in Supplementary material Appendix E, along with literature supporting the severity categorization of each. To establish the efficacy of the treatment, the average number of errors per participant was compared between groups using lower-tailed t-tests. Totals were calculated in two ways: considering minor and serious errors together, and serious errors only. Further analysis was conducted using chi-square tests to compare the frequency of each specific type of error committed in each group.

Results

Participant information

Basic participant information is in Table 1. Full participant details are in Supplementary material Appendix F.

Table 1. Participant background and demographic information.

		Treatment Group (n = 30)	Control Group (n = 30)	All (n = 60)
Age (years)	Mean	29.8	27.4	28.6
	Standard deviation	9.4	6.8	8.2
Sex	Female	20	22	42
	Male	10	8	18
English native language	Yes	23	25	48
	No	7	5	12
Ever installed a FF CRS?	Yes	15	16	31
	No	15	14	29
Ever attended car seat check?	Yes	1	5	6
	No	29	25	54

System validation

The sensitivity, specificity, and accuracy of each output sensor was calculated using the CPST's observed errors as the gold standard. The harness tension sensor had the lowest sensitivity, indicating a higher possibility of false negatives in this metric. The sensitivity of the recline angle indicator could not be calculated because none of the participants made an error in recline angle (i.e., true positive rate was zero). This is likely because the CRS was presented with the recline angle already in FF mode. The accuracy of the AFS was 88% or better in its reporting of each error (Table 2).

Table 2. Sensitivity, specificity, and accuracy of each sensor.

	Sensitivity (%)	Specificity (%)	Accuracy (%)
Recline angle	N/A	100	100
Tether removal from storage	100	100	100
Installation tension	93.1	93.5	93.3
Tether on anchor	100	87.9	88.3
Harness tension	70.0	92.0	88.3
Chest clip	100	93.2	95.0

Fixture installations

During practice installations on the fixture, the control group committed an average of 3.4 ± 2.1 errors per participant (including both minor and serious error types). The treatment group committed an average of 0.8 ± 1.1 errors per participant. The difference in average number of errors between groups was significant (p<.0001). When only serious errors were considered, the control group committed an average of 1.5 ± 1.2 serious errors per participant while the treatment group committed an average of 0.1 ± 0.5 serious errors per participant (p<.0001).

Figure 3 shows the percentage of each type of error in each group during practice installations in the seat fixture. Each error category is fully defined in Supplementary material Appendix E. The treatment group performed significantly better than the control group in terms of installation method (choosing either the LA or SB, not using both together), switching the SB into locking mode (when SB method was chosen), tensioning the installation (i.e., tightening the LA or SB to adequately pass the 1-inch test), adjusting the chest clip to the proper height, and tensioning the harness to pass the pinch test. Other error categories which were not significantly different between groups include: attaching the LA connectors to the proper LA locations (when that method was chosen), ensuring the LA or SB webbing was flat and not twisted, using the correct tether anchor, ensuring the tether webbing was flat and not twisted, tensioning the tether, and ensuring the harness webbing was flat and not twisted. Full chi-square data are shown in Supplementary material Appendix G.

Figure 3. Percent of participants committing each type error during practice installations in the test fixture.

Vehicle installations

During follow-up installations in the real vehicle, the control group committed an average of 3.5 ± 2.1 errors per participant (including both minor and serious error types). The treatment group committed an average of 1.3 ± 1.4 errors per participant. The difference in average number of errors between groups was significant (p<.0001). When only serious errors were considered, the control group committed an average of 1.6 ± 1.2 serious errors per participant while the treatment group committed an average of 0.5 ± 0.9 serious errors per participant (p=.0024).

Figure 4 shows error rates during participants’ follow-up installations in the vehicle. The treatment group committed fewer errors than the control group in choosing an installation method, putting the SB into locking mode, tensioning the tether, tension the installation (passing the 1-inch test), and tensioning the harness. The treatment group had slightly higher error rates in some categories in the vehicle compared to their practice installations in Figure 3. This is specifically true for switching the SB into locking mode and tensioning the installation. This outcome suggests that participants performed better when they were actively receiving feedback, as opposed to retaining the information to replicate it on their own later. Differences in performance in the vehicle might also be due to differences in the vehicle seats and CRS features between the two test environments. However, the treatment group still out-performed the control group in several categories, indicating that some learning was accomplished and retained within the treatment group. Full chi-square data are shown in Supplementary material Appendix H.

Figure 4. Percent of participants committing each type of error during follow-up installations in a vehicle.

Other factors

Logistic regression was used to investigate the influence of other predictive factors on the total number of misuses (minor and serious) observed in each participant’s vehicle installation. The following factors were not significant predictors of performance in either the control group or the treatment group: CRS manual use, vehicle manual use, age, sex, education level, income level, being a native English speaker, previous experience with any CRS, or previous experience with a FF CRS. The only significant predictive factor was previous attendance at a car seat check for the control group (p = 0.0003).

During the fixture installations, participants in the treatment group were significantly less likely to look at the traditional instruction manuals compared to the control group (CRS manual: Χ2 = 45.882, p < 0.0001; Vehicle manual: Χ2 = 9.231, p = 0.0024). During the vehicle installations, participants in the treatment group were still significantly less likely to use the manuals compared to the control group (CRS manual: Χ2 = 16.484, p < 0.0001; Vehicle manual: Χ2 = 6.405, p = 0.0114).

Participant feedback

A follow-up survey had participants rank their agreement with several statements from −3 (strongly disagree) to +3 (strongly agree). Participants in the treatment group were more likely than the control group to agree that they were confident they performed the tasks correctly (2.6 ± 0.6 vs. 1.2 ± 1.5; p <.0001), the tasks were easy to complete (2.1 ± 1.2 vs. 1.0 ± 1.6; p=.0028), the instructions were helpful (2.7 ± 0.5 vs. 0.4 ± 1.6; p <.0001), and they learned something new about child seats (2.2 ± 1.3 vs. 1.2 ± 2.0; p <.0233).

Discussion

An automated feedback system was designed to teach CRS installation skills to adults. Participants who had access to the automated real-time feedback produced better CRS installations compared to the control group. The treatment group had fewer errors during the installations on the feedback fixture itself and also during follow-up installations where feedback was no longer available. The AFS helped participants identify a proper method of installation (either SB or LA, not both together), put the SB into locking mode, tighten the top tether, tighten the SB or LA to pass the 1-inch test, and tighten the harness.

The control participants exhibited high error rates during both installations, suggesting that learning did not occur from simple repetition of the installation tasks without constructive feedback. Other studies agree that caregivers tend to repeat errors because they are not able to self-identify that an error has been made (Klinich et al. 2014). For the treatment group, the installations in the AFS were slightly better than subsequent installations in the vehicle environment. These results suggest that some participants struggled to independently replicate the skills that they had previously achieved. Thus, providing feedback during every installation might be the most effective way to prevent misuse. CRS manufacturers might consider incorporating feedback systems directly into their products to achieve this goal. A few CRS models currently provide different types of installation feedback, but most do not.

Three participants from the treatment group were not able to pass the acceptance thresholds for every indicator. Two out of the three made cursory efforts to re-check their work, but did not click into the “Help” menus to receive detailed instructions. One participant could not get the installation tension tight enough despite several efforts. All three participants wondered aloud whether the system was functioning correctly. These participants might have benefited from sliding scale indicators as opposed to the “Pass/fail” indicators they were shown. A sliding scale indicator could show users what the target tension is and display how close they are to reaching the target. This type of detailed feedback indicator might encourage users to exert more effort if they can see incremental progress toward their tension goal.

Unfortunately, users in the treatment group were less likely than control participants to look at the manufacturer-provided instruction manuals during either installation. If the follow-up CRS had different or confusing features, it is unclear how many of these participants would have consulted the CRS manual or other sources to learn about the new features. It is important for future versions of the AFS to encourage users to use these manufacturer-provided tools to help them navigate their own unique CRS installations.

Future work includes refinement of the AFS based on the outcomes of this proof-of-concept study. Modifications could include tutorials for RF installation or different models of CRS such as infant seats or boosters. Future work also includes usability testing in environments outside the laboratory. For example, more data is needed to understand how passersby might interact with the AFS in situations such as waiting rooms, airports, or retail stores. Attention spans and user motivations might be different in these public scenarios compared to the laboratory setting explored in this study.

It is important to note that traditional, one-on-one seat checks with the caregiver’s own product(s) and vehicle remain the best form of CRS education. Safety advocates should continue to recommend these services and encourage all caregivers to utilize them, either in-person or virtually. However, the fact remains that only 20% of the caregiver population takes advantage of these services (Greenwell 2015). With that in mind, the goal of this study was to develop effective learning opportunities in locations and situations that are convenient and likely to be utilized by a caregiver population who might not seek out traditional car seat check services. In addition, the AFS could be programmed to inform users of local seat check events and encourage them to follow up with a CPST.

This study includes several limitations. The AFS was designed to teach only one type of CRS installation (convertible CRS in FF mode). No analyses were completed for any type of RF CRS or booster. The CRS used for the installation was a standard model with no unique features. Users were not taught how to navigate any unique technology available in their child’s specific CRS. All participants completed the follow-up installation in the vehicle immediately after practicing with the fixture. There was no long-term follow up to determine if participants retained the skills over a long period of time.

Acknowledgments

We gratefully acknowledge the Battelle Engineering, Technology and Human Affairs (BETHA) Endowment for supporting this work and the research volunteers who participated.

Disclosure statement

No potential conflict of interest was reported by the author(s).