A Web-Based Team-Oriented Medical Error Communication Assessment Tool: Development, Preliminary Reliability, Validity, and User Ratings

Abstract

Background: Multiple-choice exams are not well suited for assessing communication skills. Standardized patient assessments are costly and patient and peer assessments are often biased. Web-based assessment using video content offers the possibility of reliable, valid, and cost-efficient means for measuring complex communication skills, including interprofessional communication. Description: We report development of the Web-based Team-Oriented Medical Error Communication Assessment Tool, which uses videotaped cases for assessing skills in error disclosure and team communication. Steps in development included (a) defining communication behaviors, (b) creating scenarios, (c) developing scripts, (d) filming video with professional actors, and (e) writing assessment questions targeting team communication during planning and error disclosure. Evaluation: Using valid data from 78 participants in the intervention group, coefficient alpha estimates of internal consistency were calculated based on the Likert-scale questions and ranged from α = .79 to α = .89 for each set of 7 Likert-type discussion/planning items and from α = .70 to α = .86 for each set of 8 Likert-type disclosure items. The preliminary test–retest Pearson correlation based on the scores of the intervention group was r = .59 for discussion/planning and r = .25 for error disclosure sections, respectively. Content validity was established through reliance on empirically driven published principles of effective disclosure as well as integration of expert views across all aspects of the development process. In addition, data from 122 medicine and surgical physicians and nurses showed high ratings for video quality (4.3 of 5.0), acting (4.3), and case content (4.5). Conclusions: Web assessment of communication skills appears promising. Physicians and nurses across specialties respond favorably to the tool.

Acknowledgments

The Web assessment tool reported in this study was developed as part of a research study, “Using Team Simulation to Improve Error Disclosure to Patients and Safety Culture.” This study was funded by the Agency for Healthcare Research and Quality (AHRQ) (1U18HS166580-1). We acknowledge the following Web development team members in the School of Nursing: Ashley Bond, Alan Gojdics, David Hughes, and David Jones. In addition, the following colleagues are acknowledged for their contribution to scenario development: Lianne Jeffs, Wendy Levinson, and Lorelei Lingard (University of Toronto); and Sherry Espin (Ryerson University).