Exploring Individual Opinions of Potential Evaluators in a 360-Degree Assessment: Four Distinct Viewpoints of a Competent Resident

Abstract

Background: Despite the highly acclaimed psychometric features of a 360-degree assessment in the fields of economics, military, and education, there has been increased interest in developing 360-degree instruments to assess competencies in graduate medical education only in the past recent years. Most of the effort to date, however, has focused on developing instruments and testing their reliability and feasibility. Insufficient attention has gone into issues of construct validity and particularly understanding the underlying constructs on which the instruments are based as well as the phenomena that affect ratings. Purpose: In preparation for developing a 360-degree assessment instrument, we explored variations in evaluators' opinion type of a competent resident and offer observation about evaluator's professional background and opinions. Method: Evaluators from two residency programs ranked 36 opinion statements, using a relative-ranking model, based on their opinion of a competent resident. By-person factor analysis was used to structure opinion types. Results: Factor analysis of 156 responses identified four factors interpreted as four different opinion types of a competent resident: (a) altruistic, compassionate healer (n = 42 evaluators), (b) scientifically grounded clinician (n = 30), (c) holistic, humanistic clinician (n = 62), and (d) patient-focused, health manager (n = 31). Although 72% of nurses/respiratory therapist evaluators expressed type C, 28% expressed other types just as often. Only 14% of evaluator physicians expressed type D, and the remainders were evenly split among the other types. Conclusions: Our evaluators in 360-degree system expressed four opinion types of a competent resident. The individual opinion and not professional background influences the characteristics an evaluator values in a competent resident. We propose that these values will have an impact on competency assessment and should be taken into account in a 360-degree assessment.

We are indebted to Boyd Richards, Michael Romano, and Alejandro Clavier for suggestions and support; to Bob Lynch for assistance in editing; and to faculty, nurses, respiratory therapists, and pediatric residents at Baylor College of Medicine and the University of Illinois at Chicago for their participation.

Notes

∗Significant factor loading (γ ≥ 0.43)

^aFactor definers: Q sorts loaded strongly and purely on one factor

^bFactor loaders did not total 156; some Q sorts load on more than one factor, others do not load on any of these factors