Evidence for validity of a survey to measure the learning environment for professionalism

Abstract

Background: With the emphasis on professionalism in academic health settings, including recently added accreditation requirements for US medical schools, there is a need for a valid and feasible method to assess the learning environment for professionalism.

Aim: This article describes the development and investigation of the validity of a brief measure, the learning environment for professionalism (LEP) survey, designed to assess medical student perceptions of professionalism among residents and faculty during clinical rotations.

Method: Two successive cohorts of third-year medical students completed the 22-item LEP survey at the conclusion of clerkship rotations, providing a total of 902 responses for scale reliability and principal components factor analysis, as well as assessment of changes in scores over time and correlations with a related clerkship evaluation item.

Results: The internal structure of the LEP survey was consistent with intended goals to assess both positive and negative professionalism behaviors. Acceptable internal consistency, sensitivity to change over time, and positive relationships between LEP scores and a concurrent measure of professionalism were observed.

Conclusions: Use of the instrument could help identify clinical learning environments for professionalism that represent either best practices or areas in need of improvement, assess the impact of professionalism initiatives, and help satisfy accreditation requirements.

Professionalism is a highly valued physician competency manifested in the behaviors that physicians exhibit in practice and believed to be acquired through explicit education, socialization, and the informal, “hidden curriculum” (Wear 1998; Inui 2003; Suchman et al. 2004; Papadakis et al. 2005; Cruess & Cruess 2006; Haidet & Stein 2006; Reddy et al. 2007; Hafferty & Levinson 2008; Mueller 2009; Lesser et al. 2010). Current conceptualizations of professionalism (Lesser et al. 2010), as well as new accreditation standards (LCME 2008), recognize that medical professionalism is a systems issue in which physicians’ behaviors are profoundly influenced by contextual and environmental factors (e.g., curricula, policies, role models, commercialism) (Arnold 2002; Epstein & Hundert 2002; Gofton & Regehr 2006; Stern & Papadakis 2006). Accordingly, many academic medical centers are responding with institutional initiatives tailored to their unique environments in order to promote professionalism and address unprofessional behaviors within their institutions (Brater 2007; Fryer-Edwards et al. 2007; Hickson et al. 2007; Cottingham et al. 2008). For North American medical schools, the Liaison Committee on Medical Education (LCME) introduced in 2008 an accreditation standard (MS-31A) related to the learning environment for professionalism (LEP). The standard recommends that schools “regularly evaluate the learning environment to identify positive and negative influences” on student professionalism and “develop appropriate strategies to enhance the positive and mitigate the negative influences” (LCME 2008, p. 24).

As part of our efforts to understand, analyze, and improve the LEP at our institution, we sought a survey that would be valid, brief, and easily administered. Additionally, we wanted an assessment of the professionalism environment in specific clerkships and clinical rotations rather than at the level of individual physicians. Recognizing that faculty physicians and residents are important role models for students, we wanted to assess both of these groups. Our intention was to measure the learning environment of specific departments within our medical school so that we could delineate best practices and areas in need of improvement. The surveys available at the time assessed only professional behaviors or unprofessional behaviors (but not both), had unclear ratings, measured student professionalism rather than faculty or residents, or measured attitudes about professionalism rather than witnessed behaviors (Arnold et al. 1998; Beaudoin et al. 1998; Szauter et al. 2003; Mann et al. 2005; Veloski et al. 2005; Blackall et al. 2007). Quaintance et al. (2008) have published a comprehensive assessment of professionalism with five sets of questions, two of which are similar to the survey described here, but their work was unavailable at the time. Not finding any survey suitable to our needs, we sought to develop a valid and brief measure of the LEP. This article describes the development of the LEP survey and presents evidence of its validity.

Methods

This study was approved as exempt by the Institutional Review Board at the medical school wherein it was conducted.

Instruments

The LEP survey is designed to be completed by medical students at the end of a clinical clerkship rotation to assess, in aggregate, their observed perceptions of professionalism among faculty physicians and residents. The LEP survey contains 11 items, with each one assessing behaviors of residents and attendings separately. Survey items inquire about the frequency of observed professional (5 items) and unprofessional behaviors (6 items) (Table 1). For content validity, items were adapted with minor wording modifications from three previously published scales (Arnold et al. 1998; Beaudoin et al. 1998; Szauter et al. 2003), and were informed conceptually by the work of the American Board of Internal Medicine (ABIM 1995) essential components of professionalism. Survey items are completed using a 4-point scale with a “not applicable” option that is unscored. Specifically, the positive professionalism items are scored such that consistently = 4, frequently = 3, occasionally = 2, and never = 1; whereas unprofessional items are scored in the reverse (never = 4, occasionally = 3, frequently = 2, and consistently = 1). Therefore, higher scores indicate better behaviors (more frequent desirable behaviors or fewer undesirable behaviors).

Table 1.  LEP survey items

I have observed residents/attendings

(1) Who are positive role models of effective doctor–patient relationships.

(2) Make derogatory comments about a patient or patient's family.

(3) Educating patients about their illnesses.

(4) Inappropriately withholding information or intentionally giving incorrect information to a patient.

(5) Who value human contact with their patients as an important component of patient care.

(6) Make derogatory comments about other physicians, health care providers or services.

(7) Who were concerned about the overall well-being of patients, not just their presenting complaints.

(8) Treating non-physician healthcare workers in a disrespectful or inappropriate manner.

(9) Treating patients unfairly because of the patient's financial status, ethnic background, sexual or religious preferences.

(10) Discussing confidential information in an inappropriate setting (e.g., cafeteria, crowded elevator).

(11) Place the needs of their patients ahead of their own self-interest.

In addition to our LEP survey, all clerkships require students to complete a 7-item rotation evaluation form. This form includes one item that specifically addresses professional characteristics of instructors rated on a 5-point Likert scale ranging from strongly disagree to strongly agree (“instructors demonstrated qualities such as respect for students, cultural awareness and respect for other health professionals”). Other items (not used for this study) assess clarity of goals and objectives, organization and participation in the clinical experience, informational content, and fairness of exams.

Procedures

The LEP survey was administered to two successive cohorts of third-year medical students at a medical school in the mid-south USA. To facilitate honest responses, the surveys were anonymous and students were informed that the results would be presented to the faculty in summary form at the end of the academic year. Surveys were administered to students by each clerkship coordinator on the last day of the clerkship rotation when other clerkship evaluations were administered. The survey was given during the last rotation of the fall semester in 2007, during the spring semester of 2008, and during the spring semester of 2009, and took less than 10 min for students to complete. The seven major clerkship rotations in which third-year students participate are affiliated with the clinical departments of family medicine, geriatrics, internal medicine, obstetrics–gynecology, pediatrics, psychiatry, and surgery. During the time frame of data collection, a total of 485 surveys out of a possible 586 were collected in 2008, representing an 83% response rate. A total of 417 surveys out of a possible 523 were collected in 2009, representing an 80% response rate.

To facilitate implementation of the survey and promote leadership buy-in for ongoing improvement, the College of Medicine (COM) leadership (e.g., dean, associate deans, department chairs, clerkship, and residency directors) were apprised of the plan to collect data during all planning stages, and were given opportunities to review the items and make wording suggestions. We also paid careful attention to communicating with clerkship coordinators about the data collection details, and being available and responsive to answer any questions about the study.

Context: Institutional and departmental interventions

We concluded the baseline year of survey data collection in May 2008. We believe it is important to note that beginning later that month and extending throughout the academic year 2008–2009, the COM dean launched a campus-wide professionalism initiative, as an independent initiative unrelated to the collection of data for this study. Activities associated with the dean's initiative included: (1) providing training for all hospital staff employees, medical students, resident physicians, and attending faculty physicians which focused on patient satisfaction and service, including the role of professional behavior; (2) two college-wide distinguished lectures on the aspects of professionalism given by nationally esteemed invited speakers; and (3) the adoption by the faculty of a code of conduct called the ‘College Professionalism Guideline’.

The authors reported results of the baseline year survey during the summer and fall of 2008 to the COM leaders. Administrative leaders from departments responsible for clerkships received hard copies of the survey results, identifying their individual department's data and blinding the identity of other departments. These leaders were encouraged to share the survey results with their department faculty members and residents.

In response to the 2008 LEP survey results, one department undertook multiple initiatives within the following year to raise awareness among attendings and residents in their department regarding the importance of consistently professional behavior. Specifically, this particular department engaged in an appreciative inquiry (see e.g., Carter et al. 2007) workshop and other activities led by their department chair specifically related to promoting professionalism.

Analyses

Validity characteristics discussed by Cook and Beckman (2006) were used to help guide the analysis and results presented. Briefly, the five categories of validity evidence which they outline for evaluating psychometric instruments are content validity (well-written questions that adequately represent the domain measured), response processes (the thought process of respondents), internal structure (consistency, factor structure), relations to other variables (e.g., correlations with other assessments) and consequences (do scores make a difference?). Statistical analyses were computed using SPSS 18.0 software. We conducted scale reliability analyses and principal components factor analysis using all data collected in both 2008 and 2009 (n = 902). Two principal component factors analysis procedures with varimax rotation were performed to explore the factor structure of the resident items and attending items separately. Internal reliability of the scale composite scores was assessed by computing Cronbach's alpha coefficients for each scale. To assess sensitivity of the instrument for measuring change, Mann–Whitney U-tests with corresponding probability of superiority (PS) effect sizes (Erceg-Hurn & Mirosevich 2008) were computed to assess 1-year changes in the mean LEP subscale scores (2008 vs. 2009 data). Since there was no significant difference between resident and attending scale total scores, we used an average of the parallel resident and attending items together for each respondent. We computed the 1-year change analyses for the total group combined, and by individual clerkship. To demonstrate relations with other variables, Spearman correlations were computed between LEP total scores and the professionalism item obtained from the standard end-of-clerkship evaluation forms.

Results

The factor structure of the LEP survey was consistent with intended goals, with two interpretable factors that were labeled “professional behaviors” and “unprofessional behaviors.” Table 2 shows the factor loadings, associated Cronbach's alpha coefficients, and the percent of explained variance for each factor. The resulting factor structure and item loadings were very similar for both sets of parallel resident and attending items. An item was included within a factor if its loading was 0.45 or greater. Each scale demonstrated acceptable internal reliability. Total variance explained by each factor in initial eigenvalues was also acceptably high (Table 2).

Table 2.  Item factor loadings, internal reliability coefficients, and explained variance

Item number		1	2
1	Residents	0.77	0.15
	Attendings	0.77	0.17
2	Residents	0.31	0.61
	Attendings	0.25	0.63
3	Residents	0.73	0.03
	Attendings	0.74	−0.04
4	Residents	0.09	0.72
	Attendings	−0.10	0.75
5	Residents	0.85	0.16
	Attendings	0.83	0.10
6	Residents	0.33	0.69
	Attendings	0.28	0.66
7	Residents	0.82	0.16
	Attendings	0.82	0.08
8	Residents	0.15	0.67
	Attendings	0.09	0.72
9	Residents	−0.15	0.72
	Attendings	−0.14	0.75
10	Residents	0.02	0.66
	Attendings	0.05	0.69
11	Residents	0.64	0.03
	Attendings	0.66	0.01
Cronbach's alpha residents		0.82	0.77
Cronbach's alpha attendings		0.82	0.78
Explained variance (%) residents		36	19
Explained variance (%) attendings		33	23

Note: Items that load greater than 0.45 on a factor are shown in bold face.

Table 3.  LEP survey mean scores in 2008 and 2009

Clerkship	LEP subscales	2008 Mean ± SD	2009 Mean ± SD
A	Professional	3.46 ± 0.49	3.52 ± 0.42
	Unprofessional	3.69 ± 0.40	3.81 ± 0.21
B	Professional	3.61 ± 0.39	3.69 ± 0.38
	Unprofessional	3.76 ± 0.62	3.94 ± 0.13
C	Professional	3.32 ± 0.53	3.29 ± 0.54
	Unprofessional	3.66 ± 0.27	3.71 ± 0.27
D	Professional	3.16 ± 0.63	3.46 ± 0.47
	Unprofessional	3.59 ± 0.48	3.81 ± 0.25
E	Professional	3.48 ± 0.53	3.38 ± 0.58
	Unprofessional	3.69 ± 0.51	3.69 ± 0.63
F	Professional	3.34 ± 0.53	3.57 ± 0.56
	Unprofessional	3.70 ± 0.32	3.75 ± 0.23
G	Professional	2.93 ± 0.57	3.10 ± 0.53
	Unprofessional	3.50 ± 0.38	3.56 ± 0.31
Total	Professional	3.31 ± 0.57	3.45 ± 0.52
	Unprofessional	3.64 ± 0.42	3.76 ± 0.32

Table 3 shows the LEP mean scores for each year, by clerkship. Consistent with reporting practices by Szauter et al. (2003), clerkship identities are represented by letters (A–G) to avoid inappropriate inferences about discipline-specific professionalism based on the data from a single institution. Mean change scores from year 1 to year 2 are displayed in Figure 1. Mann–Whitney U-tests comparing 2008 and 2009 LEP scores demonstrated improvement in students’ perceptions of both the increased professional (p < 0.001; PS = 0.57) and decreased unprofessional behaviors (p < 0.001; PS = 0.60) of attendings and residents for the total group. Analyses for each individual clerkship showed that one clerkship (clerkship D) improved significantly on both the professional (p = 0.003, PS = 0.65) and unprofessional behaviors subscales (p = 0.000; PS = 0.71); and another (clerkship F) significantly improved on the professional behaviors subscale only (p = 0.007, PS = 0.63). Notably, the department that improved on both scales (clerkship D) is the same department that conducted additional interventions to promote professionalism beyond those implemented at the institutional level over the study time frame.

Figure 1. LEP survey mean change on subscale scores from 2008 to 2009.

Positive relationships between the LEP scores and the concurrent measure of professionalism were observed in both years of the study but were statistically significant in the first year of data collection only (year 1, r = 0.81, p = 0.028; year 2, r = 0.46, p = 0.301).

Discussion

The study results suggest that the LEP survey can serve as a valid tool to assess medical student perceptions of the LEP. Estimates of reliability and internal structure were sound. The survey results demonstrated positive correlations with another measure of professionalism, providing evidence of concurrent validity.

A rarely reported source of validity evidence (Beckman et al. 2005) is consequences of the use of the assessment. A desired consequence of the authors in the development of this survey was to draw attention to the environment for professionalism in our clerkships and to spur clerkship, residency, and departmental leaders to make changes to improve their professionalism climate. This has been described as a “Hawthorne strategy” (Lied & Kazandijian 1998). One observed consequence occurred upon presentation of first year's survey results: the chair of the department for clerkship D announced that their department would explore the meaning of the results and determine what actions the faculty and residents would take to improve their scores. Several activities were followed including discussing survey results at a departmental meeting and an appreciative inquiry initiative conducted with all faculty and residents in that department. While college-wide professionalism activities, initiated without depending on the data collection efforts for this study, may have led to improvements overall from the first to the second year, the only clerkship that improved on both professional and unprofessional behaviors from year 1 to year 2 was this department.

Responsiveness to change over time is also an aspect of validity in psychometric instruments (Hays & Hadorn 1992). The survey was sensitive to measurement of change over time as evidenced by score improvements for all the clerkships averaged together. We believe this modest but positive change may have occurred in the context of an organized professionalism initiative that was instituted across the entire college. The survey also identified significant improvements in the scores of the only department that instituted targeted initiatives in response to the receipt of their year 1 scores (clerkship D).

There are limitations of this study. First, the data were collected at a single institution and thus may not be generalizable. The items tap only a subset of the range of professional and unprofessional physician behaviors that could potentially be assessed. However, a more comprehensive set of items comes at the expense of practicality. Future research to assess survey response process, which is the relationship between the intended construct of professionalism and the thought processes of the medical students completing the survey, would be of interest. Investigation of relationships between the LEP and other variables, such as patient and staff compliments or complaints, staff morale, malpractice claims, or medical board actions, would also be of importance. Comparing similar data from other academic medical centers for benchmarking comparison purposes and further validity testing would be beneficial. The survey is currently being used at another medical school and testing of the validity in that setting is underway.

In summary, evidence for validity of the LEP includes findings related to internal consistency, internal structure, correlations with another professionalism measure, responsiveness to change over time, and intended consequences of its use. The LEP survey is brief and easily implemented as part of standard clerkship or other clinical rotation evaluations. Use of the instrument could facilitate institutional assessment needed for accreditation as well as the evaluation of interventions to improve the LEP.

Practice points

• Measuring student perceptions of clinical learning environments for professionalism can help provide feedback to identify settings in which teachers demonstrate either best practices or areas in need of improvement.

• The LEP survey is brief and appears to be a valid and reliable tool to assess the impact of professionalism initiatives.

• Medical schools that utilize the LEP survey in an ongoing, proactive manner may find such a process useful for satisfying accreditation requirements, particularly among North American medical schools that are expected to regularly evaluate and temper factors in the learning environment that influence student professionalism.

Acknowledgments

The authors thank Dr Hugh Stoddard for his helpful comments on this study and Ms Elizabeth Hicks for research support. The institutional review board of the University of Arkansas for Medical Sciences gave ethical approval for this study. Portions of this study have been presented at the Annual Meeting of The Association of Directors of Medical Student Education in Psychiatry (ADMSEP), June 2009, Portsmouth, NH; and the Association of American Medical Colleges Annual Meeting, Group on Educational Affairs Conference on Research in Medical Education (RIME), November 2009, Boston, MA.

Funding/Support

None.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.