Professionalism in medical education: The development and validation of a survey instrument to assess attitudes toward professionalism

Abstract

Background and aim: This study examined attitudes toward professionalism in an academic medical center. The paper will describe the development and factorial validity of an instrument to measure attitudes toward professionalism in medical education among students, residents and faculty.

Methods: A factor analysis of the intercorrelations of responses to 36 items reflecting the American Board of Internal Medicine (ABIM) elements of professionalism for a sample of 765 medical students, residents and faculty was carried out. Data were collected during the spring of 2004. The study was conducted at the Penn State College of Medicine in Hershey, PA, USA.

Results: Main outcome measures include internal consistency reliability estimates (Cronbach's alpha) for each element of professionalism and a principal components analysis of the intercorrelations of responses to the 36 items in the questionnaire. Analysis of responses reveals seven identifiable factors of professionalism: accountability, altruism, duty, enrichment, equity, honor and integrity, and respect.

Conclusions: The Penn State College of Medicine Professionalism Questionnaire is one of the first valid and reliable surveys of attitudes among medical students, residents, and faculty that reflects seven elements of professionalism.

Introduction

Professionalism in medical education has become a pressing national concern. In the early 1990s articles in prominent publications began discussing the need to focus more upon professionalism education in both residency and medical school curricula (Greganti 1990; Johnson 1990; Relman 1992a; 1992b; Todd 1992; Waud 1992). During this same time period the American Board of Internal Medicine (ABIM) embarked on its ‘Project Professionalism’, which sought to examine ways in which the changing healthcare environment negatively affected the professional behavior of physicians. As part of its project the ABIM undertook an extensive literature review in order to establish a working definition of professionalism. In its report, ‘Professionalism in medicine: issues and opportunities in the educational environment’ American Board of Internal Medicine (1994b) the ABIM first offered a list of the elements of professionalism. These elements were: altruism, accountability, excellence, duty, honor and integrity, and respect for others. These elements combine to define guidelines for ideal behavior. A qualitative study aimed at conceptualizing professionalism (Van De Camp et al. 2004) presented three themes of professionalism that included interpersonal professionalism, public professionalism and intrapersonal professionalism.

Table

Practice points
•The Penn State College of Medicine Professionalism Questionnaire is a valid and reliable survey that revealed seven identifiable factors of professionalism: altruism, duty, enrichment, equity, honor and integrity, and respect.
•Use of this instrument can be helpful in assessing current professionalism curriculum and in driving curricular changes.
•This instrument can be used to assess curricular changes by tracking changes in attitudes toward professionalism across time.

In another study the Institute of Medicine issued a report entitled ‘Crossing the quality chasm: a new health system for the 21st century’ (Institute of Medicine 2001) calling for a major reform in the training of physicians and other health professionals. Responding to this report, and to ensure the highest quality of graduate medical education, the Accreditation Council on Graduate Medical Education (ACGME) initiated its ‘Outcome Project’ (ACGME n.d.), a long-term initiative designed to shift the orientation of graduate medical education accreditation from an emphasis on process measures to educational outcomes. ACGME accreditation now requires program directors to document that their residents have met pre-determined performance standards in six general competences. Professionalism is cited as one of them; therefore, teaching professionalism and evaluating the degree to which a resident meets acceptable and established criteria for professionalism is essential.

Undergraduate medical education (UME) programs also have responded by examining their own curricula in terms of successfully teaching this ACGME-identified competence and clarifying the instructional goal. One of the accreditation standards for the UME accreditation agency, the Liaison Committee on Medical Education (LCME), states:

A medical school must teach medical ethics and human values, and require its students to exhibit scrupulous ethical principles in caring for patients, and in relating to patients’ families and to others involved in patient care. Each school should assure that students receive instruction in appropriate medical ethics, human values, and communication skills before engaging in patient care activities. As students take on increasingly more active roles in patient care during their progression through the curriculum, adherence to ethical principles should be observed and evaluated, and reinforced through formal instructional efforts. (LCME 2004)

An assessment of the students’ existing attitudes and perceptions toward professionalism is essential to successfully designing an instructional plan that has the highest likelihood of improving professional behavior. This information should be taken into account in the design process. At the Pennsylvania State University College of Medicine (PSCOM), a questionnaire was developed and administered during the spring of 2004 that was aimed at assessing attitudes toward professionalism in students, residents and faculty.

The purpose of this paper is to describe the development, reliability estimates and validity of an instrument to measure attitudes toward professionalism in one medical school. Internal consistency reliability estimates (Cronbach's alpha) for each element and a principal components analysis of the intercorrelations of responses to 36 items reflecting the ABIM elements for a sample of 765 medical students, residents and faculty is reported.

Method

Item development

A nine-member task force was formed to design a method of inquiry aimed at answering two fundamental questions: (a) What is professionalism at the institutional level; and (b) How does socialization effect its development? The members of the task force (three females and six males) were on the faculty of the PSCOM, and they represented a cross-section of disciplines in the basic and clinical sciences. The criteria for inclusion on the task force were individuals who had a working knowledge of professionalism (i.e. taught, researched or published in that area) and who were interested in working on the project. With this background these individuals were familiar with the professionalism literature that was used to drive item development. This study was approved by the Penn State College of Medicine Institutional Review Board.

The Task Force members conducted an extensive review of the literature (a) to find instruments that purported to measure professionalism and that had been tested and validated; and (b) to operationally define professionalism in terms of identifying its characteristics, attributes and traits. No validated instruments were found that would adequately answer the intended research questions. Therefore, the group decided to develop its own comprehensive instrument.

The ABIM Monograph on professionalism (ABIM 1994) (a) identifies six broad elements of professionalism (altruism, accountability, excellence, duty, honor and integrity, and respect for others). Using the domain sampling approach to affective item development (Gable & Wolf, 1993) the PSCOM Task Force developed a pool of 60 items representing the most common, emergent and recurring interpretations of professionalism found in the literature based on the six ABIM elements. These items were also consistent with an internal document that delineated the goals of undergraduate medical education at PSCOM (PSCOM: Curriculum Evaluation Committee 2001). Using a modified Delphi Technique,¹ the Task Force achieved consensus on the most representative statements for each of the six elements. The group reviewed and refined the statements to eliminate any redundancy among the items, and to establish content-related evidence that the items were representative of professionalism. Items were eliminated through a judgmental process where the conceptual fit of each item was assessed. It was the consensus of the task force that deleted items were not an appropriate fit with the respective scale(s) and would not contribute positively to the overall reliability of each scale. After three rounds of deliberation, the Task Force retained 41 of the original 60 item statements.

The Task Force collectively categorized and coded each item statement into one of the elements. Some elements ended up with more item statements than others. A decision was made to use six statements for each element; thus the final instrument contained 36 items.

Description of the instrument

The PSCOM Professionalism Questionnaire was developed with four parallel forms (Medical Students; Residents; and two faculty forms—one for Clinical Science and one for Basic Science). All forms consist of a series of demographic questions specific to the group being assessed. In addition to a few differences between the demographic questions, forms were modified to show sensitivity to specific groups of educators. For example, ‘Research Subject’ was used in the items administered to basic science faculty whereas ‘Patient’ was used in the items administered to clinical faculty, residents and medical students.

Each form has a total of 36 items thought to represent the six ABIM elements: (1) Accountability, (2) Altruism, (3) Duty, (4) Excellence, (5) Honesty and Integrity, and (6) Respect, with six items representing each element. Items on the survey were clustered into groups of six, with one randomly selected item from each of the six elements comprising the clusters. Thereby, each of the six clusters of six items had an item representing each element.

Respondents were asked to rate the extent to which each statement reflected their definition of professionalism on a five-point Likert scale (1 = never, 2 = little, 3 = some, 4 = much, and 5 = great deal). In addition, respondents were also asked to rank-order the relative importance of the item within the cluster of six items (1 = highest rank, 6 = lowest rank) using each number only once within each item cluster. The rank-ordering of the item statements was intended to determine whether respondents made finer distinctions among the items when forced with a choice. No open-ended items were included on the instrument. All surveys were completed anonymously.

Sample

The survey was distributed to 1511 medical students, residents and faculty. A total of 765 completed surveys were returned, yielding a 51% return rate (56% males and 44% females). Table 1 gives the population, sample and return rates for each of the sub-groups.

Table 1.  Total population, number of respondents, and return rates by group (n = 765)

Respondent group	Total population	Respondents	Return rate (%)
Students	493	392	80
Residents	395	106	27
Faculty	623	267	43
Total^a	1511	765	51

^aThe total sample included 56% males and 44% females.

Results

Construct validity

A principal components analysis of item intercorrelations followed by oblique and varimax rotations was carried out for 765 complete sets of data to generate factors hypothesized to reflect the attitudinal elements of the instrument. The varimax rotation was found to generate the most meaningful solution. Utilizing Kaiser's Criterion, seven derived factors were found with eigenvalues greater than 1.0. The resultant factors were compared with the a priori factors (i.e. ABIM's six elements: (1) Accountability, (2) Altruism, (3) Duty, (4) Excellence, (5) Honesty and Integrity, and (6) Respect).

Although three of the six ABIM elements emerged from the analysis, the factor structure did not mirror the a priori categories as intended. Table 2 identifies the seven elements that emerged. Items were assigned a factor based on loadings of 0.40 or greater, and the item group was renamed based on the content of the items. The original elements did not cluster together precisely to emerge as factors. Rather, slightly new factors emerged in which the collective content of the items approximated five of the original elements in intent (Accountability, Altruism, Duty, Honor and Integrity, and Respect). Two additional clusters emerged that represent Enrichment and Equity. Assuming that the items generated by the Task Force represent the important issues in professionalism, these empirical data suggest that the elements identified by the ABIM may need to be fine-tuned in order to reflect more accurately how respondents perceive issues differently from the ABIM report.

Table 2.  Alpha reliability estimates and number of items by element (n = 765)

Element	Number of Items	Alpha
Accountability	7	0.77
Enrichment	6	0.78
Equity	4	0.71
Honor & Integrity	8	0.71
Altruism	3	0.73
Duty	6	0.71
Respect	2	0.51

Reliability

The level of acceptable reliability for affective measures depends in large part on the use for which the instrument is intended. Generally speaking, reliability estimates of 0.70 or better are acceptable for affective measures (Gable & Wolf, 1993). Table 2 contains the estimates of internal consistency reliability (Cronbach's alpha) for each of the derived factors (i.e. scales). As can be seen in the right-hand column, six of the seven scales have alpha reliability estimates ranging from 0.71 to 0.78. One scale, Respect, had a reliability estimate of 0.51. However, that scale contained only two items and had the weakest factor loadings of all factors. With so few items, it is not surprising that the alpha reliability estimate for the Respect factor is low. Adding items to this scale would assure that it more accurately samples from the intended domain (i.e. respectfulness). Using Nunnally's (Nunnally, 1978) formula to estimate the number of items necessary to reach a desired reliability, at least four additional items should be added, resulting in a total of six items for the Respect scale.

Discussion

This paper describes the development, reliability estimates and validity of a survey instrument to measure attitudes toward professionalism at one medical school. We believe that the development of the Penn State College of Medicine (PSCOM) Professionalism Questionnaire is important for several reasons. To the best of our knowledge, it is the first empirically valid and reliable professionalism questionnaire in US medical education. Our data suggest that the original elements of professionalism proposed by the ABIM may not adequately reflect respondents’ actual views of professionalism in medical education. While the elements in the PSCOM Professionalism Questionnaire maintain much of the ABIM's language, this instrument enhances the precision by which these elements are defined. The data presented here lend empirical support to seven elements of professionalism: accountability, altruism, duty, enrichment, equity, honor and integrity, and respect. The item statements that comprise the Questionnaire operationalize these elements, thereby deepening our understanding of respondents’ attitudes toward professionalism. Table 3 contains a sample item statement for each of the respective elements.

Table 3.  Sample item statements for each element

Element	Sample item statement
Accountability	Works collaboratively and respectfully within a team to the benefit of improved patient care or to the contribution of research
Enrichment	Shows a willingness to initiate and offer assistance toward a colleague's professional and personal development
Equity	Adopts uniform and equitable standards for patient care
Honor & Integrity	Upholds scientific standards and bases decisions on scientific evidence and experience
Altruism	Shows compassion
Duty	Participates in corrective action processes toward those who fail to meet professional standards of conduct
Respect	Avoids offensive speech that offers unkind comments and unfair criticisms to others

Another important aspect of the PSCOM Professionalism Questionnaire is that it provides an empirically sound tool to track changes in attitudes toward professionalism over time. This instrument can be used to assess attitude changes before and after educational interventions are introduced. Being able to assess attitudes toward professionalism across groups and over time is an important step toward meeting the ACGME core competence requirement in professionalism. The instrument can also assist academic health centers that are conducting needs assessments for curriculum development.

The instrument has implications for medical educators and researchers. Information can be obtained from the instrument to assess the curriculum and current practices providing information for both a whole group (programmatic assessment) and/or as a diagnostic tool for individual assessments and interventions, where appropriate. The questionnaire sub-scales could be used in experimental studies as outcome measures, blocking variables or covariates, and in descriptive research they could function as trait indicators or as predictors or criterion variables in regression analyses.

One limitation of this study is that the data were collected at one institution, thereby restricting the generalizability of the findings. An important next step would be to administer the instrument at multiple academic health centers in order to assess a broader perspective on professionalism. Additionally, this instrument is designed to measure attitudes, not behaviors. While there are links between attitudes and behaviors, caution must be exercised in that this instrument provides a snapshot of attitudes toward professionalism at one point in time. It does not identify whether the respondents actually behave in a manner consistent with the reported attitudes. Despite these limitations, the PSCOM Professionalism Questionnaire is one tool that can help medical education move forward in the efforts to enhance professionalism at all levels.

Acknowledgements

Special thanks are extended to Richard Simons, MD, for his leadership and support of this project. Financial support for this study was received from the Schreyer Institute of the Pennsylvania State University.

Notes

Additional information

Notes on contributors

George F. Blackall

GEORGE F. BLACKALL, PsyD is Associate Professor of Pediatrics and Humanities at Penn State University College of Medicine, Hershey, PA.

Steven A. Melnick

STEVEN A. MELNICK, PhD is Associate Professor of Education and Coordinator of the Graduate Program in Teaching and Curriculum at Penn State Capital College, Middletown, PA.

Glenda H. Shoop

GLENDA H. SHOOP, MEd, is Director of Curriculum Development and Evaluation at Penn State University College of Medicine, Hershey, PA.

John George

JOHN GEORGE, PhD, is Professor and Vice Chair for Educational Affairs in the Department of Family and Community Medicine at Penn State University College of Medicine, Hershey, PA.

Susan M. Lerner

SUSAN M. LERNER, MD is Assistant Professor of Surgery at Mt. Sinai School of Medicine, New York, NY.

Philip K. Wilson

PHILIP K. WILSON, PhD, is Associate Professor of Humanities and Medical Historian at Penn State University College of Medicine, Hershey, PA.

Richard C. Pees

RICHARD C. PEES, MD, is Associate Professor of Obstetrics and Gynecology at Penn State University College of Medicine, Hershey, PA.

Margaret Kreher

MARGARET KREHER, MD, is Associate Professor of Palliative Care and General Internal Medicine at Penn State University College of Medicine, Hershey, PA.

Notes

1.  The purpose of the Delphi technique is to elicit information and judgments from participants to facilitate problem-solving, planning and decision-making. It does so without physically assembling the contributors. Instead, information is exchanged via mail, fax or email. This technique is designed to take advantage of participants’ creativity as well as the facilitating effects of group involvement and interaction. It is structured to capitalize on the merits of group problem-solving and minimize the liabilities of group problem-solving. (Source: Dunham RB. 1996. The Delphi Technique. Available at: http://www.medsch.wisc.edu/adminmed/2001/orgbehav/delphi.pdf)