Psychometric analyses and internal consistency of the PHEEM questionnaire to measure the clinical learning environment in the clerkship of a Medical School in Chile

Abstract

Background and aims: The Spanish version of the Postgraduate Hospital Educational Environment Measure (PHEEM) was evaluated in this study to determine its psychometric properties, validity and internal consistency to measure the clinical learning environment in the hospital setting of Pontificia Universidad Católica de Chile Medical School's Internship.

Methods: The 40-item PHEEM questionnaire was translated from English to Spanish and retranslated to English. Content validity was tested by a focus group and minor differences in meaning were adjusted. The PHEEM was administered to clerks in years 6 and 7. Construct validity was carried out using exploratory factor analysis followed by a Varimax rotation. Internal consistency was measured using Cronbach's α.

Results: A total of 125 out of 220 students responded to the PHEEM. The overall response rate was 56.8% and compliances with each item ranged from 99.2% to 100%. Analyses indicate that five factors instrument accounting for 58% of the variance and internal consistency of the 40-item questionnaire is 0.955 (Cronbach's α). The 40-item questionnaire had a mean score of 98.21 ± 21.2 (maximum score of 160).

Conclusions: The Spanish version of PHEEM is a multidimensional, valid and highly reliable instrument measuring the educational environment among undergraduate medical students working in hospital-based clerkships.

Introduction

The learning environment is an important aspect of the curriculum that must be considered in both undergraduate and postgraduate medical education. Its impact is something well recognized and accepted because of its real influence over students’ achievement, satisfaction and success (Cavanaugh & Simmons 1997; Pimparyon et al. 2000; Genn 2001).

The Pontificia Universidad Católica de Chile Medical School (PUCMS) started an undergraduate curriculum reform in 1993 (Moreno & Velasco 1994; Rosso et al. 1997). The clerkship of the PUCMS takes place in the last 2 years of a seven-year course. It can be compared with the Foundation Programme of the UK, which is a 2-year generic training programme which forms the bridge between medical school and specialist/general practice training (Foundation Programme Website 2008). After almost 15 years from the beginning and consolidation of this curricular reform, the learning environment has been successfully evaluated among medical students in years 1–5 of the curriculum with the DREEM questionnaire (Roff et al. 2005; Riquelme et al. 2008). We considered the Postgraduate Hospital Educational Environment Measure (PHEEM) a more suitable instrument than DREEM for measuring the clinical learning environment in the PUCMS clerkship because of its content validity, high reliability and also its ability to be used in different postgraduate hospital settings (Roff et al. 2005; Aspegren et al. 2007; Boor et al. 2007). PHEEM has been used to evaluate the learning environment in Intensive Care Units of the UK (Clapham et al. 2007), validated in hospital settings in Denmark (Aspegren et al. 2007), and its psychometric properties were determined in the clerks and registrars from different hospitals and specialties in the Netherlands (Boor et al. 2007). We wished to use PHEEM in our setting, but because of possible differences between environments of the UK and Chile we decided to study the psychometric characteristics and internal consistency of the Spanish version of PHEEM before its wider use. Our research questions were as follows: Is the Spanish version of PHEEM a valid and reliable instrument measuring the educational environment of Spanish-speaker medical students? Is PHEEM a uni-dimensional or multi-dimensional instrument when looked at using statistical methods based on three different criteria?

Table

Practice points

Spanish version of the PHEEM questionnaire is a valid and reliable instrument measuring educational environment among undergraduate medical students. PHEEM is a multidimensional instrument and five factors were identified in the factor analysis. Internal consistency of the 40-item PHEEM questionnaire was high (Cronbach's α 0.955). We recommend PHEEM to measure educational environment in hospital-based clerkships of Medical Schools with Spanish-speaking students.

Methods

The 40-item PHEEM questionnaire was first translated from English into Spanish by a Chilean doctor proficient in English and then retranslated into English by a professional translator in order to ensure content and meaning. An English doctor, working in Chile, proficient in the English language, compared the two versions. This version was reviewed for content validity by a focus group of teachers involved in the clerkships and a selected group of 15 students in the context of a Quality Assurance and Accreditation process of the PUCMS in January 2007. The focus group considered the 40-item questionnaire a valid instrument measuring aspects related with educational environment and some minor differences in meaning between the two versions were adjusted. The adjusted Spanish version of PHEEM was administered to clerks in year 6 and 7 from eight different specialties (internal medicine, surgery, obstetrics and gynaecology, paediatrics, family medicine, radiology, psychiatry and emergency medicine) in four different hospitals, between January and March 2007. We evaluated their perception of this environment at the end of each clinical rotation and each student only completed the questionnaire once. They were asked to rate each item on a five-grade Likert-type scale, from 0 = totally disagree to 4 = totally agree. Because items 7, 8, 11 and 13 contained negative statements, we reverse coded the scores for these questions. To evaluate the construct validity of the three subscales of the PHEEM, we used an exploratory factor analysis followed by a Varimax rotation (Boor et al. 2007). Factors were chosen using a combination of the following three criteria: (1) Kaiser–Guttman criterion, in which all factors with an eigenvalue >1.00 were included (Field 2000, 2005); (2) Cattell criteria suggested that the inflexion point of the scree plot curve is the cut off, and all factors below it should be accepted (Cattell 1966) and (3) the factors identified in the exploratory factor analysis should capture the majority of the variance rather than less than 50% of the total. In the following sections, we used the numbering of items and subscales from the original publication of PHEEM (Roff et al. 2005). Data were analysed by the SPSS statistical programme for the exploratory analysis and internal consistency using Cronbach's α (Cronbach 1951).

Results

A total of 125 out of 220 students completed the PHEEM. The overall response rate was 56.8% and compliances with each item ranged from 99.2% to 100%. The descriptive statistics and mean item scores of each item are shown in Table 1.

Table 1.  Items subscales and descriptive statistics of the PHEEM for clerks

	Subscales^a	Compliance (%)	Mean	SD
1. I have a contract of employment that provides information about hours of work. (Tengo un programa que provee información acerca de las horas de actividad clínica)	Aut.	100	2.43	1.27
2. My clinical teachers set clear expectations. (Mis profesores clínicos establecen expectativas claras)	Teach.	100	2.42	1.13
3. I have protected time at this post. (Tengo tiempo educacional protegido en esta rotación clínica)	Teach.	100	2.08	1.27
4. I had an informative introduction programme (Tuve un programa de inducción informativo)	Aut.	99.2	2.10	1.17
5. I have the appropriate level of responsibility in this post (Tengo el nivel apropiado de responsabilidad en esta rotación clínica)	Aut.	100	2.90	0.99
6. I have good clinical supervision at all times (Tengo buena supervisión clínica en todo momento)	Teach.	100	2.48	1.31
7. There is racism in this post. (Hay racismo en esta rotación clínica)	SocS.	100	3.30	0.95
8. I have to perform inappropriate tasks. (Tengo que realizar tareas inapropiadas a mi etapa de Formación)	Aut.	100	2.61	1.17
9. There is an informative junior doctors handbook (Hay un manual informativo de la rotación clínica para los estudiantes)	Aut.	100	2.25	1.16
10. My clinical teachers have good communication skills (Mis profesores clínicos tienen buenas destrezas comunicacionales)	Teach.	100	2.60	1.09
11. I am beeped inappropriately. (Soy (“beepeado” o ubicado a mi teléfono celular) localizado de forma inapropiada)	Aut.	100	2.45	1.39
12. I am able to participate actively in educational events. (Existen las facilidades para participar en otras actividades docentes sin interferir con clases o pruebas de evaluación de otros cursos)	Teach.	100	2.03	1.28
13. There is sex discrimination in this post. (Hay discriminación por sexo en esta rotación clínica)	SocS.	100	2.88	1.19
14. There are clear clinical protocols in this post. (Tengo guías claras acerca de mis actividades clínicas en esta rotación clínica)	Aut.	100	2.27	1.12
15. My clinical teachers are enthusiastic. (Mis profesores clínicos son entusiastas)	Teach.	100	2.74	1.02
16. I have good collaboration with other doctors in my grade. (Tengo buena colaboración con otros estudiantes de mi nivel)	SocS.	100	3.25	0.67
17. My hours conform to the New Deal. (Mi horario de actividades clínicas es adecuado)	Aut.	100	2.08	1.43
18. I have the opportunity to provide continuity of care (Tengo la oportunidad de ofrecer continuidad en el cuidado de los pacientes)	Aut.	100	2.26	1.21
19. I have suitability access to careers advice. (Tengo acceso adecuado a consejería de carreras)	SocS.	100	1.23	0.96
20. This hospital has good quality accommodation for junior doctors, especially when on call. (Esta rotación clínica (hospital/consultorio) tiene una buena calidad de espacios físicos para desarrollar actividades docentes)	SocS.	99.2	2.19	1.19
21. There is access to an educational programme relevant to my needs. (Hay acceso a un programa educacional relevante con objetivos explícitos para mis necesidades)	Teach.	100	2.28	1.04
22. I get regular feedback from my seniors. (Recibo retroalimentación (feedback) regularmente de parte de los docentes)	Teach.	100	1.89	1.18
23. My clinical teachers are well organized. (Mis profesores clínicos son bien organizados)	Teach.	100	2.50	1.12
24. I feel physically safe within the hospital environment. (Yo me siento físicamente seguro en el ambiente clínico (hospitalario/ambulatorio))	SocS.	100	3.07	0.80
25. There is a no-blame culture in this post. (Hay una cultura de no-culpar en esta rotación clínica)	SocS.	100	2.27	1.12
26. There are adequate catering facilities when I am on call. (Hay facilidades adecuadas para obtener comida y bebidas (casinos-cafetería))	SocS.	100	2.54	1.07
27. I have enough clinical learning opportunities for my needs. (Tengo suficientes oportunidades de aprendizaje clínico para mis necesidades)	Teach.	100	2.65	0.97
28. My clinical teachers have good teaching skills. (Mis profesores clínicos tienen buenas destrezas clínicas)	Teach.	100	3.09	0.72
29. I feel part of a team working here. (Aquí me siento parte de un equipo de trabajo)	Aut.	100	2.59	1.23
30. I have opportunities to acquire the appropriate practical procedures for my grade. (Tengo oportunidades de adquirir los procedimientos prácticos apropiados para mi nivel)	Aut.	100	2.61	1.11
31. My clinical teachers are accessible. (Mis profesores clínicos son asequibles)	Teach.	100	2.78	0.95
32. My workload in this job is fine. (Mi carga de trabajo en esta rotación clínica es adecuada)	Aut.	100	2.30	1.33
33. Senior staff utilise learning opportunities effectively. (Los docentes utilizan las oportunidades de aprendizaje en forma efectiva)	Teach.	100	2.54	1.11
34. The training in this post makes me feel ready to be an SpR/consultant. (El entrenamiento en esta rotación clínica me hace sentir preparado para ser doctor)	Aut.	100	2.59	1.17
35. My clinical teachers have good mentoring skills. (Mis profesores clínicos tienen buenas destrezas como mentores)	SocS.	100	2.55	1.17
36. I get a lot of enjoyment out of my present job. (Tengo mucha entretención fuera de las actividades de esta rotación clínica)	SocS.	100	2.00	1.38
37. My clinical teachers encourage me to be an independent learner. (Mis profesores clínicos me fomentan el aprendizaje independiente)	Teach.	100	2.83	0.94
38. There are good counselling opportunities for junior doctors who fail to complete their training satisfactorily. (Existen buenas oportunidades de consejería para alumnos que fallan en esta rotación clínica)	SocS.	100	1.68	1.14
39. The clinical teachers provide me with good feedback on my strengths and weaknesses. (Los profesores clínicos me proveen buena retroalimentación (feedback) respecto a mis fortalezas y debilidades)	Teach.	100	1.90	1.17
40. My clinical teachers promote an atmosphere of mutual respect. (Mis tutores clínicos promueven una atmósfera de mutuo respeto)	Aut.	100	2.84	1.04

Note: Mean = mean item score; SD = standard deviation.

^aThree subscales: perceptions of autonomy (Aut), perceptions of social support (SocS) and perceptions of teaching (Teach).

Content validity of the Spanish version of PHEEM was examined by a focus group of teachers and students previously described in the methodology. Minor differences in meaning were adjusted. The item number 7 (there is racism in this post) generated some controversy because the students believed that racism was not an important issue in our school. We preferred to maintain the item, and the results supported the opinion of the students with a mean value for the item of 3.3 ± 0.95 for this item. Construct validity of the 3 subscales was examined using the exploratory factor analyses followed by varimax rotation of the data, and resulted in eight factors with an eigenvalue >1. The first factor had an eigenvalue of 15.64 (accounting for 39.36% of the variance) and the next seven factors had eigenvalues <2.26. The eight factors together explained 66% of the variance. The inflexion point of the curve was observed between factors 5 and 6 in the full scree plot (Figure 1). Therefore, we forced a factor analyses with five factors. Distribution of five factors was illustrated in the scree plot (Figure 1) accounting for 58% of the variance, evidencing the following allocation of items: Factor 1: items 5, 6, 10, 15, 16, 18, 21, 22, 23, 24, 27, 28, 29, 30, 31, 33, 34, 35, 37, 39 and 40; Factor 2: items 3, 12, 17, 19, 36 and 38; Factor 3: items 7, 8, 11, 13, 25 and 32; Factor 4: items 1, 2, 4, 9 and 14; and Factor 5: items 20 and 26. Factor 1 included the majority of the items originally allocated to the perceptions of teaching domain (12 out of 21 items). The rest of the items came from the other two domains and they were related with the perception of learning opportunities and teamwork. The items allocated to factor 2 belonged to the perceptions of teaching (2 out of 6) and social support (3 out of 6). However, most of them were related with students’ perceptions of learning and counselling opportunities. Factor 3 included items from the perceptions of autonomy (3 out of 6) and social support (3 out of 6) and those items addressed aspects related with the perceptions of fairness and culture or racial discrimination. Factor 4 was mainly composed by items originally allocated by the authors to the perceptions of autonomy (4 out of 5) and factor 5 included items from the perceptions of social support (2 out of 2) which were related with accommodation and catering facilities in the hospital.

Figure 1. Full scree plot of the factors identified in the exploratory factor analysis of the PHEEM. The arrow indicates the inflexion point of the curve between factors 5 and 6 in the scree plot.

The five factors showed a Cronbach's α of 0.952, 0.84, 0.793, 0.781, and 0.426, respectively. The 40-item questionnaire had a mean score of 98.21 ± 21.2 (maximum score of 160) corresponding to a ‘more positive than negative environment but room for improvement’ according to the criteria proposed by the PHEEM authors (Roff et al. 2005). Internal consistency of the 40-item PHEEM questionnaire was high (Cronbach's α 0.955).

The results of these analyses showed a multidimensional instrument rather than uni-dimensional instrument proposed by Boor et al. (2007). On the other hand, our findings are not consistent with a questionnaire measuring three distinct domains, as it was conceived in its original design (Roff et al. 2005). Cronbach's α for the original domains role autonomy, the perceptions of social supports and perceptions of teaching was 0.88, 0.748 and 0.935, respectively.

Discussion

The Spanish version of the PHEEM was evaluated in this study to determine its psychometric properties and internal consistency to measure the clinical learning environment in the hospital setting of PUCMS's Internship. In the first part, concerning the analyses of the psychometric properties of the instrument, five factors were identified considering three criteria (Kaiser–Guttman criterion, Cattell criterion and variance percentage). If we considered only the Kaiser-Guttman criterion, three additional factors must be included offering a mild increment of the variance (8%) comparing with five factors. The scree plot obtained by Boor and co-workers is similar to our scree plot. However, we disagree with their interpretation because they suggested that the PHEEM questionnaire was a uni-dimensional instrument based on the predominance of one factor. However, their main factor explained less than 50% of the variance (Boor et al. 2007). The current results support the thesis that content analyses of the PHEEM as performed by the original authors with three subscales cannot be replicated empirically in our study. Our results suggest that Factor 1 is the most important domain of the questionnaire including items related with the perceptions of teaching and learning. Factor 2 included items from the perceptions of teaching and social support focused on learning and counselling opportunities. The items allocated to factor 3 were related with perceptions of fairness of the programme, inappropriate tasks (item 8) and workload (item 32). Additionally, other items of the factor 3 were related with sex (item 13), culture (item 25) or racial (item 7) discrimination. Factor 4 was mainly dominated by items from the perceptions of autonomy domain and the content of the items were related with the information and organization of the programme. Only one item came from other domain (item 2 from perceptions of teaching) and it is about setting clear expectations of the clinical rotation by the clinical teachers. Factor 5 included two items and all of them came from the perceptions of social support related with accommodation and catering facilities.

Second, we confirmed the characteristic high reliability of the PHEEM questionnaire. Furthermore, for the five factors identified in the factor analyses, internal consistency was excellent for factors 1 and 2; good for 3 and 4 and poor for factor 5. For the original domains, internal consistency is considered good for the perceptions of social support domain, and excellent for the other two.

Finally, we conclude that the Spanish version of PHEEM is a multidimensional, valid and highly reliable instrument and we recommend PHEEM to measure educational environment in hospital-based clerkships of Medical Schools with Spanish-speaking students in Latin American and Spanish Medical Schools.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.

Additional information

Notes on contributors

Arnoldo Riquelme

CRISTIAN HERRERA is member of the Evidence-Based Medicine Unit and member of the Health Policy and Systems Research Unit from the School of Medicine of the Pontificia Universidad Católica de Chile.

Cristian Herrera

ARNOLDO RIQUELME, MD, MMedEd, is undergraduate and postgraduate clinical tutor in Internal Medicine and consultant in the Department of Gastroenterology at the Pontificia Universidad Catolica de Chile School of Medicine, Chile.

Carolina Aranis

OSLANDO PADILLA is statistician in the Department of Public Health particularly interested in psychometric analyses of questionnaires administered in the School of Medicine or developed in the Centre for Medical Education at the Pontificia Universidad Catolica de Chile.

Jorge Oporto

JORGE OPORTO, MD, Ophthalmology resident at Fundación Oftalmológica Los Andes, Chile.

Oslando Padilla

CAROLINA ARANÍS, MD, Otorhinolaringology resident at the Pontificia Universidad Catolica de Chile School of Medicine, Chile.