Development of ACLEEM questionnaire, an instrument measuring residents’ educational environment in postgraduate ambulatory setting

Abstract

Background: Students’ perceptions of their educational environment (EE) have been studied in undergraduate and postgraduate curricula. Postgraduate EE has been measured in hospital settings. However, there are no instruments available to measure the EE in postgraduate ambulatory settings.

Aim: The aim of this study was to develop the “Ambulatory Care Learning Education Environment Measure” (ACLEEM).

Methods: A mixed methodology was used including three stages: (1) Grounded theory (focus groups); (2) Delphi technique to identify consensus; and (3) Pilot study.

Results: Three quota samples of approximately 60 stakeholders were formed, one as Focus Groups and two as Delphi panels. Eight focus groups were carried out including 58 residents (Latin-American Spanish speakers). The results were analysed and 173 items were offered to a National Delphi panel (61 residents and teachers). They reduced in two rounds the number of important items to 54. The 54-item questionnaire was then piloted with 63 residents and refined to the final version of the ACLEEM with 50 items and three domains.

Conclusions: The 50-item inventory is a valid instrument to measure the EE in postgraduate ambulatory setting in Chile. Large-scale administration of the ACLEEM questionnaire to evaluate its construct validity and reliability are the next steps to test the psychometric properties of the instrument.

Introduction

The research related to educational environment (EE) started in the 1930s and was boosted by the work of Pace and Stern (1958). They studied aspects associated with the “atmosphere” in classrooms of primary, secondary schools and universities, utilizing qualitative research methods such as interviews or direct observation of lectures trying to capture the interactions between teachers and students. Gradually, research evolved to the use of quantitative methodologies and Hutchins (1961) created one of the first instruments developed specifically to evaluate the EE in medical education: the Medical School Environment Index (MSEI). This instrument allowed researchers to identify the US medical schools that were perceived as more aggressive and competitive from the students’ point of view.

In more recent years, several studies have been performed to develop and validate new instruments intended to evaluate the EE in different healthcare professions. Soemantri et al. (2010) conducted a systematic review that found 31 instruments published in the literature and established that the Dundee Ready Education Environment Measure (DREEM) (Roff et al. 1997), the Postgraduate Hospital Educational Environment Measure (PHEEM) (Roff et al. 2005), the Clinical Learning Environment (CLE) (Saarikosky & Leino-Kilpi 1999) and the Supervision and Dental Student Learning Environment Survey (DSLES) (Henzi et al. 2005) are likely to be the most suitable instruments for undergraduate medicine, postgraduate medicine, nursing and dental education, respectively. Specifically related with postgraduate medical education, other than the PHEEM, Soemantri et al. (2010) found eight instruments that have been used in this level: Veteran Affairs Learners’ Perceptions Survey (Keitz et al. 2003), learning environment assessment (Roth et al. 2006), questionnaire from Rotem, Godwin and Du (1995), operating room educational environment measure (Kanashiro et al. 2006), surgical theatre educational environment measure (Cassar 2004), anaesthetic theatre educational environment measure (Holt & Roff 2004), practice-based educational environment measure (Mulrooney 2005), and – despite the fact that it was designed for undergraduate medicine – the DREEM. Recently, a new instrument developed in the Netherlands, the Dutch Residency Educational Climate Test (D-RECT), emerged as a valid and reliable instrument measuring residents’ learning climate (Boor et al. 2011).

Having recognized this, and considering the wide spectrum of medical education in terms of settings where it is carried on, we noticed that all instruments, including the PHEEM, are questionnaires designed and intended to evaluate the EE in postgraduate hospital settings. Consequently, the ambulatory clinical setting of postgraduate medical education is not particularly addressed in any of the instruments mentioned before.

Ambulatory medical education is mostly conducted in primary care. Since the Alma-Ata Declaration, primary health care (PHC) has been increasingly recognized as the most important level in health systems, as we can see in documents such as the “World Health Report 2008” of the World Health Organization that was called “Primary Health Care: Now more than ever”; the “Bamako's call-to-action on research for health” of the Global Ministerial Forum on Research for Health 2008 that restated PHC as the central priority in health research (Editorial 2008a); and the series of articles in The Lancet, “Alma-Ata: Rebirth and Revision” (Editorial 2008b), and of the New England Journal of Medicine, “The Future of Primary Care” (Perspective 2008). Along with this, scientific evidence has demonstrated that health systems with stronger PHC has better health outcomes, reduced inequities and has less healthcare costs (Starfield 2002; Macinko 2003; Starfield 2005). Moreover, international agencies such as the PanAmerican Health Organization have taken similar steps recommending medical schools to orientate their curricula more strongly towards components of PHC (Borrell et al. 2008). In this scenario of relevance of PHC (therefore, of ambulatory care) and in the absence of a specific instrument to evaluate the EE in this setting, the aim of our study was to develop an inventory to measure the EE in postgraduate ambulatory medical education.

Table

Practice points

Residents’ perceptions of their educational environment (EE) have been widely recognized in postgraduate curricula. Primary Health Care has re-gained relevance and the EE of training programmes in ambulatory settings is increasingly important. ACLEEM questionnaire is a useful instrument to measure the EE in ambulatory settings in Spanish speaking countries and perhaps beyond after translation and field testing.

Methods

A mixed methodology was used including three stages: (1) Grounded theory; (2) Delphi technique to identify consensus; and (3) Pilot study.

In the first stage, the objective was to identify the aspects related with the EE in ambulatory postgraduate medical education. For this, we use a qualitative approach carrying out focus groups with residents of specialties with ambulatory activities. In each focus group, the positive and negative aspects of their ambulatory EE were explored in approximately 90 min and the conversations were recorded. The analyses of the interviews were made following the model of codification proposed by Grounded Theory (Strauss & Corbin 1998) by using the Atlas.ti® software. For the quality control of the analyses, data triangulation was performed. The results were revised considering methodological, ecological and explanatory validity.

In the second stage, we created statements addressing the most relevant aspects of the EE in postgraduate medical education in ambulatory setting that emerged from the results of the first stage. We performed a two-round Delphi technique (Hasson et al. 2000) with a panel of experts from different medical schools of Chile to prioritize the importance of each item (statement) expecting >11% of response rate in each round following a similar methodology used at this stage in the development of PHEEM (Roff et al. 2005). Respondents were asked to rate the importance of the items identified on the first stage with a Likert-type scale where 0 = without relevance, 1 = some relevance, 2 = indifferent, 3 = relevant and 4 = highly relevant. The survey was administered online. All the items with mean value ≥3 was considered relevant in both Delphi rounds and the items below this cut-off value were deleted (SPSS software).

The third stage (pilot study) was designed to test and refine the instrument created with the items selected by the Delphi panel. We piloted the questionnaire with a group of residents. They were asked to answer/report their EE perceptions for each item with a Likert-type scale: 0 = completely disagree, 1 = disagree, 2 = uncertain, 3 = agree and 4 = completely agree. We evaluated the quality of the items, identifying imprecise or ambiguous items within the questionnaire, considering the opinions of the residents.

The project was supported by the Postgraduate Director and it was approved by the Ethics Committee of the Pontificia Universidad Católica de Chile Medical School. The residents were randomly invited to participate in the focus groups and informed consent was required assuring confidentiality of their comments. Delphi panel opinions were confidential. Finally, a group of residents was randomly invited to participate in the pilot study and they anonymously answered the questionnaire.

Results

Stage 1: Grounded theory

In this first stage, 16 specialities with ambulatory setting activities were selected (family medicine, psychiatry, paediatrics, gynaecology & obstetrics, surgery, dermatology, otorhinolaryngology, ophthalmology, neurology and internal medicine with their sub-specialties) and residents were randomly invited to participate. Eight focus groups were conducted with a total of 58 participants (all from Latin American and Spanish speakers countries); 5 out of 58 (8.6%) were foreigners from Argentina, Colombia and Ecuador.

The information provided by the focus groups was open-coded generating 173 items (119 positive statements and 54 negative statements) grouped in three general domains related to relevant aspects of the EE in the ambulatory setting: Support, Clinical Teaching and Clinical Training. As a result, 173 items were identified regarding the aspects of the ambulatory EE.

Stage 2: Delphi technique

We performed a two-round Delphi process. The 173 items previously acknowledged were offered in the first round to a National Delphi panel drawn from nine Medical Schools of Chile. The survey was administered online and 61 out of 361 teachers and residents answered the survey (16.9% response rate). They considered 64 items as relevants. For the second round, 58 respondents in the Delphi panel (15.4 % response rate) reduced the number of items to 54.

The 54-item questionnaire was firstly translated from Spanish into English by a Chilean medical doctor proficient in English to be reviewed by two experts in EE (S.R. and A.R.) and then reverse translated into Spanish by a professional translator in order to ensure validity of content and meaning.

Stage 3: Pilot study

The 54-item questionnaire was piloted with a representative group 63 residents from seven specialties. Figure 1 summarizes the methodological process and findings of the three stages. As a result, four items were erased and seven were re-written because they had imprecise or ambiguous concepts. Fifty items were finally considered in the refined version of the instrument and it was translated into English to be reviewed by A.R. and S.R. following the same methodology described for the Stage 2 (see Table 1). As items 24 and 27 contained negative statements, we reverse coded the scores for these questions. The 50-item questionnaire was called the “Ambulatory Care Learning Education Environment Measure” (ACLEEM) and the items were allocated in three domains according to the content of the items: Clinical Teaching (items 1 to 16), Clinical Training (items 17 to 38) and Support (items 39 to 50).

Figure 1. Flow chart of the methodology process.

Table 1  Ambulatory Clinical Learning Educational Environment Measure (ACLEEM)

1. Working in the OPC enables me to develop my problem solving skills. (El trabajo en el consultorio/policlínico me ayuda a desarrollar mis destrezas para resolver problemas)

2. The teaching staff in the OPC have good clinical skills. (Los profesores clínicos de consultorios/policlínicos tienen buenas destrezas clínicas).

3. The teaching staff in the OPC are up to date in their knowledge and skills. (Los profesores clínicos del consultorio/policlínico están actualizados en conocimientos y destrezas clínicas)

4. My teachers in the OPC use teaching methods that are appropriate for each subject matter. (Mis profesores clínicos en el consultorio/policlínico utilizan metodologías de enseñanza adecuadas para cada uno de los contenidos impartidos)

5. I feel that my clinical teachers are appropriately qualified to carry out their teaching duties. (Siento que mis profesores clínicos están apropiadamente capacitados para realizar sus actividades docentes)

6. My clinical teaching staff is interested in improving the quality of the teaching activities in the OPC. (Mis profesores clínicos demuestran interés en mejorar la calidad de la docencia en el consultorio/policlínico)

7. I can develop my interpersonal skills in the OPC. (Puedo desarrollar mis habilidades interpersonales en el consultorio/policlínico)

8. I get my evaluations in a timely manner from the teachers in the OPC. (Conozco el resultado de mis evaluaciones de parte de los profesores del consultorio/policlínico de manera oportuna)

9. I feel that the assessment methods used in the OPC are compatible with the teaching methodology. (Siento que los métodos de evaluación utilizados en el consultorio/policlínico son compatibles con la metodología de enseñanza)

10. I have a clear idea about the objectives and learning outcomes of my educational activities in the OPC. (Tengo claros los objetivos/logros educacionales de aprendizaje de mis actividades docentes en el consultorio/policlínico)

11. I feel that the learning objectives and outcomes of the OPC are achieved appropriately. (Siento que los objetivos/logros educacionales de aprendizaje del programa de mis rotaciones ambulatorias se cumplen a cabalidad)

12. I am allowed to participate actively in external educational events and medical meetings. (En el consultorio/policlínico me dan las facilidades para participar en eventos educacionales y congresos)

13. My teachers in the OPC use teaching and learning activities effectively. (Mis profesores clínicos del consultorio/policlínico utilizan las oportunidades de enseñanza y aprendizaje en forma efectiva)

14. The allocated teaching time in the OPC is respected by the clinical teachers. (El horario de tiempo protegido para la docencia es respetado por mis profesores clínicos del consultorio/policlínico)

15. My clinical teachers provide me with feedback about my strengths and weaknesses. (Mis profesores clínicos me proveen de retroalimentación (feedback) respecto a mis fortalezas y debilidades)

16. My clinical teachers are enthusiastic about teaching. (Mis profesores clínicos son entusiastas al enseñarme)

17. Working in the OPCs gives me learning opportunities in a wide variety of diseases. (La rotación por distintos consultorios/policlínicos me permite aprender una gran variedad de enfermedades)

18. In the OPC I learn to treat patients with conditions that are specifically related to ambulatory care. (En el consultorio/policlínico aprendo a tratar enfermedades específicamente ambulatorias)

19. My clinical teachers in the OPC appropriately emphasise the doctor-patient relationship. (Mis profesores clínicos del consultorio/policlínico ponen el énfasis apropiado en la relación médico-paciente)

20. In the OPC I learn from the experience of my clinical teachers. (En el consultorio/policlínico aprendo de la experiencia de mis profesores clínicos)

21. My clinical teachers are good professional role models for me. (Mis profesores clínicos son buenos modelos profesionales para mí)

22. The clinical facilities in the OPC are suitable for working with patients in my specialty. (Los box de atención de los consultorios/policlínicos son adecuados para la atención de los pacientes de mi especialidad)

23. I have the opportunity to follow up my patients appropriately in the OPC. (Tengo la oportunidad de realizar seguimiento a mis pacientes del consultorio/policlínico)

24. I have insufficient time with each patient in the OPC. (El tiempo que tengo para la atención ambulatoria de cada paciente es insuficiente)

25. My activities in the OPC are clearly programmed. (Mis actividades en el consultorio/policlínico están claramente programadas)

26. I am able to refer my patients for evaluation by multidisciplinary teams. (En la atención ambulatoria puedo derivar a mis pacientes para evaluación por equipos multidisciplinarios)

27. There are insufficient clinical supervisors for the number of residents in the OPC. (Los supervisores clínicos son insuficientes para el número de residentes que trabajamos en el consultorio/policlínico)

28. I can obtain clinical supervision when I need it. (Puedo obtener supervisión clínica en el consultorio/policlínico cuando lo necesito).

29. I feel that I have the appropriate level of responsibility for my patients in the OPC. (Siento que tengo el apropiado nivel de responsabilidad con mis pacientes del consultorio/policlínico)

30. I feel that my clinical supervisors consider my opinions in clinical decision making about my patients. (Siento que mis supervisores clínicos consideran mis opiniones en la toma de decisiones respecto a mis pacientes)

31. I feel that I treat my patients in the OPC according to the treatment protocols for their conditions and illnesses. (Siento que trato a mis pacientes del consultorio/policlínico de acuerdo a los protocolos clínicos existentes para el manejo de sus enfermedades)

32. I am able to learn the required practical procedures in the OPC. (Tengo oportunidades para aprender los procedimientos prácticos requeridos en el consultorio/policlínico)

33. I feel that I am learning to become confident in my speciality in the OPC. (Siento que la enseñanza que recibo en el consultorio/policlínico me ayuda a desarrollar las competencias relacionadas con mi especialidad)

34. I feel that the clinical rotations in the OPC are preparing me properly for my professional future. (Siento que me preparan adecuadamente en las rotaciones ambulatorias para mi futuro ejercicio como profesional)

35. In the OPC I manage clinical problems taking into account the social and emotional aspects of my patients. (En el consultorio/policlínico manejo los problemas clínicos considerando los aspectos sociales y emocionales de mis pacientes)

36. I am able to learn to adjust my work to the resources available in the OPC. (Aprendo a trabajar adaptándome a los recursos disponibles en el consultorio/policlínico)

37. I am able to carry out health education activities in the OPC. (Tengo la posibilidad de realizar actividades de educación en salud en el consultorio/policlínico)

38. I feel that my time in the OPC is preparing me to address the health needs of the country. (Siento que en el consultorio/policlínico me preparan para atender las necesidades de salud del país)

39. The teachers in the OPC respond to my personal concerns appropriately. (Cuento con el apoyo del profesor encargado de la rotación ambulatoria para resolver mis problemas personales)

40. The workload allows me to balance the clinical care of my patients with my educational activities. (La carga asistencial me permite compatibilizar la atención de mis pacientes con las actividades docentes)

41. I can keep my work and personal life in balance when I am working in the OPC. (Puedo compatibilizar el trabajo con mi vida personal cuando trabajo en el consultorio/policlínico)

42. My working hours in the OPC permit adequate rest and eating times. (Mi jornada laboral ambulatoria considera tiempos de descanso y alimentación adecuados)

43. I feel part of the team in the OPC. (En el consultorio/policlínico me siento parte del equipo de trabajo)

44. I receive support from other OPC residents when I need it. (En el consultorio/policlínico recibo apoyo de otros residentes cuando lo necesito)

45. I feel that other members of the healthcare team are willing to help me when I need it. (Siento que otros miembros del personal de salud tienen buena disposición a ayudar cuando los necesito)

46. I have adequate access to computers and Internet in the OPC. (En el consultorio/policlínico cuento con acceso adecuado a computadores con conexión a internet cuando lo requiero)

47. The OPC provides lockers to keep my personal belongings safe. (Los consultorios/policlínicos disponen de un lugar seguro para guardar mis pertenencias)

48. There are adequate bathroom facilities in the OPC. (En los consultorios/policlínicos cuento con un baño adecuado para el uso de los residentes)

49. The OPCs have adequate supplies and instruments to render quality professional care. (En el consultorio/policlínico cuento con los insumos e implementos necesarios para prestar una atención de calidad)

50. The clinical files and/or information systems of the OPC give me adequate access to patient information. (Las fichas clínicas y/o sistema informático del consultorio/policlínico me permiten un acceso adecuado a la información del paciente)

The ACLEEM consists of 50 items with each item scored on a five-point Likert scale with 4 = Strongly agree, 3 = Agree, 2 = Unsure, 1 = Disagree and 0 = Strongly disagree. As items 24 and 27 contained negative statements, we reverse coded the scores for these questions.

The 50 items were allocated in three domains as follows: Clinical Teaching (items 1 to 16), Clinical Training (items 17 to 38) and Support (items 39 to 50).

Outpatient clinic (OPC).

Discussion

The quality assurance process of postgraduate educational programmes and residency training is increasingly important (Afrin et al. 2006). EE is one of the aspects to evaluate the quality of training programmes providing information about several domains like atmosphere, feedback and supervision in hospital and ambulatory settings (Boor et al. 2011). Several questionnaires have been developed to measure the EE and ACLEEM is the first one, particularly developed to measure aspects related to the EE in ambulatory setting. Development and validation of the 50-item inventory was based on grounded theory and a modified Delphi procedure. It is a strength of this study that Focus Groups included residents from several Latin-American countries and 16 residency programmes. Even when residents from Argentina, Colombia and Ecuador participated in the whole process, providing feedback about the content and meaning of the statements, some words in Spanish could be interpreted in a different way in other Spanish-speaking countries. e.g. outpatient clinic (OPC) in Chile could be named as consultorio or policlínico, however, in other Latin-American countries clínica externa or dispensario are more commonly used. In the future, it is important to address this issue, revising the meaning of each statement with residents of the programme, before the administration of the ACLEEM.

We need to take into account the fact that the Delphi panel included over 300 residents and teachers from several universities with 16.9% and 15.4% response rates, in the first and second rounds, respectively. This could be a source of potential bias. However, we expected >11% of response rate in each round according to a similar methodology used at this stage in the development of PHEEM (Roff et al. 2005). The 50 items were allocated in three domains according to the qualitative analysis of the data: Clinical Teaching, Clinical Training and Support. These domains must be tested by using an exploratory factor analysis. However, for a sound factor analysis a number of five subjects per item (250 residents) must be included (Streiner 1994) and the pilot study (63 residents) was not enough to carry it out. Finally, ACLEEM is a valid instrument to measure the EE in postgraduate ambulatory settings in Chile and it can be administered in Spanish speaking countries. Large-scale administration of the ACLEEM questionnaire to evaluate its construct validity (Field 2005), internal consistency and reliability including Generalisability theory (Crossley et al. 2002) are the next steps to evaluate the psychometric properties of the instrument.

Acknowledgements

This work is partially supported by grants of the National Commission for Scientific and Technological Research (CONICYT), FONDECYT no. 1 100 436 to A.R.

Declaration of interest: The authors report no conflicts of interest. The authors are responsible for the content and writing of the article.