Educational environment in intensive care medicine—use of Postgraduate Hospital Educational Environment Measure (PHEEM)

Abstract

Background: The educational climate is an important measure within medical education. This is because there are still accounts of poor teaching, humiliation, bullying and harassment of doctors in training. Deaneries and schools must be able to demonstrate to the Postgraduate Medical Education and Training Board that trainees are working and learning in a good environment.

Methods: This study used the Postgraduate Hospital Educational Environment Measure (PHEEM) to measure the educational climate in nine intensive care training schemes within hospitals in England and Scotland. 134 trainees replied out of 190 (71% response). Neither the identities of the nine units nor the trainees were known to the researchers.

Results: The results showed that there was a good overall educational climate in the intensive care units studied, with no racism or sexism, and trainees were happy with their teaching, their support and the work they did. The junior trainees (house officers and senior house officers) perceived a significantly better climate than did the senior trainees (specialist registrars). There were also significant differences in scores for the nine different intensive care units.

Conclusions: PHEEM has proved to be a reliable and consistent tool to assess educational climate with an overall Cronbach's alpha of 0.921.

Introduction

Within the United Kingdom, the Postgraduate Medical Education and Training Board (PMETB) is now established. Its role is to oversee the whole of postgraduate medical education within both primary and secondary care within the United Kingdom. There is now a need for a valid and reliable assessment of the environment in postgraduate medical education, used at a UK wide level. At the moment, postgraduate Deaneries and Royal Colleges carry out quality assurance visits, assessing the opinions of trainers and trainees of the educational environment. None of these use valid and reliable assessments.

There is a need to develop the system of using a practical, valid and reliable way of assessing educational environment, which is reproducible and transferable, and could be used both to measure the educational environment in a unit, and to compare results of educational environment measurement across different grades of junior doctors, across different Deaneries and different specialities within the United Kingdom.

What is the educational environment? It is variously referred to as climate, atmosphere or tone, is a set of factors that describe what it is like to be a learner within that organization. Chambers and Wall (2000), consider the educational climate in three parts. These are the physical environment (safety, food, shelter, comfort and other facilities), the emotional climate (security, constructive feedback, being supported, and absence of bullying and harassment) and the intellectual climate (learning with patients, relevance to practice, evidence-based, active participation by learners, motivating, and planned education).

Table

Practice points
•The educational climate or environment is an important educational measure in the quality of the doctors’ postgraduate training.
•PHEEM is simple to use.
•PHEEM is a reliable and useful tool with which to measure educational climate.
•Intensive care medicine might be perceived by some as a stressful, intensive and threatening environment within which junior doctors work.
•The educational climate in intensive care medicine is in fact good, is supportive to junior doctors and there is no racism or sexism experienced within these posts.

The good clinical teaching environment (Spencer2003) ensures the teaching and learning is relevant to patients, has active participation by learners, and shows professional thinking and behaviours. There should be good preparation and planning, both of the structure and content, reflection on learning, and evaluation of what has happened in the teaching and learning. Spencer also goes on to describe some of the common problems with teaching and learning in the clinical environment. These include a lack of clear objectives, a focus on knowledge rather than problem-solving skills, teaching at the wrong level, passive observation, little time for reflection and discussion, and teaching by humiliation.

Teaching by humiliation, bullying and harassment is a big problem, both in the United Kingdom and other countries. Much of this relates to teachers' lack of awareness of educational skills and knowledge (Lowry1992, 1993) and inability to promote a good supportive educational climate for trainees in which to learn (SCOPME1992; SCOPME1994). Lowry (1992) described disenchantment with medicine in the words of a young doctor as ‘It could have been such a wonderful thing to be a doctor – but it's not. It's just a disaster.’

There are, sadly, many examples in the literature of bullying and harassment of junior doctors and medical students. Such studies show the practice is widespread, and on the individual level, illustrate how destructive to confidence and well being bullying and harassment can really be. Wolf et al. (1991) carried out a questionnaire study of medical students in the Louisiana State University School of Medicine. Of these, 98.9% reported mistreatment, with shouting and humiliation being most frequent. Over half the sample reported sexual harassment—reported mainly by women students. There was a high level of remarks degrading doctors and medicine as a profession. Increased mistreatment was associated positively with a perceived increase in cynicism.

More recently an editorial in Medical Education (Spencer and Lennard2005) discussed teaching by humiliation. They called for an end to a culture of bullying, setting in place a self perpetuating culture of abuse, in which the victims become the perpetrators in the next round. They cited the teaching the teachers movement, to encourage teaching based on sound educational principles, monitoring of examiners’ performances, and assessment by the whole team, not just the consultants, who appear to be the main perpetrators of abuse.

What about in intensive care medicine? This would appear to be a challenging environment in which to place trainees. It can appear overwhelming, with complicated, ill patients, and complex machinery, all unfamiliar to the trainee. There are many staff, most of whom are experienced and familiar with the workings of the intensive care unit. Life or death situations occur frequently. Patients’ conditions change rapidly, and there may be little time to ponder the correct clinical decision to take. Despite this the Foundation curriculum (Foundation Programme Committee2005) includes a need to learn about the care of acutely ill patients and an increasing number of Foundation years 1 and 2 trainees are attached to ICUs.

Evaluations from trainees at senior house officer level within the West Midlands report good to excellent scores for induction, supervision and clinical experience using the post evaluation tool, developed and used every six months on all senior house officer posts since 1997 (Wall et al.2000, 2001). Also, evidence from our Deanery Quality Assurance visits to intensive care units has repeatedly shown that senior house officers value their time in intensive care medicine, feel well supported, and rate the jobs highly.

Can the educational environment be measured, using a practical, valid and reliable tool? The Dundee Ready Education Environment Measure (DREEM) was developed in Dundee by Roff et al. (1997). It is a valid and reliable measure of the perceived educational environment. It has been widely used in many countries throughout the world, including the Gulf States, Nepal, Nigeria, the West Indies, Thailand, China, and Canada and the United Kingdom. It has been used for medical students, nurses, dentists, chiropractors and other professions allied to medicine. A more detailed bibliography has been published by Roff (2005).

From this original environmental measure, further postgraduate measurement scales have been developed. The Anaesthetic Theatre Education Environment Measure (ATEEM) (Holt & Roff2004) showed that a better educational climate for first year senior house officers than for specialist registrars, and these results were statistically significant. The Surgical Theatre Environmental Education Measure (STEEM) (Cassar2004) has been developed and validated for training surgeons. The Dundee group has also developed a 40 item Inventory for the various aspects of junior doctor training in the UK and Ireland, the Postgraduate Hospital Educational Environment Measure (PHEEM) (Roff et al.2005). However their description of the tool does not contain data to show its reliability or practicality.

This study looked at the practicality of using the Postgraduate Hospital Education Environment Measure (PHEEM) to look at the education environment within Intensive Care Medicine.

Our research questions were as follows:

1. Is PHEEM a practical way to evaluate the educational climate within Intensive Care Units?

2. Is there a good educational climate in intensive care units, as the post evaluation results seem to suggest?

3. Does the educational climate vary with the different grades of junior doctor?

4. Does the educational climate vary between intensive care units?

Methods

We set out to assess the reliability and practicality of a new tool to measure the quality of the educational environment in nine intensive care training programmes in England and Scotland. We used the PHEEM (Roff et al.2005) questionnaire as a self-administered tool (Appendix 1). This has 40 statements with which the respondents are asked to indicate their agreement using a 5 point Likert scale. These range from strongly agree (4), agree (3), unsure (2), disagree (1) to strongly disagree (0). Agreement with the items indicates a ‘good’ environment giving high scores. The four negative statements (questions 7, 8, 11, and 13) were scored in reverse so that the higher the score the more positive the environment. Information on gender and seniority in terms of the grade of post were also requested as part of the questionnaire.

The intensive care programmes surveyed varied from individual hospitals to amalgamated units according to local delivery of education. These were selected on a voluntary basis by members of the Intensive Care Society Education Committee. Participation by individuals and units was entirely voluntary. Neither the individuals nor the units were identified to MC who conducted the analysis.

The data were analysed using SPSS 12.0. The reliability of the questionnaire was assessed using Cronbach's alpha both as a whole and for each item using the ‘alpha if item deleted’ to identify questions whose exclusion would improve the reliability (Turoff1970; Bowling1997; Field2000). Descriptive statistics were reported as median, mean and standard deviation. The comparative statistics used the non-parametric methods of Mann–Whitney U for two independent samples and Kruskal-Wallis for multiple independent samples (Field2000; Jamieson2004). Global mean scores, for individual respondents, were calculated with missing values scored as 2 (the midpoint on this 0–4 scale). For factor analysis, the method used was a varimax rotation, setting to accept Eigen values above 1.0 and accepting correlations above 0.5.

Results

Demographics

134 trainees, out of 190 (71%) completed the questionnaire. These were drawn from nine intensive care training programmes within England and Scotland and the numbers in each programme varied from 4–41 trainees. There were 50 female trainees (37.3%) and 80 male trainees (59.7%), and 4 missing values (3.0%). There were 3 pre-registration house officers (PRHOs), 60 senior house officers (SHOs), 68 specialist registrars (SpRs) and 3 missing values. The number of years spent at these grades ranged from 0 to 5 years.

Practicality

The questionnaires took less than 5 minutes to complete. Furthermore out of a possible 5360 responses to the 40 questions there were only 23 missing values suggesting that the questionnaire was easy to understand. Coding the questionnaires and calculating the raw scores for each individual was also quick and easy. The scores suggested by Roff et al. (2005) can be calculated by hand in less than five minutes or input into a spreadsheet in the same time. Designing the spreadsheet and calculations are more time consuming and dependent on the operator's skills.

Reliability

Cronbach's alpha scored at 0.921 for the 40 statements. When this was analysed to exclude each question in turn, using the ‘alpha if item deleted’ there was no significant improvement in the score thus confirming all questions were relevant and should be included.

Questionnaire responses

Table 1 summarizes the responses to each question. We have reported the mean and standard deviation of the results because this gives a better overall view of the results than the median values alone. Only one of the questions answered met the criteria for a normal (parametric) distribution so all comparative statistics used are non parametric in type.

Table 1.  Summary results of PHEEM questionnaire (## – questions with reverse scoring)

	Mean	Median	Std. deviation
q1 – I have a contract of employment that provides information about hours of work	2.26	3	1.24
q2 – My clinical teachers set clear expectations	2.26	3	1.05
q3 – I have protected educational time in this post	2.34	3	1.25
q4 – I had an informative induction programme	2.24	3	1.25
q5 – I have the appropriate level of responsibility in this post	3.09	3	0.79
q6 – I have good clinical supervision at all time	3.20	3	0.88
q7 – There is racism in this post ##	3.19	3	0.98
q8 – I have to perform inappropriate tasks ##	2.62	3	1.12
q9 – There is an informative Junior Doctors handbook	1.84	2	1.01
q10 – My clinical teachers have good communication skills	2.96	3	0.93
q11 – I am bleeped inappropriately ##	2.55	3	1.01
q12 – I am able to participate actively in educational events	2.68	3	0.88
q13 – There is sexism in this post ##	3.15	3	0.94
q14 – There are clear clinical protocols in this post	2.47	3	0.94
q15 – My clinical teachers are enthusiastic	2.91	3	0.93
q16 – I have good collaboration with other doctors in my grade	3.18	3	0.67
q17 – My hours conform to the new deal	2.62	3	1.02
q18 – I have the opportunity to provide continuity of care	2.49	3	0.87
q19 – I have suitable access to careers advice	2.40	3	0.91
q20–This hospital has good quality accommodation for junior doctors, especially when on call	1.74	1.5	1.15
q21 – There is access to an educational programme relevant to my needs	2.39	3	1.05
q22 – I get regular feedback from seniors	2.10	2	1.11
q23 – My clinical teachers are well organized	2.34	3	1.04
q24 – I feel physically safe within the hospital environment	2.99	3	0.78
q25 – There is a no-blame culture in this post	2.64	3	1.04
q26 – There adequate catering facilities when I am on call	1.69	1	1.29
q27 – I have enough clinical learning opportunities for my needs	2.66	3	0.93
q28 – My clinical teachers have good teaching skills	2.83	3	0.95
q29 – I feel part of a team working here	3.01	3	0.83
q30 – I have opportunities to acquire the appropriate practical procedures for my grade	2.99	3	0.84
q31 – My clinical teachers are accessible	3.01	3	0.78
q32 – My workload in this job is fine	2.90	3	0.64
q33 – Senior staff utilize learning opportunities effectively	2.35	2	0.87
q34 – The training in this post makes me feel ready to be a SpR/Consultant	2.52	3	0.88
q35 – My clinical teachers have good mentoring skills	2.51	3	0.81
q36 – I get a lot of enjoyment out of my present job	2.93	3	0.76
q37 – My clinical teachers encourage me to be an independent learner	2.68	3	0.79
q38 – There are good counselling opportunities for junior doctors who fail to complete their training satisfactorily	1.98	2	0.67
q39 – The clinical teachers provide me with good feedback on my strengths and weaknesses	2.13	2	1.03
q40 – My clinical teachers promote an atmosphere of mutual respect	2.75	3	1.05

Table 2 shows the statements which were highly rated (more than 3) or poorly rated (less than 2). There were only three questions (13, 34 and 40) with statistically significant differences with a p-value of less than 0.05 (on Mann–Whitney test) between genders. On each occasion the male trainees ranked the environment higher than did the female trainees. There were no gender differences in the aggregated scores.

Table 2.  Questions with mean scores high (more than 3) or low (less than 2) [## reverse scores high = little racism or sexism]

High mean > 3	Low mean < 2
q6 – I have good clinical supervision at all time	q38 – There are good counselling opportunities for junior doctors who fail to complete their training satisfactorily
q7 – There is (no) racism in this post ##	q26 – There adequate catering facilities when I am on call
q16 – I have good collaboration with other doctors in my grade	q20 -This hospital has good quality accommodation for junior doctors, especially when on call
q13 – There is (no) sexism in this post ##	q9 – There is an informative Junior Doctors handbook
q5 – I have an appropriate level of responsibility
q29 – I feel part of a team working here
q31 – My clinical teachers are accessible

There were significant differences between training programmes on 21 questions and all the aggregated scores. There are significant differences between trainees' grades on 17 questions and in general the Senior House Officers rated the environment more highly than did the Specialist Registrars. The order of ranking was Pre Registration House Officers, then Senior House Officers and then Specialist Registrars for all the aggregated scores except for role autonomy where the Senior House Officers ranked it highest and the Pre Registration House Officers ranked it the lowest.

Table 3 summarizes the aggregate scores for individual trainees in the way suggested by Roff et al. (2005) to identify measures of the environment globally, and in terms of teaching, role autonomy and social support.

Table 3.  Summary of aggregate scores from PHEEM questionnaire

	Mean	Median	Std. deviation
Global overview (all questions) (out of 160)	103.5	105	19.10
Perceptions of teaching (2, 3, 6, 10, 12, 15, 21, 22, 23, 27, 28, 31, 33, 37, 39) (out of 60)	38.8	40.5	9.46
Perceptions of role autonomy (1, 4, 5, 8, 9, 11, 14, 17, 18, 29, 30, 32, 34, 40) (out of 56)	35.7	35.5	7.03
Perceptions of social support (7, 13, 16, 19, 20, 24, 25, 26, 35, 36, 38) (out of 44)	28.43	29	5.20

The numbers of trainees that fall into each of the three categories are then shown in Table 4.

Table 4.  Numbers of trainees in PHEEM categories

Overall scores	<40 very poor	1
	41–80 plenty problems	8
	81–120 more + than–	110
	>120 excellent	15
Perceptions of teaching	0–15 poor	2
	16–30 need retraining	25
	31–45 moving right direction	75
	46–60 model teachers	32
Perceptions of autonomy	0–14 poor	0
	15–28 negative view of ones role	19
	29–42 more positive perception	98
	43–56 excellent	17
Perceptions social support	0–11 non-existent	1
	12–22 not pleasant	14
	23–33 more pros than cons	102
	34–44 good support	17

For the factor analysis results, the questions were grouped according to how different individuals answered the questions using Principal Component Factor Analysis. The Kaiser–Meyer–Olkin Measure of Sampling Adequacy was 0.82 (a value above 0.5 means that the sample of data is a valid matrix (Field2000) demonstrating that there are significant factors to be derived from this data). The factor analysis produced 10 factors which account for 67% of the variance. The top three factors encompass 18 of the 40 questions. The first is to do with ‘the teacher’, the second ‘learning doctoring skills in a safe environment’ and the third a ‘happiness index’. Table 5 shows these 3 main factors and the questions which are part of these three factors.

Table 5.  Factors shown in PHEEM data

The Teacher
q33 – Senior staff utilise learning opportunities effectively
q23 – My clinical teachers are well organised
q14 – There are clear clinical protocols in this post
q35 – My clinical teachers have good mentoring skills
q39 – The clinical teachers provide me with good feedback on my strengths and weaknesses
q15 – My clinical teachers are enthusiastic
q22 – I get regular feedback from seniors
q2 – My clinical teachers set clear expectations
Learning doctoring skills in a safe environment
q30 – I have opportunities to acquire the appropriate practical procedures for my grade
q28 – My clinical teachers have good teaching skills
q31 – My clinical teachers are accessible
q18 – I have the opportunity to provide continuity of care
q40 – My clinical teachers promote an atmosphere of mutual respect
Happiness
q25 – There is a no-blame culture in this post
q24 – I feel physically safe within the hospital environment
q17 – My hours conform to the new deal
q36 – I get a lot of enjoyment out of my present job

Discussion

We believe that we have shown that PHEEM has a set of reliable questions to use for measuring the educational environment within intensive care medicine (Cronbach's alpha 0.92). Furthermore the high compliance rate of filling in the questionnaire implies that the trainees found it a simple and practical proposition. We have shown that PHEEM meets some of the requirements of assessment tools namely reliability and practicality (Chambers & Wall2000).

The intensive care areas sampled provided good overall educational environments. In particular we were heartened to find that there were low levels of perceived racism and sexism. While recent studies have reported bullying and harassment is common within the NHS (Anonymous2001; Quine2002; Musselman et al.2005) it was reassuring to find that in intensive care medicine that this was not an issue. However this was a small self-selected sample of the trainees within Intensive Care Medicine, and so caution must be taken in generalising too widely from our results here. In order to confirm these findings the next stage in the process will be to undertake a ‘snapshot’ of all intensive care trainees within the United Kingdom as a whole to set benchmarks.

There were interesting differences between genders, grades of the trainees and between intensive care areas. In particular, we found that Senior House Officers experienced a better educational climate in many respects than did Specialist Registrars both when comparing grades of doctor and through hierarchical cluster analysis. Our data does not explain why this is the case, so this is an area for further study in the future. It would be valuable to understand what factors contributed to the success of the units achieving the better results.

We believe it is useful to measure the educational climate for four reasons.

1. So an individual trainee can identify what to expect.

2. So each unit can inform the teachers and the trainees of the quality of their educational climate.

3. So the educational climate can be tracked over time.

4. So that standards can be set at local, regional and national level.

Information is also useful to Deaneries, Royal Colleges and PMETB in their roles relating to quality assurance of training programmes and posts. In the future, PHEEM could be used as a quick, easy and cost effective way of measuring the educational environment. On the one hand it could be used to identify examples of excellence from which good practice could be shared with others. On the other hand it could act as a screening tool to identify areas of concern. Such areas could then be visited by Deanery teams, by Royal Colleges and by PMETB.

In conclusion PHEEM is a simple, practical and reliable way to measure the educational environment in intensive care medicine. Furthermore these results show that an intensive care area can offer a supportive and good environment for trainees, especially senior house officers and that such attachments for Foundation trainees to ICUs should be encouraged.

Additional information

Notes on contributors

Michael Clapham

DR MICHAEL CLAPHAM is a consultant anaesthetist and intensivist at the Queen Elizabeth Hospital in Birmingham, and associate dean for education in the West Midlands Deanery, United Kingdom.

David Wall

PROFESSOR DAVID WALL is deputy regional postgraduate dean in the West Midlands Deanery and professor of medical education at Staffordshire University.

Anna Batchelor

DR ANNA BATCHELOR is President of the Intensive Care Society and is a consultant anaesthetist and intensivist in the Department of Anaesthesia and Intensive Care Medicine, Newcastle-upon-Tyne National Health Service Trust, and the University of Newcastle-upon-Tyne, Newcastle-upon-Tyne, United Kingdom.

Appendix 1

PHEEM

(Postgraduate Hospital Educational Environment Measure)

Sex:□ Male □ Female

Training Grade:□ PRHO □ F2 □ SHO □ SpR

Years in present grade:□ 1 □ 2 □ 3 □ 4 □ 5

Specialty:□ Surgical   □ Medical  □ Paediatric

□ Obs & Gynae □ Anaesthetic □ Critical Care

□ Foundation  □ Other

The following items relate to your current experience. Please read each statement and rate it as it applies to your own feelings about your present position in this hospital. It is about your personal perceptions of the current post.

Please tick the appropriate box:

Table

		Strongly Agree	Agree	Uncertain	Disagree	Strongly Disagree
1	I have a contract of employment that provides information about hours of work	□	□	□	□	□
2	My clinical teachers set clear expectations	□	□	□	□	□
3	I have protected educational time in this post	□	□	□	□	□
4	I had an informative induction programme	□	□	□	□	□
5	I have the appropriate level of responsibility in this post	□	□	□	□	□
6	I have good clinical supervision at all time	□	□	□	□	□
7	There is racism in this post	□	□	□	□	□
8	I have to perform inappropriate tasks	□	□	□	□	□
9	There is an informative Junior Doctors handbook	□	□	□	□	□
10	My clinical teachers have good communication skills	□	□	□	□	□
11	I am bleeped inappropriately	□	□	□	□	□
12	I am able to participate actively in educational events	□	□	□	□	□
13	There is sex discrimination in this post	□	□	□	□	□
14	There are clear clinical protocols in this post	□	□	□	□	□
15	My clinical teachers are enthusiastic	□	□	□	□	□
16	I have good collaboration with other doctors in my Grade	□	□	□	□	□
17	My hours conform to the new deal	□	□	□	□	□
18	I have the opportunity to provide continuity of care	□	□	□	□	□
19	I have suitable access to careers advice	□	□	□	□	□
20	This hospital has good quality accommodation for junior doctors, especially when on call	□	□	□	□	□
21	There is access to an educational programme relevant to my needs	□	□	□	□	□
22	I get regularly feedback from seniors	□	□	□	□	□
23	My clinical teachers are well organised	□	□	□	□	□
24	I feel physically safe within the hospital environment	□	□	□	□	□
25	There is a no-blame culture in this post	□	□	□	□	□
26	There adequate catering facilities when I am on call	□	□	□	□	□
27	I have enough clinical learning opportunities for my needs	□	□	□	□	□
28	My clinical teachers have good teaching skills	□	□	□	□	□
29	I feel part of a team working here	□	□	□	□	□
30	I have opportunities to acquire the appropriate practical procedures for my grade	□	□	□	□	□
31	My clinical teachers are accessible	□	□	□	□	□
32	My workload in this job is fine	□	□	□	□	□
33	Senior staff utilise learning opportunities effectively	□	□	□	□	□
34	The training in this post makes me feel ready to be a SpR/Consultant	□	□	□	□	□
35	My clinical teachers have good mentoring skills	□	□	□	□	□
36	I get a lot of enjoyment out of my present job	□	□	□	□	□
37	My clinical teachers encourage me to be an independent learner	□	□	□	□	□
38	There are good counselling opportunities for junior doctors who fail to complete their training satisfactorily	□	□	□	□	□
39	The clinical teachers provide me with good feedback on my strengthens and weaknesses	□	□	□	□	□
40	My clinical teachers promote an atmosphere of mutual respect	□	□	□	□	□

Comments