The development and validation of the mini-surgical theatre educational environment measure

Abstract

Background: The operating theatre educational environment during medical school may influence student's perceptions of a surgical career. Interest in a surgical career is declining both in the USA and the UK. This study sought to develop and validate a practical questionnaire for distribution to medical undergraduates to assess their operating theatre educational environment.

Methods: A previously published questionnaire—the Surgical Theatre Educational Environment Measure (STEEM) (validated on postgraduates) was distributed to 83 final year medical students. The STEEM was found to be a reliable and valid tool for use on undergraduates (Cronbach's alpha = 0.86). From this preliminary study, factor analysis identified 13 factors covering 73% of the variance, which enabled the development of a shorter 14-item questionnaire (mini-STEEM).

Results: The mini-STEEM was distributed to 99 medical students following their final year surgical placement at a UK medical school, with a 100% response rate. The mini-STEEM was shown to be a valid, reliable (Cronbach's alpha = 0.80) and practical tool, which was easy to complete, code and analyse.

Conclusions: The results showed that the overall climate within the operating theatre for medical undergraduates in one UK Medical School was good with no gender differences in the overall scores.

Introduction

There has been mounting evidence to suggest decreasing interest in surgery as a future career by medical students (Polk 1999; Bland & Isaacs 2002; Gelfand et al. 2002; Morris 2003). This is most obviously highlighted in the numbers of unfilled posts for surgical residency programmes in the USA, which rose from 5 in 1997 to 41 in 2001 (Gelfand et al. 2002) and there continues to be a downward trend (Bland & Isaacs 2002). Within the UK, there are fewer UK graduates applying for surgical training schemes (Osborn 2006).

The reason for this change has been investigated by surgeons in the USA, whose situation mirrors, to an extent, the current emerging situation in the UK. In both countries, there have been recent changes to surgical education—not only a decrease in hours of training, but also increasing national emphasis on primary health care professions such as general practice.

Several studies have highlighted that medical students are increasingly interested in occupations within medicine with a ‘controllable lifestyle’ (Erzurum et al. 2000; Gelfand et al. 2002, Bland & Isaacs 2002; Henningsen 2002). A surgical career was perceived to be one of high pressure and stress (p = <0.001), with no control over one's time (p = <0.001). Students also felt that surgeons had inadequate family and leisure time (p = 0.001) and that income was not adequate for the level of commitment and the amount of work (p = <0.001) (Gelfand et al. 2002). These findings were mirrored by Bland (2002). The students of ‘Generation X’ (Evans & Sarant 2002) are primarily concerned about their lifestyles. However, the presence of good surgical role models early in medical training had the most profound effect on choice of future surgical career by students (Erzurum et al. 2000; Gelfand et al. 2002).

Table

Practice points
•The educational environment is an important educational measure which has a large impact on the satisfaction and success of undergraduate medical education.
•The educational environment in the operating theatre for undergraduate students is positive in one major teaching hospital in one UK medical school.
•The mini-STEEM is a reliable and valid tool with which to measure educational environment in medical undergraduates.
•The mini-STEEM is easy to use.

The lack of exposure to surgical role models, which may be due to a decreasing exposure to surgery in undergraduate medical education (Polk 1999), together with the dissolution of the historical ‘surgical firm’ due to the new shift system, has meant that medical students are less likely to meet the role model they aspire to be within the field of surgery. In this regard, the operating theatre plays a vital role in the formation of a rapport between medical students and their surgical consultant (Lyon 2004). Decreasing emphasis on the importance of the operating theatre within undergraduate surgical education, and decreased exposure to surgery in general at medical school (Polk 1999) has meant that students may never see the outcome and rewards of a career in surgery. Even though 90% of medical students in a survey in Australia agreed that ‘Going to theatre helps me to explain surgical procedures to patients’ and was a good use of their time (Lyon 2003), the operating theatre may be perceived by students to be a stressful and volatile environment, which may discourage them from attending regularly (Lyon 2003). A knowledge of surgery is also important for doctors in other areas within medicine such as emergency medicine and general practice. Doctors in these specialties will see surgical patients and will need to diagnose and refer these patients appropriately. Therefore a knowledge of and experience of surgery is important to these doctors, who are ‘non-surgeons’.

The educational environment or climate is known to have an effect on students’ satisfaction, achievement and success (Genn 2001). Within the operating theatre, this not only affects a medical student's perception of their consultant as a role model, but also their perceptions of surgery as a career.

We sought to validate a tool by which the educational environment within the operating theatre could be easily measured and used to assess students’ perceptions of surgery, their available role models and their satisfaction with their time spent in theatre.

Cassar (2004) validated and published the Surgical Theatre Educational Environment Measure (STEEM). This consisted of 40 statements with which the respondents were asked to indicate their agreement using a five point Likeart scale. These ranged from strongly agree (5), agree (4), uncertain (3), disagree (2) to strongly disagree (1). The questionnaire was piloted on 26 basic surgical trainees in Aberdeen. We used this questionnaire on 83 medical students, to ascertain whether it was also a reliable and valid tool for assessing the educational environment for medical undergraduates. The questionnaire was distributed to medical students following their final year surgical placement during the waiting periods of their surgical OSCE day. They were allocated 10 minutes to complete the questionnaire. All questionnaires were collected within this time, with a 100% response rate.

Methods

As a result of this study (Nagraj et al. 2006), we found that the STEEM questionnaire was a reliable (Cronbach's alpha of 0.86) and practical tool for assessing medical students’ perceptions of their educational environment. When the results were further analysed using an exploratory factor analysis, we found that there were 13 factors which covered 73.2% of the variance. This compared to 4 factors identified in the original paper by Cassar, which in this case would only cover 41.2% of the variance. We used these 13 factors to condense the 40-item questionnaire into a one-page 14-item questionnaire to ascertain perceptions of operating theatre learning environment of medical students. We named this questionnaire the mini-STEEM (Appendix 1).

The statements identified from the factor analysis and incorporated into the new questionnaire were:

1. My trainer is enthusiastic about teaching

2. The theatre staff are friendly

3. There are enough theatre sessions per week for me to gain the appropriate experience

4. Before the operation my trainer discusses the surgical technique planned

5. The elective operating list has the right case mix to suit my training

6. The variety of emergency cases gives me the appropriate exposure

7. I get enough opportunity to assist

8. On this unit the types of operations performed are too complex for my level*

9. The anaesthetists put pressure on my trainer to operate himself to reduce anaesthetic time*

10. I feel discriminated against in theatre because of my sex*

11. I feel discriminated against in theatre because of my race*

12. I am too busy doing other work to go to theatre*

13. When I am in theatre, there is nobody to cover the ward*

14. The atmosphere in theatre is pleasant

(Questions with a * were negative questions and will need to be reverse coded before entering into the overall scores for the surgical educational environment).

Although question 13 relates to junior doctors rather than medical students, we found that medical students did answer the question, although this question did produce the greatest number of missing values from the medical students (a total of 6 missing values out of 99). However, although medical students do have some duties on the ward, after further consideration of this question, we decided to omit this question from the questionnaire for the purposes of medical student educational climate. It is not the responsibility of medical students to cover the ward.

The modified STEEM questionnaire or ‘mini-STEEM’ was distributed to all 99 final year medical students at a UK medical school following their final year surgical attachment. The students had been allocated randomly to Heart of England NHS Foundation Trust (one of the University of Birmingham's major teaching hospitals). This group of students represented about one third of all the final year medical students at the University.

The purpose of this study was to assess whether it was a practical and reliable tool to assess the educational environment in the operating theatre for students. As a result of it fitting onto just one side of A4, we found that it was even more practical to use, taking five minutes maximum to complete. It was also easier to code, input and analyse the answers. Informal interviews were undertaken with 20 students following completion of the questionnaire to ask about ease of use and any problems interpreting the questions.

The data was analysed using SPSS version 13.0. The reliability of the questionnaire was assessed using Cronbach's alpha both for the whole questionnaire and for each item using the ‘alpha if item deleted’ to identify any rogue questions. The descriptive statistics were reported as the minimum and maximum values, the mean and standard deviation. The comparative statistics used the non-parametric methods of Mann–Whitney and the one-way ANOVA test for assessing gender differences. Exploratory factor analysis was done using principal component factor analysis, using varimax rotation, Eigen values set at 1, and accepting factor loading of 0.50 and above (Field 2000).

The analyses were also re-run using data with Question 13 (When I am in theatre, there is nobody to cover the ward) omitted for the reasons given above.

Results

Demographics

All 99 students (100%) completed the questionnaire. There were 64 females and 35 males.

Practicality

The questionnaire fitted onto a single sheet of A4 paper and students were given a total of 5 minutes to complete the questionnaire. All questionnaires were returned within this time. Informal interviews with 20 students following completion of the questionnaire, revealed that the questionnaire was easy to understand and complete, with no problems interpreting the questions. Out of a possible 6930 responses to the 14 questions, there were only 10 missing values, suggesting that the questionnaire was well understood by students. The least number of responses related to question 13: When I am in theatre, there is no one to cover the ward (n = 93 responses out of 99). Coding of the questionnaire was easy and the scores using the Likert scale were put onto a spreadsheet for further analysis. Negative questions (questions Q8, Q9, Q10, Q11 and Q12 in this study) were reverse coded.

Reliability

Cronbach's alpha scored at 0.81 for all 14 items, and was 0.80 for the 13 item amended questionnaire. This is still a high value, and compares favourably to the reliability of the original STEEM published by Cassar (Cronbach's alpha 0.88) and to the original STEEM used on medical students (Cronbach's alpha 0.86) There were no rogue questions identified using the reliability statistics, either with the 14 item or the 13 item analyses. Further analyses in this paper have been presented using the 13 item STEEM only, for the reasons discussed above.

Scores

The highest ranked statements were related to discrimination: ‘I feel discriminated in theatre because of my race’ scored the highest when reverse coded, indicating that medical students perceived no racism in the operating theatre. The lowest scoring item was: ‘I am too busy doing other work to go theatre’. The scores for all questions are shown in Table 1.

Table 1.  Scores for each of the 13 questions and overall score for the mini-STEEM

Question number and theme of each question	N	Min	Max	Mean	Std. deviation
Q1 – Enthusiastic trainer	98	1.00	5.00	3.7245	1.03327
Q2 – Theatre staff friendly	99	1.00	5.00	3.9293	0.81130
Q3 – Enough theatre sessions for getting experience	98	1.00	5.00	3.9184	0.92696
Q4 – Before operation – trainer discusses surgical technique	98	1.00	5.00	2.7755	1.09855
Q5 – Elective operating list has right case mix	99	1.00	5.00	3.3232	0.95648
Q6 – Good variety of emergency cases	99	1.00	5.00	3.0404	1.03922
Q7 – Enough opportunity to assist	99	1.00	5.00	3.0505	1.18987
Q8 – Operations too complex for my level*	99	1.00	5.00	3.0505	1.12824
Q9 – Anaesthetists put pressure on for trainer to operate*	99	1.00	5.00	3.6263	0.92124
Q10 – Discrimination in theatre because of my sex*	99	1.00	5.00	4.0909	0.94859
Q11 – Discrimination in theatre because of my race*	99	1.00	5.00	4.2828	0.90396
Q12 – Too busy doing other work to go to theatre*	99	1.00	5.00	2.5303	1.13573
Q14 – Pleasant atmosphere in theatre	98	1.00	5.00	3.7245	0.78360
Overall scores for all 13 questions	95	30.0	61.0	44.9211	7.08434

*Denotes data has been reverse coded for this question (see Results – section on practicability).

The overall mean score for the questionnaire was 45 out of 65 for the 13 item Mini-STEEM, with a mean and median score of 45. Using the five point scale, the minimum score could be 13, (1 × 13) and the maximum score could be 65 (5 7× 13). The midpoint on a 5 point scale is 3, so the midpoint score, marking the difference between a positive and a negative climate would be 39 (3 × 13). The overall scores here lie above the mid point, at 45, with a standard deviation of 7. So the overall climate was almost one standard deviation above the midpoint score here.

This overall score here for the mini-STEEM of 45 out of 65 (mid point 39 – score of 3 × 39) compares favourably to both the 40 item STEEM questionnaire distributed to final year medical students (in which the overall mean score for final year medical students was 139 out of 200 (mid point 120 overall – score of 3 × 40) and also the original STEEM paper by Cassar (2004) on basic surgical trainees (when the overall mean score was 149 out of 200 (mid point 120 overall).

All comparative statistics were performed using the Mann–Whitney and ANOVA tests, comparing each of the 13 questions and the overall scores by gender of medical student responders. There were no significant differences in any of the responses or in the overall scores.

Exploratory factor analysis

A further factor analysis of results was undertaken to reveal any subscales. Questions were grouped according to how different individuals answered the questions using Principal Component Factor Analysis with the Eigen value set at 1. The Kaiser-Meyer–Olkin Measure of Sampling Adequacy was 0.770 (a value above 0.5 means that the sample of data is a valid matrix (Field 2000) demonstrating that there are significant factors to be derived from this data). The factor analysis produced 3 factors (with rotation converging in 5 iterations) which accounted for 56% of the variance, and which may represent three subscales. All the questions except Q8 (operations too complex for my level) were captured by one of these three factors. These results are presented below in Table 2.

Table 2.  The rotated component matrix showing the loading of the questions onto the three factors

	Component
Question number and theme	1	2	3
Q5 – Elective operating list has right case mix	0.728
Q6 – Good variety of emergency cases	0.727
Q4 – Before operation - trainer discusses surgical technique	0.640
Q7 – Enough opportunity to assist	0.626
Q12 – Too busy doing other work to go to theatre	0.624
Q2 – Theatre staff friendly		0.818
Q14 – Pleasant atmosphere in theatre		0.699
Q3 – Enough theatre sessions for getting experience		0.682
Q1 – Enthusiastic trainer		0.563
Q11 – Discrimination in theatre because of my race			0.853
Q10 – Discrimination in theatre because of my sex			0.795
Q9 – Anaesthetists put pressure on for trainer to operate			0.756
Q8 – Operations too complex for my level

The three subscales have been labelled as:

Subscale 1: Good surgical operating experience Q4, Q5, Q6, Q7 & Q12

Subscale 2: Friendly atmosphere in theatre Q1, Q2, Q3 & Q14

Subscale 3: Discrimination against me Q9, Q10 & Q11

These three subscales may be used in the broad analysis of results of mini-STEEM to highlight areas of weakness and strength within a surgical placement. Specific problems within these subscales can be identified by looking at the individual statements contained therein.

Discussion

We have demonstrated that the mini-STEEM is a reliable, valid and practical tool for measuring the surgical operating educational environment in undergraduate medical education in one UK medical school. It fits neatly onto one side of A4 paper and is easy to administer. It does not take as much time to complete, to code or to analyse the statements compared to the 40-item STEEM questionnaire. Although the response rate from our questionnaire was already 100%, a systematic review of questionnaires demonstrated that shorter questionnaires had an increased response rate (p = <0.001, Edwards et al. 2002). The length of the mini-STEEM therefore, makes it easier to administer, requiring less paper and fewer data to encode for analysis. It will also help to improve the response rate from the questionnaire and encourage students to complete feedback following a surgical firm.

As discussed previously, the operating theatre learning environment plays an important role in the overall impression of surgery as a career. Although the operating theatre may be perceived to be a frightening place for medical students (Lyon 2003), the educational environment has been shown to be positive with an overall score of 45 out of 65. This is similar to the overall score seen in basic surgical trainees in Aberdeen of 149 out of 200 (Cassar 2004) and is a positive finding with regards to student's perceptions of surgery and surgical role models. Like Cassar (2004), who found that ‘… there was no significant difference in overall score between male and female trainees …’ Cassar (2004, page 261), we found the same with our final year medical students. There were no gender differences here, with regard to any of the individual questions on the mini-STEEM, or in the overall scores.

The mini-STEEM may be used as a tool by which measurements of improvements may be made to the undergraduate surgical curriculum using the three subscales identified in the mini-STEEM to identify any potential weaknesses at a particular placement. The questionnaire may also be used as a tool for providing feedback to consultant surgeons regarding their teaching within the operating theatre. It may also help to give students more positive messages about the value of attending operating theatre sessions.

The limitations of this study are that it was only carried out at one medical school in the United Kingdom, with 99 final year medical students. Further use of the mini-STEEM with other cohorts of medical students here in Birmingham, with students in other years within the medical undergraduate course, and students from other medical schools within the United Kingdom and in other countries, will help to provide a means of comparing operating theatre learning environments and developing a scale for interpretation of the overall scores.

Further studies are also required to see if the mini-STEEM could be used with postgraduate doctors such as newly qualified doctors in Foundation Year One and Foundation Year Two doctors, and doctors in specialist surgical registrar posts. Here we would want to include Question 13 again (When I am in theatre, there is nobody to cover the ward) as this is a common complaint from our young doctors in some surgical posts, when we ask this on quality assurance visits. It would also be interesting to compare mini-STEEM to the original STEEM in these groups. Another area for future study would be to make the mini-STEEM an online feedback form, and see the difference in response rates, compared with the paper version.

In conclusion, the mini-STEEM is a practical, valid and reliable tool of measuring the operating theatre educational environment for medical undergraduates in one UK medical school. The results suggest that the educational environment was perceived positively by medical students and there were no gender differences in the overall scores.

Additional information

Notes on contributors

Shobhana Nagraj

MISS SHOBHANA NAGRAJ is an Education Fellow in Surgery at Birmingham Heartlands Hospital, where her time is divided between teaching medical students from the University of Birmingham, research in Surgical Education and clinical work in the Department of General Surgery.

David Wall

PROFESSOR DAVID WALL is Deputy Regional Postgraduate Dean in the West Midlands Deanery and Professor of Medical Education at Staffordshire University.

Ellen Jones

DR ELLEN JONES is a Consultant in Emergency Medicine at the Heart of England NHS Foundation Trust and Clinical Sub-dean at the trust for the University of Birmingham Medical School, United Kingdom.

Appendix 1