Specialty Training’s Organizational Readiness for curriculum Change (STORC): validation of a questionnaire

Abstract

Background

The field of postgraduate medical education (PGME) is continuously evolving as a result of social demands and advancing educational insights. Change experts contend that organizational readiness for change (ORC) is a critical precursor for successful implementation of change initiatives. However, in PGME, assessing change readiness is rarely considered while it could be of great value for managing educational change such as curriculum change. Therefore, in a previous Delphi study the authors developed an instrument for assessing ORC in PGME: Specialty Training’s Organizational Readiness for curriculum Change (STORC). In this study, the psychometric properties of this questionnaire were further explored.

Methods

In 2015, STORC was distributed among clinical teaching teams in the Netherlands. The authors conducted a confirmatory factor analysis on the internal factor structure of STORC. The reliability of the measurements was estimated by calculating Cronbach’s alpha for all sub-scales. Additionally, a behavioral support-for-change measure was distributed as well to assess correlations with change-related behavior.

Results

In total, the STORC questionnaire was completed by 856 clinical teaching team members from 39 specialties. Factor analysis led to the removal of 1 item but supported the expected factor structure with very good fit for the other 43 items. Supportive behavior was positively correlated to a higher level of ORC.

Discussion

In this study, additional steps to collect validity evidence for the STORC questionnaire were taken successfully. The final subscales of STORC represent the core components of ORC in the literature. By breaking down this concept into multiple measurable aspects, STORC could help to enable educational leaders to diagnose possible hurdles in implementation processes and to perform specifically targeted interventions when needed.

Keywords:

• organizational readiness for change
• postgraduate medical education
• curriculum change
• questionnaire development
• change management

Introduction

For decades, the study of change, and in particular change readiness, has been one of the important research topics in social sciences because implementing change is notoriously challenging and success rates are low.^1–3 Moreover, interest in this subject is still growing in light of the increasing environmental complexity, which requires organizations to rapidly adapt to external changes in order to survive.¹ The latter is also true for health care systems as they need to be able to rapidly adapt in order to properly respond to changing societal needs, new public-health policies, and technological advances.⁴ Additionally, health care systems themselves become more and more complex as well.⁵ Consequently, the field of medical education is also continuously evolving as a result of external changes⁶ as well as advancing educational insights.^7–9 Nevertheless, attention to change readiness of health care organizations has only recently emerged.^2,10 Moreover, establishing change readiness as part of a curriculum change has unfortunately rarely been considered^2,11–14 while it could be of great value for this field. Given the substantial amount of time, energy, and resources invested in organizational efforts to implement curriculum changes, increasing knowledge about organizational readiness for change (ORC) as well as having a valid tool to assess it could optimize organizational efforts to implement curriculum changes, and ultimately, improve health care quality.²

Change readiness is the most prevalent positive attitude toward change that has been studied in organizational literature.¹ Change experts agree that ORC is a critical precursor for successful implementation of change initiatives.^3,15 Indeed, it has been suggested that failure to establish sufficient readiness for change accounts for half of the unsuccessful organizational change efforts.³ In other words, for change to take place in the desired direction, a state of readiness must be obtained. ORC is a comprehensive construct that collectively reflects the extent to which members of an organization are inclined to accept, embrace, and adopt a particular change initiative to purposefully alter the status quo,¹⁶ and provides the foundation for either resistance or adoptive behaviors.¹³ Examples of these behaviors range from active and passive resistance by showing overt or subtle behaviors intended to ensure failure, to compliance and being prepared to make sacrifices as well as enthusiastically promoting change to others.¹⁷

Not surprisingly, the relevance assigned to ORC has resulted in the development of multiple instruments for assessing this concept.^2,13 In recent years, instruments for assessing ORC in health care have also been developed, with a predominant focus on the implementation of new policies and clinical practices.^2,10,13,18 The assessment of ORC prior to curriculum change has recently been introduced in undergraduate medical education as well.^11,14

However, postgraduate medical education (PGME) is also continuously evolving, and while it would potentially benefit from a similar instrument that could help facilitate the implementation of curriculum innovation, such an instrument is currently lacking in this particular field.¹²

PGME is a unique setting in which patient care, teaching, and learning are interconnected with each other and cannot be separated.¹⁹ In teaching hospitals, PGME is completely integrated in clinical service. Therefore, any adjustments of the educational system will influence clinical service and could have consequences for working schedules, funding, and learning experiences.¹⁹ These implications of curriculum reforms justify an in-depth analysis of the implementation of these reforms, and the uniqueness of this particular setting generates the need for an ORC instrument that is adjusted to PGME. Especially, the instruments available in undergraduate medical education cannot fill this gap either, due to among other things their focus on medical faculties, to the neglect of students, with long-lasting hierarchical structures leading to a more diverse set of pressures to change.^11,12 In contrast, in PGME, trainees play a more prominent role in change processes within smaller clinical teaching teams that also tend to have more volatile composition.¹²

The assessment of ORC enables educational leaders (ie, program directors) to identify significant gaps between their own expectations and those of other members of their clinical teaching team (trainees and clinical staff) and subsequently enables them to take action on this gap in order to prevent stagnation or even failure of the change implementation.¹³

Therefore, we developed an instrument to assess ORC in PGME, the so-called Specialty Training’s Organizational Readiness for curriculum Change (STORC) questionnaire, using a Delphi study.¹² The aim of the present study was to further explore the psychometric properties of STORC.

Methods

Conceptual model

We used the conceptual model of Holt et al¹⁶ to guide the development of STORC.¹² Holt’s model defines ORC by combining psychological factors, which reflect the extent to which members of an organization are inclined to accept and implement a change, with structural factors, which reflect the extent to which circumstances under which the change is occurring either enhance or inhibit the acceptance and implementation of the change. In other words, these different factors can either function as a facilitator or barrier to change. For instance, when looking at “management support,” the implementation process is accelerated in the presence of good leaders who are seen as role models or entrepreneurs, but can be slowed down in the absence of good leaders.²⁰

Since ORC is a multifaceted construct, it can only be captured by breaking it down into measurable core components, such as “collective commitment,” the shared belief and resolve to pursue courses of action that will lead to successful change implementation, and “support climate,” that is, an intangible encouraging environment that supports the implementation of a change. As a third dimension to this framework, Holt added the level at which the analysis took place, that is, the individual level or the organizational level.¹⁶ However, in contrast to Holt et al, we focused on the assessment of ORC in PGME at the organizational level rather than the individual level, that is, at the level of the clinical teaching teams, which consist of a program director, clinical staff members, and trainees. Therefore, the following definition of ORC was adopted, “the degree to which clinical teaching team members are motivated and capable to implement curriculum change.” In this definition, “motivated” mainly refers to the psychological factors, whereas “capable” refers to the structural factors.¹²

Delphi study

As the first step toward development of a questionnaire, we conducted a Delphi study with an international sample of expert panelists. In this procedure, a 89-item preliminary questionnaire adapted from ORC questionnaire for business and health care organizations was further tailored to PGME.¹² After 2 rounds, this Delphi study was complete and resulted in a questionnaire comprising 44 items, which are divided into 10 subscales.¹² After this Delphi procedure, both the items and the additional instructions, as would be presented to the participants, were analyzed for redundancies, phrasing, and intelligibility by 6 other researchers (5 medical doctors and 1 educationalist) and 1 layperson. This analysis led to textual changes only.

Setting and selection of participants

The Dutch PGME training programs have recently been modernized according to the competency-based framework of CanMEDS.^21,22 This nationwide effort created a suitable setting for collecting empirical data to test the psychometric properties of STORC. All clinical teaching teams registered at the Dutch Federation of Medical Specialists were eligible for participation.

In the Netherlands, all teaching hospitals have a separate educational department that supports and assists the clinical departments with their educational tasks (ie, the training of different health care professionals, such as medical students, doctors, nurses, and laboratory staff). Between February 2015 and November 2015, we contacted these educational departments and asked them to discuss our study with their educational board, which comprises at least one representative (in any case the program director) of each clinical teaching team involved in PGME. If the educational board of a hospital agreed to participate, we sent an official invitation to all program directors of that hospital. Additionally, we sent a direct invitation to the program directors within our own network. Subsequently, the program directors were responsible for inviting the other members of their clinical teaching team (ie, trainees and clinical staff members) to participate. Due to this method of recruitment (ie, snowball sampling), the total number of doctors invited by the program directors is unknown to the authors.

The invitation email included an informative letter explaining the purpose of our study, the voluntary nature of participation, and the confidentiality of the contributions. The email also contained a link to the web-based questionnaire. Besides STORC, the questionnaire included an informed consent form, general questions regarding gender, age, level of training, and years registered as a medical specialist, and an instrument to measure change-related behavior.¹⁷ During the study period, several reminders to complete the questionnaire were sent to the program directors of the participating clinical teaching teams.

Ethical approval

This study was approved by the Ethical Review Board of the Dutch Association for Medical Education (NVMO). Participation was voluntary and confidentiality of the contributions was guaranteed. Written informed consent was obtained from all participants for this study.

Materials

STORC¹²

All participants were asked to rate their agreement with the 44 items of STORC on a 5-point Likert scale (1= strongly disagree, 5= strongly agree). Alternatively, they had the option to choose “not applicable.” All items are group-referenced (eg, “we need to improve…”) rather than self-referenced (eg, “I’m willing to innovate…”) in order to focus on the organizational level of readiness for change. For the purpose of this study, participants were instructed that “this change” and “this innovation” in the questionnaire items referred to the introduction of CanMEDS in PGME. Based on our conceptual model, we expect the 10 subscales of STORC to be positively related to each other.

Behavioral support-for-change¹⁷

Besides STORC, a measure for assessing change-related behavior was administered to include a variable that would be expected to have a theoretical relation with ORC in order to strengthen the validation of STORC. This “behavioral support-for-change” measure consists of a 101-point behavioral continuum reflecting 5 types of resistance and support behavior, which were made visible along the following continuum: active resistance (score 0–20), passive resistance (score 21–40), compliance (score 41–60), cooperation (score 61–80), and championing (score 81–100).¹⁷ A written description of each of these behaviors was provided. Participants were asked to indicate the score that best represented their own reaction as well as their clinical teaching team’s reaction to the introduction of competency-based medical education leading to 2 separate scores. Based on the current literature, it is expected that when ORC is high this results in more change-supportive behavior, whereas when ORC is low, resistant behavior can be expected. Therefore, we hypothesized that high scores on the separate subscales, that is, reflecting high ORC, would correlate to high scores on the behavioral support-for-change measure, that is, change-supportive behavior.

Statistical analysis

We conducted a confirmatory factor analysis (CFA) to test the factor structure of STORC, using our previous work¹² as input for which items should load on the same factor (ie, ordinal indicators of the same latent variable) with a factor loading >0.5. A priori we assumed that CFA would confirm the division of the STORC questionnaire into the 10 subscales we found in our Delphi study and which are rooted in theory.¹² The following fit indices and criteria were used: root mean square error of approximation <0.06, comparative fit index >0.9, and Tucker–Lewis index >0.9. To estimate the reliability of the measurements, we calculated Cronbach’s alpha for each subscale. We estimated standard error of measurement (SEM, <0.26) for each subscale to estimate the number of participants needed to get a reliable score. The structural equation modeling software Mplus (version 7.3; Muthén & Muthén, Los Angeles, CA, USA) was used for conducting the CFA. For all other analyses, SPSS version 23 (IBM Corporation, Armonk, NY, USA) was used.

Results

Participants

In total, 873 clinical teaching team members agreed to participate in this study. Three respondents were excluded because their specialty (ie, medical psychology) is not registered at the Federation of Medical Specialists and therefore they did not meet our inclusion criteria. Nine respondents (1.0%) were excluded due to lack of information about their current hospital (7 respondents) or age (2 respondents). Six respondents (0.7%) were excluded because they were the only respondent within their hospital, which precluded investigation of ORC at the level of their clinical teaching team. Two respondents (0.2%) could not be included because they did not complete the questionnaire.

As a result, 856 doctors (98.1% of the respondents) were included in this study: 297 (34.7%) trainees, 315 (36.8%) clinical staff members, and 244 (28.5%) program directors.

All together, the respondents represent 223 clinical teaching teams in 23 teaching hospitals thereby representing about one third of all teaching hospitals in the Netherlands, and 39 different specialties. Respondents were either working at an academic medical center (49%) or at a nonacademic teaching hospital (51%).

CFA

One item, “current pressure to implement this innovation in residency training comes from external authorities” (originally item 4) from subscale “pressure to change,” had to be removed as its loading on the expected factor, “pressure to change,” was below 0.5 (ie, −0.173) and it did not load well on other factors either. All the other items loaded on the factors as expected. As a result, the adapted STORC questionnaire consists of 43 items divided into 10 subscales (Table 1).

Table 1 STORC: final questionnaire

STORCa
Pressure to change
Current pressures to implement this innovation in residency training comes from
1. Trainees in the program
2. Clinical teaching staff
3. Program directors
Appropriateness
This innovation in residency training is appropriate for the situation being addressed.
4. This change will improve the knowledge and skills of our trainees
5. This change is tailored to the needs for change within our residency training
6. This change will be an improvement over our current practices
Necessity to change
There is a need for change
7. There is a significant difference between the current state and the desired state of residency training
8. We need to improve our residency training curriculum
9. A change is needed to improve our residency training curriculum
Management support and leadership
The educational board (hospital level)
10. Is committed to this change
11. Provides the time and resources required to implement this change
Staff culture
Clinical staff members
12. Feel a sense of personal responsibility to improve training
13. Cooperate to maintain and improve effectiveness of training
14. Are willing to innovate and/or experiment to improve training
15. Are receptive to changes in training methods
16. Share responsibility for the success of this project
17. Work together as a team
18. Discuss this change with trainees in both formal and informal situations
The formal leader of this innovation in residency training (eg, the program director)
19. Accepts responsibility for the success of this project
20. Has the authority to carry out the implementation of this change
21. Cooperates well with the clinical staff members
Involvement in this innovation in residency training
22. Formal educational leaders communicated well with us about the policy toward this change
23. Information provided about this change is clear
24. We are sufficiently consulted about the change
25. We are informed about the reasons for change
26. Trainees are willing to innovate and/or experiment to improve training
27. We have the skills that are needed to implement this change
Project resources
The following are available to successfully implement this innovation in residency training:
28. Financial resources
29. Training
30. Facilities
31. Staffing
32. Equipment and materials
33. Trainee awareness of this change
34. Incorporation of trainee needs
35. Evaluation protocol
Clarity of mission and goals of this innovation in residency training
36. We understand how this change fits in with the desired competences of trainees
37. This curriculum change has clear goals and objectives
38. Our duties are clearly related to the goals of this change
The implementation plan for this innovation in residency training
39. Identifies specific roles and responsibilities
40. Clearly describes tasks and timelines
41. Includes appropriate training
42. Acknowledges our input and opinions
43. Includes a plan for improvement based on evaluations

Notes: Bold text: subscales of the questionnaire.

a Item numbers were adapted after the removal of 1 item in subscale 1.

Abbreviation: STORC, Specialty Training’s Organizational Readiness for curriculum Change.

Table 2 provides means and standard deviations of the items along with the loadings of the items per factor and, in the footnote, fit statistics of the final model (Mplus version 7.3) as well as Cronbach’s alpha values per factor (SPSS version 23). The latter varied from 0.544 to 0.877 for the various factors. Additionally, the table shows the number of respondents needed for an SEM of ≤0.26, namely, 5–8 respondents (SPSS version 23).

Table 2 Confirmatory factor analysis and reliability analysis

Factor (F)a	Mean (SD)	Loading	α	R-SEMb
F1: pressure to change			0.609	7
Item 1	3.40 (0.94)	0.552^*1
Item 2	3.04 (0.89)	0.758
Item 3	4.04 (0.79)	0.552^*1
F2: appropriateness			0.836	7
Item 5	3.64 (0.80)	0.850^*2
Item 6	3.43 (0.82)	0.850^*2
Item 7	3.64 (0.82)	0.850^*2
F3: necessity to change			0.834	8
Item 8	3.19 (0.93)	0.725
Item 9	3.59 (0.79)	0.927^*3
Item 10	3.44 (0.86)	0.927^*3
F4: management support and leadership			0.544	7
Item 11	3.69 (0.71)	0.673^*4
Item 12	2.97 (0.93)	0.673^*4
F5: staff culture			0.877	5
Item 13	3.59 (0.81)	0.773
Item 14	3.73 (0.70)	0.847^*5
Item 15	3.62 (0.78)	0.847^*5
Item 16	3.53 (0.74)	0.847^*5
Item 17	3.46 (0.79)	0.727^*6
Item 18	3.64 (0.85)	0.727^*6
Item 19	3.63 (0.80)	0.727^*6
F6: formal leader			0.830	6
Item 20	3.96 (0.72)	0.851^*7
Item 21	3.91 (0.72)	0.851^*7
Item 22	3.79 (0.74)	0.851^*7
F7: involvement			0.797	5
Item 23	3.41 (0.80)	0.789^*8
Item 24	3.36 (0.79)	0.789^*8
Item 25	3.17 (0.88)	0.789^*8
Item 26	3.35 (0.87)	0.789^*8
Item 27	3.74 (0.67)	0.570^*9
Item 28	3.67 (0.67)	0.570^*9
F8: project resources			0.842	5
Item 29	2.85 (0.91)	0.525
Item 30	3.51 (0.76)	0.732^*10
Item 31	3.45 (0.81)	0.732^*10
Item 32	3.27 (0.88)	0.732^*10
Item 33	3.44 (0.77)	0.732^*10
Item 34	3.63 (0.72)	0.732^*10
Item 35	3.39 (0.77)	0.732^*10
Item 36	3.11 (0.85)	0.732^*10
F9: clarity of mission and goals			0.836	7
Item 37	3.51 (0.79)	0.853^*11
Item 38	3.43 (0.77)	0.853^*11
Item 39	3.29 (0.79)	0.853^*11
F10: implementation plan			0.853	6
Item 40	3.44 (0.75)	0.803^*12
Item 41	3.21 (0.81)	0.803^*12
Item 42	3.29 (0.79)	0.803^*12
Item 43	3.22 (0.85)	0.803^*12
Item 44	3.19 (0.84)	0.803^*12

Notes: Means and SD per item along with item-factor loadings and fit statistics resulting from confirmatory factor analysis, Cronbach’s alpha values (α) per factor, and raters required for a sufficient SEM for the STORC final questionnaire. CFI =0.942, TLI =0.938, RMSEA =0.052. Superscripts ^*1–^*12 denote equality constraints (ie, loadings fixed to be equal).

a Item numbers based on original questionnaire.

b R-SEM = number of raters required for SEM <0.26.

Abbreviations: SEM, standard error of measurement; STORC, Specialty Training’s Organizational Readiness for curriculum Change; SD, standard deviation; CFI, comparative fit index; TLI, Tucker–Lewis index; RMSEA, root mean square error of approximation.

Table 3 reports the correlations between factors (Mplus version 7.3). These results indicate that all factors were positively correlated to each other, with the exception of factor 3.

Table 3 Factor–factor correlations of STORC resulting from the confirmatory factor analysis reported in Table 2

Factor (F)	Correlations
	F1	F2	F3	F4	F5	F6	F7	F8	F9	F10
F1: pressure to change	–
F2: appropriateness	0.368	–
F3: necessity to change	0	0.160	–
F4: management support and leadership	0.109	0.134	−0.285	–
F5: staff culture	0.370	0.129	−0.333	0.296	–
F6: formal leader	0.225	0.143	−0.218	0.267	0.380	–
F7: involvement	0.176	0.349	−0.194	0.302	0.227	0.410	–
F8: project resources	0.219	0.233	0	0.498	0.304	0.283	0.441	–
F9: clarity of mission and goals	0.219	0.461	0	0.282	0.275	0.287	0.600	0.453	–
F10: implementation plan	0.227	0.218	0	0.276	0.292	0.298	0.501	0.508	0.591	–

Note: Correlations of “0” have been fixed to zero.

Abbreviation: STORC, Specialty Training’s Organizational Readiness for curriculum Change.

Behavioral support-for-change

An individual score for behavioral support-for-change was given by 825 (96.4%) doctors, and a clinical teaching team’s score was given by 824 (96.3%) doctors. Table 4 provides the correlations between each of the factors and each of the 2 behavioral support-for-change scores. Again, all factors except factor 3 were positively correlated to the behavioral support-for-change measure. In other words, a higher score on this continuum, that is, supportive behavior, was related to a higher level of ORC and a lower score, that is, resistant behavior, was related to a lower level of ORC as measured by the questionnaire STORC.

Table 4 Correlations between the factors (ie, subscales) of STORC and the behavioral support-for-change scores

Factor (F)	Behavioral support-for-change score
	Own score	Team score
F1: pressure to change	0.360	0.457
F2: appropriateness	0.496	0.258
F3: necessity to change	0.112	−0.178
F4: management support and leadership	0.260	0.398
F5: staff culture	0.273	0.538
F6: formal leader	0.311	0.396
F7: involvement	0.386	0.366
F8: project resources	0.303	0.378
F9: clarity of mission and goals	0.446	0.385
F10: implementation plan	0.321	0.363

Note: The correlation between resistance scores is 0.457.

Abbreviation: STORC, Specialty Training’s Organizational Readiness for curriculum Change.

Discussion

The aim of this study was to further develop and validate the STORC questionnaire based on a stepwise approach comprising a conceptual model, a Delphi procedure,¹² CFA, reliability analysis, and the inclusion of a theoretically related variable, namely, change-related behavior. This approach yielded a solid base to explore and assess the psychometric properties of STORC. Based on our conceptual model and available instruments,^17,23–27 CFA confirmed the different subscales. Indeed, both psychological and structural factors are represented in the 43 remaining items of STORC, and the 10 subscales represent most of the core components of ORC described in the literature (Tables 1 and 5).

Table 5 Final subscales, number of items per subscale, and original subscales from which STORC was adapted

Subscale	No of items	Original subscale
Pressure to change	3	Pressure for change^Citation23
Appropriateness	3	Organizational valence,^Citation24 commitment to change^Citation17
Necessity to change	3	Discrepancy^Citation24
Management support and leadership	2	Senior leadership support^Citation24
Staff culture	7	Staff culture,^Citation22 staff cohesiveness,^Citation23 implementation team roles^Citation22
The formal leader	3	Clinical champion^Citation22
Involvement	6	Involvement,^Citation25 commitment^Citation26
Project recourses	8	Project resources and context,^Citation22 general resources^Citation22
Clarity of mission and goals	3	Clarity of mission and goals^Citation23
The implementation plan	5 43	Implementation plan^Citation22

Abbreviation: STORC, Specialty Training’s Organizational Readiness for curriculum Change.

However, the 1 item that was excluded was the only item concerning “external pressure” that had been retained after our Delphi study;¹² hence, this component of ORC is now no longer represented in STORC. In PGME, external pressure refers to pressure exerted by, for example, the educational board, the hospital board, scientific societies, accreditation bodies, the Ministry of Health, or the Ministry of Education. Since current curriculum reforms are top-down driven, the exclusion of this item is surprising, but it might be the result of the specific setting STORC was developed for. The removal could simply be due to the fact that the participants in our study do not experience external pressure on a daily basis. On the other hand, it might be due to the fact that hospitals and the medical profession are organized differently than organizations in other fields. In general, hospitals are not organized according to habitual management practices (eg, formal planning and control systems).²⁸ Furthermore, during specialty training, doctors do not receive a solid foundation in management practices. The recognition of management as such, or the exertion of sufficient influence (eg, by educational board members or scientific societies), requires sufficient knowledge and information about management principles.²⁸ Alternatively, this result elucidates a more fundamental aspect of the medical profession. Medical doctors have a very prominent professional autonomy, and research in this field has shown that medical doctors resist attempts to increase management control of medical practice.²⁹ As a result, they might be less receptive to external pressure.

As we expected, based on our conceptual model, the various subscales of STORC were positively related to each other as well as to the behavioral support-for-change scores, with subscale 3 as the only exception. Subscale 3, “necessity to change,” includes items about whether or not there is a need for improvement of residency training in general, and unlike most of the other subscales, it does not refer to the particular change proposed. As organizational readiness is situational or change-specific,¹⁵ the more general focus of this subscale could explain why this subscale shows different results. The content of change matters as much as the context of change.^15,23 In other words, a clinical teaching team could have a very receptive attitude toward change in general (context) but might at the same time exhibit a low ORC toward implementing assessment method A (content) and a high ORC toward implementing assessment method B (content).

Practical implications

STORC can be used to assess ORC at several stages of the change process, that is, prior or during change, as a way to diagnose any possible or current hurdles in the implementation process in order to facilitate any corrective interventions. For instance, when team members express doubts about whether the proposed change will be an appropriate solution for the intended improvement of practice, educational leaders can try to show the relative advantage of the innovation by means of best practices. Alternatively, they can let team members adapt and refine the innovation on a limited basis to let it suit their own needs and local context. Subsequently, STORC could be used to assess the effects of these interventions.

Strengths and limitations

One of the strengths of this study was its sample size of 856 participants, which is large in relation to the number of survey items, which was 44. Furthermore, we were able to include clinical teaching teams from 39 different specialties representing about one third of all teaching hospitals in the Netherlands, which constitutes a remarkably heterogeneous sample. On the other hand, however, the number of participants per hospital varied widely.

When looking at the reliability scores, subscale 4, that is, “management and leadership,” showed a relatively low Cronbach’s alpha (0.544). The outline of STORC before the statistical analysis was the result of our previous Delphi study which had reduced the number of items about management support from 10 to 2. Since this subscale only has 2 questions, the interpretive value of a Cronbach’s alpha score is reduced. Although the correlations with the other subscales were positive and confirmed our hypothesis, this subscale should be interpreted with some caution.

In our stepwise approach to develop STORC, we first conducted a Delphi study in an international setting.¹² We subsequently collected empirical data in the Netherlands, rather than including an international sample, but the factor analysis clearly confirmed our previous results. Therefore, we expect STORC to be internationally applicable.

Conclusion

In this study, additional steps to validate STORC for PGME were taken successfully. The final subscales of STORC represent the core components of ORC in the literature. By breaking down the concept of ORC into multiple measurable aspects, STORC could help to enable educational leaders to diagnose possible hurdles in implementation processes and to perform specifically targeted interventions when needed.

Author contributions

LB and MJ participated in data collection, after which LB and JL performed the data analysis. LB drafted the manuscript. All authors contributed toward data analysis, drafting and revising the paper and agree to be accountable for all aspects of the work.

Acknowledgments

The authors wish to thank all the participating clinical teaching teams for their input and the time they invested, and Lisette van Hulst for her editing assistance.

Disclosure

The authors report no conflicts of interest in this work.