Adaptation and initial examination of the psychometric properties of the Short Supervisory Relationship Questionnaire (SSRQ) for use with general practice registrars

ABSTRACT

The relationship between general practice (GP) registrars and their supervisors underpins the training experience for the next generation of medical practitioners. Building on recent research into the development and validation of a measure of the relationship between registrars and supervisors from the perspective of the supervisor, the current study focuses on the educational alliance from the perspective of the registrar. This paper presents an adaptation and initial validation of the clinical psychology supervisory relationship measure for GP registrars in an Australian context. Following an Expert Group review and adaptation of the items, 238 GP registrars completed the adapted tool. Using exploratory factor analysis and Procrustes confirmatory rotation, an optimal four factor model of the supervisory relationship was identified, reflecting measures of Safe base (α =.93), Supervisor investment (α =.96), Registrar professionalism (α =.90), and Emotional intelligence (α =.87). The general practice supervisory relationship measure for registrars (GP-SRMR) demonstrated excellent model fit, high internal consistency, and was theoretically consistent with the original tool. Implications for clinical education and future research are presented.

KEYWORDS:

• Supervisory relationship
• educational alliance
• general practice
• clinical education

Background

The educational alliance is the basis for the clinical, educational and personal development of a general practice (GP) registrar and is central to the concepts of supervision and learning [3]. Measurement of this relationship has not featured widely in general practice but is more prominent in other disciplines, notably psychology [4]. Where the relationship between the registrar and the supervisor is less than optimal this will likely impact the educational alliance and thus the educational outcomes of the registrar [5]. The supervisory relationship has two perspectives: that of the registrar and the supervisor. Both of these perspectives are important to determine the strength of the alliance and possible areas for support. There are a number of instruments that measure the educational environment from the registrar perspective in postgraduate training in Australia and overseas [6–9]. However, there has been a lack of validated tools to measure the educational alliance or supervisory relationship within the GP training context. In addition, there is a need for the use of partner instruments to measure the educational alliance from both the registrar and supervisor perspectives. The current study reports on the adaptation and validation of a measure of educational alliance from the perspective of the registrar.

Registrar perceptions of their training experience have been widely used to examine the quality assurance processes of medical training programmes. These include general practice medicine [9], radiation oncology speciality training [6,7], anaesthetic speciality training [10], and academic training [8]. In the USA, the Clinical Learning Environment Review was introduced by the Accreditation Council for Graduate Medical Education to provide feedback to hospitals and medical centres on how successful they are at engaging residents to improve quality and safety systems in the clinical learning environment [11].

Evaluation of the supervisory relationship, or educational alliance, can be conducted from the perspective of supervisors or registrars. The Short Supervisory Relationship Questionnaire (S-SRQ) and Supervisory Relationship Measure (SRM) are partner instruments that provide the registrar and supervisor perspective on the supervisory relationship [2,4]. They were both found to be valid and reliable in the context of psychology training in the United Kingdom and share similar theoretical frameworks.

The S-SRQ is a derivative of the Supervisory Relationship Questionnaire (SRQ) [12]. The SRQ was originally developed to measure the supervisory relationship between clinical psychology supervisors and their registrars, from the perspective of the registrar. The factor structure of the SRQ provided evidence for facilitative relationship characteristics of the supervisory relationship (‘Safe base’, ‘Commitment’ and ‘Structure’), and educative and evaluative characteristics of the supervisory relationship (‘Reflective education’, ‘Role model’ and ‘Formative feedback’). Despite demonstrating good psychometric properties, test-retest reliability, and promising predictive validity, the SRQ was a substantial instrument at 67 items. To improve the utility of the SRQ in time-poor settings, the 18 item S-SRQ was developed [2], which contained fewer characteristics of the supervisory relationship (‘Safe base’, ‘Reflective education’, and ‘Structure’), yet demonstrated high internal reliability, adequate test-retest reliability and adequate convergent, divergent and predictive validity.

The Supervisory Relationship Measure (SRM) is a 51-item questionnaire which was developed as a measure of the supervisory relationship between clinical psychology supervisors and their registrars, from the perspective of the supervisor [4]. The factor structure of the SRM provided evidence for five aspects of supervisor experience and perceptions: ‘Safe base’, ‘Supervisor commitment’, ‘Trainee contribution’, ‘External influences’, and ‘Supervisor investment’. The SRM was demonstrated to be a valid and reliable measure of the supervisory relationship from the supervisor’s perspective. An independent review of supervisory relationship measures found that the SRM is ‘a sound measure of the supervisory relationship’, with a large sample used for initial verification, and application of several tests for validity and reliability [13].

Recently, the SRM was adapted for use with GP supervisors in the Australian context [1,14]. Following an expert review and psychometric evaluation, the revised General Practice Supervisory Relationship Measure for Supervisors (GP-SRMS) demonstrated excellent psychometric properties across three domains of ‘Safe base’, ‘Supervisor investment’, and ‘Registrar professionalism’. The similarity of constructs across registrar and supervisor measures in clinical psychology training and the supervisor measure in GP training is notable and provides evidence in support of the underlying theory of supervisory relationships and educational alliance.

As noted previously, it is critical to consider the supervisory relationship from the perspectives of both GP supervisors and registrars. The current study extends the GP-SRMS development and validation, through the development of a paired measure of the supervisory relationship from the perspective of GP registrars. The lack of validated tools to measure the supervisory relationship within the GP training context is a notable gap, and the current study aims to address this from the perspective of GP registrars.

Method

In this study, we adapted and validated the Short Supervisory Relationship Questionnaire (S-SRQ) for use with GP registrars. The original S-SRQ, developed for use with clinical psychology registrars [2], consists of seven-point Likert-scale items which measure the level of agreement (strongly disagree to strongly agree) with 18 statements regarding the supervisory relationship with a particular clinical supervisor, such as ‘My supervisor gave me feedback in a way that felt safe’ and ‘My supervisor had a collaborative approach in supervision’. Reversed responses for statements asked in the negative, such as ‘My supervision sessions were disorganised’ were recoded for consistency. An Expert Group reviewed and adapted items, which was followed by pilot testing. Ethical approval for the study was provided by the Monash University Human Research Ethics Committee (project number 10977).

Expert group review and pilot testing

An Expert Registrar Advisory Group was convened in November 2017, consisting of six experienced GP registrars from GPEx and General Practice Training Tasmania (GPTT; two Australian state-based GP training organisations). This group was asked to determine the appropriateness and clarity of each of the 18 statements of the S-SRQ and to suggest amendments or additions, to measure the registrar-supervisor relationship from the GP registrar perspective. A nominal group technique was used. As a result, the original 18 items were adapted, and 14 new items were developed, resulting in a 32-item instrument. The associated demographic survey was extended from 12 questions to 17 questions.

The modified S-SRQ (hereafter the General Practice Supervisory Relationship Measure for Registrars) was piloted through SurveyMonkey in December 2017 with an Expert Registrar Pilot Group, consisting of 12 registrars from GPEx and GPTT at different stages of training. Participants provided feedback on item clarity, appropriateness and time taken to complete the survey. As a result of this pilot, five items adapted from the original survey were removed from the GP-SRMR, as not being relevant for the registrar-supervisor relationship, and two items were amended, resulting in a 27-item instrument.

In addition, a review of the pilot results by the project working group identified the fact that a subscale in the GP-SRMS [1] did not appear to have a corresponding subscale in the pilot GP-SRMR. This was viewed as a shortcoming, since an aim of this project was to develop the GP-SRMR as a partner instrument to the GP-SRMS, with similar items measuring the relationship from both the registrar and supervisor perspective and allowing for more meaningful comparisons across perspectives. Although there were many statements in the pilot GP-SRMR that corresponded to the GP-SRMS domains of ‘Safe base’ and ‘Supervisor investment’, the domain of ‘Registrar professionalism’ was underrepresented in the pilot GP-SRMR.

The project team made the decision to identify and adapt appropriate statements from the GP-SRMS for the GP-SRMR that related to ‘Registrar professionalism’. This resulted in a further 17 items added to the instrument, resulting in a 44-item GP-SRMR. This instrument was reviewed by the Expert Registrar Advisory Group who approved the amended version without further change, and it was then repiloted, without issue. The 44 items are available in an online repository, which also details the source of each item; whether it was adapted from the S-SRQ or GP-SRMS, or newly developed by the Expert Registrar Advisory Group [15].

Data collection

Once the survey items were finalised, an email invitation was sent to 448 GP community-based registrars in GPEx and GPTT in May 2018 using SurveyMonkey. A total of 238 participants completed the GP-SRMR instrument (response rate 53%). Of these, 228 also completed the demographic survey (response rate 51%). The sample size was considered to be good, with a high ratio of items to factors indicating stability would be achieved with a minimum sample of less than 100 participants [16,17]. Overall, the mean missing data rate per participant across the GP-SRMS items was 2.07%, ranging from no missing data to a maximum of 3.93%. Little’s missing completely at random (MCAR) test [18] was not significant (χ² = 120.44, df = 97, p= .055), and missing values were replaced using the expectation-maximisation procedure, which provides an iterative solution to maximum likelihood estimation of missing data [17,19].

Demographic characteristics of survey participants

Demographic characteristics of registrar participants and their reported supervisors are shown in Table 1. The age distribution was positively skewed, with 91.2% of participants aged 44 years or younger. The sample included 58% who identified as female, and 72.7% of participants had completed their primary medical qualifications in Australia. 67.6% of participants had been working as a registrar for one year or less, with most registrars (70.2%) working in a full time role. The demographics of the participants is consistent with published details of Australian GP registrars [20].

Table 1. Demographic characteristics of GP-SRMR registrar-participants and their reported GP supervisors (n = 228)

Registrar characteristics	Number	Percent	Supervisor characteristics	Number	Percent
Registrar age			Supervisor gender
25–29 years	56	24.6	Male	145	63.6
30–34 years	79	34.6	Female	74	32.5
35–39 years	44	19.3	Other/Prefer not to say	9	3.9
40–44 years	28	12.3	Total	228	100
45+ years	21	9.2
Total	228	100	Time GP has been a supervisor
			2–5 years	11	4.8
Registrar gender			6–10 years	24	10.5
Male	95	41.7	11–20 years	49	21.5
Female	128	56.1	>20 years	97	42.5
Other/Prefer not to say	5	2.2	Don’t know	47	20.6
Total	228	100	Total	228	100
Registrar level of training			Supervised registrars in the past
GPT1/PRRT1	83	36.4	Yes	201	88.2
GPT2/PRRT2	19	8.3	No	19	8.3
GPT3/PRRT3	72	31.6	Unsure	8	3.5
GPT4/PRRT4	39	17.1	Total	228	100
Other	15	6.6
Total	228	100

10 participants (4.20%) did not provide any demographic information.

Statistical analysis

Building on the methodologies employed during the statistical validation of the GP-SRMS, maximum likelihood exploratory factor analysis (EFA) with direct oblimin rotation, followed by Procrustes transformation [21,22] was used. The Procrustes transformation compares the rotated solution to an ideal matrix where items either load completely or not at all; providing an estimate of how well items fit. Using SPSS version 25 [23] EFA with direct oblimin rotation was conducted with all GP-SRMR items, followed by Procrustes transformation using Orthosim version 2.01 [24]. Items with low communality, low primary loading or significant cross-loading (where the difference between primary and secondary loadings was less than .30), or poor fit were systematically removed until a stable factor structure and robust model fit was achieved. Reliability was calculated using Cronbach’s α, and scale scores were calculated by summing the items within each factor and dividing by the number of items within each factor.

Results

A Kaiser-Meyer-Olkin value of .95 [25,26] and a significant Bartlett’s Test of Sphericity χ² = 9263.15, df = 946, p< .001 [27] were found, supporting the factorability of the correlation matrix. The complete correlation matrix is available online [15]. While six factors had eigenvalues exceeding one, initially three factors were extracted consistent with the factor structure of the GP-SRMS [1]. A review of the model using Cattell’s scree test [28] and parallel analysis [29] suggested that the three factor model was appropriate, however consistent with Tabachnick and Fidell [17], alternative models were also explored.

Three, four, and five factor models of the GP-SRMR items were examined, and poor items were systematically removed. An optimal model fit was achieved with a four factor model which retained 35 items. The overall solution congruence with an ideal target matrix was .94, with values of .85 and above indicating similarity [30,31]. Two additional measures of congruence were also calculated, with the Double-Scaled Euclidean Distance (.90) and the Kernel Smoothed Distance (.87) both indicating similarity [32,33]. Factor fit was also very high. The final factor loading matrix, model fit statistics, and reliability coefficients can be found in Table 2.

Table 2. Exploratory factor analysis with direct oblimin rotation and procrustes transformation of GP-SRMR items (N= 238)

	Communalities	Supervisor investment	Registrar professionalism	Safe base	Emotional intelligence	Congruence
29. My supervisor assists me in achieving my learning goals.	.76	.90	.00	−.03	−.03	1.00
26. My supervisor engages with my learning and training needs.	.82	.90	.09	−.04	−.04	1.00
43. My supervisor helps me identify my own learning/training needs.	.70	.80	−.04	−.07	.13	.99
42. My supervisor pays close attention to the process of supervision.	.72	.79	−.11	.06	.11	.98
24.My supervisor utilises a range of current and appropriate resources.	.63	.77	.06	.00	−.01	1.00
7. My supervisor is enthusiastic about supervising me.	.73	.57	.02	.33	.10	.91
23. My supervisor’s approach to medicine aligns with my own.	.60	.56	.09	.09	.18	.97
30. My supervisor communicates clearly and effectively.	.66	.55	.17	.18	.09	.94
34. I feel my supervisor is a good role model.	.71	.54	.02	.34	.12	.90
12. My supervisor encourages me to reflect on my practice.	.59	.54	.09	.06	.22	.96
28. My supervisor is flexible in their approach to my education.	.62	.53	.09	.15	.17	.95
10. There are adequate opportunities to access my supervisor.	.53	.53	.05	.19	.10	.95
38. My supervisor has a collaborative approach in supervision.	.65	.53	.15	.04	.24	.96
15. I have a good professional approach.	.76	−.20	.87	.06	.22	.97
18. I maintain a high standard in my interprofessional communications.	.68	−.15	.81	.12	.12	.98
27. The way that I practise is safe.	.63	.11	.79	−.07	−.22	.97
17. I integrate well with others in the team.	.64	−.15	.73	.24	.10	.95
16. I take responsibility for my work.	.66	−.01	.70	.26	−.01	.95
25. I feel confident in my clinical practise.	.46	.17	.64	−.22	−.04	.96
11.My skills are appropriate for my stage of training.	.46	.15	.62	−.09	.01	.98
6. I am considerate towards others in the practice (e.g. all practice staff).	.43	−.03	.59	.18	.01	.96
20. I work hard in the practice.	.47	.12	.59	.18	−.25	.93
14. I show good organisational skills.	.36	.08	.54	−.11	.12	.96
37. I am able to manage multiple demands.	.34	.15	.52	−.15	.05	.96
1. My supervisor is approachable.	.79	.30	.08	.67	.02	.95
2. My supervisor is respectful of my views and ideas.	.80	.25	.13	.60	.16	.93
8. I feel able to openly discuss my concerns with my supervisor.	.73	.34	−.02	.52	.21	.89
31. My supervisor demonstrates professional behaviour towards me.	.56	.28	.23	.44	.01	.87
40. I feel I am able to ask for help when I am out of my depth.	.50	.22	.21	.39	.10	.87
41. My supervisor is non-judgemental in their role as a supervisor.	.65	.36	.13	.38	.17	.83
4. My supervisor acknowledges my strengths.	.63	.25	.11	.37	.30	.83
39. My supervisor is attentive to my unspoken feelings and anxieties.	.71	.23	.03	−.09	.70	.97
13. My supervisor acknowledges when I am stressed.	.71	.20	.10	.01	.66	.96
22. My supervisor shows concern for my emotional wellbeing.	.56	.19	.01	.16	.53	.93
3. My supervisor takes time to get to know me.	.72	.30	.05	.34	.39	.81
	Factor congruence	.99	.96	.75	.86
	Cronbach’s α	.96	.9	.93	.87

Items reproduced with permission. The GP-SRMR can be freely accessed online [40].

Exploratory Factor Analysis of the GP-SRMR survey resulted in four factors: ‘Supervisor investment’ (13 items; mean inter-item correlation .64), ‘”Registrar professionalism” (11 items; mean inter-item correlation .49), ‘Safe base’ (7 items; mean inter-item correlation .65), and ‘Emotional intelligence’ (4 items; mean inter-item correlation .64). The first three of these are consistent with the underlying factor structure of the GP-SRMS, while ‘Emotional intelligence’ reflects a small number of items relating to a supervisor’s attention to a registrar’s emotional needs.

Overall, factor scores were high across all factors, indicating a strong degree of satisfaction with the supervisory relationship by registrar participants. In aggregate, average subscale scores were highest for ‘Safe base’ (M= 6.52, SD = 0.71), ‘Registrar professionalism’ (M= 6.37, SD = 0.53), and ‘Supervisor investment’ (M= 6.07, SD = 0.88), and lowest for ‘Emotional intelligence’ (M= 5.62, SD = 1.15). Average scores for each factor were relatively consistent across participant demographics.

Discussion

The aim of the current study was to adapt and validate the S-SRQ [2] for use with GP registrars within the Australian GP training context. The relationship between clinical supervisors and registrars has been demonstrated to be vital in fields such as clinical psychology [4,34,35]. The therapeutic alliance between supervisors and registrars in psychology is considered to be analogous to the educational alliance in medical training [1,3,36–38] highlighting the need to consider supervisory relationships in medical training. Following an expert review, pilot study, and statistical evaluation, the revised GP-SRMR demonstrated excellent psychometric properties across four domains of ‘Safe base’, ‘Supervisor investment’, ‘Registrar professionalism’, and ‘Emotional intelligence’.

In the original S-SRQ development and validation, factor analysis identified five aspects of registrar experience and perceptions. The results of the current study suggested that a four factor model comprising ‘Safe base’, ‘Supervisor investment’, ‘Registrar professionalism’, and ‘Emotional intelligence’ was more appropriate in the general medical practice training context, following the removal of several poorly performing items. Building on the statistical methodologies which were used to develop the GP-SRMS, SRM, S-SRQ, and SRQ [1,2,4,12], the four factor GP-SRMR demonstrated excellent model fit overall, as well as within factors. The original S-SRQ demonstrated subscale reliabilities ranging from .88 to .97. The GP-SRMS subscales demonstrated comparable subscale reliabilities, ranging from .87 to .96. The factor structure of the GP-SRMR is more closely aligned with the GP-SRMS than the original SRQ and S-SRQ, reflecting the desired ability to pair and compare the relationships between GP supervisors and registrars from both perspectives.

Across the supervisory relationship measures in psychology and general practice, the ‘Safe base’ subscale consistently emerges as one of the most reliable and consistent aspects of the supervisory relationship. The final GP-SRMR ‘Safe base’ subscale retained four of the items from the S-SRQ and included three items developed by the Expert Registrar Advisory Group. S-SRQ items which were excluded include ‘My supervisor gives feedback in a way that feels safe’, ‘My supervisor is open-minded in supervision’, and ‘My supervisor gives me positive feedback on my performance’. While conceptually and theoretically related to ‘Safe base’ in a broad sense, a review of the items in the GP-SRMR suggest a theme more closely aligned with approachability, respect, and being non-judgemental. This is comparable to ‘Safe base’ in the GP supervisor version, however the supervisor version includes more explicit examples, and is longer at 17 items. Scores on the ‘Safe base’ subscale reflect an enthusiastic, open, collaborative GP supervisory relationship.

The ‘Supervisor investment’ subscale in the GP-SRMR reflects a combination of Expert Group generated items, and a smaller number of items from the S-SRQ. Theoretically, the items are aligned very well with those of the ‘Supervisor investment’ and ‘Supervisor commitment’ subscales in the SRM, and ‘Supervisor investment’ in the GP-SRMS. Similarly to that found in the development of the GP-SRMS, the further development of the ‘Supervisor investment’ domain reflects a subtle shift away from emotional investment compared to psychology training. Alternatively, emotional investment into the supervisory relationship in the medical training context may be perceived to be of less value than investing into clinical aspects such as learning needs and resources. Interestingly, elements of role modelling and reflective practice are evident in the ‘Supervisor investment’ domain across measures and perspectives. Scores on the ‘Supervisor investment’ subscale reflect a GP supervisor’s effort to support the registrar through resources, preparation, and being interested in the registrar.

As a domain within the supervisory relationship, ‘Registrar professionalism’ was included within the GP-SRMS, SRM and SRQ, however was not clearly identified in the S-SRQ. The items within the GP-SRMS were used as a model and reworded to reflect the appropriate perspective (eg ‘My registrar is considerate towards others in the practice’ was reworded to ‘I am considerate towards others in the practice’. While measuring a similar construct from different perspectives, ‘Registrar professionalism’ is more comprehensive in the supervisor version (15 items compared to 11 items), and includes more negatively phrased questions compared to the registrar version. Scores on the ‘Registrar professionalism’ subscale reflect a registrar’s perceptions of their own competence, responsibility, organisation, and commitment.

The ‘Emotional intelligence’ subscale has not previously featured as a distinctive measure within the SRM, GP-SRMS, SRQ, or S-SRQ. It is important to note that the items reflect a degree of recognising and responding to the emotional needs of registrars in the GP supervisor, as perceived by the registrar. For example, items such as ‘My supervisor is attentive to my unspoken feelings and anxieties’, as well as recognising registrar stress and emotional wellbeing, were developed by the Expert Registrar Advisory Group in response to an identified need for greater recognition of registrar distress. Conceptually, these items share the most similarities with ‘Safe base’, however the results of the factor analysis provided evidence for a separate factor. It is likely that the separation of ‘Safe base’ as a measure of having an enthusiastic, open, and collaborative GP supervisory relationship, compared to ‘Safe base’ as including the recognition and response to emotional needs of the registrar, reflects a different culture with medical training compared to psychology training. It is noteworthy that emotional investment was also less emphasised in the adaptation from the SRM to the GP-SRMS [1]. Given the limited number of items within the ‘Emotional intelligence’ subscale, and the identified need for this aspect by the Expert Registrar Advisory Group, it is likely that this subscale is underrepresented and would require further development and validation.

The current study is not without limitations. With a response rate of approximately 53%, there remains a question about the representativeness of the sample used, particularly with respect to drawing conclusions about the distribution of the participant responses. The extent to which the results generalise beyond Australian GP registrars is also not able to be determined. However, response-rate bias is not overly problematic for quantitative analyses such as scale adaptation and validation, suggesting that the results are robust [39]. Registrar selection of the GP supervisors that they reported on was also not randomised or stratified, which may have impacted on the distribution of responses. Test-retest reliability has not yet been established, nor the convergent validity between GP supervisor reports and GP registrar reports, however this research is currently ongoing.

Conclusions

The aim of the current study was to present the adaptation and initial validation of the Supervisory Relationship Measure for use with general practitioner registrars. Following an expert review and psychometric evaluation, the revised GP-SRMR demonstrated excellent psychometric properties across four domains of ‘Safe base’, ‘Supervisor investment’, ‘Registrar professionalism’, and ‘Emotional intelligence’. Given the lack of validated tools which can be used to explore the supervisory relationship and educational alliance from the perspective of both supervisors and registrars in general practice medical training, it is hoped that the GP-SRMR and its partner GP-SRMS will be of significant utility in clinical education settings. The paired GP-SRMR and GP-SRMS are expected to be used for systemic training evaluation as well as individual intervention when difficulties in supervisory alliance are experienced. Future research will focus on determining test-retest reliability of the GP-SRMR; predictive validity with external outcomes; convergent validity between supervisor and registrar versions of the tool; further exploring and expanding the utility of the ‘Emotional intelligence’ domain; and identifying the training needs and guidelines for most effective use of the GP-SRMR in clinical education.

Ethics approval and consent to participate

Completing the anonymous survey implied consent by the participants. Ethical approval for the study was provided by the Monash University Human Research Ethics Committee (project number 10977).

Authors’ contributions

All authors were involved in the study design and data collection. RK conducted the initial data preparation, and SC conducted the statistical analyses. All authors were involved in constructing the manuscript. All authors read and approved the final manuscript.

Acknowledgments

The authors would like to acknowledge the support and assistance of the Expert Group; Pilot Group testers; and partner organisations GPEx, GPTT, GPSA, and Monash University.

Disclosure statement

The authors declare that they have no competing interests.

Data availability statement

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Additional information

Funding

This research project was supported by the Royal Australian College of General Practitioners with funding from the Australian Government under the Australian General Practice Training programmes (ERG007).