Development and validation of the ExPRESS instrument for primary health care providers’ evaluation of external supervision

Figures & data

Table 1. Phases and steps in the development of ExPRESS.

	Step	Objective	Description
Phase 0	1. Qualitative studies	Increase understanding of external supervision	Seven focus group discussions with providers and supervisors to understand experiences. Reported elsewhere.
	2. Instrument search	Identify supervision measurement instruments	Systematic search for instruments to measure supervision. No existing tool found applicable to external supervision context.
Phase 1	3. Conceptualisation I	Develop model for questionnaire	Defining construct. Categorisation in normative, formative and restorative functions. Division in section A (individual supervisor) and section B (supervision overall)
	4. Item development I	Develop item pool and adapt relevant items	Of > 400 items, 122 retained in item pool, of which six used directly, 22 modified or inspirational and eight new items added for the first version.
	5. Translation I	Prepare for tests in Rwanda	Forward and backward translation into Kinyarwanda
	6. Interviewing I	Cognitive testing of items	Individual cognitive interviews of 10 providers and one information expert. Two group discussion with five providers and six supervisors. 17 items modified, 10 items removed, 3 items added.
	7. Field test I	Factor structure and reliability	134 respondents, 58 retest. Exploratory factor analysis and test–retest reliability
Phase 2	8. Conceptualisation II	Refine conceptual model	Systematic refinement of conceptualisation using multiple sources on supportive supervision
	9. Item development II	Adapt to refined model	11 items modified, 1 item removed, 5 items added
	10. Interviewing II	Lexical test	Two interviews with professional native English linguist to test lexical qualities of English version
	11. Relevance assessment I	Content validation by international expert	Relevance assessment of items by four international experts on supportive supervision analysed via the Content Validity Index. 14 items modified, 5 items removed, 6 items added
	12. Item development III	Review and revise prior to new translation	12 items modified, 4 items added.
	13. Translation II	Prepare final Rwandan version	Renewed translation and back-translation of all items due to several changes and modifications
	14. Relevance assessment II	Content validation by regional experts	Relevance assessment of items by five providers and five external supervisors in Rwanda, Uganda, Kenya and Nigeria analysed via the Content Validity Index
	15. Interviewing III	Testing response scale	Two group discussions with five providers on the response scale. Response scale changed for section B.
	16. Item development IV	Adding latent variable	Adding three items of a latent variable ‘Solving problems jointly’
	17. Translation III	Translate added items	Translation and back-translation of added items for the ‘Joint Problem Solving’ latent variable.
	18. Field test II	Confirmatory factor analysis	154 respondents. Confirmatory factor analysis and Differential Item Functioning

Table 2. Data quality and characteristics presented as range and (mean) across all items within a section.

	Field test I (N = 134)		Field test II (N = 158)
Parameters	Section A	Section B	Section A	Section B
Item mean	2.6–4.0 (3.5)	2.8–4.0 (3.5)	2.6–4 (3.3)	2.8–4.3 (3.7)
Item SD	0.9–1.2 (1.1)	0.8–1.2 (1.0)	0.9–1.2 (1.1)	0.9–1.4 (1.1)
Item median	3–4 (3.7)	3–4 (3.7)	3–4 (3.4)	2–5 (3.8)
% with lowest response in item	1–25 (6)	1–12 (4)	2–25 (10)	1–29 (5.8)
% with highest response in item	6–33 (17)	6–31 (19)	3–29 (11)	9–60 (35)
Item kurtosis	1.8–3.6 (2.7)	2.0–3.3 (2.5)	1.9–4.0 (2.7)	1.6–4.0 (2.7)
Item skewness, absolute values	0.2–0.9 (0.5)	0.1–0.7 (0.3)	0.0–0.9 (0.5)	0.2–1.3 (0.8)

OBS: Items of field test I and II are different

Table 3. Test–retest differences and kappa values of field test I (N = 58).

	Differences (Retest minus test)										Weighted Kappa
Item	−4	−3	−2	−1	0	1	2	3	4	Missing	Modified	Linear	Quadratic
a1			1	12	31	12		1		1	0,63	0,47	0,59
a2		1	3	15	27	8	1			3	0,57	0,47	0,60
a3			4	11	32	8	1			2	0,63	0,58	0,71
a4			4	17	27	7	2			1	0,64	0,60	0,72
a5			1	8	40	7	1			1	0,75	0,68	0,78
a6			4	15	26	10	3			0	0,45	0,39	0,55
a7		1	1	13	29	11	1			2	0,61	0,48	0,63
a8			5	10	31	11	1			0	0,59	0,53	0,67
a9		1	3	17	24	8	3	2		0	0,47	0,39	0,52
a10			3	12	30	7	4			2	0,50	0,46	0,57
a11		1	1	13	28	7	1	2		5	0,61	0,50	0,59
a12	2		1	13	32	8	1			1	0,66	0,53	0,57
a13	1	1	4	8	31	11		1		1	0,61	0,53	0,60
a14			3	12	36	6	1			0	0,65	0,58	0,69
a15			3	20	28	7				0	0,59	0,45	0,62
a16		1	2	17	30	6	1			1	0,60	0,50	0,63
% of total	0%	1%	5%	23%	53%	15%	2%	1%	0%
b1			4	10	32	6	4		1	1	0,49	0,47	0,51
b2			4	13	23	12	5		1	0	0,39	0,33	0,45
b3			1	12	27	13	2	1		2	0,55	0,42	0,55
b4		1	4	11	35	5	1			1	0,59	0,55	0,61
b5			6	12	27	10	2			1	0,42	0,38	0,50
b6		1	5	10	29	10	3			0	0,48	0,44	0,55
b7			5	12	32	6	3			0	0,47	0,46	0,55
b8		1	3	13	30	9	1			1	0,56	0,47	0,59
b9			2	17	29	6	3			1	0,54	0,46	0,60
b10		1		11	32	11	2			1	0,55	0,43	0,50
b11			4	17	27	5	3	1		1	0,42	0,37	0,48
b12			2	17	35	3				1	0,72	0,61	0,74
b13			4	15	33	6				0	0,57	0,49	0,59
% of total	0%	1%	6%	23%	53%	14%	4%	0%	0%

Item numbers correspond to field test I version, non-corresponding with item numbers of field test II

Table 4. Confirmatory factor analysis in field test II. Model fit of section A and B.

Model	Chi2*	df*	p*	CFI*	TLI*	RMSEA*	RMSEA (90% CI)		SRMR
A: 1 factor	375	153	0,00	0,81	0,78	0,11	0,13	(0,12; 0,15)	0,08
A: 2 factors	363	153	0,00	0,82	0,79	0,11	0,13	(0,12; 0,14)	0,08
A: 3 factors	266	153	0,00	0,89	0,88	0,08	0,10	(0,09; 0,12)	0,08
A: 4 factors	206	153	0,00	0,94	0,93	0,06	0,09	(0,07; 0,10)	0,06
A: 4 factors, exclude A2	177	136	0,00	0,95	0,94	0,06	0,08	(0,07; 0,10)	0,06
A: 4 factors, exclude A2, move A9	162	136	0,00	0,96	0,95	0,05	0,08	(0,06; 0,09)	0,05
A: 4 factors, exclude A2, A13, move A9	149	120	0,00	0,96	0,95	0,06	0,08	(0,06; 0,10)	0,05
A: As above + error correlations**	119	120	0,05	0,98	0,98	0,04	0,06	(0,04; 0,08)	0,04
B: 1 factor	182	105	0,00	0,86	0,83	0,08	0,10	(0,08; 0,12)	0,07
B: 2 factors	168	105	0,00	0,88	0,85	0,08	0,09	(0,08; 0,11)	0,07
B: 3 factors	150	105	0,00	0,90	0,88	0,07	0,09	(0,07; 0,11)	0,06
B: 4 factors	146	105	0,00	0,90	0,88	0,07	0,09	(0,07; 0,11)	0,06
B: 4 factors, exclude B3	107	91	0,00	0,94	0,92	0,06	0,07	(0,05; 0,09)	0,05
B: 4 factors, exclude B3 B15	86	78	0,01	0,95	0,93	0,06	0,07	(0,04; 0,09)	0,05
B: 4 factors, exclude B3 B15 B5	69	66	0,03	0,96	0,95	0,05	0,07	(0,04; 0,09)	0,05
B: 3 factors, exclude B3 B15 B5	76	66	0,01	0,95	0,94	0,06	0,07	(0,04; 0,09)	0,05
B: 3 factors, exclude B3 B15	75	66	0,02	0,95	0,94	0,06	0,07	(0,04; 0,09)	0,06

Df: Degrees of freedom, CFI: Confirmatory fit index, TLI: Tucker-Lewis Index, RMSEA: root mean square error of approximation, SRMR: standardized root mean square residual.

*Based on Satorra-Bentler estimation (not available for confidence intervals of RMSEA nor for SRMR).

** Error correlations: A3–A4 and A16–A17. Bold model: the final selected model.

Table 5. Final items of ExPRESS. Items removed following field test II are indicated with *.

	Latent variable
Section A. During the most recent supervision, the supervisor…
A1. Communicated in a friendly way	GC
*A2. Explained the purpose of the supervision visit
A3. Wanted to know my opinions	GC
A4. Listened to me attentively	GC
A5. Treated me with respect	GC
A6. Observed how I carry out specific tasks of my work	UW
A7. Spend enough time discussing my work tasks with me	UW
A8. Was familiar with my area of work	UW
A9. Showed appreciation for my work	GC
A10. Asked me what problems I experience at work	SPJ
A11. Engaged me in discussions to examine problems at work	SPJ
A12. Involved me in deciding how to handle problems at work	SPJ
*A13. Followed up on previous discussions
A14. Encouraged me to ask questions	BC
A15. Gave useful feedback about my work	BC
A16. Asked me what I need to learn more about	BC
A17. Discussed next steps	BC
A18. Checked to make sure I understood everything we discussed	BC
Section B. In general, supervisors…
B1. Keep their supervision appointments	C
B2. Try not to disturb patient care	C
*B3. Treat women and men equally
B4. Gather me and my colleagues for discussing as a group, when needed	C
B5. Maintain proper confidentiality of work-related information	C
B6. Strengthen the teamwork at my facility, when needed	C
B7. Explain the criteria used when assessing my performance	AP
B8. Assess my performance in a fair way	AP
B9. Give useful feedback after assessing my performance	AP
B10. Have sufficient clinical skills and knowledge	CT
B11. Explain difficult issues in a clear way	CT
B12. Update me when there are major changes in guidelines	CT
B13. Help make sure my needs for training are met	CT
B14. Help me feel confident at work	CT
*B15. Conduct supervision in a way that makes me provide better care

The instrument has the following latent variables:

GC: Generating comfort, UW: Understanding work, SPJ: Solving problems jointly, BC: Building capacity, C: Collaborating, AP: Assessing performance, CT: Supervisor capacity to teach.

Figure 1. Final structural equation model for section A with standardised factor loadings, error terms and error correlations.

GC: Generating comfort, UW: Understanding work, SPJ: Solving problems jointly, BC: Building capacity

Figure 2. Final structural equation model for section B with standardised factor loadings and error terms.

C: Collaborating; AP: Assessing performance; CT: Capacity to teach.