Tracking trainee development: preliminary validation of a tool designed to evaluate clinical psychology competencies over time

Figures & data

Figure 1. Example section of the PsyCET-S administered via Qualtrics.

Table 1. Interrater agreement between experts (N = 12) in ordering the five behavioural vignettes in each domain.

		Agreement in rank-ordering behavioural vignettes			Range of ratings (0–100)
Domain	Abbrev	W^a	χ2	p-value	Range
Knowledge of the Discipline	Knowledge	.956	45.87	<.001	20–95
Ethical, Legal and Professional Matters	Ethics	.972	46.67	<.001	15–100
Psychological Assessment and Measurement	Assessment	.969	46.53	<.001	15–100
Intervention Strategies	Intervention	.985	47.27	<.001	15–100
Research and Evaluation	Research	1.000^b	24.00	<.001	20–100
Communication and Interpersonal Relationships	Interpersonal	1.000^c	24.00	<.001	20–100
Reflective Practice	Reflective	.967	42.54	<.001	15–100

^aW scores of 0–.20 = no agreement, .21–.39 = minimal, .40–.59 = weak, .60–.79 = moderate, .80–.90 = strong, and above .90 = almost perfect agreement.

^bRevised descriptors – Kendall's W for initial vignettes was 0.649, X2 = 31.1, p < .001, M = 70.9 (range = 25–100).

^cRevised descriptors – Kendall's W for initial vignettes was 0.839, X2 = 40.3, p < .001, M = 71.5 (range = 25–100).

Table 2. Descriptive statistics for supervisor and trainee ratings on the PsyCET-S for each competence domain.

	Supervisor ratings					Trainee ratings
	% scores					% scores
Domain	Min	Max	>50	Mean	SD	Min	Max	>50	Mean	SD
Knowledge	20	100	77.8	72.1	19.4	27	89	65.6	61.2	18.6
Ethics	35	100	84.8	77.6	18.5	30	100	73.4	66.9	17.3
Assessment	30	100	77.8	71.7	19.5	27	91	67.2	61.4	18.4
Intervention	16	100	78.6	71.6	20.4	9	90	62.5	58.7	21.0
Research	29	100	85.9	73.9	16.9	20	98	71.9	64.6	18.7
Interpersonal	8	100	84.8	77.0	20.6	20	92	74.6	66.7	19.0
Reflective	37	100	87.8	79.8	17.2	30	93	76.2	67.0	16.7

Knowledge = Knowledge of the Discipline; Ethics = Ethical, Legal and Professional Matters; Assessment = Psychological Assessment and Measurement; Intervention = Intervention Strategies; Research = Research and Evaluation; Interpersonal = Communication and Interpersonal Relationships; Reflective = Reflective Practice.

Figure 2. PsyCET-S competence ratings from supervisors across placement levels and domains.

Table 3. Changes in supervisor ratings on PsyCET-S between successive placements.

	Changes in scores between successive placements
	1st v 2nd placement (n = 14)			2nd v 3rd placement (n = 18)			3rd v 4th placement (n = 13)
Domain	M Diff (SD)	t	sig	M Diff (SD)	t	sig	M Diff (SD)	t	sig
Knowledge	29.5 (16.4)	6.96	<.001	12.0 (11.2)	4.67	<.001	1.1 (8.4)	.51	.309
Ethics	24.5 (23.7)	4.01	<.001	10.2 (15.0)	2.96	.004	0.6 (15.0)	.14	.445
Assessment	27.2 (18.7)	5.62	<.001	9.8 (11.2)	3.83	<.001	0.7 (18.0)	.15	.442
Intervention	31.1 (11.3)	10.61	<.001	11.7 (9.8)	5.24	<.001	.6 (11.6)	.21	.420
Research	21.1 (21.7)	3.76	.001	8.4 (9.6)	3.81	<.001	1.1 (15.5)	.28	.393
Interpersonal	31.3 (22.6)	5.38	<.001	9.1 (12.2)	3.23	.002	2.6 (13.8)	.70	.248
Reflective	21.6 (24.7)	3.28	.003	7.7 (9.6)	3.50	.001	0.8 (13.2)	.22	.413

Knowledge = Knowledge of the Discipline; Ethics = Ethical, Legal and Professional Matters; Assessment = Psychological Assessment and Measurement; Intervention = Intervention Strategies; Research = Research and Evaluation; Interpersonal = Communication and Interpersonal Relationships; Reflective = Reflective Practice.

Figure 3. PsyCET-S competence ratings from trainees across placement levels and domains.

Table 4. Changes in trainee self-ratings on PsyCET-S between successive placements.

	Changes in scores between successive placements
	1 v 2 (n = 9)			2 v 3 (n = 9)			3 v 4 (n = 7)
Domain	M Diff (SD)	t	sig	M Diff (SD)	t	sig	M Diff (SD)	t	sig
Knowledge	23.3 (17.3)	4.27	.001	11.6 (7.5)	4.88	<.001	7.4 (6.1)	3.42	.006
Ethics	18.6 (17.3)	3.39	.004	16.2 (13.9)	3.70	.002	1.7 (10.8)	.46	.330
Assessment	20.2 (13.7)	4.70	<.001	17.2 (10.6)	5.11	<.001	2.2 (9.2)	.69	.257
Intervention	26.5 (15.8)	5.31	<.001	15.7 (11.2)	4.43	<.001	4.8 (9.1)	1.52	.086
Research	23.3 (13.2)	5.57	<.001	13.0 (13.1)	3.15	.006	8.7 (6.0)	4.11	.002
Interpersonal	18.1 (22.8)	2.51	.017	17.3 (11.8)	4.65	<.001	1.1 (7.0)	.45	.332
Reflective	15.2 (18.9)	2.54	.016	12.5 (6.9)	5.70	<.001	4.6 (7.9)	1.52	.089

Knowledge = Knowledge of the Discipline; Ethics = Ethical, Legal and Professional Matters; Assessment = Psychological Assessment and Measurement; Intervention = Intervention Strategies; Research = Research and Evaluation; Interpersonal = Communication and Interpersonal Relationships; Reflective = Reflective Practice.

Data availability statement

The data that support the findings of this study are available from the corresponding author, [KL], upon reasonable request.