Investigating the validity of oral assessment rater training program: A mixed-methods study of raters’ perceptions and attitudes before and after training

Figures & data

Table 1. Rater background characteristics

Raters	N	Male	Female	Mean age	Rating experience	Teaching experience	Rater training	Relevant courses passed
NEW	10	5	5	41.2	0.8	3.7	0.3	2.4
OLD	10	5	5	31.7	3.4	14.2	4.1	4.7

Table 2. Exploratory factor analysis of raters’ attitudes and perceptions (pre-training)

	Factor
	1	2	3
1. I feel comfortable with trying a face-to-face rater training program	0.353	0.812^a	−0.196
2. I think I am going to enjoy the rater training experience	0.254	0.837^a	−0.381
3. Generally, I support the notion that we need to assess the language proficiency of students’ speaking	0.860^a	−0.305	−0.208
4. It’s not necessary to have some sort of formal assessment and training process to ensure comparability of standards	0.783^a	0.328	0.302
5. Do you anticipate any problems with the training program?	0.035	0.136	0.048
6. How effective do you think the rater training program will be?	0.694^a	−0.397	−0.355
7. I am flexible and I do accept authorities’ comments in rating even if they are against mine	0.336	−0.313	0.510^a
8. It is difficult for me to notice my mistakes in rating	0.332	0.161	0.715^a

Note: Extraction method: Principal axis factoring.

^a Three components extracted.

Bold values indicate the factor under which each item of the questionnare has had significant loading.

Table 3. Principle axis factoring of raters’ perceptions of the training program (pre-training)

	Initial Eigenvalues
Factor	Total	% of Variance	Cumulative %
1	2.860	42.127	42.127
2	1.594	19.925	69.451
3	1.103	13.792	83.243
4	0.706	8.831	92.074
5	0.586	6.752	94.334
6	0.350	4.378	96.451
7	0.197	2.461	98.913
8	0.087	1.087	100.000

Note: Extraction method: Principal axis factoring.

Table 4. ANOVA analysis of factor scores of raters’ perceptions of the training program (pre-training)

	Sum of squares	df	Mean square	F	Sig.
Between groups	78.844	2	39.422	4.842	0.019
Within groups	170.965	21	8.141
Total	249.809	23

Table 5. Confirmatory factor analysis of raters’ attitudes and perceptions (pre-training)

	Factor
	1	2	3
1. I feel comfortable with trying a face-to-face rater training program		0.851
2. I think I am going to enjoy the rater training experience		0.896
3. Generally, I support the notion that we need to assess the language proficiency of students’ speaking	0.947
4. It’s not necessary to have some sort of formal assessment and training process to ensure comparability of standards	0.731
5. Do you anticipate any problems with the training program?
6. How effective do you think the rater training program will be?	0.801
7. I am flexible and I do accept authorities’ comments in rating even if they are against mine			0.634
8. It is difficult for me to notice my mistakes in rating			0.765

CFI: 0.893.

TLI: 0.886.

RMSEA: 0.591.

SRMR: 0.790.

Table 6. Interfactor correlation coefficient of the raters’ perceptions of the training program (pre-training)

		F1	F2	F3
F1	Pearson Correlation	1	0.077	0.001
	Sig. (2-tailed)		0.856	0.998
	N	8	8	8
F2	Pearson Correlation	0.077	1	0.138
	Sig. (2-tailed)	0.856		0.745
	N	8	8	8
F3	Pearson Correlation	0.001	0.138	1
	Sig. (2-tailed)	0.998	0.745
	N	8	8	8

Table 7. Exploratory factor analysis of raters’ attitudes and perceptions (post-training)

	Factor
	1	2	3
1. Altogether how effective did you find the face-to-face training program?	0.876^a	−0.159	−0.262
2. Overall, how friendly (trainer-trainee) did you find the face-to-face training program?	0.339	0.778^a	0.268
3. The rater trainer was tense	0.377	0.655^a	−0.262
4. I could talk to the trainer easily	0.327	0.591^a	0.291
5. The trainer used understandable language	0.788^a	0.188	−0.163
6. How much did you enjoy your face-to-face training experience?	0.331	−0.387	0.469^a
7. How much do you think the face-to-face training program achieved its purpose?	0.882^a	−0.024	−0.144
8. How much material descriptors, (e.g. scripts, notes, etc.) was used in the program?	0.693^a	−0.518	−0.236
9. How much, do you think, your method of rating changed as a result of face-to-face training program?	0.670^a	−0.510	−0.010
10. How effective did you find the feedback given to you during the norming session?	0.516^a	0.187	0.204

Note: Extraction Method: Principal Axis Factoring.

^a Three components extracted.

Bold values indicate the factor under which each item of the questionnare has had significant loading.

Table 8. Principle axis factoring of raters’ perceptions of the training program (post-training)

Factor	Initial Eigenvalues
	Total	% of Variance	Cumulative %
1	4.930	46.306	46.306
2	1.545	15.455	64.755
3	1.188	11.878	76.633
4	0.704	7.035	83.668
5	0.480	4.804	88.472
6	0.397	3.971	92.443
7	0.308	3.083	95.526
8	0.208	2.079	97.606
9	0.130	1.296	98.902
10	0.110	1.098	100.000

Note: Extraction Method: Principal Axis Factoring.

Table 9. ANOVA analysis of factor scores of raters’ perceptions of the training program (post-training)

	Sum of squares	df	Mean square	F	Sig.
Between groups	1.877	2	0.938	7.114	0.004
Within groups	3.165	27	0.132
Total	5.042	29

Table 10. Confirmatory factor analysis of raters’ attitudes and perceptions (post-training)

	Factor
	1	2	3
1. Altogether how effective did you find the face-to-face training program?	0.908
2. Overall, how friendly (trainer-trainee) did you find the face-to-face training program?		0.851
3. The rater trainer was tense		0.688
4. I could talk to the trainer easily		0.613
5. The trainer used understandable language	0.741
6. How much did you enjoy your face-to-face training experience?			0.574
7. How much do you think the face-to-face training program achieved its purpose?	0.819
8. How much material descriptors, (e.g. scripts, notes, etc.) was used in the program?	0.723
9. How much, do you think, your method of rating changed as a result of face-to-face training program?	0.746
10. How effective did you find the feedback given to you during the norming session?	0.552

CFI: 0.911.

TLI: 0.906.

RMSEA: 0.579.

SRMR: 0.781.

Table 11. Interfactor correlation coefficient of the raters’ perceptions of the training program (post-training)

		F1	F2	F3
F1	Pearson Correlation	1	0.839**	0.675
	Sig. (2-tailed)		0.009	0.066
	N	10	10	10
F2	Pearson Correlation	0.839**	1	0.365
	Sig. (2-tailed)	0.009		0.374
	N	10	10	10
F3	Pearson Correlation	0.675	0.365	1
	Sig. (2-tailed)	0.066	0.374
	N	10	10	10

** Correlation is significant at the 0.01 level (2-tailed).

Table 12. Relationship between raters’ questionnaire and interview responses and their rating behavior change

	Severity		Bias		Consistency
	N	%	N	%	N	%
Feedback successful/positive about feedback	13	65	14	70	14	70
Feedback successful/negative about feedback	0	0	3	15	5	25
Feedback unsuccessful/positive about feedback	0	0	0	0	0	0
Feedback unsuccessful/negative about feedback	7	35	3	15	1	5