An examination of IELTS candidates’ performances at different band scores of the speaking test: A quantitative and qualitative analysis

Figures & data

Table 1. Mock IELTS Test candidates’ demographics

Demographics	Categories	Numbers
Gender	Males	19
	Females	40
Age-groups	21–29	45
	30–35	9
	35–39	5
Language Levels	B2	53
	C1	6

Table 2. Intra-rater correlations—First rater

		FlC (2^nd time)	LR (2^nd time)	GRA (2^nd time)	Pro (2^nd time)
FlC (1^st time)	Pearson Correlation	.997**	.441**	.116	.212
	Sig. (2-tailed)	.000	.000	.380	.107
	N	59	59	59	59
LR (1^st time)	Pearson Correlation	.439**	.997**	.276*	.214
	Sig. (2-tailed)	.001	.000	.034	.104
	N	59	59	59	59
GRA (1^st time)	Pearson Correlation	.113	.291*	.995**	.393**
	Sig. (2-tailed)	.394	.025	.000	.002
	N	59	59	59	59
Pro (1^st time)	Pearson Correlation	.230	.217	.364**	.991**
	Sig. (2-tailed)	.080	.098	.005	.000
	N	59	59	59	59

**Correlation is significant at the 0.01 level (2-tailed).

*Correlation is significant at the 0.05 level (2-tailed).

Table 3. Intra-rater correlations

		FlC (2^nd rater)	LR (2^nd rater)	GRA (2^nd rater)	Pro (2^nd rater)
FlC (1^st rater)	Pearson Correlation	.981**	.435**	.095	.209
	Sig. (2-tailed)	.000	.001	.474	.112
	N	59	59	59	59
LR (1^st rater)	Pearson Correlation	.407**	.989**	.279*	.223
	Sig. (2-tailed)	.001	.000	.032	.089
	N	59	59	59	59
GRA (1^st rater)	Pearson Correlation	.108	.260*	.988**	.386**
	Sig. (2-tailed)	.417	.046	.000	.003
	N	59	59	59	59
Pro (1^st rater)	Pearson Correlation	.240	.223	.379**	.977**
	Sig. (2-tailed)	.067	.090	.003	.000
	N	59	59	59	59

**Correlation is significant at the 0.01 level (2-tailed).

*Correlation is significant at the 0.05 level (2-tailed).

Table 4. Descriptive statistics of measured variables

	N	Minimum	Maximum	Mean	Std. Deviation
FlC	59	4.00	7.50	5.6610	.82232
LR	59	4.00	7.00	5.4831	.85584
GRA	59	4.00	7.00	5.7458	.93483
Pro	59	5.00	7.50	6.2712	.69059
Total	59	5.00	7.00	5.8305	.62014
Valid N (listwise)	59

Table 5. Descriptive statistics of tests of normality

	Skewness		Kurtosis
	Statistic	Std. Error	Statistic	Std. Error
FlC	.303	.311	−.643	.613
LR	.417	.311	−.585	.613
GRA	.030	.311	−1.293	.613
Pro	−.294	.311	−.966	.613
Total	.403	.311	−.575	.613

Figure 1. Normal P-P of the Regression Standardized Residual.

Table 6. Correlation between the Variables

		Total speaking score	FlC	LR	GRA	Pro
Pearson Correlation	Total speaking score	1.000	.671	.701	.653	.622
	FlC	.671	1.000	.445	.110	.225
	LR	.701	.445	1.000	.275	.227
	GRA	.653	.110	.275	1.000	.389
	Pro	.622	.225	.227	.389	1.000
Sig. (1-tailed)	Total speaking score	.	.000	.000	.000	.000
	FlC	.000	.	.000	.203	.043
	LR	.000	.000	.	.018	.042
	GRA	.000	.203	.018	.	.001
	Pro	.000	.043	.042	.001	.
N	Total speaking score	59	59	59	59	59
	FlC	59	59	59	59	59
	LR	59	59	59	59	59
	GRA	59	59	59	59	59
	Pro	59	59	59	59	59

Table 7. Residuals statistics

	Minimum	Maximum	Mean	Std. Deviation	N
Predicted Value	4.9701	7.0315	5.8305	.60747	59
Std. Predicted Value	−1.416	1.977	.000	1.000	59
Standard Error of Predicted Value	.023	.063	.036	.010	59
Adjusted Predicted Value	4.9676	7.0379	5.8282	.60907	59
Residual	−.34104	.28394	.00000	.12468	59
Std. Residual	−2.639	2.197	.000	.965	59
Stud. Residual	−2.752	2.367	.008	1.016	59
Deleted Residual	−.37091	.32936	.00231	.13838	59
Stud. Deleted Residual	−2.941	2.476	.009	1.037	59
Mahal. Distance	.785	12.644	3.932	2.710	59
Cook’s Distance	.000	.179	.023	.037	59
Centered Leverage Value	.014	.218	.068	.047	59

^aDependent Variable: Total speaking score.

Table 8. ANOVA Test

Model		Sum of Squares	df	Mean Square	F	Sig.
1	Regression	21.403	4	5.351	320.462	.000^b
	Residual	.902	54	.017
	Total	22.305	58

^aDependent Variable: Total speaking score.

^bPredictors: (Constant), Pro, FlC, GRA, LR.

Table 9. Model summary

Model	R	R Square	Adjusted R Square	Std. Error of the Estimate
1	.980^a	.960	.957	.12922

^aPredictors: (Constant), Pro, FlC, GRA, LR.

^bDependent Variable: Total speaking score.

Table 10. Coefficients in multiple regression analysis

Model		Unstandardized Coefficients		Standardized Coefficients	t	Sig.	95.0% Confidence Interval for B		Correlations			Collinearity Statistics
		B	Std. Error	Beta			Lower Bound	Upper Bound	Zero-order	Partial	Part	Tolerance	VIF
1	(Constant)	−.468	.186		−2.518	.015	−.840	−.095
	FlC	.308	.023	.408	13.186	.000	.261	.354	.671	.874	.361	.782	1.279
	LR	.248	.023	.343	10.834	.000	.202	.294	.701	.828	.296	.748	1.336
	GRA	.264	.020	.398	13.077	.000	.224	.305	.653	.872	.358	.808	1.237
	Pro	.268	.027	.298	9.817	.000	.213	.322	.622	.801	.269	.812	1.231

^aDependent Variable: Total speaking score.

Table 11. Frequency of key factors in Coherence

			Categories of Criteria		Total
			FlC1	FlC2
Band score	5	Count	47	4	51
		% within Band score	92.2%	7.8%	100.0%
		% within Categories of Criteria	61.0%	30.8%	56.7%
		% of Total	52.2%	4.4%	56.7%
	6	Count	20	4	24
		% within Band score	83.3%	16.7%	100.0%
		% within Categories of Criteria	26.0%	30.8%	26.7%
		% of Total	22.2%	4.4%	26.7%
	7	Count	10	5	15
		% within Band score	66.7%	33.3%	100.0%
		% within Categories of Criteria	13.0%	38.5%	16.7%
		% of Total	11.1%	5.6%	16.7%
Total		Count	77	13	90
		% within Band score	85.6%	14.4%	100.0%
		% within Categories of Criteria	100.0%	100.0%	100.0%
		% of Total	85.6%	14.4%	100.0%

^aCriteria = FlC (Coherence Only).

Table 12. Frequency of key factors in Lexical Resource

			Categories of Criteria				Total
			LR1	LR2	LR3	LR4
Band score	5	Count	107	36	62	0	205
		% within Band score	52.2%	17.6%	30.2%	0.0%	100.0%
		% within Categories of Criteria	58.2%	69.2%	70.5%	0.0%	62.9%
		% of Total	32.8%	11.0%	19.0%	0.0%	62.9%
	6	Count	60	9	17	2	88
		% within Band score	68.2%	10.2%	19.3%	2.3%	100.0%
		% within Categories of Criteria	32.6%	17.3%	19.3%	100.0%	27.0%
		% of Total	18.4%	2.8%	5.2%	0.6%	27.0%
	7	Count	17	7	9	0	33
		% within Band score	51.5%	21.2%	27.3%	0.0%	100.0%
		% within Categories of Criteria	9.2%	13.5%	10.2%	0.0%	10.1%
		% of Total	5.2%	2.1%	2.8%	0.0%	10.1%
Total		Count	184	52	88	2	326
		% within Band score	56.4%	16.0%	27.0%	0.6%	100.0%
		% within Categories of Criteria	100.0%	100.0%	100.0%	100.0%	100.0%
		% of Total	56.4%	16.0%	27.0%	0.6%	100.0%

^aCriteria = LR.

Table 13. Frequency of key factors in Grammar Range and Accuracy

			Categories of Criteria				Total
			GRA1	GRA2	GRA3	GRA4
Band score	5	Count	144	1	26	0	171
		% within Band score	84.2%	0.6%	15.2%	0.0%	100.0%
		% within Categories of Criteria	57.1%	16.7%	47.3%	0.0%	54.1%
		% of Total	45.6%	0.3%	8.2%	0.0%	54.1%
	6	Count	82	2	19	0	103
		% within Band score	79.6%	1.9%	18.4%	0.0%	100.0%
		% within Categories of Criteria	32.5%	33.3%	34.5%	0.0%	32.6%
		% of Total	25.9%	0.6%	6.0%	0.0%	32.6%
	7	Count	26	3	10	3	42
		% within Band score	61.9%	7.1%	23.8%	7.1%	100.0%
		% within Categories of Criteria	10.3%	50.0%	18.2%	100.0%	13.3%
		% of Total	8.2%	0.9%	3.2%	0.9%	13.3%
Total		Count	252	6	55	3	316
		% within Band score	79.7%	1.9%	17.4%	0.9%	100.0%
		% within Categories of Criteria	100.0%	100.0%	100.0%	100.0%	100.0%
		% of Total	79.7%	1.9%	17.4%	0.9%	100.0%

^aCriteria = GRA.

Table 14. Frequency of key factors in Pronunciation

			Categories of Criteria			Total
			Pro1	Pro2	Pro3
Band score	5	Count	95	35	55	185
		% within Band score	51.4%	18.9%	29.7%	100.0%
		% within Categories of Criteria	61.3%	53.8%	67.1%	61.3%
		% of Total	31.5%	11.6%	18.2%	61.3%
	6	Count	43	13	19	75
		% within Band score	57.3%	17.3%	25.3%	100.0%
		% within Categories of Criteria	27.7%	20.0%	23.2%	24.8%
		% of Total	14.2%	4.3%	6.3%	24.8%
	7	Count	17	17	8	42
		% within Band score	40.5%	40.5%	19.0%	100.0%
		% within Categories of Criteria	11.0%	26.2%	9.8%	13.9%
		% of Total	5.6%	5.6%	2.6%	13.9%
Total		Count	155	65	82	302
		% within Band score	51.3%	21.5%	27.2%	100.0%
		% within Categories of Criteria	100.0%	100.0%	100.0%	100.0%
		% of Total	51.3%	21.5%	27.2%	100.0%

^aCriteria = Pro.

Table 15. Monte Carlo test to compare the band scores 5, 6, and 7 in terms of the frequency of Coherence error types

	Value	df	Asymp. Sig. (2-sided)	Monte Carlo Sig. (2-sided)
				Sig.	99% Confidence Interval		Cramer’s V
					Lower Bound	Upper Bound	Value	Approx. Sig.
Pearson Chi-Square	6.225^b	2	.044	.039 ^c	.034	.043	.263	.044
Likelihood Ratio	5.567	2	.062	.076 ^c	.070	.083
Fisher’s Exact Test	5.844			.040 ^c	.035	.045
Linear-by-Linear Association	5.961^d	1	.015	.016 ^c	.013	.020
N of Valid Cases	90

Table 16. Monte Carlo test to compare the band scores 5, 6, and 7 in terms of the frequency of error types in Lexical Resource

	Value	df	Asymp. Sig. (2-sided)	Monte Carlo Sig. (2-sided)			Cramer’s V
				Sig.	99% Confidence Interval
					Lower Bound	Upper Bound	Value	Approx. Sig.
Pearson Chi-Square	13.792^b	6	.032	.033 ^c	.029	.038	.145	.032
Likelihood Ratio	13.891	6	.031	.029 ^c	.025	.033
Fisher’s Exact Test	12.718			.035 ^c	.030	.040
Linear-by-Linear Association	1.157^d	1	.282	.289 ^c	.277	.300
N of Valid Cases	326

Table 17. Monte Carlo test to compare the band scores 5, 6, and 7 in terms of the frequency of error types in Grammar

	Value	df	Asymp. Sig. (2-sided)	Monte Carlo Sig. (2-sided)
				Sig.	99% Confidence Interval		Cramer’s V
					Lower Bound	Upper Bound	Value	Approx. Sig.
Pearson Chi-Square	30.844^b	6	.000	.000 ^c	.000	.000	.221	.000
Likelihood Ratio	21.477	6	.002	.002 ^c	.001	.003
Fisher’s Exact Test	19.866			.001 ^c	.000	.002
Linear-by-Linear Association	8.703^d	1	.003	.003 ^c	.002	.005
N of Valid Cases	316

Table 18. Chi-Square test to compare the band scores 5, 6, and 7 in terms of the frequency of error types in Pronunciation

	Value	df	Asymp. Sig. (2-sided)	Cramer’s V
				Value	Approx. Sig.
Pearson Chi-Square	6.225^b	2	.044	.137	.024
.024Likelihood Ratio	5.567	2	.062
Linear-by-Linear Association	5.961	1	.015
N of Valid Cases	90

Table

Band	Fluency and Coherence	Lexical Resource	Grammatical range and accuracy	Pronunciation
9	• speaks fluently with only rare repetition or self-correction; • any hesitation is content-related rather than to find words or grammar • speaks coherently with fully appropriate cohesive features • develops topics fully and appropriately	• uses vocabulary with full flexibility and precision in all topics • uses idiomatic language naturally and accurately	• uses a full range of structures naturally and appropriately • produces consistently accurate structures apart from “slips” characteristic of native speaker speech	• uses a full range of pronunciation features with precision and subtlety • sustains flexible use of features throughout • is effortless to understand
8	• speaks fluently with only occasional repetition or self-correction; hesitation is usually content-related and only rarely to search for language • develops topics coherently and appropriately	• uses a wide vocabulary resource readily and flexibly to convey precise meaning • uses less common and idiomatic vocabulary skillfully, with occasional inaccuracies • uses paraphrase effectively as required	• uses a wide range of structures flexibly • produces a majority of error-free sentences with only very occasional inappropriacies or basic/non-systematic errors	• uses a wide range of pronunciation features • sustains flexible use of features, with only occasional lapses • is easy to understand throughout; L 1 accent has minimal effect on intelligibility
7	• speaks at length without noticeable effort or loss of coherence • may demonstrate language-related hesitation at times, or some repetition and/or self-correction • uses a range of connectives and discourse markers with some flexibility	• uses vocabulary resource flexibly to discuss a variety of topics • uses some less common and idiomatic vocabulary and shows some awareness of style and collocation, with some inappropriate choices • uses paraphrase effectively	• uses a range of complex structures with some flexibility • frequently produces error-free sentences, though some grammatical mistakes persist	• shows all the positive features of Band 6 and some, but not all, of the positive features of Band 8
6	• is willing to speak at length, though may lose coherence at times due to occasional repetition, self-correction or hesitation • uses a range of connectives and discourse markers but not always appropriately	• has a wide enough vocabulary to discuss topics at length and make meaning clear in spite of inappropriacies • generally paraphrases successfully	• uses a mix of simple and complex structures, but with limited flexibility • may make frequent mistakes with complex structures though these rarely cause comprehension problems	• uses a range of pronunciation features with mixed control • shows some effective use of features but this is not sustained • can generally be understood throughout, though mispronunciation of individual words or sounds reduces clarity at times
5	• usually maintains flow of speech but uses repetition, self correction and/or slow speech to keep going	• manages to talk about familiar and unfamiliar topics but uses vocabulary with limited flexibility	• produces basic sentence forms with reasonable accuracy • uses a limited range of more complex structures, but these	• shows all the positive features of Band 4 and some, but not all, of the positive features of Band 6
	• may over-use certain connectives and discourse markers • produces simple speech fluently, but more complex communication causes fluency problems	• attempts to use paraphrase but with mixed success	usually contain errors and may cause some comprehension problems
4	• cannot respond without noticeable pauses and may speak slowly, with frequent repetition and self-correction • links basic sentences but with repetitious use of simple connectives and some breakdowns in coherence	• is able to talk about familiar topics but can only convey basic meaning on unfamiliar topics and makes frequent errors in word choice • rarely attempts paraphrase	• produces basic sentence forms and some correct simple sentences but subordinate structures are rare • errors are frequent and may lead to misunderstanding	• uses a limited range of pronunciation features • attempts to control features but lapses are frequent • mispronunciations are frequent and cause some difficulty for the listener
3	• speaks with long pauses • has limited ability to link simple sentences • gives only simple responses and is frequently unable to convey basic message	• uses simple vocabulary to convey personal information • has insufficient vocabulary for less familiar topics	• attempts basic sentence forms but with limited success, or relies on apparently memorised utterances • makes numerous errors except in memorised expressions	• shows some of the features of Band 2 and some, but not all, of the positive features of Band 4
2	• pauses lengthily before most words • little communication possible	• only produces isolated words or memorised utterances	• cannot produce basic sentence forms	• Speech is often unintelligible
1	• no communication possible • no ratable language
0	• does not attend