Identifying fluency parameters for a machine-learning-based automated interpreting assessment system

Figures & data

Figure 1. A model of identification of interpreting fluency parameters.

Table 1. List of 15 selected temporal measures of fluency and their brief definition.

Variables	Fluency Parameters	Definition and Calculation
Independent variables	Number of unfilled pauses* (NUF)	No. of unfilled pauses equal to and longer than 0.25 seconds, excluding the first pause at the very beginning of interpreting.
	Mean length of unfilled pauses (MLUP)	Sum of length of unfilled pauses / No. of unfilled pauses equal to or longer than 0.25 seconds
	Number of relatively long unfilled pauses* (NRLUP)	No. of unfilled pauses duration larger than Q3 + 1.5 * IQR and smaller than and equal to Q3 + 3 * IQR
	Number of particularly long unfilled pauses* (NPLUP)	No. of unfilled pauses duration larger than Q3 + 3 * IQR
	Number of filled pauses (NFP)	No. of filled pauses
	Mean length of filled pauses (MLFP)	Sum of length of filled pauses / No. of filled pauses
	Number of relatively slow articulation* (NRSA)	No. of the duration per syllable larger than Q3 + 1.5 * IQR and smaller than and equal to Q3 + 3 * IQR
	Number of particularly slow articulation* (NPSA)	No. of count of the duration per syllable larger than Q3 + 3 * IQR
	Number of relatively quick articulation* (NRQA)	No. of the duration per syllable smaller than Q1 − 1.5 * IQR and larger than and equal to Q1 − 3 * IQR
	Number of particularly quick articulation* (NPQA)	No. of the duration per syllable smaller than Q1 − 3 * IQR
Dependent variables	Phonation time ratio (PTR)	Total length of speaking time / (total length of speaking time + total length of filled and unfilled pauses)
	Articulation rate (AR)	No. of syllables / the duration of utterance, excluding pause time
	Speaking rate (SR)	No. of syllables in each sample speech / speaking duration, including pause time
	Mean length of runs (MLR)	Number of unfilled pauses equal to or longer than 0.25 seconds / number of syllables
	Average length pauses (ALP)	Sum duration of pauses equal to or longer than 0.25 seconds / the number of pauses.

Table 2. Data distribution based on a five-number summary.

Category	Mean	Std	Min	X25.	X50.	X75.	Max	IQR
Duration per Syllable	0.207	0.080	0.060	0.150	0.195	0.245	2.680	0.095
Unfilled Pause	0.454	0.591	0.030	0.080	0.250	0.610	8.990	0.530

Table 3. Thresholds to define outliers.

Category	Q1-3 times IQR	Q1-1.5 times IQR	Q3 + 1.5 times IQR	Q3 + 3 times IQR
Syllable Speak Duration	−0.135	0.007	0.388	0.530
Unfilled Pause	−1.510	−0.715	1.405	2.200

Figure 2. Correlation matrix chart. Note * p < 0.05, **p  < 0.01, ***p < 0.001.

Table 4. Regression models predicting overall assessment SCORE.

No.	Variables	Predictors	R^2	Adjusted R^2
1	Independent variables	MLFP + NPLUP	0.201	0.166
2		NRLUP + MLUP + MLFP	0.215	0.163
3	Dependent variables	SR	0.086	0.066
4		AR + PTR + MLR	0.124	0.066

Table

• Fluency and good pace • Articulation and pronunciation • Ability to engage with the audience • Stamina

Very Good (70–100) Engaging and communicative performance which conveyed the source speaker's emotions and intentions. Clear articulation and projection with little to no sign of nerves.

Good (60–69) Confident performance. Little to no hesitation or fillers in delivery. Clear articulation with good expression and projection. Frequent and appropriate eye contact.

Pass (50–59) Some hesitation but does not interrupt flow of performance. Clear articulation with only minor/infrequent fillers. Good frequency and length of eye contact. Good voice projection.

Poor (40–49) Some hesitancy in performance. Use of fillers and unclear articulation. Insufficient eye contact or engagement with the audience.

Very Poor (0–39) Hesitant performance. Frequent use of fillers. Lack of eye contact. Poor intonation/voice modulation. Inappropriate projection (too quiet/too loud). Lack of clear articulation.

Table

Parameter	Mean	SD	STDERR	Skewness	Kurtosis	Shapiro_Wilk statistic^2	sw_p_value
SCORE^♣	58.709	7.349	0.724	0.214	−1.061	0.956**	0.002
NFP^♣	6.359	8.899	0.877	3.295	15.223	0.660**	0.000
NPLUP^♣	7.942	7.746	0.763	1.508	1.874	0.821**	0.000
NRLUP^♣	12.854	8.365	0.824	1.188	2.416	0.922**	0.000
NUP^♣	117.825	43.058	4.243	0.651	0.488	0.967*	0.011
NPSA^♣	1.689	2.249	0.222	2.218	6.158	0.731**	0.000
NRSA^♣	22.262	14.198	1.399	0.815	−0.070	0.929**	0.000
MLUP	0.858	0.163	0.016	0.475	0.048	0.978	0.081
MLFP^♣	0.457	0.394	0.039	0.816	0.345	0.920**	0.000
ALP^♣	0.836	0.175	0.017	0.576	−0.334	0.958**	0.003
MLR	0.086	0.021	0.002	0.216	−0.512	0.987	0.394
AR^♣	4.371	0.434	0.043	0.300	−0.935	0.959**	0.003
SR^♣	3.270	0.469	0.046	0.284	−1.023	0.960**	0.003
PTR	0.705	0.058	0.006	0.224	−0.485	0.981	0.148

Note. ♣ Violated normality assumption, based on Shapiro-Wilk test; **p < 0.01; *p < 0.05.

Table

No.	Approach	Predictors	R^2	Adjusted R^2	Standard error
1	Best Subset Selection	NRSA	0.415	0.403	0.298
2		NPSA	0.264	0.248	0.421
3		NPSA + NRSA	0.429	0.404	0.299
4	Stepwise Selection	NFP + NUP + NRSA + MLUP	0.523	0.480	0.282

Table

No.	Approach	Predictors	R^2	Adjusted R^2	Standard error
1	Best Subset Selection	NRSA	0.417	0.404	0.324
2		NRLUP	0.358	0.344	0.339
3		NUP	0.346	0.333	0.344
4		NPLUP	0.319	0.305	0.384
5		NRSA + NRLUP	0.541	0.521	0.294
6		NRSA + NPLUP	0.527	0.506	0.297
7		NRSA + NUP	0.474	0.451	0.312
8		NPLUP + NUP	0.465	0.441	0.319
9		NPLUP + NRLUP	0.449	0.425	0.321
10		NRLUP + NUP	0.418	0.393	0.336
11		NRSA + NPLUP + NRLUP	0.584	0.556	0.290
12		NRSA + NPLUP + NUP	0.553	0.523	0.292
13		NRSA + NRLUP + NUP	0.543	0.512	0.293
14		NPLUP + NRLUP + NUP	0.491	0.458	0.318
15		NRSA + NPLUP + NRLUP + NUP	0.585	0.547	0.290
16	Stepwise Selection	NFP + NRSA + MLUP + MLFP + NUP	0.634	0.591	0.266

Table

No.	Approach	Predictors	R^2	Adjusted R^2	Standard error
1	Best Subset Selection	NUP	0.460	0.449	0.040
2		NRLUP	0.452	0.441	0.042
3		MLUP	0.410	0.398	0.043
4		NPLUP	0.393	0.380	0.044
5		MLUP + NUP	0.680	0.666	0.033
6		NPLUP + NUP	0.597	0.579	0.035
7		NPLUP + NRLUP	0.561	0.541	0.040
8		NRLUP + NUP	0.541	0.521	0.040
9		MLUP + NRLUP	0.530	0.509	0.041
10		NPLUP + MLUP	0.431	0.406	0.041
11		NPLUP + MLUP + NUP	0.685	0.664	0.030
12		MLUP + NRLUP + NUP	0.681	0.659	0.033
13		NPLUP + NRLUP + NUP	0.627	0.602	0.036
14		NPLUP + MLUP + NRLUP	0.561	0.532	0.040
15		NPLUP + MLUP + NRLUP + NUP	0.689	0.661	0.030
16	Stepwise Selection	NUP + NRSA + MLUP + MLFP	0.738	0.714	0.028

Table

No.	Approach	Predictors	R^2	Adjusted R^2	Standard error
1	Best Subset Selection	MLUP	0.840	0.836	0.067
2		NPLUP	0.659	0.652	0.114
3		NPLUP + MLUP	0.841	0.834	0.067
4	Stepwise Selection	MLUP	0.840	0.836	0.066

Table

No.	Approach	Predictors	R^2	Adjusted R^2	Standard error
1	Best Subset Selection	NUP	0.555	0.546	0.016
2		NRLUP	0.267	0.251	0.018
3		NRLUP + NUP	0.555	0.536	0.017
4	Stepwise Selection	NUP + MLFP	0.582	0.564	0.014