An investigation of a nationwide exam from a critical language testing perspective

Figures & data

Table 1. Structure and themes of students’ questionnaire

Structure	Contents	Items
Theme 1	Demographic features: • Age, sex, major, place of university • Participation in the exam	5
Theme 2	The amount of information about the: • test makers, the time that the test was made, the main sources of the exam questions, the degree of their involvement and control in test development, provision of information about the circumstances of different universities by the ministry of higher education	6
Theme 3	Students’ viewpoints: • Whether they prefer to be tested by their own university as opposed to a centralized testing method • Whether or not the exam questions should match their learning styles and individual differences • Whether they prefer the present testing method (multiple –choice) as opposed to open-ended questions • Their overall satisfaction with the quality of study materials in BA courses as the main sources for the exam, the time required to complete the test, the feedback they receive on their test results, the quality of knowledge they gain from the test • The extent to which they thought the test provided a true and correct reflection and indication of their actual knowledge • Aspects of the TEFL MAEE that can be improved	13

Table 2. Major categories, themes, and coding schemes

TEFL MA students’ viewpoints about:	Theme code
Test makers, the time that the test was made,	T1
The main sources of the exam questions	T2
The involvement and control of different test parties	T3
The present testing method of the exam	T4
(multiple-choice) items as an appropriate method	T5
Improvement of any aspects of the TEFLMAEE	T6
Being tested by their own university as opposed to a centralized testing method	T7

Table 3. Test statistics: awareness of the main sources of the exam questions

	Observed value	Expected value	Percent
Have no idea	2	12	3.3
Never/strongly disagree	30	12	50.0
Seldom/slightly agree	11	12	18.3
Frequently/often agree	8	12	13.3
Always/strongly agree	9	12	15.0
Total	60		100
χ^2	37.50

Notes: d.f = 4; p = 0.05; χ² = 9.49.

Table 4. Test statistics: percentage, appropriateness of multiple choice format

	Observed value	Expected value	Percent
Have no idea	12	12	20.0
Never/strongly disagree	10	12	16.7
Seldom/slightly agree	22	12	36.7
Frequently/often agree	9	12	15.0
Always/strongly agree	7	12	11.7
Total	60		100
χ^2	11.50

Notes: d.f = 4; p = 0.05; χ² = 9.49.

Table 5. Test statistics: the match between exam questions and learning styles and individual differences

	Observed value	Expected value	Percent
Have no idea	7	12	11.7
Never/strongly disagree	21	12	35.0
Seldom/slightly agree	11	12	18.3
Frequently/often agree	12	12	20.0
Always/strongly agree	9	12	15.0
Total	60		100
χ^2	9.67

Notes: d.f = 4; p = 0.05; χ² = 9.49.

Figure 1. Percentages, information about test makers.

Figure 2. Percentages, information about the time that the test was made.

Table 6. Test statistics: involvement and control of different test parties (students and university teachers) in test development

	Observed value	Expected value	Percent
Have no idea	8	12	13.3
Never/strongly disagree	2	12	3.3
Seldom/slightly agree	9	12	15.0
Frequently/often agree	21	12	35.0
Always/strongly agree	20	12	33.3
Total	60		100
χ^2	22.50

Notes: d.f = 4, p = 0.05; χ² = 9.49.

Table 7. Test statistics: preference to be tested by their own university as opposed to a centralized test

	Observed value	Expected value	Percent
Have no idea	2	12	3.3
Never/strongly disagree	5	12	8.3
Seldom/slightly agree	16	12	26.7
Frequently/often agree	23	12	38.3
Always/strongly agree	14	12	23.3
Total	60		100
χ^2	24.17

Notes: d.f = 4; p = 0.05; χ² = 9.49.

Table 8. Test Statistics: overall satisfaction with the test

	Observed value	Expected value	Percent
Have no idea	3	12	5.0
Never/strongly disagree	6	12	10.0
Seldom/slightly agree	9	12	15.0
Frequently/often agree	15	12	25.0
Always/strongly agree	27	12	45.0
Total	60		100
χ^2	30.00