Artificial intelligence pedagogical chatbots as L2 conversational agents

Figures & data

Table 1. Demographic information.

Age		Duration of University Education		L2 Proficiency
Years	N (%)	Years	N (%)	Proficiency	N (%)
18	15 (10)	First	28 (20)	Low-intermediate	10 (7)
19	17 (12)	Second	37 (26)	Intermediate	49 (34)
20	35 (24)	Third	63 (44)	High-intermediate	62 (43)
21	46 (32)	Fourth	10 (7)	Advanced	22 (15)
22	18 (13)	Fifth	5 (3)
23	6 (4)
24-29	6 (4)

Figure 1. Study procedures.

Figure 2. Learners’ perspectives on the pedagogical chatbot.

Table 2. Descriptive statistics of learners’ perspectives on the pedagogical chatbot.

Item	Strongly Agree N (%)	Agree N (%)	Neutral N (%)	Disagree N (%)	Strongly Disagree N (%)	M	SD
1. I like chatting with the chatbot.	31 (22)	79 (55)	11 (8)	20 (14)	2 (1)	3.82	.976
2. The chatbot is a good conversational partner.	18 (13)	73 (51)	20 (14)	30 (21)	2 (1)	3.52	1.006
3. The chatbot can successfully keep a conversation going for a long time.	16 (11)	26 (18)	31 (22)	50 (35)	20 (14)	2.78	1.224
4. Using the chatbot in L2 is interesting.	44 (31)	66 (46)	17 (12)	14 (10)	2 (1)	3.95	.974
5. Chatting with the chatbot can improve my L2.	50 (35)	60 (42)	15 (10)	15 (10)	3 (2)	3.97	1.034
6. Chatting with the chatbot helps me to practice my L2.	64 (45)	59 (41)	12 (8)	8 (6)	0	4.25	.835
7. Chatting with the chatbot can help me to improve my L2 writing.	76 (53)	52 (36)	8 (6)	6 (4)	1 (1)	4.37	.828
8. I can learn many aspects of L2 by the chatbot’s responses to my language-related questions.	28 (20)	67 (47)	23 (16)	23 (16)	2 (1)	3.67	1.012
9. The chatbot can effectively explain L2 grammar and vocabulary.	14 (10)	58 (41)	39 (27)	30 (21)	2 (1)	3.36	.968
10. Chatting with the chatbot is more effective for L2 learning than chatting with my classmates.	14 (10)	36 (25)	30 (21)	31 (22)	32 (22)	2.78	1.312
11. The chatbot is a valuable L2 learning tool.	23 (16)	82 (57)	20 (14)	16 (11)	2 (1)	3.76	.906
12. I can ask the chatbot questions that I cannot ask people.	46 (32)	37 (26)	19 (13)	34 (24)	7 (5)	3.57	1.292
13. I feel more comfortable and less stressed chatting with the chatbot than chatting with humans.	29 (20)	38 (27)	24 (17)	32 (22)	20 (14)	3.17	1.358
14. I do not worry about making writing mistakes when I chat with the chatbot.	59 (41)	54 (38)	8 (6)	16 (11)	6 (4)	4.01	1.141
15. Using the chatbot motivates me to practice my L2 writing.	58 (41)	60 (42)	8 (6)	16 (11)	1 (1)	4.10	.984
16. I am willing to continue using the chatbot to learn and practice my L2.	33 (23)	44 (31)	28 (20)	23 (16)	15 (10)	3.40	1.290
17. It is easy to chat with the chatbot.	67 (47)	65 (45)	5 (3)	6 (4)	0	4.35	.744
18. The chatbots’ responses were relevant to the text I wrote.	6 (4)	48 (34)	39 (27)	38 (27)	12 (8)	2.99	1.055
19. The chatbot can understand my L2.	46 (32)	72 (50)	12 (8)	12 (8)	1 (1)	4.05	.899
20. The chatbot’s language is easy to understand.	74 (52)	55 (38)	9 (6)	5 (3)	0	4.38	.759
Overall Mean						3.71	1.029

n = 143.

Table 3. Ranks of the usefulness of the chatbot-mediated interactions for L2 practice.

Proficiency group	Mean Rank	Sum of Ranks
Intermediate	43.87	2149.50
High-intermediate	55.13	2701.50

n = 49 for each group.

Table 4. Thematic categories and corresponding frequencies of responses in the first theme.

Theme	Category	Students N (%)
The pedagogical chatbot and L2 learning	Usefulness	9 (69)
	L2 writing practice	10 (77)
	L2 improvement	10 (77)
	Attention to linguistic forms	9 (69)
	Attention to linguistic forms (advantage)	3 (23)
	Willingness to use	7 (54)
	Unwillingness to use	4 (31)

Table 5. Thematic categories and corresponding frequencies of responses in the second theme.

Theme	Category	Students N (%)
Affordances of the chatbot	Effortlessness	13 (100)
	Responsiveness	6 (46)
	Novelty	11 (85)
	Interest	5 (38)
	Language quality	12 (92)
	Acceptable conversational ability	5 (38)
	Acceptable instructional ability	4 (31)

Table 6. Thematic categories and corresponding frequencies of responses in the second theme.

Theme	Category	Students N (%)
Limitations of the chatbot	Unresponsiveness	5 (38)
	Decrease in motivation	5 (38)
	Limited conversational ability	8 (61)
	Limited instructional ability	6 (46)
	Questions formation	5 (38)
	Irrelevant responses	4 (31)
	Heightened focus on linguistic forms	4 (31)

Table 7. Thematic categories and corresponding frequencies of responses in the third theme.

Theme	Category	Students N (%)
The chatbot and L2 proficiency	Minimum L2 proficiency needed	6 (46)
	Suitability for beginners	3 (23)
	Suitability for intermediate learners	6 (46)