Designing public financial management systems: exploring the use of chatbot-assisted case studies

Figures & data

Table 1. Number of messages sent to the chatbot.

Number of messages sent to the chatbot	Number of students	Student clue
1	3	s1/male, s2/male, s3/male
2	1	s4/female
3	2	s5/male, s6/male
4	2	s7/male, s8/male
6	1	s9/male
8	1	s10/female
14	1	s11/female
20	1	s12/male
21	1	s13/male
	13

Table 2. Questions posed by category of areas of the problem space, with examples of questions asked by students and total number of questions.

Category No.	Category of the problem space	Examples of questions asked	No. of questions
(i)	General features of the problem scenario	What are sources of dissatisfaction with the payment process? What would you like to be done or improved?	9
(ii)	Identification of the entities involved in the problem scenario	Who are the parties involved in the payment process? What is the treasury division?	10
(iii)	Features of the invoice payment process	How does the payment process work? Who makes the transfer orders?	18
(iv)	Reasons and explanations for the activities performed in the invoice payment process	Why does the Junta of Andalucía hold accounts in several banks? Why did it take so long to make payments?	3
(v)	Modalities of execution of the activities	How are invoices matched with orders? How are payments tracked?	13
(vi)	Timing and duration of activities	How long on average does it take to pay invoices? How long does it take for the banks to make payments?	3

Table 3. Questions unanswered by the chatbot by type of failure, with examples of questions asked by students and total number of questions.

Issue type No.	Type of issue	Examples of questions asked	No. of questions
(i)	Use of pronouns	Why is this done in this way? How can this be more efficient?	3
(ii)	Incomplete sentence	PO? Chiefs?	2
(iii)	Phrasing of the sentence	Who can make a transfer order? (The question was then rephrased as: Who makes the transfer orders?) Sources of dissatisfactory performance? (The question was then rephrased as: What are the sources of dissatisfaction?)	5
(iv)	Question outside the knowledge domain of the chatbot	How can the process be made more efficient? Why wasn’t the payment process fixed before?	2