How to provide automated feedback on the writing process? A participatory approach to design writing analytics tools

Figures & data

Table 1. Example of the use case for an intervention provided to the participants.

Question	Example
Context of the intervention	I have to complete a writing assignment within a specified word limit
When	I am working on the assignment, and I exceed the word limit
What	An automatic tool within the word processing software
Who (Addresses)	Me
How (By)	Providing a pop-up stating that I exceeded the word limit and have to cut some words before submitting the assignment
Why (Outcome)	To make sure I will not submit a writing assignment which is too long.

Table 2. Topics identified in the bachelor students focus group (N = 5).

Topic	Sticky note example	Who	Number of sticky notes (participants)	Number of votes (participants)
Typing patterns	Information on typos, grammatical issues	S, T	7 (3)	3 (3)
Planning	What my first ideas were, before I started, to understand it better	T	6 (3)	3 (3)
General structure	Number of words	S	3 (2)	3 (3)
Draft version	The parts (paragraphs) that I struggled with	NT	3 (2)	2 (2)
Sentence structure	How often do you change your sentences	S	3 (2)	2 (2)
Analysis of text	Which mistakes are being made the most	T	8 (2)	1 (1)
Register	Highlight words that are part of the academic register	S	2 (1)	1 (1)
Time	How much time I spend making on making my assignment	NT	4 (3)	0 (0)
Errors and judgement	How many mistakes you make in one writing assignment	NT	3 (3)	0 (0)
Uncategorized	Summary graphs, statistics	S	1 (1)	0 (0)

Note. ‘Who’ indicates who should (not = N) have access to the data: T = teacher, S = student.

Table 3. Topics identified in the PhD students focus group (N = 6).

Topic	Sticky note example	Who	Number of sticky notes (participants)	Number of votes (participants)
Planning time	What time of day do I do my best writing and is this consistent/predictable for me	S, T	2 (1)	4 (4)
Number of revisions	Number of revisions per paragraph	NT, S, T	8 (4)	2 (2)
Supervisor effect	How affected am I by previous reviews	NT, T	6 (3)	2 (2)
Empty screen time	Total time spent in general (staring at it)	NT	5 (4)	2 (2)
Speed of writing	How much time did I spend on each sentence	S, T	4 (4)	2 (2)
Content of the revisions	Type of revisions in terms of sentence structure or paragraph structure, content	S	3 (3)	2 (2)
Planning structure	In which order do I write the typical research paper?	S, T	5 (5)	1 (1)
Change over time	Change in writing over time, does it speed up/improve	S	1 (1)	0 (0)
Uncategorized	How did I decide what info made it to the text (esp. literature section)?	T	1 (1)	2 (2)
Uncategorized	Style/discipline style. More or less passive.	T	1 (1)	1 (1)

Note. ‘Who’ indicates who should (not = N) have access to the data: T = teacher, S = student.

Table 4. Topics identified in the teachers focus group (N = 4).

Topic	Sticky note example	Who	Number of sticky notes (participants)	Number of votes (participants)
Making changes in language	Show instances where the student can improve language that is not necessarily incorrect	S	7 (3)	3 (3)
Linearity of the writing process	How linear their writing process is	T	3 (3)	3 (3)
Use of feedback during revision	To what extent students make significant changes to draft versions	T	5 (4)	2 (2)
Just do it!	Never start with the opening sentence. Just write!	S	5 (2)	1 (1)
Making changes in content	Where and how their argument works/does not work	S	3 (3)	1 (1)
Students’ perceptions of writing	Which aspects of writing they enjoy	T	3 (2)	1 (1)
Using peer feedback	How to give correct feedback on texts of other students	S	1 (1)	1 (1)
How students start the writing process	Do writing plans really help the student?	T	6 (4)	0 (0)
The pre-writing stage	How to start writing. E.g. finding resources and how to start	S	3 (2)	0 (0)
Using genre conventions	How would students define an academic writing style?	T	1 (1)	0 (0)

Note. ‘Who’ indicates who should have access to the data: T = teacher, S = student.

Table 5. Topics identified in the professional development staff focus group (N = 6).

Topic	Sticky note example	Who	Number of sticky notes (participants)	Number of votes (participants)
Reading into writing	How do students integrate the literature/evidence?	T	7 (6)	7 (6)
Using evidence	How to synthesize sources into writing?	S	4 (4)	3 (3)
Understanding the task	Is the student answering the question in the assignment?	S	2 (1)	3 (3)
Understanding the process	What steps do I go through when writing an assessment?	S	8 (5)	2 (2)
Planning, organizing ideas	How students’ ideas/concepts develop as they plan, draft and revise?	T	2 (2)	2 (2)
Time	How long does it take to write an assessment from start to finish?	T	4 (2)	1 (1)
(Pre-)reading	What and how do students read?	T	5 (4)	0 (0)
Editing	Where they need to proofread and edit?	S	4 (3)	0 (0)
Writing, drafting, revising process	When does most in depth revision occur? (between 1st/2nd, etc.)	T	4 (3)	0 (0)
Structure	How they approach the structure, i.e. which section do they address first?	T	3 (3)	0 (0)
Appropriate use of resources	What kinds of readings/texts are students using (e.g. published or other students’ assessments)?	T	1 (1)	0 (0)
Planning	How they plan or didn't, e.g. overall structure of the document	S	1 (1)	0 (0)
Addressing the task	The extent to which students understand and address the assignment questions	T	1 (1)	0 (0)

Note. ‘Who’ indicates who should have access to the data: T = teacher, S = student.

Table 6. Topics identified in the writing researchers focus group (N = 4).

Topic	Sticky note example	Who	Number of sticky notes (participants)	Number of votes (participants)
Being stuck	Where students struggle, e.g. answering the question, formulating an argument	T	3 (3)	3 (3)
Time	How much time students spend on a particular assignment	T	3 (3)	3 (3)
Revision types, levels	Suggested revision types and examples in their own text of where they've done them	S	3 (2)	3 (3)
Planning	Did they plan well enough before starting to write?	S, T	3 (2)	1 (1)
Macro edits	Moving text to build cohesion	S, T	4 (2)	1 (1)
Critical thinking	Where the core thesis changed substantially	S	4 (2)	0 (0)
Academic integrity	Alerts to large chunks of text posted in from somewhere - cheating?	T	1 (1)	0 (0)
Uncategorized	Their writing behavior/patterns + how this can be improved	S	1 (1)	1 (1)

Note. ‘Who’ indicates who should have access to the data: T = teacher, S = student.

Figure 1. Overview of the topics of desired indicators discussed by all stakeholders.

Notes. Topics are mapped to the process they address in Flower and Hayes’ (1981) model of writing processes (x-axis) and level of the indicators (behavioral versus cognitive; y-axis). Topics related to concepts in the writing environment (Hayes, 2012) are grouped in the rectangular boxes. Letters in the circles (T = teacher, S = student, NT = not teacher) indicate the stakeholders who should have access to these data.

Hayes, J. R. (2012). Modeling and remodeling writing. Written Communication, 29(3), 369–388. doi:10.1177/0741088312451260

 Web of Science ®Google Scholar