Making internal feedback explicit: harnessing the comparisons students make during two-stage exams

Figures & data

Figure 1. Internal feedback mechanism of two-stage exams.

Table 1. Reflective questions designed to make internal feedback explicit.

Reflective questions after individual stage If you are aware of any weaknesses in your answer, or if there are aspects of your answer that were still troubling you when you handed it in, could you explain what the issues were in a few sentences. If you could have talked to an expert while writing your answer what feedback would you have asked for (i.e. on what aspects of your work would you have asked for feedback)? Explain. Reflective questions after group stage How did the group answer differ from your own answer? Note two differences and clearly explain what they were. What did you learn about your own answer from the differences mentioned above? Please explain. Overall, which was better the group answer or your own answer? Why do you think this? If you think they were both the same, explain why you think this. Did the group discussion make you aware of any weaknesses or strengths in your answer that you were not aware of at the time you wrote it? If yes, explain what you became aware of? Can you explain exactly how you would improve your answer now, if you had the opportunity to do so?

Figure 2. Timeline of two-stage exam how the research questions were addressed.

Table 2. Coding categories for student-generated feedback comments.

Category of internal feedback	Definition and scope
Content	Generates feedback about subject content, the correctness or accuracy, about strengths, weaknesses, errors, incorrect responses etc.
Writing Process	Generates feedback about quality of writing, its logic, structure, convincingness, balance or quality of argument
Self-regulatory	Generates feedback about own thinking, abilities, learning strategies (effectiveness or not), skills especially group working and writing insofar as this went beyond the immediate task and had a future orientation.

Table 3. Number, categorisation and examples of students’ feedback comments after individual submission.

	No. comments	Mean (SD)	Examples
Content	31	1.35 (0.71)	I would have asked [the expert] the way the MPI was made to see whether the way the measure has been created may imply some issues. I made a suggestion about having more indicators for calculating MPI but did not know whether this would work.
Writing	32	1.39 (0.72)	…how to put my argument across so he understands what I am trying to say. I would have asked how to write my answer – should it be written by importance of weaknesses, should it include less or more points with less or more explanation. …not sure how convincing my argument was and how to present it well.
Self-regulation	4	0.17 (0.38)	I couldn’t gather my thoughts before I started writing.
Total	67	2.91 (1.25)

Table 4. Number, categorisation and examples of students’ content and writing feedback after the group submission.

	No. comments	Mean (SD)	Examples
Content	45	1.96 (0.95)	I learned that there was more to add to the improvements I had already come up with and that there are many more flaws with the MPI than I originally thought.
Writing	46	1.96 (0.81)	I learned that my own answer was fairly scruffy and badly structured. [I learned] a better way to structure my answer and present my ideas because I was really impressed by the way we did it together… [in the group]. Completeness: I didn’t go to that depth as they did. To get the best answer, be specific on one point rather than coming across many points but not in depth.
Self- regulation	53	2.35 (1.31)	See Table 5 for further analysis and examples.
Totals	144	6.26 (1.77)

Table 5. Number, categorisation and examples of students’ self-regulatory feedback comments after group submission.

Self-regulatory feedback categories	No.	Examples
Writing Process	14	We discussed each point before it was written we clarified and fully answered each point, whereas when I am writing my own answers I may speed through without completely explaining what I mean.
Different Perspectives/ approaches	13	I learned to always explore other perspectives and evaluate every aspect of an answer or problem.
Group skills/ benefits and importance	11	I think I learnt how valuable it is to talk about your ideas with others. I could apply this idea to essay writing too.
Language Skills	7	…weakness of my writing maybe the language, perhaps my vocabulary is not as diverse as others my age.
Planning	5	In exams it is difficult to have the time to plan the answer. So, I often write what comes to my mind as I am writing. This can make the point I want to get across difficult to do so, even though I may know the material well.
Wider reading	3	I would probably include some alternative outside hypothesis other than Amartya Sen. Therefore, engage more outside research and book reading.
Total	53

Table 6. Mapping of feedback concerns raised at the individual stage with students’ elaborations at the group stage.

Category	Individual stage concern		Group stage solution
	Example	No.	No.	Examples
Content	…slightly frustrated by my lack of concrete information about other alternative measures to the health/ education/standard of living deprivation measures.	31	23	the group answer was more comprehensive and well-informed as we were able to fact-check with each other… considerations about how a generic poverty measure may not be as useful when applied to a broad spectrum of societies as social context and social norms should not be undervalued in how they impact the experience of and perspective on poverty, as I came to realise during the discussion part of the question.
Writing	I was not sure how in detail the problems and improvements should be described. I would ask if the explanation is in enough detail and if he [the expert] would write more critically or just like I did.	32	25	I need to be more critical, explain my answers in more detail and also try to build in relevant examples.
Totals		63	48

Table 7. Feedback Comments: Teacher versus Students.

	Teacher Comments		Student Comments
	No.	Mean (SD)	No.	Mean (SD)
Content	41	1.78 (0.51)	45	1.96 (0.95)
Writing	30	1.31 (0.46)	46	2.00 (0.83)
Self-regulation	0	0	53	2.30 (1. 30)
Total	71	3.09 (0.65)	144	6.26 (1.75)

Table 8. Teacher Feedback compared with Students’ Feedback commentary.

	Examples of comments
Category	Teacher	Student
Detailed and specific understanding of how to improve on the content.	Few very good points, scope for many more.	I would add the importance of social needs like the access to clean water, proper food, electricity and so on. I think that those have to have more weight in the measure, rather than having a bike or a motorcycle. Also, I would like to explain better the differences between the countries.
	Scope for some more thinking and more effective discussion.	The group answer included even more weaknesses. MPI only focuses on health, education and objects when it comes to the standard of living and fails to include aspects of democracy, freedom of speech, freedom to do what you wish, opportunities and social exclusion. These are all aspects that play a more and more important role when it comes to determining how poor people are.
Realisation of misunderstandings and flaws.	The discussion should have been more about the make-up of the index itself. More scope for discussion.	I would remove my comments about people being off the grid - they would make up a really small percentage. The mentally ill with eating disorders should be counted as unhealthy, no matter how wealthy they are. I would add points about liberty, freedom, mental health and happiness.
Learnt how arguments could be stronger, explanations and examples.	Reasonably clear and effective (i.e. room to be even better).	It was also more eloquently written as we collaborated to come up with the best wording and way to describe, was much more in depth. I learned how to express my ideas more concisely. I would try to explain myself better.
	Should explain/justify more effectively.	The main difference was the in-depth answers with clear explanations and justification whereas I wrote important points and short descriptions. I need to be more critical, explain my answers more and also try to build in relevant examples.
Learnt a better way to structure and present the answer	You have introduced the discussion effectively. But you have to explain the points you make in more detail.	I did not come up with the structure my teammates did. I didn’t go to that depth as they did. To get the best answer, be specific on one point rather than coming across many points but not in depth. Be specific and get into a point comprehensively rather than coming across many points in shallow.
	Since this is a poverty measure trying to measure depravation of education, what you write first seems more appropriate.	Structure could have been better. It also felt like a more mature answer. I wish I had included correlation, outdatedness of assets and mortality rate of curable diseases. These were all things I had thought about but maybe couldn’t think how to write them in the most effective way.